In clinical practice, magnetic resonance (MR) imaging is often scanned with large slice thickness due to many limiting factors such as scanning time. The acquired images are thus anisotropic, with much lower inter-slice resolution than the intra-slice resolution. For better coverage of the organs of interest, multiple anisotropic scans, each of which focus to a certain scan direction, are usually acquired per patient. In this work, we propose a 3D deep learning based super-resolution (SR) framework to reconstruct the isotropic high-resolution MR images from multiple anisotropic scans. In particular, we employ the spatially sparse fidelity loss to the locations acquired in anisotropic inputs, such that their intensities keep the same before and after the reconstruction. Meanwhile, the adversarial regularization is adopted to make sure that the entire reconstructed image owns consistent appearance perceptually. Different from other SR methods, our approach fuses inputs of multiple anisotropic images, instead of a single one. Moreover, our reconstruction is attained without using any supervision from the isotropic high-resolution images, making it unique among early works and highly applicable to many real clinical scenarios.