Magnetic resonance imaging (MRI) has become necessary in clinical diagnosis for knee osteoarthritis (OA), while deep neural networks can contribute to the computer-assisted diagnosis. Recent works prove that instead of only using a single-view MR image (e.g., sagittal), integrating multi-view MR images can boost the performance of the deep network. However, existing multi-view networks typically encode each MRI view to a feature vector, fuse the feature vectors of all views, and then derive the final output using a set of shallow computations. Such a global fusion scheme happens at a coarse granularity, which may not effectively localize the often tiny abnormality related to the onset of OA. Therefore, this paper proposes a Local Graph Fusion Network (LGFNet), which implements graph-based representation of knee MR images and multi-view fusion for OA diagnosis. We first model the multi-view MR images to a unified knee graph. Then, the patches of the same location yet from different views are encoded to one-dimensional features and are exchanged mutually during fusing. The local fusion of the features further propagates following edges by Graph Transformer Network in the LGF-Net, which finally yields the grade of OA. The experimental results show that the proposed framework outperforms state-of-the-art methods, demonstrating the effectiveness of local graph fusion in OA diagnosis.