Background The NSD family of histone lysine methyltransferases have emerged as important biomarkers that participate in a variety of malignancies. Recent evidence has indicated that somatic dysregulation of the nuclear receptor binding SET domain-containing protein 1 (NSD1) is associated with the tumorigenesis in HCC, suggesting that NSD1 may serve as a prognostic target for this malignant tumor. However, its mechanism in human hepatocellular carcinoma (HCC), the major primary malignant tumor in the human liver, remains unclear. Hence, we investigated how NSD1 regulated HCC progression via regulation of the Wnt/beta-catenin signaling pathway. Methods Reverse transcription quantitative polymerase chain reaction (RT-qPCR) and Western blot analysis was performed to identify the expression of NSD1 in HCC cells and clinically obtained tissues. The relationship between NSD1 expression and prognosis was analyzed by Kaplan-Meier survival curve. Further, a NSD1 knockout cell line was constructed by CRISPR/Cas9 genomic editing system, which was investigated in a battery of assays such as HCC cell proliferation, migration and invasion, followed by the investigation into NSD1 regulation on histone H3, Wnt10b and Wnt/beta-catenin signaling pathway via ChIP. Finally, a nude mouse xenograft model was conducted in order to assess tumorigenesis affected by NSD1 knockout in vivo. Results NSD1 was overexpressed in HCC tissues and cell lines in association with poor prognosis. Knockout of NSD1 inhibited the proliferation, migration and invasion abilities of HCC cells. CRISPR/Cas9-mediated knockout of NSD1 promoted methylation of H3K27me3 and reduced methylation of H3K36me2, which inhibited Wnt10b expression. The results thereby indicated an inactivation of the Wnt/beta-catenin signaling pathway suppressed cell proliferation, migration and invasion in HCC. Moreover, these in vitro findings were reproduced in vivo on tumor xenograft in nude mice. Conclusion In conclusion, the study provides evidence that CRISPR/Cas9-mediated NSD1 knockout suppresses HCC cell proliferation and migration via the NSD1/H3/Wnt10b signaling pathway, suggesting that NSD1, H3 and Wnt10b may serve as potential targets for HCC.