Paeonol (2-hydroxy-4-methoxyphenylacetophenone) is a natural phenolic component isolated from the root bark of peony with multiple pharmacological activities and has been proven to have anti-cancer effects. The objective of this study is to investigate the influence mechanism of paeonol on the proliferatory and apoptotic activities of ovarian cancer (OC) cells by modulating the transforming growth factor beta (TGF-β)/Smad3 pathway. The SKOV3 cells were pretreated with various concentrations of paeonol (0, 25, 50, 100, 200, 400 μg/mL) for 48 hours to determine the optimal experimental concentration of paeonol. Following this, the TGF-β overexpression vector was constructed and transfected into the SKOV3 cells. The assessment of cell proliferation, invasion, and migration was conducted through MTT, colony formation, flow cytometry, transwell, and wound-healing experiments. The detection of TGF-β/Smad3 pathway-related proteins and apoptosis-related proteins (B-cell lymphoma (Bcl-2) Bcl-2-associated X protein (Bax)) was performed using Western blot analysis. Paeonol exhibited a significant inhibitory effect on SKOV3 cell viability when administered at concentrations ranging from 50-400 μg/mL, with an IC50 value of 200 μg/mL. Within the concentration range of 50 to 200 μg/mL, paeonol exhibited a dose-dependent effect on the progression of SKOV3 cells, including a reduction in the anti-apoptotic protein Bcl-2, an increase in the pro-apoptotic protein Bax (P<0.05), inhibition of cell migration and invasion (P<0.05), and promotion of cell apoptosis (P<0.05), particularly at a concentration of 200 μg/mL. These effects were found to be more pronounced. The aforementioned effects of paeonol can be ascribed to its inhibition of the TGFβ/Smad3 pathway, according to a mechanistic viewpoint. It is noteworthy that the inhibitory impact of paeonol on SKOV3 cell progression is counteracted by the elevation of TGF-β levels following overexpression. We conclude that paeonol exerts regulatory effects on the TGF-β/Smad3 pathway, leading to the inhibition of proliferation, migration, and invasion of OC cells, thereby attenuating malignant behavior of cancer cells.