(1) Aim: We aim to evaluate the association between arousals during sleep and subclinical coronary atherosclerosis detected by coronary computed tomography angiography (CTA) in patients with obstructive sleep apnea (OSA). (2) Methods: This was a cross-sectional study. Consecutive newly diagnosed OSA patients, who underwent coronary CTA examinations within 3 months of the sleep study, were eligible. We used the arousal index (ArI) derived from polysomnography to assess arousals during sleep and a semi-automated plaque quantification software to characterize and quantify the subclinical coronary atherosclerosis. Multiple regression models were used to evaluate the associations of the ArI with the coronary atherosclerotic plaque presence, volume, and composition. (3) Results: A total of 99 patients with OSA were included in the study. In the multivariable models, patients with a high ArI (ArI > 32.2 events/h) were more likely to have coronary plaques compared to those with a low ArI (ArI ≤ 32.2 events/h) (OR: 3.29 [95% CI: 1.284 to 8.427], p = 0.013). Furthermore, the ArI exhibited significant associations with total (β = 0.015), noncalcified (β = 0.015), and low-attenuation (β = 0.012) coronary plaque volume after accounting for established risk factors (p = 0.008, 0.004, and 0.002, respectively). However, no association between the ArI and calcified plaque volume was found. (4) Conclusion: Repetitive arousals during sleep are associated with an increased coronary plaque burden in patients with OSA, which remained robust after adjusting for multiple established cardiovascular risk factors.