The objective of this study is to develop a deep-learning-based detection and diagnosis technique for carotid atherosclerosis (CA) using a portable freehand 3-D ultrasound (US) imaging system. A total of 127 3-D carotid artery scans were acquired using a portable 3-D US system, which consisted of a handheld US scanner and an electromagnetic (EM) tracking system. A U-Net segmentation network was first applied to extract the carotid artery on 2-D transverse frame, and then, a novel 3-D reconstruction algorithm using fast dot projection (FDP) method with position regularization was proposed to reconstruct the carotid artery volume. Furthermore, a convolutional neural network (CNN) was used to classify healthy and diseased cases qualitatively. Three-dimensional volume analysis methods, including longitudinal image acquisition and stenosis grade measurement, were developed to obtain the clinical metrics quantitatively. The proposed system achieved a sensitivity of 0.71, a specificity of 0.85, and an accuracy of 0.80 for diagnosis of CA. The automatically measured stenosis grade illustrated a good correlation ( r = 0.76) with the experienced expert measurement. The developed technique based on 3-D US imaging can be applied to the automatic diagnosis of CA. The proposed deep-learning-based technique was specially designed for a portable 3-D freehand US system, which can provide a more convenient CA examination and decrease the dependence on the clinician's experience.