Echocardiography is the first-line diagnostic technique for heart diseases. Although artificial intelligence techniques have made great improvements in the analysis of echocardiography, the major limitations remain to be the built neural networks are normally adapted to a few diseases and specific equipment. Here, we present an end-to-end deep learning framework named AIEchoDx that differentiates four common cardiovascular diseases (Atrial Septal Defect, Dilated Cardiomyopathy, Hypertrophic Cardiomyopathy, prior Myocardial Infarction) from normal subjects with performance comparable to that of consensus of three senior cardiologists in AUCs (99.50% vs 99.26%, 98.75% vs 92.75%, 99.57% vs 97.21%, 98.52% vs 84.20%, and 98.70% vs 89.41%), respectively. Meanwhile, AIEchoDx accurately recognizes critical lesion regions of interest along with each disease by visualizing the decision-making process. Furthermore, our analysis indicates that heterogeneous diseases, like dilated cardiomyopathy, could be classified into two phenogroups with distinct clinical characteristics. Finally, AIEchoDx performs efficiently as an anomaly detection tool when applying handheld device-produced videos. Together, AIEchoDx provides a potential diagnostic assistant tool in either cart-based echocardiography equipment or handheld echocardiography device for primary and point-of-care medical personnel with high diagnostic performance, and the application of lesion region identification and heterogeneous disease phenogrouping, which may broaden the application of artificial intelligence in echocardiography.