In ultrasound video, the tracking and segmentation of carotid plaques arc prerequisite for the analysis of plaque properties. The problem at hand is quite challenging as some issues such as low ultrasound image resolution, large variation between frames and low tracking efficiency need to addressed. Our method, Siamese automatic tracking and segmentation network (SiamATSN), is an end-to-end deep learning method. Multiple dual attention region proposal network (DARPN) blocks are developed to integrate multi-level features for better target detection. The DARPN block is composed of spatial-wise attention module, channel-wise attention module, and Region Proposal Network. Moreover, a fusion module is further introduced to capture long-range contextual clues. At last, concat module which combines low and high resolution features is embedded in the decoder path to attain a more precise segmentation. Extensive experiments are conducted and the experimental results showed that our approach outperforms the state-of-the-art deep learning based methods.