The low accuracy of unilateral upper limb multitask motor imagery (MI) classification is a significant issue hindering the development of brain-computer interface-based rehabilitation training. The low spatial resolution of electroencephalogram (EEG) in stroke patients and the complexity of multitask classification are the main reasons for this issue. The objective of this study is to develop an algorithm for decoding multiclass motor imagery tasks of unilateral upper limbs of patients, which is used to improve the classification accuracy. The algorithm is named temporal-spatial domain adaptation (TSDA), which is based on self-attention convolutional neural networks (SACNN) as the backbone network and combines the advantages of model transfer and domain adaptation. The base model of healthy subject training is used as the initialization model for patient domain adaptation training in TSDA. In addition, we also propose two domain distance loss functions, temporal multi-kernel conditional maximum mean discrepancy, and spatial multi-kernel conditional maximum mean discrepancy, which used to constrain the optimization direction of spatial-temporal features during model training. In this study, we collected EEG data from 16 healthy subjects and 20 stroke patients for four types of unilateral upper limb MI tasks to verify the classification performance of TSDA and compare it with other advanced algorithms. The base model trained on EEG data of 16 healthy subjects by SACNN is used as the initialization model. The initialization model is further trained in TSDA with one patient as the target domain and another patient as the source domain. Each of the 20 patients served as a source domain and a target domain. The classification accuracy of TSDA is 51.7% +/- 0.08, which is significantly higher than that of the benchmark algorithms (P < 0.05). The technical verification and visualization results show the stability and reliability of the TSDA model. This paper provides more theoretical basis for MI multitask classification of stroke patients based on transfer learning.