In the therapist-centered rehabilitation program, the experienced therapists can observe emotional changes of stroke patients and make corresponding decisions on their intervention strategies. Likewise, robotic-assisted stroke rehabilitation systems will be more appreciated if they can also perceive emotional states of the stroke patients and enhance their engagements by exploring emotion-based dynamic difficulty adjustments. Nevertheless, few research have addressed this issue. A two-phase pilot study with anxiety as the target emotion state was conducted in this article. In phase I, the motor performances and the physiological responses to the stroke subject's anxiety with high, medium, and low intensities were statistically analyzed, and anxiety models with three intensities were offline developed using support vector machine-based classifiers. In phase II, anxiety-based closed-loop robot-aided training task adaptation and its impacts on patient-robot interaction engagements were explored. As a comparison, a performance-based robotic behavior adaptation was also implemented. Experimental results with 12 recruited stroke patients conducted on the Barrett WAM(TM) manipulator verified that the rehabilitation robot can implicitly recognize the anxiety intensities of the stroke survivors and the anxiety-based real-time robotic behavior adaptation shows more engagements in the human-robot interactions.