Background Trait self-esteem reflects stable self-evaluation, and it affects social interaction patterns. However, whether and how trait self-esteem can be expressed through behaviors are controversial. Considering that facial expressions can effectively convey information related to personal traits, the present study investigated the three-dimensional (3D) facial movements related to self-esteem level and the sex differences therein. Methods The sample comprised 238 participants (46.2% males, 53.8% females). Their levels of trait self-esteem were evaluated by employing the Rosenberg Self-Esteem Scale (SES) (47.9% low self-esteem, 52.1% high self-esteem). During self-introductions, their facial movements in 3D space were recorded by Microsoft Kinect. Two-way ANOVA was performed to analyze the effect of self-esteem and gender on 3D facial movements. Additionally, Logistic regression models were established to describe the relationship between 3D facial movements and self-esteem levels in both genders. Results The results of two-way ANOVA revealed a main effect of trait self-esteem level for cheeks and lips' movements. Meanwhile, there was a significant interaction between trait self-esteem and gender on the variability of lips' movements. In addition, the combination of facial movements can effectively identify trait self-esteem in men and women, with 75.5 and 68% accuracy, respectively. Conclusion The present results suggest that the 3D facial expressions of individuals with different trait self-esteem levels were different, and such difference is affected by gender. Our study explores a possible way in which trait self-esteem plays a role in social interaction and also provides the basis for automatic self-esteem recognition.