Background Social attention ability is crucial for human adaptive social behaviors and interpersonal communications, and the malfunction of which has been implicated in autism spectrum disorder (ASD), a highly genetic neurodevelopmental disorder marked by striking social deficits. Methods Using a classical twin design, the current study investigated the genetic contribution to individual variation in social and non-social attention abilities, and further probed their potential genetic linkage. Moreover, individual autistic traits were further measured in an independent group of non-twin participants to examine the hypothetical link between the core social attention ability and ASD. Results We found reliable genetic influences on the social attentional effects induced by two distinct cues (eye gaze and walking direction), with 91% of their covariance accounted for by common genetic effects. However, no evidence of heritability or shared genetic effects was observed for the attentional effect directed by a non-social cue (i.e. arrow direction) and its correlation with the social attention ability. Remarkably, one's autistic traits could well predict his/her heritable core social attention ability extracted from the conventional social attentional effect. Conclusions These findings together suggest that human social attention ability is supported by unique genetic mechanisms that can be shared across different social, but not non-social, processing. Moreover, they also encourage the identification of 'social attention genes' and highlight the critical role of the core human social attention ability in seeking the endophenotypes of social cognitive disorders including ASD.