Objective: The associations of vascular risk factors (VRFs), apolipoprotein E (APOE), and translocase of outer mitochondrial membrane 40 (TOMM40) with cognitive function have been investigated mostly in western societies. In the present study, we sought to examine the associations of VRFs [i.e., current smoking, current drinking, physical inactivity, obesity, total cholesterol (TC), triglycerides (TG), low-density lipoprotein cholesterol (LDL-C), high-density lipoprotein cholesterol (HDL-C), diabetes, and hypertension] and variants located in APOE (epsilon 2/3/4) and TOMM40 (rs2075650) with global cognitive function in Chinese older adults, with a focus on their potential interactions. Methods: This is a cross-sectional study that included 422 permanent residents (mean age 69.2 years, 54.3% female) living in Beijing, who were free of dementia. Data were collected through interviews, clinical examinations, and laboratory tests. The two genetic polymorphisms were genotyped, and participants were dichotomized as carriers vs. non-carriers of APOE epsilon 4 or TOMM40 G. Global cognitive function was assessed with the Mini-Mental State Examination (MMSE). Data were analyzed with multivariable linear regression models. Results: Physical inactivity and diabetes were independently associated with a lower MMSE score (all p < 0.05). When four putative VRFs (i.e., current smoking, physical inactivity, high LDL-C, and diabetes) were aggregated, an increasing number of having these factors was associated with a decreasing MMSE score in a dose-response manner (p = 0.001). TOMM40 polymorphisms, independent of the APOE epsilon 4 allele, interacted with aggregated VRFs to influence cognitive performance, such that having one or more of these VRFs was particularly detrimental to the cognition of TOMM40 carriers. Further analyses revealed interactions of the TOMM40 polymorphism with (i) physical inactivity and (ii) diabetes, such that having either physical inactivity or diabetes in combination with carrying a TOMM40 G allele, compared to having neither, was significantly associated with a markedly lower MMSE score (all p < 0.05). Conclusion: This study provides some evidence supporting the association of vascular risk factors with poor cognitive performance among dementia-free Chinese older adults and further revealed their interactions with the TOMM40 polymorphism. The results underscore the vulnerability of global cognitive function to VRFs, which could be reinforced by carrying the TOMM40 rs2075650 G allele. These findings have potential implications for developing tailored intervention programs to maintain cognitive function.