The possibility of genotype–environment interaction for memory performance and change was examined in 150 monozygotic (MZ) twin pairs from the Swedish Adoption Twin Study of Aging (SATSA). We used an MZ twin pair difference approach to examine the possibility that genotype was associated with intrapair variability and thus suggestive of genotype–nonshared environment interactions. Multiple ‘variability genes’ were found for longitudinal change in a semantic memory task including candidates coding for apolipoprotein E (APOE) and estrogen receptor alpha (ESR1) as well as serotonin candidates (HTR2A and 5HTT). One candidate also related to variability in change in episodic memory (5HTT). Of the significant associations observed, generally results indicated that MZ pairs who carry putative risk alleles were less variable than noncarriers, suggesting that noncarriers may be more sensitive to environmental contexts. We sought to ‘contextualize’ the possible nonshared environmental influences for found gene–environment (G × E) effects by considering intrapair differences in measured social and stress factors, including social support, life events and depressive symptoms. Results suggested that nonshared environmental influences associated with depressive symptoms may moderate the G × E relationship observed for ESR1 and APOE and longitudinal semantic memory change whereby noncarriers of putative risk alleles may be relatively more sensitive to depressionevoking environmental contexts than carriers of the risk allele. Thus, the contexts that facilitate or reduce depressive symptoms may affect semantic memory resiliency dependent on genotype. Further work ought to consider larger sample sizes as well as consider additional social and contextual factors.