Human dimensions research has proposed a multitude of variables impacting the viability of wildlife populations. Extant approaches to systematizing these variables have mostly focused on human relations to only one animal species or taxon and are largely descriptive, rather than explanatory. In this study, we provide a three-layer framework for understanding people’s responses to a variety of human–wildlife encounters. We conducted a comparative qualitative study, interviewing 20 stakeholders on one of three ecologically disparate model animals. Through thematic analysis, we identified person-specific, species-specific, and overarching factors whose interplay shapes people’s reactions to encounters with wildlife. The person-specific factors, individual people’s biographic backgrounds and life themes, fuel the polarization of stances towards wildlife. The species-specific factors, people’s mental images of wild animals, explain the particular character of different human–wildlife relations. The overarching factors, fundamental questions regarding the place of humans in nature or motivations of control over animal agents, stir the intensity inherent in human encounters with wildlife. This three-layer framework amends existing proposals by providing a cohesive system and an in-depth portrayal of shared and specific factors and processes in various human–wildlife relations and by elucidating their interaction in influencing people’s responses to encounters with wild animals.