While the relationship between economic factors and "cultural backlash" remains highly contested in political science, studies show that both anti-immigration sentiment and radical right support vary considerably within countries. Regional economic conditions seem to correlate with both anti-immigration attitudes and radical right voting but the causal mechanisms underlying these relationships remain largely unknown. In this paper, we argue that the interplay of individual skill sets and differences in regional labor markets helps explain these phenomena. Following a literature on geographical concentration of firms in economics, we assume that it is efficient for firms to rely on highly specific skills to concentrate geographically, thus generating skill-based population clusters and raising the bar to inter-territorial relocation. We expect that the shrinking or collapse of regional economic clusters has distinct effect on anti-immigration sentiment and radical right support, over and beyond residential self-selection into such clusters that may explain initial regional patterns. The shrinking or collapse of regional sectors induces acute economic stress for affected workers. Whereas workers with high skill transferability and high employability can mitigate the medium-term consequences through occupational change or relocation, others are left in precarious positions. We test our argument using geo-referenced panel data from the GSOEP in combination with detailed information on the dynamics of regional (county-level) labor markets, obtained through original aggregations of data from the German microcensus. This rich and novel data base also allows us to contrast our proposed mechanism with alternative economic or psychological explanations.