Canonical models of congressional oversight suggest that members of Congress (MCs) rely on 'sentinels' to monitor and influence the actions of bureaucratic agencies. Individuals or interest groups with a stake in agency decisions can pull 'fire alarms' to signal their opposition to certain policies, prompting Congress to respond. However, there is limited evidence on who pulls these fire alarms and which signals are most effective in getting Congress to take action. Using new geocoded data on 13 million public comments submitted during agency rulemaking, I show that MCs respond selectively to fire alarms. When large numbers of individuals or interest groups from their districts oppose a proposed rule, MCs are more likely to i) publicly criticize the agency on social media, ii) privately seek information through informal communications, and iii) summon agency officials to testify before congressional committees. These findings show how partisan and geographic factors shape the ability of Congress to oversee the bureaucracy.