The responsiveness of political parties to voters' policy preferences is a core feature of democracies. A growing number of studies analyze this phenomenon, although it is unclear whether and to whom parties are responsive. This paper discusses the different implications of the proximity and directional model of voting for parties' responsiveness to position shifts of the electorate. It argues that parties are responsive to the position of specific segments of the electorate that are more likely to vote for them, and does not respond to the median voter. Taking this into account, this paper analyzes and compares the responsiveness of parties to the median and the party voter. It relies on the unique setting of direct democracy in Switzerland to estimate the ideological position of the median voter, the party voter, and the different political parties over time. The paper uses a Bayesian Item Response Theory model to operationalize the ideological direction of ballot proposals and uses it to evaluate parties' responsiveness to the median and the party voter. The results show that parties are not responsive to the median voter's position but seem to follow the position of the party voter.