Open-Ended Survey Questions: A comparison of information content in text and audio response formats
Open-ended questions provide an important and rich source of data in addition to closed-ended questions to measure the preferences, attitudes and motivations of individuals in surveys and experiments. However, despite their usefulness, they pose challenges to respondents, such as increased response burden and related phenomena such as a decrease in response length and response quality. Consequently, the question arises as to how researchers should design open-ended questions so that answers contain a high degree of information.
Our study examines the effect of requesting respondents to answer questions via voice input in comparison to text input. Using a U.S. representative quota sample (N=1,500) and questions adapted from popular social survey programs, we experimentally vary the response format and examine which format produces answers with a higher amount of information. Information content is operationalized by answer length, the number of topics derived from topic models, and an assessment of document similarity using cosine similarity. Preliminary results show that oral responses are longer but results with regard to information content are mixed. Open audio responses do not seem to be the supposed secret weapon to learn more about individuals' motives and preferences.
Our study examines the effect of requesting respondents to answer questions via voice input in comparison to text input. Using a U.S. representative quota sample (N=1,500) and questions adapted from popular social survey programs, we experimentally vary the response format and examine which format produces answers with a higher amount of information. Information content is operationalized by answer length, the number of topics derived from topic models, and an assessment of document similarity using cosine similarity. Preliminary results show that oral responses are longer but results with regard to information content are mixed. Open audio responses do not seem to be the supposed secret weapon to learn more about individuals' motives and preferences.