Survey Comment Nonresponse and the Characteristics of Nonresponders

Abstract

This study explored survey commenting behavior and investigated the two broad questions of “who†comments and “how†respondents comment. Participants were military personnel (N=419) who were given the opportunity to respond to an open-ended question on a training evaluation survey after completing a foreign language course. The first set of hypotheses examined the question of “who†comments. As expected, results showed that dissatisfied individuals and those interested in the topic were especially inclined to comment. Additionally, partial support was found for a positive relationship between education and commenting behavior, such that those with some college education but no four-year degree or higher were more likely to comment than those with no college experience. Unexpectedly, however, no significant difference was found between respondents with a four-year degree and higher than those with either some or no college experience. In contrast with the study hypotheses, conscientious respondents were not especially inclined to comment. The second set of hypotheses explored the question of “how†respondents comment. As hypothesized, dissatisfied respondents tended to provide negative comments, and conscientious respondents provided specific comments. In contrast to the study hypotheses, respondents scoring high on agreeableness and emotional stability did not provide particularly positive comments, those who were satisfied with course materials and the learning environment were more (not less) likely to provide specific comments, and those high in extraversion and conscientiousness were not particularly likely to provide suggestions. The vividness of verbatim comments may lead managers and training evaluators to place more weight on comments than quantitative data (i.e., ratings) when following up on survey results. This study suggests that those who do so may be basing their decisions on input from an unrepresentative segment of respondents. Steps should be taken to inform managers and training evaluators of the potential for nonresponse bias in open-ended comments. In addition, survey research and practice would benefit from interventions aimed at increasing the likelihood of hearing from the voices that often go unheard.