Steps in validating a questionnaire
I use the Total Survey Error Framework (Pennay 2014) as the conceptual and practical framework to assess and reflect on improving the quality of the questionnaire and its results, and to reduce the possibility of any errors (Whiteley 2015).The framework (as shown below) consists of both errors of representation (sampling errors) and errors of measurement (non-sampling errors).Pilot testing tends to be used more to help with predicting the analysis of a large-scale survey.For one of the face-to-face surveys I designed, the survey was piloted face-to-face with 22 volunteers, and the non-verbal behaviour of respondents was observed alone.I use the verbal probing method alongside the thinking aloud method to detect potential sources of response error.This is because when the thinking aloud method is used on its own, useful information is not always gained, since although it can highlight that there is a problem with a question, it cannot always tell you what the problem is.Furthermore, some participants may find it difficult to think aloud, and you may also want to focus on specific areas.
In the former, the respondent is asked to vocalise his or her thought process as he or she is filling in the questionnaire.The former occurs as part of the selection of the sample from the sampling frame.The latter occurs when designing and conducting the survey.(2009) and other survey researchers recommend the use of ‘expert reviews’ as the first step to validate the questionnaire (De Maio and Landreth 2004).Expert reviews are where both questionnaire design experts and subject matter experts evaluate the content, cognitive, and usability standards of a questionnaire (Groves et al. This can highlight any problems the questions and scales may have in relation to its scope for example, or the way the questions may be understood or perceived by its participants, and evaluate whether the questions are clear and easy for participants to understand and answer (De Maio and Landreth 2004; Groves et al. Cognitive interviewing is a process in which participants respond to draft survey questions, and are asked to reveal their thought processes at the same time (Farrall et al. The premise is that knowing these thought processes and cognitive understanding can help the researcher evaluate the quality of the survey questions and responses. (2005) and Willis and Artino (2013), there is a growing body of literature that demonstrates that only a very few cognitive interviews can allow for the identification of problems with questions, and help in making revisions, so that the quality of the survey data can be greatly improved.
Search for steps in validating a questionnaire:
The researcher then reads or listens to the transcript to understand the respondent’s thought process and understanding, and uses this to remove or change any difficult questions or phrasing.