Questionnaire pretesting: cognitive interviews, pilot surveys and what each one finds

1. The problem: questions that are clear to authors and ambiguous to respondents

A health services study asks: "In the last 12 months, have you consulted your family doctor?" The team that wrote the question had visits in mind. Some respondents count a phone call for a prescription, others do not; some include their child's visit, others only their own; a few older people count any contact with the clinic. The final percentage looks precise, comes with a margin of error and goes into the report. But it does not measure one single thing. Questionnaire pretesting is the stage that uncovers problems like this before they are multiplied across 1,000 interviews.

Pretesting is not a luxury reserved for large projects. Experienced researchers have said the same thing for decades: no questionnaire should be used in the field without adequate testing, and even the best-written instrument improves once it has been tested. The drafting rules covered in our article on questionnaire design reduce errors but cannot eliminate them, because an author cannot read their own questions through a respondent's eyes.

2. What can break between question and answer

To answer a question, a respondent goes through four steps, usually within a few seconds. Each step can fail independently, and pretesting looks for failures at every one of them.

• Comprehension. What does the question mean, and what does each word in it mean? Does "household income" include the pension of a grandmother who lives with the family? Remittances from relatives abroad?
• Retrieval. Does the respondent have the information, and can they recall it for the period asked? The number of times you bought bread last year cannot be recalled, only estimated.
• Judgement. How does the respondent turn what they remember into an answer: do they count, estimate, compare themselves with others, or adjust the answer to what seems acceptable?
• Response. Does their answer fit the options offered? If the scale lacks the right option, the respondent picks something close and the information is lost.

A questionnaire that is formally well written can fail at any of these steps. That is why questionnaire pretesting is not done by reading the questionnaire, but by watching people complete it. Comprehension problems are the most dangerous, because the respondent does not know they have understood something different: they answer confidently, and the answer looks perfectly valid in the dataset.

3. Questionnaire pretesting methods, from desk to field

Pretesting is not a single activity but a sequence of methods, each good at something different. Their order has a logic: first remove the errors visible at the desk, then investigate understanding with real respondents, and finally check how the instrument performs in field conditions. Skipping a stage pushes its problems into the next one, where they are harder to diagnose.

Expert review

Researchers who did not write the questionnaire go through it question by question with a checklist: double-barrelled questions, hidden assumptions, unbalanced scales, faulty skip logic, length. It is quick and cheap, but it only finds the problems an expert can anticipate, not the ones real respondents produce.

Cognitive interviews

A cognitive interview is a one-to-one session of 45-90 minutes in which a trained researcher administers the questionnaire to a respondent and explores how they arrive at each answer. Two techniques are used, usually in combination:

• Think-aloud: the respondent is asked to say everything that goes through their mind while answering. The researcher notes hesitations, interpretations and calculations.
• Verbal probing: after the answer, the researcher asks follow-up questions. "What does 'family doctor' mean to you here?", "How did you arrive at the number 4?", "Can you put the question in your own words?", "How sure are you of that answer?"

The cognitive interview is a qualitative method: it does not look for frequencies but for kinds of problems. If two or three respondents out of eight understand a term differently, the problem is real and the question gets rewritten. The work is done in small rounds, with corrections between rounds. The researcher's skill matters as much as in an in-depth interview: probing must be neutral and must never hint at the "right" interpretation.

The pilot survey

A pilot survey runs the near-final questionnaire on a small sample under the study's real conditions: the same collection method, the same or similar interviewers, the same tablet or telephone script. It no longer investigates the thought process but the mechanics: actual length, skip logic, questions with many refusals or "don't know" answers, fatigue towards the end, reactions to the introduction. At the end of the interview, respondents can be told it was a test and asked what they found unclear or difficult.

Interviewer debriefing

After the pilot, interviewers meet for a structured discussion: where they had to repeat or explain questions, where they sensed discomfort, which instructions were missing. Experienced interviewers easily pick up confusion and resistance; newer ones reveal problems with training and with how instructions are written.

4. Who to test, and how many

The basic rule of questionnaire pretesting: test respondents must come from the same population as the study's respondents, with the same relevant characteristics, meaning familiarity with the topic, education, age, language, and urban or rural residence. Testing a questionnaire meant for older rural residents on office colleagues in the capital tests nothing.

The numbers are small but carefully chosen. Cognitive interviews are usually run in rounds of a handful of interviews, with deliberately varied profiles in each round, and continue until rounds stop producing new problems. For an initial pilot survey, 15 to 30 respondents are generally enough, more if the population is heterogeneous or if the pilot is repeated in several stages. A study with very different subgroups, for example young urban adults and older villagers, needs each subgroup represented in the test.

5. Pretesting in more than one language

In Moldova, most national studies are fielded in Romanian and Russian, and some in other minority languages too. Each language version is in effect a separate instrument and is tested separately. Typical problems include terms with different connotations in the two languages, scales whose labels do not carry the same intensity ("fairly satisfied" is not necessarily the exact equivalent of its translation), and instructions that become longer or clumsier once translated.

Good practice includes cognitive interviews in each language, run by native-speaking researchers, and a comparison of interpretations across versions. If Russian-speaking respondents understand a question differently from Romanian-speaking ones, differences in the results will reflect the translation rather than reality. A multilingual team that works on both versions from the outset reduces these risks before pretesting even starts.

6. From findings to corrections

Questionnaire pretesting should produce a documented list, not just a feeling that the test went well. Findings are gathered in a question-by-question table: what problem appeared, with how many respondents, at which step of the answering process, and what change is proposed. Some corrections are minor: a word replaced, a reference period clarified, a response option added. Others are structural: a question split in two, a section moved, a new filter, a different scale.

Every significant revision is followed by a new test, with different respondents. A single test is the absolute minimum; proper pretesting has several stages and continues until no further changes are needed. For tablet or telephone studies, each corrected version of the script is checked again along every possible route, including with automatically generated test data, to confirm that skips and validations work. More on these methods on our CAPI and CATI page.

Finally, pilot responses are coded and analysed as if they were real data. If an answer cannot be linked to any planned analysis table, the question is probably redundant; if a planned table stays empty, a question is missing. This is the last cheap opportunity to confirm that the instrument produces exactly the information the decision needs.

7. Limits and common mistakes

• Testing on colleagues. People who know the project and its terminology cannot spot what is unclear.
• A pilot without observation. An online pilot with no debriefing questions shows length and drop-out, but not their causes.
• Too little time in the schedule. A pretest scheduled the day before fieldwork leaves no room for any serious correction.
• Untested corrections. A question rewritten after the pilot is a new question, and new questions need testing.
• Overconfidence in the pretest. A good pretest reduces measurement error, but it does not guarantee valid answers on sensitive or rare behaviours, and it is no substitute for a well-built sample.

8. Pretesting methods compared

9. What to ask the agency about pretesting

• Which pretesting methods are included in the proposal, and in what order?
• How many cognitive interviews and pilot interviews are planned, and with which respondent profiles?
• Are the Romanian and Russian versions tested separately, by native-speaking researchers?
• Does the pilot use the same collection method and the same script as the main study?
• How much time does the schedule leave between the pilot and the start of fieldwork for corrections?
• Will you receive a pretest report listing the problems found and the changes made?
• Are substantially revised questions tested again?
• Are pilot data checked against the planned analysis tables?

Questionnaire pretesting costs little relative to the whole study, and it is the only stage at which measurement errors can be fixed cheaply. Cognitive interviews show what people understand, the pilot survey shows what happens in the field, and together they turn a plausible questionnaire into a trustworthy instrument. If you are preparing a survey and want the instrument tested before launch, our quantitative research team can run the whole process, and the differences between collection methods are explained in our article on face-to-face interviews.

See also: Quantitative research CAPI and CATI Multilingual team How to design a questionnaire Face-to-face interviews (CAPI) In-depth interviews

The Perspicuus research team · 17 December 2025