The Delphi method and expert panels: reaching a collective judgement when the data are missing

1. The problem: decisions for which the data do not yet exist

A producers' association has to decide which technologies to invest in over the next ten years. A ministry is preparing a vocational training strategy and must anticipate which occupations will be in demand. A development fund wants to rank ten possible interventions in a sector where statistics are patchy. None of these questions can be answered by a population survey: ordinary people do not have the information, and historical data do not describe a future that is changing. The best available source is the judgement of people who know the field. The difficulty is gathering it without letting it be bent by whoever speaks loudest. The Delphi method was developed for precisely this problem.

The usual way of consulting experts, a round table or a workshop, has well-known flaws. Higher-status participants dominate the discussion. Some shift position to avoid contradicting a superior or a client. Others will not back down from a view once they have stated it in public. The outcome reflects the dynamics of the room as much as the knowledge of the people in it.

2. What the Delphi method is

The Delphi method is a structured technique for consulting an expert panel through several rounds of questionnaires, with four defining features.

• Anonymity: experts do not know who gave which answer, and sometimes not even who else is on the panel. Each can hold or change a position at no social cost.
• Iteration: the questionnaire runs over two, three or four rounds, and experts can revise their answers.
• Controlled feedback: between rounds every expert receives a summary of the panel's responses, typically the median and the range covering the middle half of answers, together with anonymised arguments, especially from those holding outlying views.
• A statistical group response: the final result is not a conclusion written up by the moderator but the distribution of answers, with measures of central tendency and spread.

The logic is straightforward. Each expert first forms a judgement alone, then sees where it sits relative to colleagues and what arguments they offer, and decides whether to hold or adjust. Change comes from arguments, not from pressure in the room. Positions that remain apart after several rounds are valuable information in their own right.

3. When it is the right choice, and when it is not

The Delphi method fits when the question calls for specialist knowledge rather than the views of the general public; when objective data are missing or insufficient; when the future cannot be extrapolated from the past; and when the relevant experts are spread across places or institutions, so that a joint meeting is hard to arrange or would be dominated by a few. Typical uses: technology forecasts, ranking needs within a sector, setting criteria for a programme, validating a set of indicators, estimating parameters for which no statistics exist.

It does not fit when the answer lies in data or can be obtained through a representative survey: experts are no substitute for measuring the behaviour or opinions of a population directly. It is also the wrong choice when the decision needs an open exploration of a new topic, where in-depth interviews with experts yield more. And it does not suit a one-week deadline: several rounds, with time to respond and to analyse in between, usually take a few weeks.

4. The expert panel: who, how many, how

Panel composition is the decision that matters most. A panel assembled from whoever is available, or only from people who share the client's outlook, produces a predictable consensus of no value. Selection starts from a map of the relevant kinds of expertise: for a vocational training strategy, for example, employers from several industries, education specialists, representatives of public authorities, researchers and employee organisations. For each kind, explicit criteria are set: experience, role, publications or projects, first-hand knowledge of the subject.

There is no standard size. Homogeneous panels can work with a small number of experts, while panels that must represent several perspectives need more, so that each group carries weight. Size is planned with attrition in mind: some experts stop responding after the first round, and losses are greater on long studies. Recruitment is done in person, explaining the purpose, the number of rounds and the time required, because early commitment reduces dropout.

In a small country such as Moldova, experts in a given field often know each other. Anonymity needs extra protection: arguments are reworded so their author cannot be recognised, and the panel list is not published before the study ends unless the participants agree.

5. The rounds: how the questionnaire is built

Round one can be open or structured. The classic version begins with open questions: which factors will shape the sector, which interventions should be considered. The answers are analysed and turned into clear statements or items. The modified version starts directly from a list built from desk research and interviews, which shortens the study at the risk of locking in the researcher's framing from the outset.

Round two asks experts to rate each item, on importance, likelihood, timing or feasibility, usually on numerical scales, with a short justification for extreme ratings. Items follow the same rules as any good questionnaire: one idea per item, defined terms, no wording that hints at the answer, as set out in our guide to questionnaire design.

Round three, and possibly a fourth, sends each expert their previous answer, the panel's median and range and the anonymous arguments, and asks them to reconsider. Experts who stay outside the central range are asked to explain why. Administration is usually online, which lets experts respond when they have time and protects anonymity, with personalised reminders to limit dropout.

6. Consensus and the stopping rule

Two things are fixed before the first round: what counts as consensus, and when the study stops. Consensus is defined operationally, for example as a minimum share of experts placing an item in the same zone of the scale, or as a spread of answers below a given threshold. The exact threshold depends on the scale and the stakes, but it must be set in advance: chosen after the results are visible, it can be tuned to confirm what someone wants.

The stopping rule combines consensus with stability: if answers no longer change meaningfully from one round to the next, another round brings no new information, only fatigue and dropout. Items that are stable but lack consensus are reported as such. One variant used in public policy explicitly aims not at consensus but at mapping divergent positions and the arguments behind each, because the decision-maker needs to know which options are contested.

7. Analysis: from responses to decisions

Analysis is primarily descriptive: for each item, the median, the spread, the movement between rounds and, where the panel contains distinct groups, a comparison between them. A systematic difference between employers and education specialists, for instance, is a finding in itself, not noise to be averaged away. The qualitative arguments in the justifications are coded by theme and explain why experts think as they do.

The output is a list of priorities or forecasts, with the level of agreement for each, the contested items and the main arguments on each side. For the decision-maker this gives three useful categories: points experts agree on and that can be built on; contested points that need further data or an explicit policy choice; and points judged unlikely or unimportant, which can be set aside. This kind of synthesis is part of our decision support services.

8. Limits and common errors

The most important limit is that agreement does not guarantee accuracy. Experts in one field may share the same assumptions, and a homogeneous panel can converge quickly on a shared mistake. Panel diversity is the main defence.

The second limit is pressure to conform, which does not vanish even under anonymity: seeing the median, some experts move towards it without new arguments, simply to avoid standing apart. That is why consensus is reported together with how it evolved and the justifications given, not only as a final figure.

The common errors are a panel chosen by availability rather than criteria; vague or double-barrelled items that experts read differently; a consensus criterion set after the results; unreported attrition that changes the make-up of the panel between rounds; and a moderator who summarises arguments selectively, tilting towards one position. Ethics require that experts are told the purpose, the confidentiality terms and how their answers will be used, in line with Moldova's Law no. 133/2011 and the GDPR.

9. The Delphi method compared with other ways of consulting experts

Combinations are common: a few expert interviews prepare the item list, a Delphi study rates the items, and a closing workshop takes the contested items to the decision-makers. In development programmes the method sits well alongside a needs assessment, adding the specialists' view to that of the community.

10. What to ask the agency

• A map of the kinds of expertise needed and written criteria for selecting experts.
• The planned panel size, with expected attrition and a plan for keeping participation up.
• The chosen variant (classic, with an open first round, or modified, with a starting list) and the reason.
• The items and scales, and how they will be piloted with a few experts outside the panel.
• The definition of consensus and the stopping rule, fixed before round one.
• How anonymity will be protected, including rewording of arguments.
• Response rates reported by round and by expert group.
• A report that sets out agreement, disagreement and the arguments, not just medians.

The Delphi method does not remove uncertainty, but it makes it visible and structured: it shows where specialists converge, where they part ways and why. For an organisation that has to decide before the data exist, that is the difference between a strategy built on the views of the most influential person in the room and one built on the collective judgement of the people who know the field. To discuss an expert study that fits your problem, see our focus groups and in-depth interviews and strategic marketing consulting services.

See also: Decision support Focus groups and IDI Strategic marketing consulting In-depth interviews with experts and decision-makers How to design a questionnaire Community needs assessment

The Perspicuus research team · 12 August 2025