After our last post we have received a lot of comments and questions via email. Thank you very much for all your feedback! Today we’d like to discuss the one question we have been asked most during the last years: What sample size is representative? And how many interviews are recommended to obtain representative results?

What makes a sample representative?

Let’s start with a story that actually has become the founding myth of market research. A century ago, the American journal The Literary Digest started to conduct opinion polls among their ten million readers to predict the results of the presidential elections. In five successive elections their predictions were absolutely right until they massively failed in 1936, even though they conducted about 2.4 million interviews among their readers. To their surprise, George Gallup was able to predict the result of this election correctly with “only” 50,000 interviews.

So, what happened? The Literaray Digest’s sample failed, because their readers weren’t representative for the general population. They had a different age structure, a different average income – and, apparently, different political preferences. On the contrary, Gallup understood, that representativeness is not so much about the sample size but depending on the right composition of the sample. He simply used quotas to make sure, that every group of people was correctly represented in his sample. This break-through discovery was the starting point for market and opinion research as we know it today.

For representativeness it is not the size that matters but the right composition. But is that plausible? In the 1960s, A.C. Nielsen Jr. gave an interesting answer to those, who believed that a higher sample size would increase its representativeness.

“If you don’t believe in random sampling, the next time you have a blood test, tell the doctor to take it all.” – A.C. Nielsen Jr.

Despite its undeniable sarcasm, this quote provides us with a very comprehensible analogy. It doesn’t matter if you analyse a drop of blood or if you take a whole litre of it: the analysis findings will always be the same. One drop of blood perfectly represents all of it.

Why does sample size matter?

Obviously, sample size is still important. But why exactly does it matter? We have to become a bit technical here. Whenever you have a representative sample for a population, by chance some of the target variables may be over- or underrepresented in your sample. Unfortunately, “by chance” means, there is really nothing you can do about it, when collecting the data.

At least, statistical calculations can help you to estimate the likelihood that your error is within a certain margin, e.g. that such deviations from the real value are less than x% at a confidence level of 95%.

• For opinion researchers, a confidence level of 95% is the most common option. Here, your risk is less than 5% that the real value is outside the corresponding margin of error. However, in other disciplines, a confidence level of 99% might be the standard (e.g. in the pharmaceutical industry, as statistical errors can be a question of life and death).
• Given the confidence level, you can calculate the margin of error for each value of a distribution. Let’s say your survey result gives you a market share of 50% and your corresponding margin of error is 3% (at a 95% level), then your risk is less than 5% that the real market share is lower than 47% or higher than 53%.

If you want to reduce the margin of error (given a certain confidence level), you basically have only one choice: you have to increase the sample size. As a rule of thumb: if you want to reduce your margin of error by half, you have to quadruplicate your sample size.

In our example: if you want to reduce your margin of error to 1.5% (instead of 3.0%, all at a 95% level), you have to increase the sample size from 1,000 to 4,000 interviews. These additional 3,000 interviews would narrow the confidence interval. Now, your risk would be less than 5% that the real market share is lower than 48.5% or higher than 51.5%.

By the way, there are many Sample Size Calculators available in the web. For example, have a look at this one.

How good is good enough?

And this leads us to a very important business question: How good is good enough? There is definitely no general answer to it, but we’d like to discuss three scenarios to illustrate possible ways of thinking about it:

• Concept Test: Let’s assume that a company has two alternatives for an advertisement campaign. But which one works better? You would just need to identify the winner and go with it! Assuming that the outcome is not to tight, about 500 interviews can be sufficient (which corresponds to a margin of 4.3% at a 95% level – so the best option should lead with at least 9%).
• Election research: When forecasting the popularity of political parties at elections you’re probably interested in more than individual ratings. You will wonder about which parties could form a coalition to gain a majority. If you have two parties with a 3% margin of error each, it will become quite hard to predict it, especially if the outcome is expected to be tight. In this case you should increase the sample size to reduce the margin of error.
• Subgroups: Very often, in addition to overall statistics you want to analyse subgroups of your sample: Who are these heavy users exactly? How do men differ from women? What kind of products prefer readers of a certain magazine? If you just use a smaller subset of your main sample, the available number of interviews for your subsequent analysis will be reduced as well. In this case you should work with an increased sample size, too.

At the end of the day, the art consists in having enough interviews that allow you to draw dependable conclusions and still being reasonable with the overall costs of fieldwork.

Summary

So how many interviews are recommended to obtain representative results? This question simply cannot be answered. You can have small samples that are very representative and large samples that are not representative at all (very often: “Big Data”).

• Representativeness is about the right composition of your sample. It indicates if your sample gives you the right picture about reality. If it is a bit blurry, it will still allow you to get the big picture correctly.
• The size of a sample defines how clear you can see. If your sample is not representative, a large size will enable you to see very clearly – but it will be a false picture, a misrepresentation of truth.

Speaking in the analogy of a blood test: for a simple test, a single drop may be enough – but if you want to do a complex analysis and perform a series of tests, you may require a bigger sample.