Is your company also among those whose future decisions are made more and more depending on survey results? If so, have you asked your online panel provider whose opinion you are using to make those decisions? And whether it is really reliable?

Competition of (online) panels in our markets is certainly on the rise. And without doubt, to build a “panel” has become relatively simple. It is also very easy to get lost in big words and flashy presentations. They all sound similar, so it’s time to get curious.

I am going to point out the four most relevant questions you should ask, in order to assess the quality of a panel. These questions define who is in this particular panel and what makes the difference between a real panel and an amateurish address list. I am also providing you with Norstat’s answers to these questions, as also stated in the ESOMAR 28.

Which type(s) of sample sources do you use from which you get respondents?

We place great emphasis on the quality of our recruiting sources and are happy to pay a little more in order to maintain the superior standards you have come to expect from us. Respondents can join on invitation only, which reduces the risk of panel overlap and of attracting “professional” respondents. Members of our panels are recruited for market research purposes only using diverse online and offline methods.

If you provide samples from more than one source: How are the different sample sources blended together to ensure validity?

Every person who is invited to join our panel gets a personalized invitation-code which allows them to register at our panel. Instant quality checks during the recruitment process enable us to exclude peculiar applicants or shut down single recruitment sources as a whole.

Do you have a confirmation of respondent identity procedure? Do you have procedures to detect fraudulent respondents?

The fact that we have an “on invitation only” registration method drastically reduces the risk of forged identities from the start. Before becoming an active panel member, all candidates are required to complete the double-opt-in process. This ensures that the person in question really is who he says he is or is at least the owner of the e-mail address. Having completed the mandatory demographic registration variables, thorough duplicate checks are made.

What steps do you take to achieve a representative sample of the target population?

Since we’re recruiting actively, we have full control over the subscription of single target groups in our panel. We define the required targets thoroughly comparing the sociodemographic structure of our active panel with the general population. In order to avoid source induced biases we diversify our recruitment sources over different channels.

If we understand our research participants, we collect better quality data in a more engaging and motivating way for the respondent.

We cannot pass the topic of data quality

Online panels without diverse recruitment usually consist of members of certain internet sites, users of a particular service and “professional respondents”. Meaning their profiles (could be) similar and their views and opinions on certain topics alike. Plus, they are not motivated in anything else than reaching the easy money via incentives.

A happy panelist is actively taken care of, is invited to different types of surveys within a reasonable timeframe and is motivated to give an honest and direct opinion. Panels with high quality consist of members with different backgrounds, from all parts of the country. Your panelist should range from business owners to kindergarten teachers and from stay-at-home moms to grandmas. It should be representative for the whole population.

This is the kind of panelists and panel you need, to help you set the next steps of your business and base your decisions on. To make it crystal-clear, panels recruited from websites about craftsmanship and hardware do not provide you any knowledge of, let’s say, women who work in services or education sector.

So, be curious and ask your provider rather more than nothing at all. The word “representative” does not mean anything unless it is validated by hard facts. But if the answers are shady, the data is probably skewed and so are your conclusions.