As a qualified survey statistician, when I made the move from research to business I was flabbergasted by how many surveys people in business were conducting, how appallingly useless the data was that they collected, and how oblivious these people were to this problem.

 

Surveys must be designed with the same kind of rigour that scientists design experiments – but don't freak out! Surveys can be far less complex and intricate and dire than scientific experiments. It's just that if you want to draw conclusions from your survey data, that data must be objective, and objectivity is the product of rigour. A survey will produce objective data if it is designed with all of the attributes of data integrity in mind:

The steps you take in designing a survey are paramount to the degree of integrity of the data you end up with. So do what you can to avoid making the 8 mistakes discussed here (unless you actually do want to invest in a useless survey).

 

 

 

The first way to make sure you waste the time and money you invest in your survey is to avoid at all costs defining or stating the purpose or reason for conducting the survey. If someone asks you why you want the survey, change the subject to how many multiple choice questions you'd like to see on the questionnaire. If you were to make your purpose clear, you would run the risk of having to drop some of those interesting questions you want to ask, and limit the survey just to those questions that would provide useful data.

 

Don't bother listing and defining the specific objectives of your survey – because if you do, you'd have to check if the questions you want on the questionnaire are actually going to collect the right data to meet those objectives. To ensure your survey turns out to be as irrelevant and uninformative as possible, use brainstorming to generate your list of questions, or copy the questions from another survey.

 

 

 

If it isn't possible for you to make mistake 1, there's still hope for sabotaging your survey. Leave who or what your target population is undefined and vaguely described. Avoid listing the criteria or specific characteristics that define that target population. If you are as specific as the following example is, then you might have to put effort into working out which people to survey and which to not survey.

 

Isn't it easier just to survey everyone in whatever list you already conveniently have? Or just those that are the most obvious? Or those that you “choose” to survey? It sure is. And if you don't articulate your target population, who's really going to know anyway?

 

 

 

It couldn't be any easier or cheaper to render your survey data useless than to exploit the biasing influence that mail out surveys offer. Mail out surveys are usually posted or emailed to respondents, and they are thus very inexpensive. Because of the low response rates typical of volunteer surveys (particularly when no incentive is offered), mail out surveys result in large bias in the data, if those that respond are in some way different to those that don't respond.

 

The methods that allow you to control for and reduce bias are more expensive anyway. Well, at least it looks that way when you measure cost by dollars per respondent. If you measure cost by the more complex ratio of the value of the data relative to the total cost of collecting it, you'd run the risk of having to adopt one of the following methods instead:

If you find yourself in the position of not being able to do the mail out option, you can still stuff up your survey quite successfully by choosing one of the above methods without considering the type of respondents in your target audience, the type of questions you will ask, the number of questions you will ask, your budget and the level of confidence and precision you want in your survey data. Just make a wild choice and hope for the worst.

 

 

 

If you were a survey goody-goody, then you'd no doubt translate your survey objectives into a suite of questions for respondents that will most likely get the data you really needed to meet those objectives. But if you're a survey rebel, you'd instead ask any question that was in some way interesting or potentially useful, and you'd be obliging to anyone else who wanted to tack their interesting questions onto the questionnaire too.

 

In the quest to produce the worst survey with the least effort, you'd completely ignore the impact that any of the following factors have on how logical, intelligible or possible it is for respondents to answer those questions:

Design your questionnaire with the aim to fit as much information on each page as possible, if your intention is to render the data next to useless. For example, you can more easily destroy the readability of the collected data by not allowing enough space for respondents to provide answers to the questions (this is how many questionnaires magically fit onto a single page). Also, try sequencing your questions illogically, making everyone answer all the questions, even those that are irrelevant. And if you use multiple-choice, make sure that you frustrate respondents with options that you came up with off the top off your head. Don't explain the meaning of acronyms or special terminology you use. A really good trick is to ask a question like “Do you prefer A or B?” with instructions to tick “yes” or “no”.

 

 

 

Whenever a survey is based on a sample, the data collected cannot tell you the exact answer - only an estimate. The confidence you can put in that estimate increases as you increase the sample size. Most people are conscious of this. But if you are like most people, your beliefs about how big the sample size needs to be will be wrong.

 

That offers up another strategy for engineering your survey to waste your money. Sample sizes have virtually nothing to do with population size, and almost everything to do with how much reliability you want in the survey estimates, how much budget you have, and what kind of data those estimates are based on (eg percentages versus continuous data like income or revenue). To avoid making wise decisions about sample size (it's going to take a little bit of effort, after all), you can instead adopt one of the following dubious approaches to decide who to send the survey to:

The most useful data comes from samples that have been randomly selected. Random selection means that every potential respondent in your target population has the same chance of being selected into your sample. This means you won't get bias in your sample design. So avoid random selection and sensible sample size design if a ridicule-worthy survey is your goal.

 

 

 

The way a survey is conducted has a significant impact on the integrity of data you end up with. To flush this integrity down the toilet, you can fail to specify how the following activities are to be performed and just let it happen “naturally”:

 

If you want to save time and don't care about the usefulness of your survey data, then don't bother testing your survey questionnaire with a small number of your target population to fill it out "as if for real". You see, if you were to pilot test your survey, you might have to go back and change things if it didn't run as you wanted or deliver the kind of data you needed.

 

There's more you can do to warrant your survey data useless. Yes, you can pretend it is perfect when you analyse it, and take its analysis as gospel. Firstly, don't do any of the following things to make it analysis-ready:

Secondly, to get the least information as possible from your survey data, analyse the survey data using just averages or percentages and don't bother with other statistics that give you more context (such as ranges, standard deviations, modes and medians).

 

 

 

It really isn't too hard at all to produce a survey that produces nothing of value. In part it is because most people have really no idea that survey design and implementation is a science, and there are scientists that specialize in how to do them well. It isn't rocket science; it's the field of statistics. Doing surveys well means adopting a few simple principles:

If you have any doubt about how to put these principles into practice, seek some help from a statistician, or read more about simple sample surveys. You'll be glad you did because your resulting survey results will be defensible, cost effective, and actually answer your objectives.

 

 

 

Stacey Barr is a specialist in performance measurement, helping people to move their business or organisation's performance from where it is, to where they want it to be.