Happy Christmas!

Happy Christmas to you all from Austin Research.

Here’s hoping for a prosperous 2010.

Even Santa can't escape!

Picture shamelessly stolen from an email from my friends at Protel Fieldwork.

Beware Randomness! A Warning for Market Researchers

As regular readers of my blog will know I have recently immersed myself in two books that discuss how randomness affects our everyday lives (Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets by Nassim Nicholas Taleb and The Drunkard’s Walk: How Randomness Rules Our Lives by Leonard Mlodinow).

These books have prompted me to consider how randomness affects the world of market research. As researchers our raison d’etre is to make sense of data but if there is an element of randomness, as there is bound to be with any dataset, how does it affect what we do?

When reporting data we use the concept of a statistical difference to ascertain whether or not the disparity between 2 figures is a real difference or due to chance. However, we commonly test for significance at the 95% level meaning that for every 20 times a difference is shown as significant, 1 of these differences is a mirage, due purely to chance. Therefore, 1 time in 20 randomness in the data misleads us.

In my agency days I can remember a few times when I have been preparing a report or presentation and came across figures which seemed to be anomalies compared to the rest of the results. We’d verify the data again to make sure it was OK and look at the question wording and its order within the questionnaire to check if there were any reasons why the data could be different from the norm. When these checks were made and didn’t unearth anything we’d then discuss possible reasons for the inconsistency and invariably come up with a theory to justify the figures to the client. Although, between us, we sometimes theorised that the strange figure could be due to randomness this isn’t something that tends to go down well with clients. We are paid to explain what data means, not make excuses, but our inability to acknowledge the possibility of randomness in our data has the potential to lead to flaky interpretation.

Another danger of randomness is the human tendency to try to make sense of it. People like to be in control and, by definition, we are not in control of randomness and try to construct theories to make sense of it. This can be especially dangerous for market researchers. When interpreting a set of data there is a danger that researchers can, either consciously or sub-consciously look for patterns that provide evidence of their hypotheses and ignore contradictory data (what psychologists call the “confirmation bias”). This applies to both quantitative and qualitative data. Indeed, it could be argued that qualitative research is more likely to be subject to this sort of confirmation bias due to the smaller number of respondents typically involved in a study. It may also be the case that moderators or interviewers are sub-consciously looking for information during the fieldwork phase that reinforces their or their client’s hypotheses around the subject they are researching.

Market researchers therefore need to beware of randomness. It is vitally important that researchers have the ability to keep an open mind and recognize that chance events produce patterns that are open to misinterpretation. We should question our perceptions and theories, spending as much time looking for evidence that they are wrong as we spend searching for reasons they are correct.

Football Management – A Matter of Luck?

I have just finished reading Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets by Nassim Nicholas Taleb and am currently reading The Drunkard’s Walk: How Randomness Rules Our Lives by Leonard Mlodinow. Both books, as the titles suggest, look at how random events impact on both our personal and professional lives. Taleb is a stockmarket trader and part of his book focuses on how traders may become successful through luck rather than judgement in the short-term but can be undone by “unforeseen” events, as has been proved by the recent global financial meltdown. Taleb argues that most traders tend not to make allowances for extreme events (e.g. the terrorist attacks of 9/11) which, although unlikely to happen, will happen every now and then. Taleb calls these events “Black Swans” – seeing thousand of white swans and no black ones does not prove that all swans are white but seeing one black swan proves that not all swans are white.

Another profession where random events can heavily influence success is football management. There are many factors that managers have little or no influence over at any point in time – injuries to players, refereeing decisions, playing squads inherited from previous managers and even the presence of beach balls on the pitch. A manager’s career can therefore be heavily reliant on luck. Young managers enduring a period of bad luck at the start of their careers can quickly find themselves under pressure and out of a job despite having the talent required to be successful in the long term. The stigma of failure can be hard for a manager to shake off making it difficult to find another club prepared to take a chance on them. Conversely, new managers can enjoy a period of good luck at the start of their careers building themselves a reputation that can sustain them through future periods of poor performance.

Iain Dowie

Iain Dowie is manager for whom success came quickly but who’s career in football management has now dramatically tailed off, just a few years after it started. Dowie’s first club in football management was at Oldham Athletic. In May 2002 he stepped up from assistant manager following the sacking of Mike Wadsworth. In his first season he led the club into the Division Two play offs after spending heavily on players. However, the club then hit financial trouble leading to the loss of many first team players and Dowie and his squad were not actually paid for several months. Dowie eventually resigned from the position in December 2003.

In his time at Oldham, Dowie benefitted from the misguided decision of the board to spend heavily. This allowed to him to build a successful side. The club were then unable to sustain this investment but Dowie left with his reputation intact as the club’s financial difficulties rather than his own ability were seen to be the cause of the decline of the club’s fortunes.

A couple of days after leaving Oldham, Dowie joined Crystal Palace, a team occupying 19th place in the Championship. Under Dowie’s leadership Palace won 17 out of the next 23 games reaching the play-offs. They then beat West Ham in the final earning a place in the Premiership.

Palace lasted just a season in the Premiership although they were only relegated on the last day of the season. Back in Division One they managed to reach the play-offs again in the following season although they lost heavily to Watford in the semi-finals (managed by another up and coming young manger, Aidy Bothroyd).

Dowie’s success at Crystal Palace can be linked to one player in particular. Up until Dowie arrived at Selhurst Park Andy Johnson’s career had failed to take off with previous managers at the club often playing him on the wing. At centre forward, Johnson scored 32 goals in Dowie’s first season followed by 21 goals in the Premiership, making him the second highest scorer in the league that season. Johnson stayed for another season at Palace after relegation but eventually left to join Everton for a fee of £8.6 million. Johnson has since established himself as a Premiership centre forward with Fulham reportedly paying over £10 million to secure his services.

Dowie has called Johnson the best player he has worked with and was fortunate that Johnson was part of the Palace squad when he arrived. However, Dowie must take credit for galvanising the player and getting the most from him.

Dowie’s reputation was on the up. He was seen as a progressive manager employing a fitness trainer and unafraid of using new techniques which he felt might give his team an advantage. Dowie was one of the few managers who had a degree (a masters in Mechanical Engineering) which helped to build an image of a man more intelligent than his peers. He was also seen as a manager who was a great motivator of players, taking inspiration from books such as Beyond Winning and Chicken Soup for the Soul.

In May 2006 he left Crystal Palace “by mutual consent” and 8 days later he was back in the Premiership as Charlton Athletic’s manager. The Palace chairman, Simon Jordan, was furious, claiming that Dowie had misled him about his reasons for leaving, stating he wanted to be closer to his family in the north of England. Simon Jordan issued a writ, winning a court case against Dowie. Dowie appealed against the court’s ruling and eventually the two parties settled out of court.

Despite having more money to spend on players than any previous manager at the Valley, Dowie only lasted 15 games at Charlton following a disastrous start to the season. As well as the club’s poor form there were also rumours of Dowie falling out with the Charlton board over his methods.

Following 3 months out of work Dowie then joined Championship club Coventry City in February 2007. Despite the club being in a dire financial state Dowie started well, keeping them in the Championship. However, in the following season results tailed off dramatically after a good start. Dowie, was though, working at a club that found itself close to administration. He joined on the understanding that new owners were ready to take over the club. This deal never materialised leaving Dowie with little money to work with in the transfer market.

In February 2008 with the club on the brink of administration, a private equity company called SISU agreed a deal to take over Coventry City. The new chairman, ex-footballer Ran Ranson, quickly released Dowie from his contract after disagreements with the manager over how the club should move forward.

Dowie was undoubtedly unlucky at Coventry. Although the promised investment never materialised the team won a similar proportion of games as Crystal Palace had under his stewardship. His sacking appeared to be more as a result of a clash of personalities with Ranson rather than a reflection on his performance as manager. However, this was the third consecutive job where Dowie had clashed with the board.

At the beginning of the 2008/9 season Dowie found himself in charge at Queens Park Rangers, a club recently bought by the multi-millionaire Formula One tycoons Bernie Ecclestone and Flavio Briatore. Dowie only lasted 15 games and was sacked with the club in 9th place in the league. Dowie’s problems with authority figures seemed to continue with rumours of clashes with Briatore over the club’s transfer policy.

It was another 6 months until Dowie found another job, becoming Alan Shearer’s assistant at Newcastle United on April Fool’s Day 2009. The club won just 1 game in 8 under Shearer’s reign which ended with them dropping into the Championship.

Since then Dowie has not found work at another club and is rarely mentioned when vacancies arise. He is now regularly employed as a pundit on Sky Sports (a sort of poor man’s Chris Kamara) and it may be that his future is more likely to be in the media than in football management.

Dowie’s career illustrates how luck helped to give him the reputation as one of the most promising managers in England and how the lack of it seems to have destroyed that reputation in just 3 years. From his record it’s difficult to tell if Dowie is actually a good manager or a bad one. However his recent associations with failure do not make him an attractive managerial prospect for chairmen or fans.

The big, successful clubs tend to have managers with impressive track records built over long periods of time. Ferguson, Wenger and Ancellotti have proved they are good managers; the length of their careers meaning that their success is due to more than luck. The FA also appears to have been justified in employing Fabio Cappello as England manager. Despite his large salary his record at club level is second to none, proof of his undoubted ability as a manager. In contrast Steve McLaren had spent relatively little time in the game as a manager before becoming England boss and thus represented a big gamble on the part of the FA, one that didn’t pay off.

For smaller clubs unable to attract or afford managers with highly successful long-term careers how should they decide who to employ? For one I think they should be wary of managers who have been successful over a short period of time. Reputations in football can be built very quickly but short term achievements can be due to a large slice of luck. More seasoned managers such as Gary Johnson at Bristol City and Neil Warnock at Crystal Palace with long track records of achievement seem to be over-looked for jobs at bigger clubs, making less appeal to chairmen and fans than younger managers perceived to be on the up. However, the length of time they have been in the game proves their success is due to talent rather than luck. There’s no substitute for proven experience. For every young manager who will become the next Brian Clough there are many who will turn out to be the next Bryan Robson. Chairmen need to look beyond recent results in order to identify real talent.

The relationship between attendances at football matches and ticket prices – some thoughts

I have recently been working on a project for Coventry City which, although not directly related to ticket pricing, has prompted me to think about the relationship between ticket prices and attendances at football matches.

Coventry City’s ticket prices are some of the lowest in the Championship (the second tier of English football). An adult season ticket for the current season cost as little as £259 which works out at a little over £11 a game for 23 home league matches. To attend an individual match costs from £19 for an adult if the ticket is bought before the day of the match.

Despite this keen pricing strategy the club have not seen any uplift in attendances this season. Season ticket sales were slightly higher than last season but it’s fair to say that the crowds for individual games have been disappointing.

A sparsely populated Ricoh Arena

Many clubs, including some in the Premier League, are currently finding it difficult to attract supporters against the background of the longest recession on record. On phone-ins and on websites many fans suggest clubs slash prices to fill the empty seats at grounds. During a survey, when asked what else Coventry City could do to encourage them to attend more home games, one supporter said:

“Reduce the ticket prices. Its not rocket science to work out that 30,000 tickets sold at around £24 is much better than 16,000 sold at £32! Show the fans that they matter and are not just a source of funding for the club.”

I’m not so sure that this “charge less and they will come” school of thought is correct. To begin with, for most supporters the ticket price is only part of the cost of attending a football match. Additional funds need to be found for transport, car parking, refreshments, programmes and perhaps even merchandise. Even if ticket prices are low supporters need to find the money to pay for these extras, meaning that lower ticket prices may only have a marginal effect on their total cost of attending a game.

We also recently asked Coventry City fans without season tickets if they planned to see more or less games this season than last season. Encouragingly, a slightly higher proportion said they are planning to see more games (26% of fans) than said they are planning to see less (19%). Interestingly, when asked why they are planning to see more games few fans mentioned the lower prices, rather supporters were most likely to say that their individual circumstances had changed allowing them to attend more games. These changing circumstances included having more disposable income, more time to go to the football, moving nearer to the Ricoh Arena (Coventry City’s home ground) and their children getting older and becoming more interested in football. These are all factors that the marketing department at the club would find difficult or impossible to influence.

However, when those supporters who planned to attend fewer games were asked why, the most often given reason was the cost of going to matches with the perceived poor performance of the team the next most frequently mentioned.

It seems that Frederick Herzberg’s Motivation-Hygiene theory of job satisfaction may be relevant to attending football matches. From my research price seems to be a “hygiene factor” rather than a motivation to attend games. Increasing ticket prices causes dissatisfaction but reducing them does not provide long-term motivation to attend games.

I don’t have any empirical evidence but the primary motivator to attend games must be the performance of the team. The product is a more important part of the marketing mix than price reflected in the fact that the successful teams in the Premiership still sell out their home games despite the current economic situation. Since moving to the Ricoh Arena in 2005, Coventry City have only managed to completely sell out the stadium once, for an FA Cup quarter final tie against Chelsea in 2009, even though the game was shown live on terrestrial TV. In this instance ticket prices were almost irrelevant; fans were attracted by a combination of the success of the home team, the profile of the visitors and the occasion.

Apart from success on the pitch other motivating factors almost certainly exist, perhaps playing in a new stadium with modern facilities (most teams have an uplift in attendances when they move to a new stadium) but a winning team will be far the most important factor in attracting supporters to games.

The life of a football marketer is a difficult one. Although ticket pricing is within their control (unlike, for example, team performance, supporters’ disposable incomes and the location of the stadium) it appears to be a blunt weapon to use to attract supporters. Price cuts may lead to temporary improvements in gates but they will be unsustainable without success on the pitch. They can also create an expectation of low prices and promotions with supporters becoming resentful if they don’t get them.

Austin Research and the new Dyson Air Mulitplier Fan

On Monday 12th October, to great expectation, Dyson unveiled their latest innovation, the Dyson Air Multiplier Fan. It is poised to revolutionise the market for fans as it has no blades or grille and produces a constant stream of air, unlike conventional fans.

Are you a fan?

During the development of the product Dyson commissioned Austin Research to undertake various quantitative usage and attitude studies with fan users in the UK, US, Australia, China and Japan. The research was carried out online and designed to understand how people used their fans and the frustrations they had about using them. The results from the surveys were fed into the design process and used to help shape the development of the Air Multiplier Fan.

For more details about the fan go to http://www.dyson.co.uk/fans/ where you can find a great video of people’s reaction to the fan when they see it for the first time.

The Eventually Event Driven Survey (courtesy of Severn Trent Water)

 

On the 16^th May 2008 I had a problem with my water and an engineer from Severn Trent Water came to look at the supply to my house to check that everything was working OK at their end. Despite it now being August 2009, 15 months after the event, I know the exact date as I have just received 2 identical letters from the water company asking me to fill in a questionnaire about the May 2008 appointment. Funnily enough, due to the time lag I can’t remember if the engineer showed his company ID or if he had a friendly and polite manner. As a result I am having trouble completing the survey.

As a market research consultant and as a customer of Severn Trent Water I am angered by the pointlessness of this exercise. I would confidently guess that the response rate to this survey will be incredibly low due to the delay in sending out the questionnaire and any responses received will be worthless as recall of the event will be minimal. Through my bill I am paying for the management of this survey and the processing of the resulting data yet Severn Trent have announced that they will raise average water prices by 4% above inflation by 2015. Perhaps a quick look at their internal processes might reveal some areas where efficiencies can be made.

Other gripes I have with the survey are:

• The letters sent were addressed to “The Occupier” rather than to me directly which makes them look like junk mail. They know that I am the account holder (they use my name for the bills) so why not address it to me in particular so that I will be more likely to both open and respond to the letter.
• You have to write your reference number from the letter sent to you on to the postal questionnaire along with your name and address. In this day and age I don’t think it’s asking too much for these details to be pre-printed onto the paper questionnaire.
• There is no incentive to complete the questionnaire. There isn’t even any explanation as to what the resulting data will be used for. Therefore, why would anyone bother returning the survey?
• The questionnaire looks as though it was designed by someone on a week’s work experience rather than by anyone with any knowledge of market research:

In the interests of balance, there are things that are done relatively well. A pre-paid envelope is included so that the completed questionnaires can be returned without cost to the respondent and there are also options for completing the survey online or by phoning Severn Trent’s customer service centre.

However, timing is all with this type of event driven research. Any survey, no matter how well designed, will be a worthless exercise if not sent out very soon after the event. As such, a delay of more than a year is a brazen waste of bill payers’ money.

Getting the best out of Survey Monkey – 10 tips for better surveys

 

Survey Monkey has now been in existence for over 10 years and has allowed a wide range of people and organisations to carry out market research they might not have otherwise been able to afford. Although the software definitely has its limitations I have successfully used it for relatively simple surveys up to around 20 questions in length.

Below are 10 tips for getting the best out of Survey Monkey in terms of making your surveys as respondent friendly as possible. The more your surveys are designed with respondents in mind, the higher the rate and better the quality of response.

1. Subscribe to the Professional version of Survey Monkey. It costs a bargain $19.95 a month (less than £15 at current exchange rates) and, amongst other features, allows you to include skip logic in your questionnaires, add a logo to the survey template and cross-tabulate the resulting data. In any case the Basic version only allows a maximum of 10 questions and 100 responses so any serious research will need the paid for version. You can subscribe on a month by month basis so you only pay when you are using the software. (By the way, I have no connection with Survey Monkey and, unfortunately, do not get any commission on any sales I may generate!).

2. Improve the look of your questionnaire. It never fails to amaze me that large companies who have all their communications professionally designed with a consistent feel are happy to send out questionnaires designed in Survey Monkey that look awful and reflect badly on their organisation. Questionnaires should be as clear, uncluttered and as easy to read as possible.

Although you don’t have full flexibility to make the questionnaire look exactly as you want within Survey Monkey you can edit the themes available to you if you subscribe to the Professional version. You can change a number of aspects of the way your survey looks including the font size and background colour. You can also add your company logo to the top of each page. If you have some basic HTML knowledge you can also format text within a question, for example underlining or emboldening certain words within a question.

You can find the “Edit themes” button if you click on “Edit Survey” when within the “Design Survey” area.

3. Split your questionnaire into different pages. I see a large number of Survey Monkey questionnaires where the survey is presented on one page and the respondent has to scroll down to continue with the questionnaire. As well as putting the respondent off, answers to questions may be influenced by respondents looking down the survey at future questions. Ensure that the questions on each page fit onto a screen so that respondents do not have to scroll down. As a guide no more than 3 questions should be on each page.

When designing your questionnaire you should see a button called “Split Page Here” after each question. Make sure you use it. This feature is available in the Basic version of the software.

4. Use Skip Logic so that respondents only answer questions relevant to them. Something else that infuriates me is a long questionnaire that includes instructions such as “If you have answered No to Q5 then go to Q7”. Respondents should be concentrating on answering the questions not guiding themselves through the survey. These sorts of instructions are likely to confuse respondents causing them to answer irrelevant questions, miss out ones they are supposed to answer or to close down the questionnaire completely.

The logic you can use within Survey Monkey is relatively unsophisticated. Questions, or blocks of questions, can only be filtered from the previous question. Combinations of answers from one or more questions cannot be used as filters at any time.

Splitting the questionnaire into pages means that respondents can skip pages without ever seeing the questions they should not be answering.

You can add skip logic for each question when designing the questionnaire. Skip Logic can only be used by subscribers to the Professional version of the software.

 5. For open-ended questions increase the size of the box where respondents type their answers. By default the box that appears when adding a “Single Textbox” or “Comment/Essay box” question is fairly small. Small text boxes encourage respondents to give short, shallow answers.

Clicking on “Edit question” for an individual question presents you with a number of options. Scroll down to the “Change question size and placement” option and check the box next to it.  This brings up a number of options. Using these ensure the box is as wide as possible (100 chars wide) and increase the number of lines to a reasonable amount, I would suggest 5 or 6. This feature is available in the Basic version of Survey Monkey.

6. With a few exceptions, make each question a must answer question. If this is not done respondents can skip questions as they wish making the data incomplete. However, it may be the case that you are happy for respondents to skip some questions where they might not have an answer i.e. is there anything else you’d like to tell us about product X?

Clicking on “Edit question” for each question allows you to check the box “Require Answer to Question”. The default is for this box to be unchecked. Again, this feature is available in the Basic version.

7. Include a “Thank You” page at the end. Once the respondent reaches the end of the questionnaire they should reach a page that thanks them for completing the questionnaire and provides details for someone they can contact if they have any queries about the survey. Including a “Thank You” page is good manners and reassures respondents that the survey is a genuine piece of market research.

These contact details should also be provided in any introduction to the survey either in an email invite or at the start of the survey. The introduction should also explain the purpose of the survey in order to maximise response.

To create a “Thank You” page add a new question and choose the question type “Descriptive text”. You can then type text as you wish.

8. Make sure the survey links to a relevant site on completion. By default when respondents finish the survey they land on the Survey Monkey website. Professional subscribers can change this within Survey Monkey to any website of their choice. This is a great chance to guide respondents to a relevant website and, at the very least, increase traffic to that site.

The facility to change the end page link is available in “Change Settings” when in the “Collect Responses” area.

9. Unless there are good reasons not to, allow only one response from an individual PC. This will stop rogue respondents going into a survey a number of times to complete it. Respondents may be tempted to do this if the survey has some sort of incentive e.g. entry into a prize draw, or if an individual respondent cares so much about the subject of the survey that they wish to bias the results in some way. However, if you expect different respondents to complete the survey from the same PC then you should allow multiple responses from one PC.

 In “Change Settings” within the “Collect Responses” area there is “Allow multiple responses”. By default this does not allow multiple responses per computer but it is worth double checking this before sending out a survey.

10. Test, test and test the survey repeatedly. Mistakes in questionnaires reflect badly on the organisation associated with the survey and impact negatively on response rates.

Go through the survey many times yourself before allowing real respondents to access it to check everything makes sense and is working as it should do. You will need to temporarily allow multiple responses from one PC to do this. Also, get others to go through the survey, preferably, those for whom the survey is relevant e.g. if you are sending a survey out to football fans, get a handful of football fans to go through it before you send it out.

Once you are happy with the survey make sure you clear any data you have collected through testing (an “eraser” button is shown next to all of your surveys when you are in the “My Surveys” area. Click on this button to clear data for a particular survey). Also, make sure that multiple responses are not allowed from the same PC before sending.

 

You can access a short , simple example survey which has incorporated these tips by clicking here. This survey does allow multiple responses from one PC so you can go into it as many times as you like.

It should be remembered that these tips are specifically related to Survey Monkey and do not cover the specialist areas of sampling, questionnaire design and data analysis. If you do not have expertise in these areas then it is unlikely that any survey you design will meet your objectives and probable that resulting data will be useless or even dangerously misleading. If you do not have market research expertise but budgets are tight consider hiring a market research consultant who can help design your survey (perhaps someone like myself – details at www.austinresearch.co.uk!).

Does the Market Research industry do enough to help itself?

 

Spend on market research has traditionally held up well during times of economic downturn. However, most forecasts and indicators suggest that the current recession is having a significant negative impact on spend on research.

Why should this be? One of the reasons given for market research previously escaping the ravages of recession was that the information provided by research was even more valuable during uncertain times where it is crucial that the correct business decisions are made. Surely this still applies during this downturn so what has changed?

With regard to quantitative research I believe that the rise of DIY online research has been a factor in making clients’ research spend ever more vulnerable. Before internet research clients had little option but to hire agencies to collect data using traditional methods such as telephone and face-to-face interviewing as they did not have these capabilities in-house. Now, as long as an email sample for the target audience exists, it is straightforward and, most importantly, exceptionally cheap to carry out an online survey using tools such as Survey Monkey. Bearing in mind the ever-increasing pressure on businesses to cut costs it is not be surprising that increasing numbers of surveys are carried out this way.

If my hypothesis that research agencies and consultants are losing out to DIY research is correct this suggests that the aspects of service outside of data collection that agencies bring to projects, such as survey design, questionnaire design and data interpretation are not valued. I believe that the market research industry needs to engage in a concerted exercise to raise the profile of its practitioners and educate businesses, both large and small, as to the benefits that researchers provide.

Businesses need to be made aware that the consequences of amateur designed research can be more dangerous than not carrying out any research at all. The survey and questionnaire have to be designed properly to ensure that the resulting data is meaningful and without bias. Data interpretation is also an art and is best carried out by experienced, objective researchers. The message needs to be that bad research is, at best, a waste of time and resources and, at worst, a threat to your business.

If we accept that there is a need for promotion of the industry to be stepped up who should be responsible? The Market Research Society already labels itself “the voice of research” and states on its website that “Through its media relations and public affairs activities, MRS aims to create the widest possible understanding of the process and value of market, social and opinion research, and to achieve the most favourable climate of opinion and legislative environment for research.” As the industry body the MRS is in a unique position to raise the profile of the industry in a coherent, credible fashion. I believe though that their current efforts in this area should be stepped up.

 Possible new initiatives could include:

• A PR campaign aimed at relevant businesses explaining the value of properly designed market research while highlighting the pitfalls of DIY research
• A website aimed at those with little knowledge of market research explaining the key principles and providing a how to guide for conducting various types of market research. The website could include case studies showing where good market research has been a major contributor to the success of a business and where bad research and interpretation has resulted in business failure
• Most ambitiously, the appointment of a market research “tsar” who would be responsible for devising a strategy for promoting the industry and implementing it.

Both the PR campaign and website could be introduced relatively quickly and cheaply and be part of a long term effort to change perceptions of market research.

I would be very interested to hear others’ views on this issue. Do you agree with my prognosis? In what other ways would you like to see the industry promote itself?

Avoiding the dangers of DIY market research – a low cost solution

The rise and rise of DIY market research

As online surveys continue to grow in popularity and greater numbers of people become aware of no or low cost data collection tools such as Survey Monkey and Google Docs more and more companies feel that they are able to bypass the market research agencies and carry out their own studies in-house.

In one way this is a good thing – it raises the profile of market research with surveys increasingly used to inform decision making. If a company has internal market research expertise then this is a justifiable means of carrying out cost-effective surveys.

However, the drawbacks of carrying out market research without using specialist market research practitioners outweigh the advantages. Market research is difficult to do properly. Training and experience is required to design a survey that meets business objectives while being a positive experience for respondents. Too many organisations are sending out bad surveys giving a poor image of their company to respondents who can include customers, potential customers, employees and other stakeholders. Most worryingly bad research provides bad data which feeds into bad decision making. Although you’ve spent nothing on the survey it could end up costing a fortune if you make a wrong business decision because of it.

Bad practice in DIY surveys

Increasing numbers of DIY questionnaires are sent to my inbox from a variety of sources. Below are some examples of the bad practice in questionnaire design that I commonly see in these homemade surveys:

Irrelevant questions – questions asked of respondents that assume a certain level of knowledge or experience rapidly turn off respondents and lead to them choosing answers that don’t apply to get through the questionnaire or dropping out of the survey completely.

Non exhaustive answer lists – answer lists which don’t cover all eventualities again lead to poor quality data and drop out

Question bias ­– questions which lead respondents to answers. This can be due to a variety of factors, including question wording and the order in which questions are presented.

Ugly questionnaires ­– many times it seems that questionnaires are put together as quickly as possible and no thought is put into how the questionnaire looks. Clear, clean questionnaires get the best quality of response. A real bug-bear of mine are questionnaires where you have to scroll down the screen for what seems like an age to see all the questions (hint – you can split the questions between different pages in Survey Monkey!).

Long, boring questionnaires – respondents have given up some of their time to complete a survey. Don’t repay them by making the survey long and tedious, they will repay you by not considering their answers and possibly dropping out of the survey.

Using jargon – impenetrable industry jargon is often used alienating the respondent. Those who work in an industry often forget that the public do not speak in the same way as they do about their products and services.

A cost effective solution

Bearing in mind the potential hazards of DIY research it is, in my view, essential to involve a specialist market researcher to steer the project away from these dangers. However, market research agencies are relatively expensive. It’s not really surprising that many companies choose to spend little or nothing on a DIY survey instead of thousands of pounds on a market research agency.

However, there is another way. Market research consultants can be hired at key stages of the project. They can help with survey design, giving advice on who should be interviewed, how many should be interviewed and how they should be interviewed, as well as advising on questionnaire design ensuring the questionnaire meets business objectives while avoiding the common pitfalls. They can also be a valuable resource in interpreting the final data. Accurate and insightful data interpretation is an art and essential if decisions are to be made as a result of the survey findings.

The good news is that market research consultants can cost a fraction of the price that agencies charge for managing a project. By involving them at the points of the survey where their expertise is most needed they can be hired for a few days or even hours if required. A relatively small investment can really maximise the value of your surveys, helping you to avoid the adverse consequences of bad market research.