Survey Science

Effective Exhibitor Surveys

The exhibitor survey is an art unto itself, determining not only the effectiveness of an exhibitor's presence, but whether the exhibitor should participate in the trade show, as well as the demographics and needs of attendees.

Exhibit Surveys, based in Red Bank, N.J., is a leader in the field, and Ian Sequeira, the company's vice president, teaches a course required for those seeking the Certified Trade Show Marketer designation at the annual Exhibitor Show.

In the course, Sequeira stresses the importance of determining attendees' "product interest," because it establishes the size of the exhibitor's potential buying audience at the show. Sequeira recommends creating a list of products and asking attendees to note which ones interest them and which they expect to buy in the near future.

Sequeira says exhibition attendee surveys also should include questions about what role attendees play in purchasing for their companies; what other meetings they attend; what industry brands they recognize; which exhibits attendees recall seeing or liking; and what interaction they had with other exhibitors, along with basic demographic information.

The results should be shared with exhibitors to provide guidance on larger goals: Should we exhibit? What should our show strategy be? What products should we emphasize? How many staff members do we need? What should our graphics say?  

How satisfied are meeting planners with the feedback they get from pre- and postmeeting surveys? Choose one: Extremely satisfied, mostly satisfied, neutral or not at all satisfied.

Survey experts would immediately spot at least five things wrong with that question. Planners who can't identify the five common survey errors -- or, indeed, are dissatisfied with their ability to elicit actionable feedback from meetings-related surveys -- should brush up on survey best practices, particularly given the importance now placed on measuring meeting results and proving the value of meetings, which often is determined by attendee surveys.

Stumped? First, the question asks a respondent to speculate how "meeting planners" in general feel about surveys, not about the respondent's firsthand experience. Such questions that rely on hearsay typically result in bad data and are discouraged by researchers.

Second, the question is vague. What is meant by being "satisfied" with survey feedback? Is the question asking whether planners receive positive feedback that gives them professional satisfaction (i.e., the meeting was a success)? Or is it asking if they're satisfied with the usefulness of the feedback, be it positive or negative? Questions in mail or Internet surveys in particular cannot be ambiguous, because there is no opportunity to clarify a question once the survey has been sent out.

Third, the question is "double barreled," meaning it asks about two things at once: pre- and postmeeting surveys. Imagine that a respondent is satisfied with premeeting surveys but dissatisfied with postmeeting surveys; how can he or she answer the question accurately? This should be two separate questions.

Fourth, the response scale is skewed toward positive responses. Respondents do not have the same gradation of choices to express dissatisfaction as they do to register satisfaction (there's no "mostly dissatisfied" to counterbalance "mostly satisfied"). Response scales should be balanced to get the most objective data.

Finally, the question doesn't allow respondents to indicate that they don't know the answer or the question is not applicable to them. If the question doesn't exhaust the entire spectrum of responses, some respondents will be forced to choose a disingenuous answer, spoiling the data.

"We find that planners, generally speaking, are not research experts," says Ira Kerns, managing director of MeetingMetrics in New York City, which offers survey and other measurement services to clients. That's not a criticism, he quickly adds, it's reality and unrealistic to expect otherwise. But planners who don't hire experts like MeetingMetrics or Red Bank, N.J.-based Exhibit Surveys often end up creating their own paper surveys to hand out on-site, or they use free or low-cost web tools, like SurveyMonkey.com or Zoomerang.com. In those cases, Kerns says, planners' lack of knowledge of survey science results in "a certain amount of ineffectiveness" of the endeavor.

Following are 15 more tips from various experts for creating better surveys.

1. Set objectives first. You have to know what your goals are before you start formulating questions, says Ian McGonnigal, an executive director for George P. Johnson, headquartered in Auburn Hills, Mich., who has helped conduct meetings-related surveys for IBM and Motorola. "A survey must be designed against objectives, because if it isn't, you're spending a lot of time and energy on something that's going to give you very little return," he says.

Figuring out specifically what information you want to obtain also helps keep the survey focused. If planners want to know how to improve next year's meeting, they should ask only about things they're willing to try or change. If, for example, the meeting must be held at the same venue next year, it's a waste of time to ask attendees to assess the location ­of this year's meeting -- especially if nothing can be done about it. If next year's venue hasn't been determined, asking about location could be valuable.
 
2. Ask what you truly want to ask. This sounds obvious, but Kerns says one common mistake planners make is they don't articulate the question that will elicit the data they really want. For example, if an association planner is considering several different special events to add to next year's meeting and intends to charge extra fees for admission to those events, the survey should ask "for which of the following events would you be willing to pay?" instead of "which of the following events would you attend?" or "which additions would you like to see at next year's meeting?" The planner doesn't want to identify events that sound good but rather events that sound like they're worth the price of admission.

Similarly, getting attendees to rate satisfaction with various meeting components -- the registration process, for example -- isn't useful if the survey doesn't also provide a way for dissatisfied attendees to explain their complaints. Dissatisfaction with registration could stem from a technical problem, or lack of support staff or difficulty finding the registration desk. Planners shouldn't simply ask whether attendees liked or disliked various meeting elements -- they also should ask why.

3. Stick to the point. According to Zoomerang's online survey tips, abandonment rates skyrocket after 11 minutes. There isn't widespread agreement about the optimal length of a survey, however. "It depends on how engaged people are," says Nancy Scherden, market research manager for Warrenville, Ill.-based Navistar Inc. "Is it meaningful and relevant to them?" People who have more invested in the survey's subject will be willing to spend more time answering questions.

General wisdom has it that surveys should be as short as possible, with each question correlated to an objective. "Nothing more, nothing less," McGonnigal says. "All nice-to-know questions should be eliminated."

But Kerns says some planners suffer from excessive self-censorship. "Too many untrained meeting planners make their surveys too short and lose many opportunities to gain key insights about needed actions," he notes.

4. Use branch logic. One way to make online surveys shorter -- or at least appear shorter -- is to employ a technique called "branch log­ic," which adds or eliminates questions based on earlier responses. For example, if an attendee indicates that she didn't attend the general session, a web-based survey using branch logic can jump ahead to the next subject, skipping over questions about the quality of the general session speakers. On the other hand, if another respondent indicates that he attended the session and then gave the speakers poor marks, branch logic can insert a question asking for an explanation -- a question that wouldn't appear if the respondent gave the speakers high marks. Hiding questions until they're necessary to ask can give the survey the illusion of brevity, and in some cases, those extra questions won't be necessary to pose at all.

5. Organize well. William M. Trochim, author of The Research Methods Knowledge Base, an online textbook with a chapter on surveys (see resources sidebar at right), says surveys should have a logical flow, investigate one topic at a time and never start with open-ended questions that require survey takers to write out their own response. Kerns adds that planners should put the most important questions first, in case people abandon the survey in the middle. When asking multiple questions about a topic, Kerns suggests starting with general questions and then getting more specific.

6. Handle demographic questions carefully. Eric Eden, vice president of marketing for McLean, Va.-based Cvent, which offers a survey product to planners, says respondents tend to be shy about entering identifying demographic information about themselves. If planners already have a profile of the survey taker in a database, they needn't bother with demographic questions. If such questions must be asked, put them at the end of the survey, Kerns advises.

7. Get intensity data. Many surveyors make the mistake of asking yes-or-no questions instead of "intensity questions," according to Zoomerang. For example, if a planner wants to know what topics to cover at the next training session, surveys should ask how valuable attendees would find a proposed session ("extremely valuable," "somewhat valuable," etc.), not just whether a session would be valuable. That way a planner would know not only which topics garner interest but to what degree attendees want to learn about those topics. A session that gets fewer but more passionate votes might rightfully top a planner's priority list instead of another session that gets weaker support from more people. Without intensity data, planners might choose the session that simply got more votes.

8. Offer a midpoint. For survey designers, asking for answers based on a five-point scale vs. a 10-point scale isn't arbitrary. Odd-numbered scales will have a true midpoint (three on a five-point scale, for example), whereas even-numbered scales will force respondents to edge closer to one end of the scale or the other (choosing between two or three on a four-point scale, for instance, or five or six on a 10-point scale). Kerns says respondents who otherwise would choose a true midpoint tend to opt for the slightly more positive answer when a midpoint isn't available.

9. Achieve question balance. Surveys should have a mix of structured questions (multiple choice, yes or no, rank on a scale) and unstructured questions (write a response), Eden says. The best surveys will have more structured than unstructured questions.

10. Include a proper introduction. Accord­ing to "What Is a Survey," an online resource for nonsurvey experts edited by Fritz Scheuren of the National Opinion Research Center at the University of Chicago (see "More Survey Resources"), all questionnaires should have an introduction that explains what the survey is about and why it's being conducted. Getting respondents to understand the value of taking the survey or the importance of the resulting research will help increase response rates.

11. Avoid junk filters. Designers of paper surveys don't have to worry about this, but creators of online surveys need to make sure their e-mails asking for participation don't get quarantined by spam filters. Nathaniel Estes, a Cvent account executive who hosts webcasts on survey design, says planners can test subject lines and e-mail invites to different e-mail addresses to see if the messages get intercepted. Avoid using words in capital letters in the subject line, as well as promises of "free gifts," dollar symbols, or multiple exclamation points -- all trademarks of spam.

12. Offer incentives and reminders. Gifts or cash giveaways will boost response rates. A common snail-mail survey technique is to send a dollar bill with the survey, guilting respondents into returning the questionnaire. Promises of a Starbucks gift card, for example, upon completion of the survey is another option. Surveyors also can enter all respondents into a drawing for a prize, rather than promising each individual a gift.

Yet another technique to boost response rates is to send a reminder e-mail or postcard. Estes says he's seen examples of response rates doubling after sending out follow-up notes, which are another opportunity to remind the target of the importance of their input.

13. Use premeeting surveys. "The typical view of meeting measurement is the postmeeting survey," says Kerns. "The problem is, that tells you where you are, but it doesn't tell you where you were." Premeeting surveys set the baseline to measure the effect of the meeting. Suppose 90 percent of attendees indicate in postmeeting surveys that they have a positive impression of a certain product brand. What does that tell a planner about the meeting? Did the same percentage have that impression going in, meaning the meeting didn't cause a net change in opinion? Maybe only 75 percent thought well of the brand before the meeting, meaning the event was successful. On the other hand, maybe 98 percent of attendees had a positive brand impression going in, and that sentiment changed after brand managers revealed disappointing new product features. Postmeeting surveys only tell half of the story.

14. Send regret surveys. Eden recommends setting up a one- to two-question survey for people who decline invitations to meetings or events. Eden says Cvent uses such "regret" surveys to find out if people are still interested in the topic or product or service and simply can't attend the meeting, or if they aren't interested at all. "We get 50 percent more leads for our sales team because we offer that," he says.

15. Don't jump to conclusions. Eden warns against reaching erroneous conclusions. If only a small sample of attendees are surveyed, or if response rates are exceedingly small, planners shouldn't assume survey responses are indicative of the attitudes of the entire attendee base. If fewer than 10 percent of all attendees return surveys, it would be a mistake for planners to make changes based solely on that data, Eden says. There's no absolute cutoff for what percentage constitutes a reliable sample. "The higher percentage, the better," Eden says.