Defending Your Event

Measuring the Metrics

MeetingMetrics is software that guides planners in developing events with measurable value. While it offers end-to-end assistance in identifying objectives and planning, measuring and reporting on a meeting, on a basic level it's a survey tool. Customizable questionnaire templates help planners determine attendee needs before the event, measure responses immediately afterward, and calculate business impact and ROI several months later.

New York City-based GuideStar Research, the developer of the software, rolled out lower pricing this spring for small meetings and also introduced an ROI diagnostic service for MeetingMetrics customers who don't have the time and/or expertise to do a full ROI study. Managing director Ira Kerns doesn't expect to make much of a profit on this new service, which runs between $2,500 and $2,900. "My interest is to get it out there," Kerns explains, "and get people thinking about measuring the value of meetings."  

Events often show returns of several hundred percent, according to Kerns, who notes, "Wouldn't you like to go to the CEO or CFO and say, 'We got $6 back for every $1 we spent'?"

For those first setting out to measure, the key is to ask attendees the right questions. "Make them tied back to the objectives," advises Coredelia Hecko, CMP, a project manager with Concepts Worldwide in Carlsbad, Calif. "Most importantly, make sure you have that discussion before the meeting, about really figuring out what those objectives are, so you know what you're measuring."

In early March, Las Vegas reported 340 group cancellations for the year, at an estimated cost to the city of more than $131 million. That's but one city. Seventy-six percent of meeting planners polled by M&C in late March have had to defend meetings or events scheduled to take place this year or in 2010. (See News Research for the full survey results.)

The need to prove a meeting's value clearly has been established, and not just for the 500-plus recipients of Troubled Asset Relief Program funds, which must adhere to Department of the Treasury guidelines and set policies on "reasonable" spending for entertainment, meetings and incentives.

At a time when all spending is facing intense scrutiny, measuring return on investment is increasingly crucial. However, relatively few meeting planners calculate ROI by any formal methods, as M&C's recent survey also illustrates. One possible reason for their reluctance: It's extra work they hadn't planned on doing, says Dr. Jack Phillips, chairman of the Birmingham, Ala.-based ROI Institute and co-developer of the Phillips ROI methodology. For many, he says, the process "takes them out of their comfort level. They're not so good at this, and haven't thought about it, and don't really want to go there."

Others are afraid of the consequences. "There's always this fear of 'Oh my God, our event is going to be killed,' " Phillips says. Planners worry that measuring value will only prove that the investment in the meeting doesn't pay off. But in the vast majority of cases, he says, "there's more value than the organizers think there is. In about 80 percent of the studies we do, we find a positive ROI." In other words, the dollar equivalent of the meeting's benefits exceeds the cost.

Furthermore, planners shouldn't assume a negative ROI is the kiss of death. "Of the 20 percent that are negative, we estimate that only about 10 percent of those are discontinued," adds Phillips. Rather, negative results reveal opportunities to  make adjustments and changes.

Parsing the process
A positive ROI is like a stamp of approval, a tangible number that helps substantiate an event's purpose. "I don't want to be the voice of doom, but I do think that the industry is at a very precarious place," says Ira Kerns, managing director of GuideStar Research and developer of MeetingMetrics software, a measurement tool used by the industry (see "Measuring the Metrics," below). People must think about measuring meeting value to protect and preserve meetings, notes Kerns.

The ROI Institute offers certification courses, so planners can become adept at measuring ROI. Or, a full ROI analysis can be outsourced, either to consultants, a meetings management company or an organization like GuideStar. But regardless of whether a comprehensive study is planned, the ROI Institute recommends certain steps be taken for every meeting to ensure that it drives value.

The following process is based on an ROI Institute hierarchy that places business impact and ROI at the top, above more easily measured parameters. A number of software tools, designed to help collect and process the data, are based on this methodology and hierarchy -- including MeetingMetrics, Knowledge Advisors' Metrics that Matter and Gaelstorm's SeseiROI.

Think strategically. Gauging a meeting's value requires identifying needs and objectives -- and doing so from the outset. The earlier this begins, the easier it is to measure the results. In some cases, this can be a challenge.

"Not all meeting planners have the wherewithal, or even the opportunity, to question the strategy," Phillips says. This is particularly true of those whose roles are logistical rather than strategic. His advice: Try to change that. In his seminars, Phillips typically finds only two or three people out of a few hundred who say they have no influence at all over the meeting content or objectives. "If you're in a position where you cannot affect either one of those, then you're going to have a hard time driving business value, let alone evaluating it," Phillips says. (For advice on this challenge, see "Get Strategic.")

Identify the need. For corporate gatherings in particular, make sure there's a legitimate reason for meeting. Too often, the reason is "we meet every year."

Define objectives. Determine specific goals for the meeting, not just the topics that will be covered. The ROI Institute breaks down these objectives into four categories, which correspond to the levels of their measurement hierarchy.

Particularly important at this stage is defining success, explains Cordelia Hecko, CMP, a Certified ROI Professional and project manager with Concepts Worldwide, a strategic meetings management provider in Carlsbad, Calif. "What would success look like?" asks Hecko. "If we had 50 percent of the people saying they felt motivated, would that be success? It's really getting a more specific goal." Objectives break down into four levels, as follows.

• Level 1: Reaction objectives. How do you want the audience to react? Do you want them to feel motivated? Do you want them to deem the content relevant and useful to their jobs?

• Level 2: Learning objectives. What knowledge will attendees gain? This could be company information, an understanding of new technology, new skills or even meeting new people. The takeaways are the real business value.

• Level 3: Application objectives. What do you want attendees to do the day after the meeting, or the next week or the following month? Often, solid expectations are associated with a meeting, according to Phillips, but they might not be clearly written or defined as such. "So let's put it in there as an objective," says Phillips. "That keeps the speakers focused on that application; it keeps the participants clearly informed about what you expect them to do; and then it gives us a hint about what [data] we might need to capture if we want to follow up and see what they've done."

• Level 4: Impact objectives. If attendees do what you want them to do, what impact would that have on business? The objective for a sales meeting would likely be for the salespeople to sell more. In a customer meeting it might be for customers to buy more. These objectives often relate to sales data, market share data and the like. Or, they could involve less obviously quantifiable data, such as employee productivity or brand awareness. Hecko notes, "If you've really got clarity on your objectives, then you're going to do a good meeting, whether you measure it or not." The measurement, she adds, only proves it.

Goals become dollars
Postmeeting surveys reveal how well the first three objectives are met, while business impact can be measured with follow-up research several months later. To measure ROI, the dollar amount of the business impact -- in sales, productivity or whatever the original impact objectives were -- must be compared to the meeting's total cost.

The further one decides to take the study, the more complex it gets and the more helpful training in the process becomes. Meeting Professionals International and the ROI Institute have established guidelines for converting results to dollars, for instance. But great value can be derived simply by identifying the first two or three levels of objectives, says Phillips, and evaluating success in achieving them after the meeting.

Todd Hanson, president and founder of Catalyst Performance Group in Appleton, Wis., is measuring nearly all of the events his company designs and plans. "We're doing it at different levels," he explains. "So, in some cases we're measuring just reaction. It's not easy; it takes a lot of time and effort to measure a return on investment that's credible."

Jack Phillips agrees that not every meeting needs a full ROI study. "A very complex, complicated, comprehensive study to show the impact and ROI of meeting and events should be reserved for a very few select events," he advises. Meetings that lend themselves to quantifiable measurement, such as sales meetings, work best, he notes, adding, "It should be perhaps the most expensive event you have -- or maybe it's a very strategic one, or maybe it's the one that touches the most people in terms of audience."

Real-world success
Software tools offer questionnaire templates designed to facilitate ROI studies. MeetingMetrics, for example, pulls from a database of about 1,000 customizable survey questions. Ideally, the measurement process begins several months in advance, with needs analysis and pre-event surveys -- what GuideStar's Kerns refers to as the "market research" approach to meeting planning.

But often, the scramble to prove ROI comes too late in the game to set up the ideal framework for measurement. Cordelia Hecko conducted a full ROI study for a client last year, a technology company for which her firm has planned the annual sales meeting for several consecutive years. But the plan to measure came late in the process.

"Ideally, we would have done a survey ahead of time, a pre-event survey to establish some baseline data and also to help shape the agenda," Hecko explains. "It didn't work out that way. So we didn't have as much input into the content of the meeting, but we did do a postevent survey immediately after the program, and a follow-up ROI survey."

Even given the shorter time frame, Hecko believes the process itself was a great service to her client. "I think what we were able to find out for them was where they could make changes and make the meeting even better, which was just as valuable [as the actual ROI]," she points out. "Had they not gone through this process and gathered that information from their attendees, they wouldn't have had that insight."

Hecko's client did have objectives, she says, "but we helped solidify them and make them a little more measurable."

Hecko used MeetingMetrics for the surveys and to gauge the client's success. In many ways, she says, the event was a perfect candidate for a full ROI study. "We have a long-standing history and relationship with the top executives, so they really understood the overall goal of this meeting. As a sales meeting, the whole objective was to impact their bottom line. So connecting those dots between their investment and return was a lot simpler line to draw. We found they were able to add an additional $5 million in revenue" as a result of the meeting, Hecko reports. "It gave them a 200 percent ROI."

Royally complicated?
Clearly, calculating ROI can be a daunting task for the uninitiated. "People throw ‘ROI' around a lot," says Kerns, "but if you had to defend the method in front of the CEO, or in front of a group of financial people who really understood how finance works, I don't know that [many approaches by meeting planners] would be very credible. Jack Phillips' is defensible. You have to learn a little more; you don't need to be a financial guru, but you're going to understand the intricacies of what you have to do to get efficacy." A scientific methodology, he adds, is essential if meetings are to be taken as seriously as, say, advertising or public relations as a way of driving organizational performance.

"A lot of people in the industry measure satisfaction and call it ROI," says Todd Hanson of Catalyst Performance Group. "Well, it really isn't. The ROI Institute methodology is extremely well refined. It's an extremely linear and disciplined way in which to measure results on complicated programs. And in short, it's doable."

Justification for attendees
Meeting Professionals International has been using the MeetingMetrics platform to measure its events for some time, according to Dr. Graydon Dawson, MPI's director of global training systems. For its upcoming events, the association is using the software to do a full ROI study. "Our motive is to prove the value of our meetings in these tough economic times, to ourselves and to the individual conference attendee, by providing attendees with personalized ROI reports," he explains.

To do so, MPI will define areas of measurement for the meeting and, in some cases, also ask attendees to define their own ROI measures. "There is one basic rule about selecting meeting ROI measures," says Dawson. "The measure or measures have to be based on acquired knowledge and skills taken from the meeting experience of the participant that can translate into improved individual performance on the job after the meeting."

Participants will then answer survey questions that quantify their achievements, and rate their confidence level in these responses. They also will be asked to list all costs associated with attending the meeting -- including travel, lodging, meals, transportation, registration and other fees, as well as estimate the cost of their out-of-office time to their organizations.

Calculating and providing the results will really help attendees make the case for meetings, says Dawson.