We Have Always Done It That Way

I am going out on a limb here, because I am not an expert in research methods, but I think it is time that we stopped using the standard evaluation forms at conferences. I started my career in the conflict resolution training business, and at the end of each of our training events, we gave our participants an evaluation form. It asked five or ten questions about the quality of the event, the instructors, etc., using a five-point Likert scale. We compiled the scores and included the data in our report to the funders.

Can anyone guess what the scores were? Around 4. There was slight variation (down to maybe 3.5), as some groups were more or less impressed with the venue or the instructors or the content. I learned early on that the instructors derogatorily referred to these evaluation forms as “happy sheets.”

Flash forward fifteen years, and here I am speaking for the association community, anxiously waiting to get back my evaluations to see if I am staying above the magic 4.0 line.

Stop. Throw these forms away. Never use them again. Go back to the drawing board and ask again (or maybe for the first time?), WHY are you using these sheets? I am guessing that the standard answers are things like, we need to compare the quality of different speakers (or the same speaker over time), or we want to know if participants are satisfied with their experience.

Those are laudable goals, but are the happy sheets really getting you there?

Quality of speakers should be based on the impact they have on participants, which is not always measurable at the end of the session (or even a few days later). Some speakers (dare I say, some of my fellow authors!) design sessions specifically to provoke new thinking in the audience. This can be uncomfortable for the participant in the moment, but immensely valuable over the long term. Other speakers need to deliver specific content to help people accomplish a specific task. One person can get tremendous benefit from both sessions, but in the moment rate the provocative session lower than the “just what I needed” session, whose impact is more immediately apparent. Happy sheets don’t tell you who your good speakers are.

Happy sheets don’t tell you much about the participants’ real experience either. As Amy Smith wrote in the first edition of the book, there are critical questions you need to ask when designing your learning experience in the first place, like what are the business problems of your participants that you can help to solve. With those in place, you should design some research to see whether or not your event made progress against those goals. You may find that more qualitative tools will be more effective than happy sheets. Do interviews with participants. Sit in and observe the speakers. Have a session at the conference where a facilitator can have a back-and-forth conversation with participants about what is working or not working at the conference (entice them with some good break food!). Gather data six months later in addition to right after the meeting. If nothing else, you should at least experiment with some of these methods.

The happy sheets provide relatively instant feedback, so I know they are “satisfying” to an organizer. But that is about you. Your event evaluation should be about the event and the customer, and it should be focused on learning, not earning a score. Approach this as a research issue, and design your evaluation research so it will generate learning, which then leads to experimenting and changing the way you do things.

Tags: The Way We Execute, We've Always Done It That Way // 2 Comments »