#7 in our training evaluation blog post series:

I was reading an interesting article recently called, “Are you too nice to train?” by Sarah Boehle and she included an interesting case that I’d like to share:

Roger Chevalier, an author and former Director of Information and Certification for ISPI, joined the Century 21 organization as VP of Performance in 1995. The company trained approximately 20,000 new agents annually using more than 100 trainers in various U.S. locations. At the time, the real estate giant's only methods of evaluating this training's effectiveness and trainer performance were Level 1 smile sheets and Level 2 pre- and post-tests. When Chevalier assumed his role with the company, he was informed that a number of instructors were suspect based on Level 1 and 2 student feedback. Chevalier set out to change the system.

His team tracked graduates of each course based on number of listings, sales and commissions generated post-training (Level 4). These numbers were then cross-referenced to the office where the agents worked and the instructor who delivered their training. What did he find? A Century 21 trainer with some of the lowest Level I scores was responsible for the highest performance outcomes post-training, as measured by his graduates' productivity. That trainer, who was rated in the bottom third of all trainers by his students in Level I satisfaction evaluations, was found to be one of the most effective in terms of how his students performed during the first three months after they graduated. "There turned out to be very little correlation between Level I evaluations and how well people actually did when they reached the field," says Chevalier, now an independent performance consultant in California. "The problem is not with doing Level 1 and 2 evaluations; the problem is that too many organizations make decisions without the benefit of Level 3 and 4 results."

Industry studies appear to support his words. A 2009 ASTD Value of Evaluation research study found that 91.6% of the organizations in the study evaluated training at level 1, 80.8% at level 2, 54.6% at level 3 and 35.9% at level 4. 4.1% did no evaluation at all! Of the 91.6% that evaluate at level 1, only 35.9% said this level had high or very high value. Yet of the 36.9% of organizations that evaluated results (level 4), 75% said this level had high or very high value.

ASTD’s findings are somewhat alarming because they suggest that the majority of these organizations are going no further than level 1 evaluation, if they’re evaluating at all. We could assume from this data that the level 1 information gathered by these organizations’ training teams is the primary or maybe the only measurement used to justify their training efforts. Qualitative data and comments get rolled up into an overall total and used as a benchmark to measure the effectiveness of the trainers and the training programs being offered. Level 1 is used in isolation with no knowledge or thought about how the training programs address (or don’t address) key business needs. So why do companies do this?

I agree with Boehle’s theory that it comes down to two factors. First, Level 1 “smile sheets” are easy to do while levels 2, 3, 4 and 5 may appear to be costly, time consuming and potentially confusing (where do we start? How do we do it?). Secondly, if stakeholders (e.g. CXOs, internal clients and business partners) don’t demand accountability, why evaluate further? Digging in further may uncover negative results - if all appears to be working well on the surface, no one is asking questions and learners are happy, why rock the boat?

It’s been our experience that the best practice training and development teams recognize that they have a responsibility to ensure that the programs they produce and deliver are aligned with the organization’s needs – to demonstrate how training is contributing to the success of the organization. They need to show proof that training is really making a difference - clearly identifying how organization’s bottom line is being positively impacted and how business needs and issues are being addressed. Using only level 1 data to measure training and trainer effectiveness is dangerous and tells very little about how much learning is actually taking place on the job and how business results are truly being impacted. And sooner or later, this will catch up to the training providers and ultimately to the organizations they work for. Training budgets will be cut, work will be outsourced, and organizations will struggle to keep up with their competition in a tight and highly competitive economy.

Be sure to check out our other evaluation blog post in this series: