8 tips for creating a successful evaluation strategy

#4 in our training evaluation blog post series:

In this tight economy, organizations want to know that they’re getting their money’s worth wherever it’s spent – and training is no exception. But when it comes to training, many organizations are unable to clearly identify what they are getting in return for their dollars. Research shows Levels 4 (Results) and 5 (ROI) are the two training evaluation levels least integrated into organizations, and yet they can provide the greatest value, as they measure the impact that learning has on the business and determine the return on investment.

So how do you get your company headed in the right direction and where do you start the evaluation process? A good first step is to create an overall evaluation strategy for training. Use it as a roadmap to help you stay focused on the big picture while implementing the details. When creating a strategy for your organization, consider the following tips:

1. Flip the Kirkpatrick evaluation model upside down and start with Level 4 evaluation (Results). Identify key corporate goals and strategies and determine how training aligns with them. Identify stakeholder expectations. What are they looking to achieve as a result of training? This information will drive your evaluation strategy.
2. Get buy-in from the top and commitment from appropriate stakeholders.
   • What strategies can you put into place to "sell" senior leaders and stakeholders on the benefits that evaluation will have to the organization?
   • What senior leaders and stakeholders could be strong champions providing support and commitment?
3. Determine who owns the evaluation process.
4. Identify what training course(s)/program(s) will be evaluated and at what levels. For each course or program, answer the question, "Is there a strong business need for this training?"
5. Identify how each level of evaluation will be measured. What tools will be used to gather the information needed for your evaluation strategy?
6. Determine what resources will be required for each level of evaluation. Levels 1 and 2 can be directly controlled and managed by a training team while participation, time and resources will be required from employees, managers, senior leaders, stakeholders and business partners for levels 3, 4 and 5. Are they willing and able to provide it?
7. Identify challenges and risks for each evaluation level.
8. Talk to other departments to determine what information is currently being tracked in your organization that can be used for levels 4 and 5 evaluation.  Don't reinvent the wheel. Use what's available then figure out if you need to fill gaps with further information.

Evaluation doesn’t need to be complicated. There are a number of fairly simple ways to implement evaluation in your organization that can provide a lot of value. Start by creating your evaluation strategy then stay tuned here for future tips and ideas that will help you implement evaluation successfully in your organization.

Be sure to check out our other evaluation blog post in this series:

• Measuring the Value of Learning
• Our favourite training evaluation resources
• Refreshing the Kirkpatrick Four Levels evaluation model – Kirkpatrick Then and Now

Refreshing the Kirkpatrick Four Levels evaluation model - Kirkpatrick Then and Now

#3 in our training evaluation blog post series:

As I’ve written in past evaluation blog posts, more and more organizations are now focusing on how training impacts business results, wanting to know if they are getting the most for their return on training investment. If you follow evaluation blogs and read articles and research on evaluation, you’ll notice that Training and Development professionals are now looking to improve their learning process by “beginning at the end”. They are first defining business outcomes and determining the desired Level 4 results then working down the model rather than starting with traditional level 1 evaluation and working upwards.

Don Kirkpatrick’s concept of the 4 levels of evaluation - Reaction, Learning, Behavior and Results - was first introduced in the 1950’s during and after his Ph.D. dissertation.  But it wasn’t until after numerous articles and speaking engagements at national conferences over the years that his landmark book, Evaluating Training Programs: The Four Levels was published in 1994. It quickly became a cornerstone for T&D professionals providing a logical structure and process for measuring learning. One interesting note I learned:  Don never called his evaluation concept the “Four Levels”.  Someone else did and it caught on!

I recently picked up the Kirkpatrick Then and Now book by Jim Kirkpatrick and Wendy Kayser Kirkpatrick (2009). Taking the original 4 levels concepts developed by Don, Jim (Don’s son) and Wendy (Don’s daughter-in-law) have created an updated and fresh version of Don’s original model.  The book is an Interesting read, covering the “Then” – a description of Don’s 4 levels as well as his first-hand account of how he developed his model, testimonials from his colleagues and a photo gallery. The “Now” part of the book defines the new five Kirkpatrick Foundation Principles and provides practical advice on how to implement the updated Kirkpatrick model successfully starting from the top and working on down.

So grab a copy of the book, have a read and enjoy learning how to turn the Kirkpatrick model on its head!

Be sure to check out our other evaluation blog posts in this series:

• Measuring the value of learning
• Our favourite training evaluation resources

Musings on mobile learning

Just this morning I attended a thought-provoking webinar with Gary Woodill, author of Mobile Learning Edge.

Near the beginning of the session, Gary mentioned the Marshall McLuhan quote “We look at the present through a rear-view mirror. We march backwards into the future,” and discussed how this is apropos for learning in general and mobile learning in particular. This really struck a chord with me. When I think about it, even in modern times we are always trying to fit the latest advances that come to learning into the box of what we already know. For instance, eLearning can too often be like the PowerPoint presentation we knew from classroom training, trying to recreate that classroom experience in an online environment instead of taking advantage of the capabilities of technology to create truly engaging and interactive experiences that relate to the learning material. Now we see the same happening as mobile learning gains more traction: looking in the rear-view mirror at eLearning and trying to repurpose it for a smaller and more portable delivery mechanism, instead of embracing the true potential at our fingertips.

Gary also talked about the changing definition of mobile learning over the last 10 years. I think that it’s not only the definition of mobile learning that is undergoing change, it’s also our deep-down understanding of learning itself. Yes, as modern people involved in adult learning, we understand that learning is not tied to a classroom, or even a desktop computer, but do we (and the C-levels in our organizations) truly accept that a valid learning event can take place in two minutes using Twitter – for example, to poll contacts for the best way to do solve a problem?

One of the examples Gary gave that really resonated for me, and shows the power we have with mobile learning, is the story of a doctor travelling in Africa who came across a teenager with a severely infected arm due to a hippopotamus bite. The doctor recognized the need for an amputation at the shoulder to give the teen a chance at survival, but he’d never performed such a surgery before. After unsuccessfully trying to reach help by phone, he ended up connecting to a colleague on vacation by text message.  The doctor learned to perform the operation through a series of text messages from his colleague, and was able to successfully amputate the limb. Now if that’s not a poignant example of the value of mobile learning and the need to embrace the tools and technology that can make it happen, I don’t know what is.

Read about Gary Woodill on his website here. I'm looking forward to reading his book!

Our favourite training evaluation resources

 #2 in our training evaluation blog post series:

Just do a Google search for “training evaluation” and you’ll find over 11,000 websites and articles at your fingertips. But you probably don’t have the time to sort through all the information to find what you need. Here at Limestone we’ve run into the same challenges so over time, we’ve created a “favourite” list of evaluation resources that we’d like to share with you:

Evaluation websites/blogs/discussion groups

• www.kirkpatrickpartners.com  Registration is free. Includes access to a weekly newsletter, white papers and articles, free podcasts and webinars, a quick tips podcast series, certification training and other evaluation resources.
• www.greatleadershipbydan.com  An interesting blog post on training evaluation.
• www.roiinstitute.net  Good information on level 5 ROI created by respected author Dr. Jack Phillips.  Includes access to learning opportunities, webcasts and audio conferences, podcasts, articles and tools.
• www.bdld.blogspot.com  Thoughts on instructional design and performance including training evaluation.
• www.astd.org  We highly recommend becoming a member. There are extensive high-quality resources available for both evaluation and T&D in general, and great member discounts apply on courses and publications. Some of the offerings for evaluation include white papers and articles, best practice research and discussion groups.        
• Kirkpatrick Evaluation group on Linkedin.  Some thought-provoking discussion threads on evaluation.       
• www.evaluatetraining.com/blog  A lot of great evaluation blog posts.       

Books/reference material

• Evaluating Training Programs: The Four Levels, Donald Kirkpatrick (the original publication of Kirkpatrick’s 4 levels of evaluation – a “must” read)
• Kirkpatrick Then and Now, Jim and Wendy Kirkpatrick, Kirkpatrick Institute (the 4 levels updated with new strategies and tools)
• The Six Disciplines of Breakthrough Learning, Calhoun Wick, Roy Pollock, and Andrew Jefferson (excellent practical guidelines, ideas and examples on measuring learning from a business results perspective)
• ASTD Handbook of Measuring and Evaluating Training, Patricia Phillips (a great, practical book on learning evaluation)
• ASTD InfoLine issues on a variety of level 1 – 5 topics (an excellent choice of topics)
• ROI in Action casebook, Patricia Pulliam Phillips

Higher learning

• Kirkpatrick Partners - Four levels certification and Four levels certificate programs
• ROI Institute- ROI certification
• ASTD - Essentials of Learning Transfer on-line certificate and ROI Basics on-line certificate

 Be sure to check out our first evaluation blog post in this series:

• Measuring the value of learning

Measuring the value of learning

#1 in our training evaluation blog post series:

Good training evaluation techniques identify and measure what learning has occurred during and after learning, whether job performance improvements have been realized, and most importantly, what the cost-benefits are to the organization. When it comes to training evaluation techniques, Kirkpatrick's four level evaluation model (including level 1-reaction, level 2-learning, level -behaviour and level 4-results) is the most widely used and respected model for evaluation and measurement of business value. Level 5 evaluation, Return on Investment (ROI), is an additional step to training evaluation that has been developed by respected author and founder of the ROI Institute, Dr. Jack Phillips.

I found an interesting statistic in a 2009 ASTD study on The Value of Evaluation: Making Training Evaluations more Effective. 92% of respondents said they measure at least Level 1 of the Kirkpatrick/Phillips module of learning evaluation, but the use of the model drops off dramatically with each subsequent level. The study suggests that organizations may not fully grasp how evaluation should be used. I have to agree with this insight.  Many people I’ve talked to and worked with during my career don’t realize that good evaluation tools and techniques are available that can be used to make their organization stronger. There are ways to implement evaluation that can have a positive impact without being overly complicated or time-consuming. During our recent Kirkpatrick certification, I came across many T&D and HR practitioners who were looking for tools that could successfully work in their organization and processes to make evaluation “easier” to implement.

We, at Limestone, are passionate about the value of training evaluation and would like to share some of our thoughts, strategies, and ideas with you through a series of “evaluation” blog posts.  So stay tuned.

Free June webinars

The month of June offers a great variety of free T&D webinars on some interesting topics including mobile learning which has been gaining ground in the industry and garnering a lot of interest lately. 

Thursday, June 2, 2011: How to Implement Mobile Learning: Practical Considerations

Wednesday, June 8, 2011: How Learning Technologies impact training, sales and marketing functions

Wednesday, June 15, 2011: Supporting a Mobile Learning Workforce: Challenges and Solutions

Thursday, June 16, 2011: Performance "To Go": Where is Mobile Learning Going, How do We Go Too?

Tuesday, June 21, 2011: The Serious Role of Play in Learning

Wednesday, June 22, 2011: How to select an LMS for your small/medium business

Wednesday, June 29, 2011: How to Develop Elegant, Enthralling Learning Experiences

Wednesday, June 29, 2011: Best Practicies in Mobile Learning Authoring

Thursday, June 30, 2011: Using Courage to Transform the Workplace (click on View All Webinars)

Stephanie and I will be signing up for a number of the June seminars.  Hope to "see" you there.

Our favorite T&D/HR best practice and industry research sites

Companies are looking more closely at the bottom line than they ever did in the past.  In today’s economy, it’s vital that a company proactively looks at ways to continually improve their business. In a 2010 Bersin & Associates High-Impact Learning Culture report, findings show that “…there is a direct positive correlation between a strong culture of learning and business performance improvement; cultivating a strong value of learning is essential to business success. HR and L&D need to win over leadership and management, because it is these groups that will affect the most change. They can train, facilitate and encourage leadership and management, but cannot fully own the quality of an organization’s learning culture”. Best practice companies recognize the importance of continuously developing and training their employees to acquire the key competencies needed to be competitive and flexible in business.

So how does your learning organization measure up to best practice organizations?  Not sure?  We get a lot of requests from clients and colleagues about where to find best practice and industry research reports and I’d like to share our favourite sites with you.  Be aware that while some articles are free for the taking, other articles have fees attached. But with some diligent research and patience, you should find plenty of information to get you started.

Our favourite sites for best practice research:

• ww.brandon-hall.com(Brandon Hall Group)        
• www.astd.org (American Society for Training and Development        
• www.aberdeen.com (Aberdeen Group)        
• www.bersin.com (Bersin & Associates)        
• www.conferenceboard.ca (Conference Board of Canada)        
• www.trainingmag.com (Training Magazine Online)        
• hbr.org (Harvard Business Review)       
• www.i4cp.com (Institute for Corporate Productivity)   

If you have favourite sites for best practice and industry research that I haven’t listed here, please pass them along.  

Could the latest tablet from Apple be the ultimate mobile learning device?

It was hard to miss that Apple’s iPad 2 was released internationally last Friday. If you braved the lines and the cold, you might have been lucky enough to snag one for yourself. Not me yet, but it’s definitely on my wish list.

I wasn’t so sure about a tablet device until we got our 1^st gen iPad, but now I’m sold on how comfortable it makes ‘couch computing’. It’s an easy way to check email, access the Internet and watch videos, whether I’m working in comfort outside of normal office hours or using it for fun. And let’s not forget the apps—hundreds of thousands of software applications that can do everything from balance your bank account to manage your to-do list to playing some Angry Birds or Plants vs. Zombies (I admit it, I’m guilty).

With its even thinner profile and 10 hour battery life, plus the added front and back facing video cameras, I’m wondering if the iPad 2 can finally be a perfect platform for mobile learning. There are already rumours about Apple releasing an LMS for the iPad 2, which would take advantage of the unit’s fast processor, speech-to-text integration and video conferencing.

Think about the possibilities here: having your remote workers take a tracked mobile learning module, then getting on a video call with a supervisor to demonstrate their learning in their workplace. Later, they check into a module wiki to see how others have put their learning into action, or even see or record videos to facilitate knowledge sharing. All from the same device. Pretty cool stuff.

Even without a supported LMS from Apple, educators can take advantage of existing apps to communicate directly with students, post messages and discussions on virtual whiteboards, and conduct “virtual workshops”. As a learning support tool, the iPad is ideal.

I think the main barrier to implementation would be price, as the low-end iPad 2 version costs a steep $519 (16GB of storage and Wi-Fi only). But when you think of the savings of no longer needing to gather the workforce in to a central location and backfill during training sessions, it starts to become a more practical option for the long term. And given we still have a perfectly functioning 1^st gen iPad in our house, I think there are good prospects for a long life from any product from Apple. It’ll be interesting to see what the future holds…

Check out these free T&D webinars

Training Magazine’s online website has a number of free webinars coming soon:

Tuesday, March 22, 2011: Creating High-Impact, Ah-Ha! Learning Moments with Videos, Slides, Webinars and Games. Join Ray Jimenez to learn proven techniques that can be used in preparing and organizing videos, slides, webinars and games to immediately create long-lasting learning experiences and Ah-Ha! moments.

Tuesday, March 29, 2011: The New Science of Leadership Advantage.  Join Judith Blair and Nancy Maresh to learn how leaders expecting more from their workforce need to know what works and what doesn't for creating motivation and replenishing energy. Keys for program design include brain-based skills for overcoming resistance, sparking fresh thinking and resilience and helping to ensure peak performance results.

Wednesday, March 30, 201: Apples & Oranges: Employee Skills Assessment Driving Business Performance. Join Ajay Pangarkar as he explain how within organizations there is an expectation for learning professionals to connect their training efforts (Kirkpatrick Level 2) to business objectives (Kirkpatrick Level 3 and 4).  This session directly addresses the need to develop and implement real-time employee training assessment strategies specifically designed to meet targeted business objectives.

Tuesday, April 4, 2011: Change Happens - Be Ready for It! Change Management for Learning Professionals. Join Lance Dublin to learn or review the basic principles and techniques of change management and communication as they apply to the learning environment. You will take away a working knowledge of the proven I-3 Change Implementation model that draws upon concepts and proven practices from change management, communications and consumer marketing.

If you’re not already a member, all you need to do is sign up at http://www.trainingmagnetwork.com.  Once you're signed in, click on the Home page to view the list of free webinars.  The site also includes links to archived webinars if you've missed past sessions.