Torey Quinn's List: Evaluate Learning

"Managers are fond of the maxim “Employees are our most important asset.” Yet beneath the rhetoric, too many executives still regard—and manage—employees as costs. That’s dangerous because, for many companies, people are the only source of long-term competitive advantage. Companies that fail to invest in employees jeopardize their own success and even survival. In part, this practice has lingered for lack of alternatives. Until recently, there simply weren’t robust methods for measuring the bottom-line contributions of investments in human capital management (HCM)—things like leadership development, job design, and knowledge sharing. That’s changed. Over the past decade, we have worked with colleagues worldwide to develop a system for assessing HCM, predicting organizational performance, and guiding organizations’ investments in people.

Using the framework we describe here has the obvious and immediate practical benefit of improving organizational performance. More broadly, though, as the links between people and performance come into focus, organizations will also begin to appreciate the long-term value of investments in human capital—and the folly of dwelling on narrow, near-term goals."

"Perhaps the best known evaluation methodology for judging learning processes is Donald Kirkpatrick's Four Level Evaluation Model that was first published in a series of articles in 1959 in the Journal of American Society of Training Directors (now known as T+D Magazine). The series was later compiled and published as an article, Techniques for Evaluating Training Programs, in a book Kirkpatrick edited, Evaluating Training Programs (1975). However it was not until his 1994 book was published, Evaluating Training Programs, that the four levels became popular. Nowadays, his four levels remain a cornerstone in the learning industry.

While most people refer to the four criteria for evaluating learning processes as “levels,” Kirkpatrick never used that term, he normally called them “steps” (Craig, 1996). In addition, he did not call it a model, but used words such as “techniques for conducting the evaluation” (Craig, 1996, p294).

The four steps of evaluation consist of:

Step 1: Reaction - How well did the learners like the learning process?
Step 2: Learning - What did they learn? (the extent to which the learners gain knowledge and skills)
Step 3: Behavior - (What changes in job performance resulted from the learning process? (capability to perform the newly learned skills while on the job)
Step 4: Results - What are the tangible results of the learning process in terms of reduced cost, improved quality, increased production, efficiency, etc.?"

"There are two parts to a survey question: the question stem and the scale for response choices. In many situations, entirely too much time is given to deciding which scale to use. In fact, if the stem is poorly written, then the scale means very little."

"The main goal of training is to ensure that learning is applied to work. But for that to happen, learning must occur at the individual level in the training session. It is the job of the instructor or facilitator to see that learning on this level actually happens. "

"Although it’s likely that most readers are familiar with Kirkpatrick’s four levels of evaluation, a review is sometimes in order. Here’s a look at the advantages and limitations of each level, as well as a case study. "

"Many modern learning programs must show a return on investment to senior leaders or risk being eliminated. Here are 10 tips that provide a basis for measuring ROI.


Learning programs unable to prove their worth are unlikely to survive in today’s complex business environment. Learning is almost always funded on the expectation that its benefits will outweigh its costs, which makes it ever more vital that return on investment (ROI) be measured."

"The argument could be made that much research is being conducted but not always used when conducting ROI studies.

Workplace trainers have called for more applied research and more extensive use of case studies. Although the field is fortunate to have a large and growing body of literature in ROI case studies, there is little guidance on how to benefit from these case studies to improve practice. 

This article provides a rationale and framework for analyzing ROI case studies. The article first reviews the case study format and benefits of using case studies, then examines a recommended framework for case study analysis. Examples of participant reactions to case study analysis are followed by conclusions. "

"When teaching the ASTD E-Learning Instructional Design Certificate programs, I come in contact with countless instructional designers, both inexperienced and veteran, sincerely attempting to create the best e-learning modules possible. Usually, I notice two things:
Their instinct is to do the standard design activities that are suggested by tradition, by the authoring tools, and by example.
Following tradition, doing what is recommended by many authoring tools, and patterning one’s work after many examples in the workplace is going to result in pretty ineffective e-learning."

"Did anyone else read the recent Huffington Post article, “The Surprising Secret to Employee Engagement,” which examines the relational aspects required for sustained engagement?
The author, Matt Tenney, who also wrote Serve to Be Great, made the admittedly apt observation in such a way that it seemed the concept of meeting the emotional needs of individuals had not been previously identified as critical. The article was good, and the concept was valid, but I found myself thinking: What about Maslow? Why is this surprising? "