What's the problem with Kirkpatrick?

I attended a session at the CIPD's HRD 2008 show in London last week entitled The Value of Learning. The session was presented by the CIPD's Martyn Sloman, John Castledine from Pfizer and Ian Tegerdine from Birmingham Childen's Hospital NHS Foundation Trust.

The aim of the session aim was to 'present a fresh perspective on evaluation', based on work carried out for the CIPD by the University of Portsmouth. While acknowledging that Kirkpatrick's four levels of evaluation had proved valuable over more than thirty years in helping to measure instructor-led, content-based 'training' interventions, Sloman felt that a new approach was necessary to support a more self-directed, work-based 'learning' process.

Now, first of all let me say that I am fully supportive of self-directed, work-based learning, not least because that's what I prefer; but let's not for one moment think that this means there has been any radical departure from the practice for organisations to deliver good old-fashioned top-down training programmes. Surely what's needed is a combination of the two.

It's worth checking first of all whether Kirkpatrick's model is still useful for evaluating the top-down stuff. My assertion is that it is, even though in practice, we don't do that much of it - according to an ASTD survey, 91% of organisations evaluate at level 1 (reactions), 54% at level 2 (learning), 23% at level 3 (behaviour change) and 8% at level 4 (impact on results).

Let's start with level 1. We all deride the 'happy sheets ', but that doesn't mean they are of no importance. If your customers don't like what you deliver for them, then they will be reluctant to participate in future events and they will tell their friends. If you ignore negative reactions, you will be without a job and out on your ear in no time at all. And level 2 is important too. There's no point in having objectives for learning, however loosely defined, if you don't measure whether they are accomplished. And in any assessed course - and however informal and learner-centred our strategy becomes these will still be needed, not least to become qualified as a CIPD member - assessment of learning is a given.

Levels 3 and 4 may not be so important when learning is the primary goal, as it is for education, but in a training context, where the goal is to change behaviour in order to improve performance, then they are clearly vital. Learning is of no use if it doesn't get applied (level 3); nor is it any use if when it is applied it makes no positive difference (level 4). Now I know that evaluating at levels 3 and 4 is hard, and if you were going to do it properly you'd need pre and post measures and control groups, but it's rarely necessary to go this far. If you're proposing a highly expensive new intervention or looking to make a major change in existing practice, then it's only right and proper that you evaluate it properly; but even Kirkpatrick doesn't propose that you undertake major studies routinely - he proposed merely that you can argue for a reasonable chain of evidence, i.e. we did this, they learned that, they seem to be putting it to work and it seems to have made a positive impact.

ROI was mentioned frequently in the session in the context of Kirkpatrick's model, an association which I don't understand. Kirkpatrick didn't talk about ROI - my understanding is that ROI was added much later as a fifth level in Jack Phillips' extended model. Not that ROI is in any way a red herring. I know there are some l&d professionals whose numeracy doesn't rise to such lofty heights, but in certain circumstances ROI is important - if we're investing so much money in this effort, we want to know what we're getting back in return.

So does Kirkpatrick's model apply as well in a learner-centred context? I think it does. If I'm engaging in any sort of learning activity, I want to know:

• Did I enjoy it?
• Did I learn what I wanted?
• Was I able to put it into practice?
• Has it made any difference to my performance?
• And, as a form of ROI measure, do these improvements in performance justify the effort I put in?

Sloman contrasts ROI with ROE, the return on expectations, 'which assesses the extent to which training has delivered the benefits expected'. But surely all evaluation is a measure of ROE:

• I expected to enjoy the course.
• I expected to learn something.
• I expected to be able to apply it.
• I expected all this to impact on my performance.
• I expected the effort put in to give me a payback.

What the session did rightfully acknowledge was that effectiveness is not the only useful measure of learning and development performance - efficiency matters too:

• How many people took up the opportunities provided?
• How long did it take for them to learn what they needed?
• Did this learning happen in a timely manner?
• How much did it cost?
• What proportion of our capacity was taken up?
• And if we charge for delivering training services, did we receive the income that we require?

I'm going to take some convincing that Kirkpatrick isn't a reasonable starting point for assessing effectiveness, both qualitative and quantitative. Supplemented with measures of efficiency, I think we get all the indicators we need; and implemented sensibly, at a level of detail which is proportionate to the investment, evaluation is within the reach of any organisation.