Friday 3 May 2013


Evidence based learning



Evidence is crucial to learning. You can't learn from an experience, unless there is evidence of what happened and why. Evidential learning is becoming big in the public sector at the moment - evidence-based policy, evidence based practice, evidence based nursing, evidenced based policing; all of these offer the opportunity for evidence based learning as well.

However the mechanics of learning varies, depending on who has the evidence, and how it is gathered.

The simplest form of evidence based learning comes from the immediate review of activity by the team involved. In the After Action review, we have the questions
  1. What was supposed to happen in the activity we are reviewing? 
  2. What actually happened?  
  3. Why was there a difference? 
  4. What have we learned from this, and 
  5. What will we do about it?
This is an immediate learning loop, the team experience the outcome of the activity, and they discuss their own evidence in question 2, analyse it in question 3, derive the learning in question 4, and determine their own action in question 5. Question

A more complex form comes in the Retrospect, which is a later review by the team involved. The questions are similar

  1. What was the learning point?
  2. What actually happened?
  3. What were the root causes?
  4. What is the learning?
  5. (Optional) What action needs to be taken to embed this?
Again the team discuss the evidence, analyse it and derive the learning. This is done with the help of a facilitator, who's role is to ensure the evidence is objective and that root cause has been determined. Sometimes the team discusses question 5 (the action), sometimes the action is determined by another more senior group.


An even more complex form of evidence based learning is found in the Learning History. Here a historian, or historian team, interviews many people and looks for observations and insight, and pulls these together into lessons, which are presented to management for action.

In analyses of major incidents, at the far end of the scale, a special incident investigation team is tasked with collecting (and often making) the observations, the insights, the learnings, and even the recommendations for action.

In each of these approaches, evidence is gathered, root cause determined, learnings derived and actions assigned. The flow is the same, but the actors differ.

Each organisation seeking to do "evidence based learning" will need the same flow, and will need to design a process, and assign actors, to drive this flow.

9 comments:

Anonymous said...

Nick - great stuff.
You are right that the public sector is putting ever more emphasis on evidence. The challenge is that for many parts of the public sector (including international development where I work) the kinds of approaches you describe here are often not recognized as being evidence based.
For many, evidence based means coming from tools such as formal research, quantitative project monitoring, independent evaluation, or experimental design approaches such as randomized controlled trials (like clinical trials but in the field).
I personally believe the types of slef or group reflective tools you mention are extremely valuable as they are relatively quick and cheap and bring learning into practice in a way that "impartial" external techniques might not and are much better at capturing tacit knowledge - but I think we still have some way to go to get the public sector to take these approaches more seriously.

Nick Milton said...

So for your sector, Ian, how does it work?

The Observations and the Insights are collected by third parties (formal research, independent evaluation etc)

Who then looks at these, derives the lessons, and identifies the changes that have to be made?

Ian said...

Good question. Usually any area of work has a headquarters technical advisory team - they will usually be the ones to update guidance and training at the organizational level. At country level the project managers are the ones to identify lessons and decide on changes just as with reflective processes. I'm sure they also apply their own practical experiences in making these changes but usually this will be without the benefit of knowing and using the types of techniques you describe in your post.
In the area of evaluations it is becoming more common to require that the office produces a management response i.e. a formal documented response on what the office intends to do to follow up on the evaluation recommendations.

Bruno Winck, Kneaver said...

What suprises me is that outside of Learning History it is the same agent for the 4 first steps and in two cases managers for taking action. This means that for organizations subject to closed mindset or denial it will be difficult to learn and in the case of Retrospect and Learning History if managers are just executives with no field experience decisions to act will be more based on other considerations like costs or politics. We have seen that often.

Are there documented processes where users, customers, owners of the organization are directly part of the loop. This means analyzing evidences from the view a broader system where the organization is embedded.

Nick Milton said...

If managers don't want to make changes (for whatever reason - politics or cost), then it's hard to see how any organisation can learn. If that's the issue, then it needs to be dealt with.

I don't know of organisations that include users, customers and owners in their learning cycle, except by including customer feedback in the "observations" step. Do you know of any?

Lisandro Gaertner said...

There is the official organization and the de facto organization. The second one always learn in spite of the first one's efforts not to. But if the first one doesn't learn, it will fail and the people who made the second one work will take their knowledge with them.

After a lot of experiences with the first one, I am inclined to help people in the second one learn no matter what. People over power. That's my motto. :-)

Nick Milton said...

Knowledge Management is one of the few initiatives that delivers value both to "the people" and "the management" (not forgetting that management are people as well).

However KM introduced from the top down may be rejected by "the workers", while KM from the bottom up may be rejected by "the management." The result is as you explain Lisandro - the people leave, or remain frustrated.

However the solution is not to continue the battle, but to find the win win solution, and to engage both "workers" and "management" in something that benefits them both. This requires extensive communication, but benefits everyone.

(I use quote marks, because in the knowledge economy, workers are managers and managers are workers)

Lisandro Gaertner said...

I really wish there was a simple win win situation. But a don't believe in it. There is only, in my humble opinion, a solution where people agree to lose some in order to achieve a new goal and change themselves. But, in this mediation between haves and have-nots, the cards are on the haves side. And, most of the time, the haves don't want to change themselves or the system. So, if we don't want to rely on guerrilla style techniques, we depend upon a high level of consciousness and, why not, moral from the haves side to things to happen. It is not impossible to happen, I have seen it happen, but it is not commonplace.

Sure the haves are people too, but all the pantomime around them is created to make them believe they are the best among equals and that creates a great problem on the Knowledge and People managing issues. Maybe it is a matter of education, the basic one kind, or, maybe, of changing how people learn to become "leaders", another kind of discrimination in my opinion. Maybe when we get to the offices and workplaces, the battle is already lost or won. People, and what we believe they should be, dream and aspire is both the problem and the solution.

I don't have answers, but I have my opinions and strategies created based on 10 years on the trenches, and, let me tell you, Knowledge Management is becoming more a matter of philosophy than business to me. It expresses and exacerbates a lot of prime human issues concerning learning, power, social formation, ethics and such. So, you must understand why sometimes I just don't want to objectify KM to answer to power, money and vanity "management" issues. It is a crucial matter and what we do in this arena can change the world mentality in the next years.

A little too tragic, right? Sure. Maybe this is just my upbringing by a hippie mother talking. But maybe she is right too. ;-)

Nick, I really appreciate this conversation we have on your blog. Keep up with the good work.

Nick Milton said...

Lisandro, there may be something probably going on here in terms of national culture.

In Brazil, you have a high level of "Power Distance" (http://www.clearlycultural.com/geert-hofstede-cultural-dimensions/power-distance-index/) which results in fixed powerful heirarchies, as well as a high level of Uncertainty Avoidance (http://www.clearlycultural.com/geert-hofstede-cultural-dimensions/uncertainty-avoidance-index/).

In Europe, power distance is mostly much lower, heirarchies are more fluid, and uncertainty avoidance is mostly low as well.

The European conditions are more supportive for KM, the Brazilian conditions less so. High Power distance makes it difficult to challenge or influence hierarchy, high uncertainty avoidance makes people cling to old rules rather than seek new solutions. It is easier to sell the concept of win-win in Europe than it is in South America.

This does not mean that win-win is impossible in South America, but it means that it is counter-cultural, and therefore requires far more careful change management, and will take much longer.

Blog Archive