How to calculate the transmission efficiency of lessons learned

The lesson learning system should be a supply chain for new knowledge. But how do you calculate its effectiveness?

We can look at lesson learning as a supply chain; identifying new pieces of knowledge and supplying them to other knowledge workers so they can improve their work. If it works well, knowledge is gathered, accumulated, synthesised into guidance, and used to inform future operations.

But how well does this supply chain work, and how can you calculate its efficiency?

This is something I tried to measure recently with a client, using a survey.

I divided the lesson supply chain into three steps, shown below

1. Capture and documentation of lessons from project experience;
2. Update of guidance documents based on new lessons;
3. Review of guidance documentation in future projects.

Now this particular client  did not have a clear supply chain for lessons, so many people accessed lessons through search.

I therefore asked survey respondents to estimate in what percentage of cases the following happened, giving them the options to selecting 0%, 20%, 40%, 60%, 80% and 100%. The average estimate from the survey responders is shown below.

• Any given project documents lessons (46%)
• Any given lesson is used to update guidance documents (32%)
• Any given project makes use of guidance documents (43%)
• Any given project seeks for documented lessons (44%)
• Any given search finds useful lessons (36%)

We can then use these figures to estimate the effectiveness of lesson transmission, as shown in the diagram above, and described below.

The effectiveness of transmitting lesson through guidance is 46% (capture) x 32% (update) x 43% (review) = 6%.

The effectiveness of transmitting lesson through search is 46% (capture) x 44% (search) x 36% (find) = 7%.

These are pretty low figures! Even if both routes were independent and there was an overall success rate of 13%, that still means that 87% of project knowledge was never transmitted other than via human memory.

Let's compare that with an organisation that treats lessons seriously. The diagram and numbers below are not from a survey, but from published statistics in another organisation. This organisation does not require users to search for lessons, but has a well-resourced supply chain to embed lessons into procedures and guidance.

• Any given project documents lessons (100% - lesson capture is mandatory part of every project, and supported by dedicated resources)
• Any given lesson is used to update guidance documents (93% - this figure was tracked on a quarterly basis and the "closure rate" for lessons varied between 88% and 95%)
• Any given project makes use of guidance documents (100% - all activity was directed by operational procedures which were reviewed and use to make the action plans)

The overall transmission efficiency is 93%

Please note that these figures only reflect the documentation of lessons and their transmission to future knowledge workers; they don't include the loss of knowledge in the documentation process, or the issues of transmission of understanding from writtenprocedires into the human braid - they only look at the effectiveness of the supply chain of written lessons.

This effectiveness can be measured, through surveys or through lesson tracking, and we can see that is can vary between very ineffective (as low as 6%) or extremely effective (as high as 93%).  If you are applying lessons learned, then aim for high effectiveness!