The link between Big Data and Knowledge - the classic Walmart story

This is a short analysis of the story of WalMart's preparation for hurricanes, in order to explore the link between Big Data and Knowledge.

Hurricane Irma food distribution, by City of St Pete on Flickr

One of the earliest and most famous "big data" stories is around Walmart and it's analysis of sales data related to hurricanes.  Let's look at this story through the lens of "Observation, Insight, Lesson" to see the link between Big Data and Knowledge.

According to the New York Times in 2004 -

A week ahead of (Hurricane Frances) landfall, Linda M. Dillman, Wal-Mart's chief information officer, pressed her staff to come up with forecasts based on what had happened when Hurricane Charley struck several weeks earlier. Backed by the trillions of bytes' worth of shopper history that is stored in Wal-Mart's computer network, she felt that the company could "start predicting what's going to happen, instead of waiting for it to happen," as she put it.

The experts mined the data and found that the stores would indeed need certain products - and not just the usual flashlights. "We didn't know in the past that strawberry Pop-Tarts increase in sales, like seven times their normal sales rate, ahead of a hurricane," Ms. Dillman said in a recent interview. " 

And the pre-hurricane top-selling item was beer."Thanks to those insights, trucks filled with toaster pastries and six-packs were soon speeding down Interstate 95 toward Wal-Marts in the path of Frances. Most of the products that were stocked for the storm sold quickly, the company said. Such knowledge, Wal-Mart has learned, is not only power. It is profit, too.

This story is a classic example of the power of Big Data, but let's look at it through a Knowledge lens.  The Military model of knowledge development is that it comes through a 3-step process of Observation, Insight, Lesson, and that the Lesson then leads to Action. The New York Times suggests that the big data algorithms delivered knowledge, but I suggest that what they delivered was Insight, and that the Knowledge is applied when you know what to do with those insights.

• The "trillions of bytes' worth of shopper history" is a collection of Observations
• The correlation of shopper behaviour with imminent hurricane landfall is an Insight
• The lesson, on which action will be based, comes from an analysis of the insights with a background understanding of Walmart's business and strategy. 

I would disagree with the New York Times. I think that what the data-mining experts found was Information in the form of a Correlation. The Knowledge would come in knowing what to do with that information (Knowledge, after all, being what enables correct action).

What would you do, dear reader, if you were a Walmart Executive who had been given this information on beer and pop tart sales? What action would you take?

• Should you (as the New York times suggests) stock your stores with extra supplies to increase profit?
• Should you (as this article suggests) put Pop-Tarts near the store entrances during hurricane season so that sales can "soar"?
• You could even consider raising the margin on these items, following the principles of supply/demand pricing?

All of these solutions will increase sales and improve profit, but would Walmart really want to use a hurricane solely as an opportunity to increase profit? That might not go down so well with the public.  As Peter Drucker said, "Information only becomes knowledge in the hands of someone who knows what to do with it" and luckily Walmart has the knowledge they need to use this information wisely.

What Walmart actually do 

Walmart have built their own "best practice" for disaster relief (here, from page 177)  In the US this includes

• Making sure their own stores are reopened as soon as possible 
• Providing merchandise for emergency responders, such as bottled water for first responders, office supplies for command posts, flashlights and batteries for search and rescue crews
• Donating pallets of dry food, soft drinks, flashlights, clothes, blankets and childrens board games
• Hosting massive barbecues in their parking lots, to feed citizens with no domestic power

 As a result WalMart is seen as a lifesaver, and it's disaster response procedure has been compared favourably with that of FEMA. They have won the hearts of the public, and of the administrators.

Walmart are winning on three counts. They have plenty of observations in the form of shopper data, they analyse this data to derive insights, and they have knowledge (based on experience and codified in best practice) that allows them to take the correct actions. 

It is that knowledge that allows them to know what to do with the insights they mine from Big Data