“Every end user was the captain of his or her own ship — soon there were boats driving all over the ocean, and traffic was so choked that no one ended up going anywhere. There ensued what can be termed “spreadsheet Hell”. In spreadsheet Hell, there was no shortage of data, but there was a real drought of credible data” (Turning Spreadsheets into Corporate Data, Bill Inmon)
Anyone who worked in a large organization in 2017 (and earlier) when Bill Inmon wrote the above, is probably nodding their head in agreement while painfully grimacing with the veracity of the quote. Mr. Inmon created the data warehouse (and is one of the seminal figures in both the history of databases as well as of all computing) — and was undoubtedly right about the challenges of spreadsheets 6 years ago.
To overly simplify a complex history, essentially the viable commercialization of computers and improvements in the microprocessor chip industry beginning over 50 years ago, were coupled with the rise of databases to store all this data. Mr. Inmon’s data warehouse contribution developed a novel way to transform and integrate data from relational databases (and other sources) into a centralized location for analyzing data and generating business insights.
From the 1990s or so until the advent of cloud-based databases (AWS, Google Cloud, etc) in the past decade, the technology simply didn’t exist for general consumption of corporate data outside of a data warehouse in a governed, controlled manner. The problem, as Mr. Inmon aptly described it, was that “in order to have true meaning, there must be both context and values. Interwoven with the recognition of the importance of context is the notion that lineage of spreadsheet data is of the utmost importance.” During these years, users would regularly copy/paste data into locally-stored spreadsheets making it impossible to know where and how the data got from the data warehouse to the spreadsheet.
The spreadsheet Hell that Inmon saw in 2017 was very real, but it has turned into a new, potentially more expensive Hell 2.0 facing most companies today — at best, dashboard Hell and at worst, inability to find any data whatsoever. Dashboard Hell is even more expensive than spreadsheet Hell both because of the sheer amount of personnel time devoted to making and maintaining hundreds of thousands of dashboards, but more important — the promised benefits of full trust and reliability in data, fully optimized business decisions, control tower view of overall performance and all the investments made to get there — are simply not realized.
Reliance on dashboards to monitor a business is by definition too rigid — prevents users from self-serving analytics. More importantly, it not only reduces curiosity and intimacy with data — but also ensures a fully passive data consumption experience. If all filters are pre-defined and built into a dashboard, it implies the operator is robot-like in identifying pre-prescribed levers to pull/push to maximize profits..
As bad as dashboard Hell may be for organizations, there is an even deeper, more problematic place to be — that’s where there’s no graphical user interface at all for users to transform, slice and combine data, perform simple functions, extrapolations and graphics. While some technologies and vendors will pitch themselves as offering this experience, the reality is that spreadsheets are the only proven consumption layer that actually enable this kind of active, continuous learning. Measuring learning (vs performance) is inherently ambiguous — one of the best ways I’ve found is to simply ask the leaders of an organization whether they believe they have an active, curious learning environment.
Mr. Inmon was clearly correct that organizations need governed metrics layers — single sources of truth built and maintained by data professionals — but once companies have this prerequisite (whether via corporately governed LookML, etc), they actually need spreadsheets and the learning culture it engenders.
Success in a competitive, challenging macroeconomic environment requires set of tools to:
- Flexibly select data need across domains
- Analyze, pivot, compute ad hoc values and visualizations
- Explore and analyze data in self–service manner — requiring no additional engineering or analytics support
- Slice/dice data in new (e.g. not pre-defined ways)
Good news, between Google Sheets data connectors, Omni (founded by former creators of Looker), and others, there’s increasing awareness that spreadsheets are not only here to stay — but are in fact the past, present and future of corporate analytics.
© Ben Ganzfried 2023, All Rights Reserved.