Hypothesis: It is not possible to quantitatively measure value in knowledge work
An uncomfortable truth or do we just need to keep trying?
The year of the open-minded Chris
I have declared 2021 to be the year of the open-minded Chris. As part of my commitment to personal growth am trying hard not to have opinions that I am emotionally attached to, but rather to have hypotheses that I hold lightly. I do believe that having a position on a subject is important and valuable. “You can’t steer a parked car” as the adage has it. So don’t worry dear readers you will still be getting a healthy dollop of opinionated verbiage here :)
First things first, quantitative vs qualitative
I often get these confused so let’s start with an explanation. Quantitative data can be counted, measured, and expressed using numbers. Qualitative data is descriptive and conceptual.
So back to the hypothesis then
If I had a dollar for every hour of meeting time I have spent in working groups, focus groups and the like trying to figure out how we measure the value of a particular endeavor I would … well… have a few dollars. It is hard. This stuff is complex. Taking software as an example — how can we measure the value of a particular project, or initiative, or feature? We could try to use money, that seems the most direct. But how can you measure how much money a particular feature is bringing in, or (even more difficult) will bring in? Let’s take an online store checkout page already developed and live in production, surely the value of this feature could be measured? Yes you could measure conversion rates, or revenue flowing through the page. But would the user have checked out if the product item photos were amateurish? Or if the site did not build trust? Or if the company’s SEO strategy had not led them there? What about a purely back-end feature, say a messaging sub-system. Any guesses on the value?
Value and systems thinking
We try to pull things apart and measure in isolation. But an understanding of systems thinking tells us we are on the wrong track.
So back to these meetings …
… we sit around the table, or the whiteboard, or the flip-chart and break out the (real or virtual) sticky notes and we collaborate on how we can measure the value of this next thing we are considering doing. But not once have I heard anyone ask the question: “can we measure the value of this?” It would be a mike-drop moment. The assumption is that the value can be measured, we just need to figure out how. Now, with many (many!) years of business experience behind me, I hear a small whisper in the back of my mind…. what we are all trying so earnestly to achieve — what if this can’t be done :-O ?
Value, like beauty is in the eye of the beholder
If I showed you a painting I had drawn (I can’t even draw stick figures properly) and a great piece of art, say the Mona Lisa. You could give me your opinions, you would say Leonardo Da Vinci’s work was better — and you would be right. But could you quantify this? What number would you give my work and what would you give to the most famous painting in the world? Let’s make it a bit harder. What about one of Salvador Dali’s works — The Persistence of Memory. Out of 10 is the Dali work an 8 and the Mona Lisa a 10? Its a matter of opinion, some hold that Dali’s work is the more progressive, the more imaginative, more evoking of emotion. Let’s take this to the extreme, how much do you love your kids? Your parents? Your best friend? Simply measure your love and then we can rank these relationships in order of love and start prioritizing the investment you should put into each relationship. Its ridiculous of course. Love cannot be expressed as a number or series of numbers. Love cannot be graphed or charted. Neither can value.
So value cannot be measured. But surely output can be measured?
This is an interesting one. Here we are saying we won’t attempt to measure the value, but we want to know the output of a particular team, or project. There may be some value (excuse the pun) in measuring output. Its a debatable point. But can it be done in a meaningful way? In considering this question I always come back to the seventh principle of the agile manifesto:
Working software is the primary measure of progress
Framing the question this way the question becomes “can we measure the output of working software?” Various measures suggest themselves: story point sprint velocity (applicable to many teams doing Scrum) and end user usage metrics. My current hypothesis here is that like value, output cannot be measured in a meaningful way. But I am holding this hypothesis quite lightly.
So what then can we do?
The best explanation of business metrics and measurement I have read comes from Jurgen Apollo’s Management 3.0. Jurgen describes using proxy metrics. These proxy metrics are pointers to the real thing. The real thing can’t be measured but the proxy can. An electron microscope measures objects so tiny they are literally imperceptible. Electrons are bounced off this tiny object and observed (an over-simplification but good enough for our analogy). Here we aren’t looking at the thing itself, we’re looking at the electrons that hit the thing and bounce off. These signals act as proxies to the object.
The most basic proxy measure for value could be revenue. Ultimately people will pay for what gives them value. Net Promoter Score is another good candidate as a proxy measure. However it is important to understand these are very broad brushstroke proxies — e.g. with Net Promoter Score you are often measuring the entire software experience, the user friendliness of the screens, the support experience, the online help and so on. These measures make for good proxies on a system level, not so much for a part of a system.
This article has focused on quantitative measurements, numbers that can be compared, trends derived and so on. A qualitative approach is different, and could provide rich ground for discussion and framing approaches, even perhaps drawing conclusions. It would be a different approach, one beyond where I am going to go with this blog.
Prove me wrong
In scientific thought, a hypothesis can never be 100% proven true, but it can be proven to be false. Take the theory of relativity, Einstein’s famous hypothesis. It has been proven to hold many times, and it has never been proven not to explain the phenomena it describes. But in theory there could one day be an observation that disproves the theory of relativity. Disproving the hypothesis that “It is not possible to quantitatively measure value in knowledge work” would be quite simple. Simply describe a method to measure value. Where the measurement is quantitative, i.e. using numbers not words. What steps to we take? What past measures of value in a complex knowledge work domain have proven to be accurate? Where are they documented?
Of course I am not the first person who has put a hypothisis like this out in to the world. Martin Fowler put his thoughts out there in the good ‘ole days of 2002
So let’s see, one day I may come across a credible, reliable way of quantitatively measuring the value of knowledge work. You are most welcome to offer your suggestions in the comments. Until I find this “way” I will be holding on (lightly) to this theory.
