The Dangers of Local Optima
Recently we encountered a strange situation. One of our clients had a top of the funnel KPI that looked like it was going sideways with regards to efficiency, but their revenues were going gangbusters. We use this metric for optimization and it is the core number by which everyone from their Board of Directors on down judge the efficiency and success of the digital campaigns.
Suffice it to say, this was not good.
Interestingly enough, the revenue numbers were setting new records, which made us question what was happening with our long standing and go-to KPI. So we started checking off the boxes. Was the revenue coming from the current marketing investment? Check. Were we paying what our financial models said was appropriate to the audiences? Again check. Was the top of the funnel KPI consistent over time and audiences?
Hmm. No, no it wasn’t. Aha! Changes to user engagement paths had affected the top of the funnel over time, and the average KPI from the past was no longer aligned with revenue and profit.
Our go-to KPI had become a Local Optimum, and we needed to address it quickly.
I first came across the term Local Optima in Eliyahu Goldratt’s classic novel on business optimization The Goal. Essentially, a local optimum is a metric that may apply to a “neighborhood” of criteria, but is uncorrelated with the entire system. Goldratt used the uptime of a single machine in a factory as his example, where running any machine at 100% capacity other than the one machine that throttles throughput for the entire factory will cause inefficiencies and losses for the entire system.
Goldratt and his Theory of Constraints thinking dictate that “Measurements of local optimum behavior should be abolished and replaced with holistic measurements.” If we had a Local Optimum on our hands, attempts to “correct” for this KPI would not be mildly bad, it would almost assuredly hurt the profitability of the entire company.
So we threw out the KPI and went back to first principles: “Marketing is a financial investment in a financial outcome”. What could we look at that would be correlated to revenue production over time?
It turned out that slightly down-funnel was another metric that turned out to be 100% correlated to revenue production and, in fact, was the one to which we calibrated our bid models. We were able to show that the sideways direction of the top-of-funnel KPI wasn’t a problem, but was actually a natural side-effect of using the “true” KPI to back-calculate to the top of the funnel. By shifting the conversation from our most visible KPI, the conversation then quickly shifted to how to take advantage of this knowledge. How could we change our reporting, conversation, and client-side metrics to align? We lucked out. Our client is extremely savvy in understanding their own cohort data and financial targets and rapidly escalated the conversation to the C-suite and their Board.
We talk consistently about using Profit as a KPI. In reality, as digital marketers we often have to use early engagement numbers as proxies for true Profit. It is always good to step back and assess the relationship to Profit in case a Local Optimum has snuck into the room as it did in this situation.