Posts

When KPIs are Blinders: The Dangers of Local Optima

/in , , /by

Recently we encountered a strange situation.  One of our clients had a top of the funnel KPI that looked like it was going sideways with regards to efficiency, but their revenues were going gangbusters. We use this metric for optimization and it is the core number by which everyone from their Board of Directors on down judge the efficiency and success of the digital campaigns.

Suffice it to say, this was not good.

Interestingly enough, the revenue numbers were setting new records, which made us question what was happening with our long standing and go-to KPI. So we started checking off the boxes. Was the revenue coming from the current marketing investment? Check. Were we paying what our financial models said was appropriate to the audiences? Again check. Was the top of the funnel KPI consistent over time and audiences?

Hmm. No, no it wasn’t. Aha! Changes to user engagement paths had affected the top of the funnel over time, and the average KPI from the past was no longer aligned with revenue and profit.

Our go-to KPI had become a Local Optimum, and we needed to address it quickly.

I first came across the term Local Optima in Eliyahu Goldratt’s classic novel on business optimization The Goal.  Essentially, a local optimum is a metric that may apply to a “neighborhood” of criteria, but is uncorrelated with the entire system. Goldratt used the uptime of a single machine in a factory as his example, where running any machine at 100% capacity other than the one machine that throttles throughput for the entire factory will cause inefficiencies and losses for the entire system.

Goldratt and his Theory of Constraints thinking dictate that “Measurements of local optimum behavior should be abolished and replaced with holistic measurements.”  If we had a Local Optimum on our hands, attempts to “correct” for this KPI would not be mildly bad, it would almost assuredly hurt the profitability of the entire company.

So we threw out the KPI and went back to first principles: “Marketing is a financial investment in a financial outcome”. What could we look at that would be correlated to revenue production over time?

It turned out that slightly down-funnel was another metric that turned out to be 100% correlated to revenue production and, in fact, was the one to which we calibrated our bid models.  We were able to show that the sideways direction of the top-of-funnel KPI wasn’t a problem, but was actually a natural side-effect of using the “true” KPI to back-calculate to the top of the funnel. By shifting the conversation from our most visible KPI, the conversation then quickly shifted to how to take advantage of this knowledge. How could we change our reporting, conversation, and client-side metrics to align? We lucked out. Our client is extremely savvy in understanding their own cohort data and financial targets and rapidly escalated the conversation to the C-suite and their Board.

We talk consistently about using Profit as a KPI. In reality, as digital marketers we often have to use early engagement numbers as proxies for true Profit. It is always good to step back and assess the relationship to Profit in case a Local Optimum has snuck into the room as it did in this situation.

What Magritte Has to Tell Us About Marketing Data

/in , , , /by

In the winter of 1928-29 a Belgian painter living in Paris painted what looked like an advertisement for pipe tobacco. The painter, René Magritte, painted a caption below the large pipe stating “This is not a pipe” creating confusion for viewers and a bit of a stir in the Paris art scene at  the time.

The title of the painting is “The Treachery of Images” and Magritte intended it as a reminder that representational art is a reflection of the artist’s view of the object,  not the object itself. His interpretation was that  the painting of the pipe is not actually a pipe. You can’t smoke it, fill it with tobacco, or put it in your pocket; it is a representation of reality.

Magritte’s message is particularly apt in our view of marketing data. As marketers, we use our marketing data all the time in measuring and assessing user behavior related to digital advertising and engagement. But it is extremely important to remember that marketing data is an imperfect representation of user behavior and not a perfect simulation.

There are three main reasons that marketing data is imperfect. The first is in the nature of tracking. Tracking is technically limited in its scope and reach.  At best, tracking can measure engagement from the same device over time, or the same multi-device account over time. This only works if the tracking is implemented correctly in the first place and is not disabled by the end user. Because of the technical limitations of tracking there will always be engagement that is not tracked because of multi-device use, the amount of time for which the tracking is active (cookie window), or because of personal opt-out at the user level either by choice or by browser pre-set settings.

The second main source of imperfection in marketing data is in user behavior.  As humans we have a wide variety of choices and methods of interacting with advertising. Some of us choose to avoid interacting with advertising as much as possible while some of us behave in the opposite fashion. Some of us will choose not to click on ads at all, some of us don’t hesitate. Some of us need to heavily research purchases, some don’t. There is not one engagement path that is adhered to by all users yet often our data is interpreted through a “single funnel” lens that introduces inaccuracies in interpreting data.

The last main source of data imperfection is in data integrity. The cleaning and analyzing of data must align with the knowledge being sought. De-duping rules will differ by business model. Attribution will vary based on external factors such as affiliate payment rules (and will still never tell a 100% accurate story). Data dropout can cause interpretation issues.

So what is a marketer to do?

The best thing you can do is to fully utilize data to test your own model of reality.  The best data firms don’t blindly use data but instead use it to inform their perception of reality.  For example, if you know that a percentage of social users engage with your brand without clicking on ads and instead show up as brand search or No Referrer traffic, you should try to assess: how much, how can that magnitude be assessed and what impact does it have on my decisions as a marketer if I can hypothesize about non-measured (but real!) activity.

Magritte’s point  is well-made. Our data is not reality, but an imperfect reflection of reality biased by user behavior, technical limitations, and process. Our best response is to acknowledge and accept this and use it to our advantage.