Are you Measuring your Marketing ROI Effectively?

By Scott McDonald, The Advertising Research Foundation

John Wanamaker’s quip “Half the money I spend on advertising is wasted; the trouble is don’t know which half” takes on new meaning with the measurement options available today. But which work best in quantifying the impact of marketing activities?

The ARF’s Scott McDonald tackled the topic by providing tips on the best use of A/B tests vs. Market-Mix Models (MMM) vs. Multi-Touch Attribution (MTA) to measure marketing ROI in an article published in ForbesMagazines. His overall recommendation: “all three of these techniques can provide value, but none should be treated as ‘the answer.’”

CEO’s struggle to quantify marketing impact has been around for decades. In the absence of trusted tools, they often made decisions from their “gut” and used “anecdotes or vanity metrics to justify the results.” But this approach no longer works in the era of data-driven marketing.

In examining three approaches that use data to assess marketing campaigns, Scott summarized what each does, the scenarios in they work best, and their pros and cons.

    All three of these techniques can provide value, but none should be treated as ‘the answer’
    — Scott McDonald, CEO & President

A/B Testing

A/B Testing allows marketing professionals to assess and test distinct variables by comparing a “test’ cell to a ‘control’ cell.” It is typically used in situations in which “the marketer can read the result of the test through direct feedback from the market” such as in direct marketing, website optimization, and e-commerce. Common test scenarios include assessing “different messages, offers, color schemes, creative elements, mailing lists or marketing targets.”

Although A/B testing offers a lack of complexity relative to other tools, there are limitations and cautions to this approach, including:

•     Difficulty in establishing the right variable to be monitored
•     Does not provide answers to large strategic questions, e.g., “how much to spend, on which channels [and] toward what kind of brand strategy?”.
•     Pressure to use smaller size cells or cheaper inventory to save money, which detracts from test quality and validity
•     Poor experimental design plays an important role by minimizing the “number of cells needed to isolate the incremental effect of each variable of interest”.

Despite A/B testing’s advantages, Scott argues that marketers are “unlikely to come to a satisfactory answer to Wanamaker’s question, especially in today’s complex market environment, and even more especially if they are not direct marketers,” if they choose to use this approach exclusively. Market-Mix Models and Multi-Touch Attribution, he states, can provide better answers.
Market-Mix Models (MMM) and Multi-Touch Attribution (MTA)

MMM and MTA share several characteristics. They:

•     Use large volumes of data, including information on exposure to marketing messages and subsequent impact
•     “Employ advanced statistical techniques (mostly econometric, regression-based techniques) to assess the relationship between marketing inputs (e.g., advertising, promotion, discounts) and outputs (sales, share of market)”
•     “Depend largely on correlational analyses though, occasionally, experimentally-derived factors or results from in-market tests are taken into account in the models”.

Yet, there are important differences:

•     Market-Mix Models rely on market-level data, which are analyzed after a campaign is finished. “The models generate strategic-level findings that take account of a very wide variety of market inputs and media channels, as well as exogenous variables (like the economy) that affect consumer demand”. The downside: it can take a long time to assemble the data, so results often come in after a campaign has finished.
•     Multi-Touch Attribution Models analyze digital touchpoints at the individual level and allow “marketer[s] to make changes to a campaign in progress, not merely apply lessons to the next campaign. Combining MTA with A/B tests can provide valuable findings while a campaign is in progress. However, MTA’s confinement to digital also comes with the liability of exposure to the high levels of fraud, bots and non-human traffic that have bedeviled the digital advertising supply chain”.

Traditionally analysts relied on MMM or MTA exclusively. Integrating findings from the two is becoming more common. However, the “black boxes” used to perform these analyses can create an ironic opacity, when the goal of using these tools is transparency around results.

Recommendations

All three methods are valuable and provide important information to better assess marketing ROI. However, Scott McDonald urges that “none should be treated as ‘the answer’”. Instead, they should be combined with a set of actions:

•     Understand statistics enough to be comfortable with relevant terminology and key principles
•     Go beyond thinking of just your campaign, and consider how the consumer relates to your brand: “try to convert the results of your ROI studies into a narrative about the customer’s motivations and drivers, and use that to frame further variables to test going forward.”
•     Identify metrics that matter to your CFO

When choosing the most effective approach for your business, Scott McDonald urges marketing professionals to avoid looking exclusively at short-term results. Instead, he advises to examine “are you building brand equity and customer life-time value.”