Down with BASES!

Nielsen BASES. Whether you swear by it or swear at it, if you work in consumer products innovation, chances are Nielsen BASES has consumed – and perhaps cost you – more than a few days of your life.

Founded nearly four decades ago to help large consumer packaged goods (CPG) companies mitigate the risks of new product development, BASES professes to be the “gold standard in forecasting and predicting in-market success,” leveraging an extensive database of historical product testing data to tell you just how many new widgets you’ll sell in year one – all before you’ve even manufactured the thing.

In theory, this sounds too good to be true – a Magic 8-Ball that helps product, marketing, and innovation managers ensure they’re allocating scarce development resources to the best ideas. In theory, you might be tempted to let this magic tool decide which new products consumers will ever see on shelves, which new initiatives will live, and which will die.

Unfortunately, in practice, that’s exactly what happens. BASES deems a new product concept “market ready” and forecasts that it will exceed its manufacturer’s first year sales hurdle and will be green-lit and resourced for further development. those concepts deemed “risky” or forecasted to generate too low a sales volume will be deprioritized, shelved, or killed outright.

Why is this unfortunate? Because here’s the dirty (not so) secret about Nielsen BASES: in the world of innovation, it’s best defined as a necessary evil. Without fail, every CPG client I have worked with both mandates BASES concept testing as part of their product development process and laments their own reliance on it.

I would argue that, for the following reasons (among others), it is past time for us to collectively and meaningfully challenge the status quo.

It is difficult – if not impossible – for consumers to understand and assess a breakthrough innovation on a piece of paper (or screen).

Imagine this thing. it’s not like anything you’ve ever seen or used before. Well, it’s kind of like a broom, but imagine a broom with no bristles. Or a mop with no strings. Just a stick with a flat, cloth-covered panel on the end that magically picks up all the dirt and dust from your floor in a fraction of the time it usually takes.

How much would you pay for it? How often would you use it? How much do you like it better than the broom or mop it would replace in your household cleaning routine? Hard to say, right? I bet you wish you could see it, hold it, test its features in your own kitchen after your two-year-old decides dinner will be eaten off the floor tonight.

This is oversimplified, of course, but after reading this new product description, it may not surprise you to learn that Swiffer – a product that today rakes in more than half a billion dollars annually – flopped in BASES.

Products and brands that buck category norms or (re)define new categories altogether are difficult to test because consumers have no precedent against which to imagine and assess the idea.

Stripped of the real-world context in which they actually make buying and usage decisions, consumers are left to invent what they think this new widget is, what it does, and the value it provides.

We can do our best to describe features, use contexts, and value propositions in a concise, single paragraph write-up. But at the end of the day, the consumer’s honest response to many of these questions is likely to be “I haven’t a clue.”

While derivative “er” (smaller, faster, lighter, healthier, easier) products that offer incremental or evolutionary improvements to their predecessors may be testable via a written concept, for truly innovative new products, BASES falls short.

Consumers aren’t all that good at telling you what they’re going to want.

Let’s pretend we’ve solved for issue #1. Consumers can now accurately interpret and assess a breakthrough idea via a brief written description. Do we really trust these respondents’ own estimations of their future behavior and preferences?

I remember being introduced to Facebook back in 2005, just a year after its launch. A friend trying to convince me to sign up enthusiastically pitched it as “an online version of that book you get when you arrive at school that has everyone’s names and pictures so you can check out other people at your university.” My response was, “Why don’t I just meet them in person?” I swore I would never use Facebook. I couldn’t see how or why this new website could ever play a meaningful role in my life, because social media wasn’t a thing yet. Never could I have forecasted that this “useless” platform would one day define my and others’ everyday social interactions.

BASES asks consumers to tell you how much they might like, value, buy, and use your hypothetical widget roughly 18-24 months from now, when it’s likely to actually hit markets. But while we consumers have a great sense of our own behaviors and preferences today, we cannot accurately or reliably anticipate how they might change over time, thanks to the introduction and adoption of products, services, trends, and norms that do not yet exist.

BASES relies on historical data that fails to accommodate the incredible and unprecedented velocity of change in today’s markets.

By definition, historical data is limited by what has happened in the past. But we no longer operate in a world where tomorrow will look just like yesterday. We now operate in the world of VC-backed startup proliferation, of social media, of Amazon, of bloggers, of wearable technology, of Kickstarter, of direct-to-consumer and subscription models, of 3D printing,of social/environmental accountability.

Even with faster product development and launch cycles – or, perhaps, partly because of them – it is highly likely that the channel dynamics, shopping behaviors, consumption patterns, consumer prefer- ences, technologies, category definitions, and/or competitive landscapes surrounding your new product and influencing its in-market success will have changed (potentially drastically) from the time BASES’s historical data was collected, to the time you test your concept against that data, to the time your product actually hits shelves.

Had Yoplait brought a new yogurt concept into testing back in 2007, would BASES have predicted that this product’s in-market sales would be severely hampered by what was at the time an unknown startup: Chobani? Of course not, because none of BASES’s historical data would have accounted for Greek yogurt, which had yet to upend and redefine a long-standing and stable category of yesterday.

CPG innovators who are reliant on BASES are trusting outdated and increasingly irrelevant data to drive difficult and critical strategic and financial decisions.

Everyone games the system, and everyone knows it.

Do something enough times, and you’re bound to get the hang of it. BASES is no different. So perhaps it should come as no surprise that when i Google “BASES concept testing,”two of the top three search results are titled “How to Beat BASES” and “Winning with BASES.” these articles and numerous others detail how to craft a high-scoring BASES concept – from structure to phrasing to the definition of a “benefit” – that consumers are sure to respond well to, sharing tips and tricks that innovation and market managers know all too well.

Once, while working with a client to write a concept for BASES testing, he suggested we describe a new food product as “the perfect balance of health and convenience.” When challenged, given that describing this product (which offered a multitude of other benefits) as “healthy” was tenuous at best, he replied, “i agree, but we know that phrase tests well in BASES.”

This is not to blame the client. if i were in his shoes, with my project’s survival (and likely my own performance review) on the line, i’d be tempted to rig the odds in my favor, too. But how reliant can or should we really be on research results we knowingly helped manipulate?

So how do we get to the next base?

Why do we, as an industry of thoughtful, inquisitive, and restless innovators, allow ourselves to be held hostage by such a broken system? Because BASES provides two things yet to be adequately supplanted by a competitive solution:

/  Quantitative concept testing data that can easily be dropped into a business case or financial projection model to make the case for further investment and development. Understandably, those in charge of resource allocation want and need an assessment of economic risk and impact. They need data.

/  CYA (Cover Your…Behind) Protection – BASES serves as the ultimate stamp of approval in new product development. Whether or not its forecast or recommendation is “right,” no manager will fault, scold, or punish you for heeding its advice.

Alas, consumer products innovators find themselves trapped in a system with more detriments than benefits, but benefits powerful enough to convince us all to turn a blind eye to its faults. Well, maybe not all of us.

Luckily, consultancies are having more and more conversations with clients about alternative testing options, which can be taken as a sign of the tides beginning to change. Depending on the nature of their concepts, testing challenges, and business requirements, forward-thinking clients are ready to explore: from immersive, contextual “pop-up” or in-home/in-situ qualitative testing that brings brands, products, and experiences to life; to limited pilot launches in test markets (which provide powerful, real, relevant data against which one can extrapolate full-market performance forecasts); to Delphi testing, which puts mathematics against the future by relying less on defined- by-today consumers and more on defining-tomorrow experts and trailblazers who understand where the market is going.

Has any one of these techniques yet earned a place as the new default, one-size-fits-all concept testing platform? No, but maybe one-size-fits-all simply doesn’t fit anymore.

the author

Michelle Jacobs

Michelle Jacobs is co-head of IC New York and a senior innovation strategist at Idea Couture. See her full bio here.