Net Promoter Score. To be or not to be? That is the ultimate customer experience question

By Dale Wolf

For now, I say the answer is "yes."  Net Promoter Score will improve an organization's transformation into one that sees customer experience as its main purpose. Advocates and growth will follow.

Net Promoter Score, a measurement methodology espoused by Satmetrix and Fred Reichheld, remains the Ultimate Question when evaluating customer advocacy. ‘In 2003, a Harvard Business Review article –The one number you need to grow’ – introduced the world to Net Promoter. Based around the idea that customer referral is the key metric, rather than the likes of customer satisfaction or customer retention, Net Promoter claimed to provide a score that would accurately predict a company’s ability to impress customers, turn customers into advocates, and -- in turn -- become an indicator of potential business growth.

The reason this is a question at all is that there are detractors to this methodology.

Here's an example found on MyCustomer.com:

Timothy Keiningham, senior vice president and head of consulting at IPSOS Loyalty, recently set out to re-examine Reichheld’s research and findings on NPS. Using industries cited as exemplars of the NPS metric, and longitudinal data from 21 companies and 15,500 interviews, it was assumed that the findings would replicate those of Reichheld in 2003 and Satmetrix in 2004. But there was a shock in store. Not only did they not replicate the findings, but when comparing the results to the American Customer Satisfaction Index (ACSI), Keiningham and his team also found that Net Promoter had no clear superiority to other measures. Full Report: Download keiningham_nps_analysis.pdf

So, you might ask, in the light of a survey of 15,000 interviews saying NPS is not the "end all-be all" why, Dale, do you cling to this "overly-simplistic" measurement?

"Good question!" I say.

Because I see too many examples of corporate leaders who have made a commitment to NPS who are betting their own compensation on this. When you put your own money behind something, you catch my attention. For example:

“NPS is the most powerful tool we have ever deployed at GE.” This evaluation comes from Dan Henson, a 20-year GE veteran who is now the company’s chief marketing officer. His comments, based on several years of experience implementing NPS across a wide range of businesses, stand in stark contrast to a recent article in an academic market-research journal. That article, written by two professors, not only concluded that there is no connection between NPS and business growth or profits, but also cautioned that using NPS “is misguided and potentially harmful.” Meanwhile, NPS has become a centerpiece of GE CEO Jeff Immelt’s growth agenda, and his company is rolling it out to every one of its businesses around the world."

The alternative is to develop and deploy a more complex tool. "Fine," I say, "Get all the data you can get."

But, I'm a simple guy. CEOs need something simple that will help them guide their businesses.

The important thing about NPS is that it becomes a tool that drags (or catapults) a company to focus on the customer. This shift in thinking is what is important. It is hard to rally around a 50-question customer loyalty survey. But a simple measure ... would you refer us to your friends or colleagues ... can change a culture. If such a measure is adopted across a company and if compensation is driven by it, the entire organization will magically transform itself into a customer-centric animal seeking reward.

So the debate can roar on. And while the pundits are roaring their opinions, facts and conclusions about NPS, those companies who adopt it are transforming -- like a caterpillar into a Monarch butterfly -- into organizations focused on making customers happy.

Happy customers beget more happy customers.

And growth will follow.

Comments

[Mark Gregory]

Dale I couldn't agree more with your response following your original posting of NPS. I'm a fan of its simplicity and the fact that so many respected organisations find success in its use says a lot.

Regards,

Mark

[Amy Madsen]

Hi Dale.

I couldn't provide a better rebuttal than you when asked, as you ask above,

"...why, Dale, do you cling to this 'overly-simplistic' measurement?

'Good question!' I say.

Because I see too many examples of corporate leaders who have made a commitment to NPS who are betting their own compensation on this..."

My apologies if I am re-posting the same information, but we like to point to the successes companies - like GE, Philips, HSBC, Swiss Re, Lego, T-Mobile, Intuit, Schwab, and more - are having with Net Promoter.

Here is a link to Net Promoter success stories (most are blog write-ups from the Net Promoter Conferences):

http://www.netpromoter.com/success-stories/index.php

We also maintain a link on the Net Promoter discussion forum called "Companies Using NPS" that list dozens of publicly referenced cases of NP usage such as in press releases or financial earnings reports - here is that link (brief registration process required for viewing):

http://netpromoter.groupee.net/eve/forums/a/tpc/f/2731073251/m/6661046912

[Dale Wolf]

Tim,

I hope the spirit with which I responded did not imply that challenging Fred Reichheld was inappropriate. Even God allows us to challenge Him.

What I think you opened up here and what I can do in some small way is to stimulate the discussion. There are too many times when we let opinions get in the way of making decisions, but it is facts that should guide us.

The entire Customer Experience Movement makes sense intuitively. That is one of the reasons as a direct marketer that I gravitated to it so strongly. I learned long ago that the success of any direct marketing initiative depended upon context. I live this creed. It is my car license plate number! Context is at the foundation of CEM.

But so should be strong, undeniable and convincing metrics. As you suggest, there may be no silver bullet here. There seldom are in the complex world of marketing. But silver bullets have a way of catching us. The sound bites of marketing! It turns out that most soundbites are only partially true.

I notice even Satmetrix, which espouses Net Promoter as a major element in customer loyalty measurement, even they collect additional data beyond The Ultimate Question.

Maybe this discussion will encourage them or others to join in this discussion.

Again, for all of us, thanks for taking the time to start the debate.

Dale Wolf

[Tim Keiningham]

Dale --

I do not enjoy challenging the claims attributed to Net Promoter. All of us in the loyalty space owe Mr. Reichheld a debt for his early efforts to promote the importance of customer loyalty. Even though many of the early claims did not hold under more rigorous research, this early work certainly advanced our thinking.

Nonetheless, this good does not give a free pass to make claims in a prestigious journal for Net Promoter that are not supported by the research. Research bias contaminates both management science and practice. It must be addressed.

I too believe that loyalty consultants and researchers have over-complicated the message (and the analyses) with more advanced statistics than it took to get the Apollo space missions to the moon. It makes it impossible for management to understand, communicate, and rally support. This is ridiculous!

With regard to offering a "better" way, I am trying very hard to avoid any appearance that I am "one witch doctor exposing another, in order to replace him" (to use the words of one business writer).

Regardless, I believe that there will never be a one size fits all solution. If there were, everyone would already be using it.

Sincerely,

-- Tim Keiningham

[Dale Wolf]

Thanks so much for this response. I can feel your passion for scientific findings upon which we all make important decisions.

I am not a researcher so there's no way I can wade into the discussion from a scientific approach. It appears that your research has been well founded and I will find time to read it in more depth.

If you do not mind, I will post your response as an article in this blog ... where it will get more attention.

Then I offer just a suggestion about this whole discussion (debate).

Unless it is your strategy that attacking Reichheld will create more visibility for your position (a perfectly reasonable strategy), I suggest that rather than attack Reichheld, approach this from a more positive, educational point of view.

And somehow find a way to bring your validations into a simple to understand and simple to execute message and methodology. Every CEO I have worked with is stretched beyond personal capacity. They yearn for and demand simple approaches. Simple approaches create change.

I would further re-state that one of the advantages of Reichheld's approach is that it meets the CEO test. It may or may not be accurate, but it is causing organizations to become more customer centric ... and that is a good thing.

If you can present an approach that is more accurate and also something that senior executives can get behind, then we will all be better off.

To me, the important thing behind all this measurement is that it result in cultural shift from corporate-centric to customer-centric. Only then can we begin building unique and valued experiences that will delight customers and move away from me-too product features and a decline in pricing and profit.

[Tim Keiningham]

Dear Dale --

I would like to respond to your comments regarding to the research I conducted with Professors Bruce Cooil (Vanderbilt University), Tor Wallin Andreassen (Norwegian School of Management), and Lerzan Aksoy (Koç University) that appears in the current issue of the Journal of Marketing ["A Longitudinal Examination of Net Promoter and Firm Revenue Growth," http://tinyurl.com/2om9hs]

FIRST, Managing Service Quality has just released another paper we co-authored that investigates other aspects of the Net Promoter research conducted by Reichheld and Satmetrix. The research examines different customer satisfaction and loyalty metrics and tests their relationship to customer loyalty behaviors. The goal was to test the robustness of the customer-level analysis conducted by Reichheld and Satmetrix, which served as the foundation of the Net Promoter research. Contrary to Reichheld's assertions, the results indicate that recommend intention alone will not suffice as a single predictor of customers' future loyalty behaviors. Use of multiple indicators instead of a single predictor model performs better in predicting customer recommendations and retention. The fact is, customer loyalty behaviors (i.e., word of mouth, retention, and share of wallet) are distinct and best predicted using different measures.

Because of the importance of the work, Managing Service Quality is making the paper available for free for download: ["The Value of Different Customer Satisfaction and Loyalty Metrics in Predicting Customer Retention, Recommendation, and Share-of-Wallet," http://tinyurl.com/2aq3aj]

These findings, taken in conjunction with the findings of our research reported in the Journal of Marketing regarding the relationship between Net Promoter and growth, call into question the robustness of the entire analysis conducted by Reichheld.

SECOND, with regard to our Journal of Marketing paper [http://tinyurl.com/2om9hs], there are two key findings from the research:

1) We did not find Net Promoter to be a good predictor of growth at all.

2) We found very strong evidence of research bias in the research reported by Reichheld in support of Net Promoter.

In the Harvard Business Review article that introduced Net Promoter, in the book, The Ultimate Question, and in presentations regarding Net Promoter, the American Customer Satisfaction Index has been specifically mentioned as not linking to firm growth by Reichheld. The book, The Ultimate Question, argues that the ACSI does not yield much insight into loyalty or growth, noting that “investors rarely waste money on standard satisfaction surveys” as a result (The Ultimate Question, p. 86).

Similarly, an article in the Harvard Business Review states (p. 49): “Our research indicates that satisfaction lacks a consistently demonstrable connection to actual customer behavior and growth. This finding is borne out by the short shrift that investors give to such reports as the American Customer Satisfaction Index. The ACSI, published quarterly in the Wall Street Journal, reflects customer satisfaction ratings of some 200 U.S. companies. In general, it is difficult to discern a strong correlation between high customer satisfaction scores and outstanding sales growth.”

Furthermore, in a web-based presentation, Mr. Reichheld states that a “Bain team looked at the correlation between growth and customer satisfaction, and found there is none.” A scatter diagram was shown with the X-axis labeled “American consumer satisfaction index annual growth” and the Y-axis labeled “Sales annual growth.” The R-square reported was 0.00, indicating no correlation whatsoever.

Given that our findings show that Net Promoter was not superior to the ACSI when using Reichheld’s best-case scenarios, it is virtually impossible to imagine a scenario other than research bias as the cause. This is a VERY SERIOUS problem. We expect research published in our most prestigious journals to be free of bias in management science, just as we do in all other fields of study. We would not consider this kind of problem acceptable had the research been conducted in medical, psychological, or physics research; the same standards apply in management science.

Managers have adopted Net Promoter based upon the belief that solid science underpinned the claims attributed to the metric. In fact, there would have been no Harvard Business Review paper introducing Net Promoter without the research. This also has serious implications regarding the credibility of Reichheld’s book, The Ultimate Question. Additionally, biased “research” that is published in a prestigious management journal contaminates not only management practice, but also management science, as it will be used by scientists as a basis for future research.

It is vital that we not be apologists or revisionists when it comes to issues of research bias. Our credibility must never be in question regarding the research we publish in prestigious journals; the truth matters. Therefore, discussions about Net Promoter by researchers (practitioner and academic) must first adequately address this issue. If not, then why do any research at all, as we can simply present the answers we want to believe as supporting evidence and be done with it? Given the evidence we uncovered, however, we seriously doubt that there will be an acceptable answer to the issue of bias in Reichheld's reported research. [Ironically, in The Ultimate Question, Reichheld emphasizes the importance of eliminating bias from research (pp. 106-111).]

While we can all agree for the need to have measures that are easily understood and used by managers, that is completely irrelevant to the issue at hand. Regardless of whether or not one chooses to believe in Net Promoter, we all must insist that the evidence used to support the metric be unbiased. This issue is so overridingly important to very core of what we as researchers (academic and practitioner) do and stand for that it must be addressed. Our credibility in science and in practice is based upon honest and fair evaluations of data...if this is contaminated and unchallenged, then there is no reason to believe anything we say.

Sincerely,

Tim Keiningham