Why your onboarding survey lies to you

A friend running customer success at a Series A company sent me a screenshot last year: NPS 41, up four points on the quarter, with a little green arrow. Two weeks later she sent a second message. Six of the twenty accounts from the Q1 cohort had not renewed.

Her question was reasonable. How does a healthy score and a dying cohort coexist?

They coexist easily, because the survey was never measuring what she thought it was. It was doing exactly what it was designed to do. The design has three structural problems, and none of them are fixed by better questions or better timing.

Problem one: you are sampling the people who stayed

This is survivorship bias, and in onboarding feedback it is close to total.

Run through the mechanics. You send a satisfaction survey at day 30 or day 60. It goes to active users — people with a working login, who open your emails, who are still in the product often enough for an in-app prompt to catch them. Every one of those conditions is a filter, and every filter selects for the same thing: the customer who is doing fine.

The customer who is not doing fine has already stopped opening the emails. That is the observable behaviour of stalling. So the population most likely to churn is the population least likely to be in your sample, and the correlation is not weak — it is nearly mechanical. The people your instrument is quietest about are customers who already cost you the full acquisition price.

The response rates make it worse. Email NPS response rates have fallen substantially over the past several years; the practitioner benchmarks now cluster around 10 to 15 per cent for email, with in-app somewhat better at roughly 20 to 35 per cent. Those are vendor-published figures rather than peer-reviewed ones, so treat the exact numbers as soft. The direction is not in doubt, and the arithmetic is the point regardless: at a 15 per cent response rate, an NPS of 41 describes fifteen out of a hundred customers, pre-filtered for engagement. You know nothing about the other eighty-five, and the ones you most needed to hear from are disproportionately among them.

There is a widely quoted claim that only one in twenty-six unhappy customers complains and the rest simply leave. It is attributed to Esteban Kolsky, a former Gartner analyst, from around 2015. I have not been able to trace it to a published methodology — it appears to originate in conference material, and it is now cited so widely that the citations mostly point at each other. I mention it because you will encounter it, and because the underlying intuition matches what churn interviews consistently show, but I would not put it on a slide. Silence is not satisfaction; that much is well established without needing a specific ratio.

Problem two: the instrument measures an attitude, not an event

Net Promoter asks how likely you are to recommend the product to a colleague. That is a summary judgement — an attitude, formed after the fact, compressed into a single integer.

What you need to know is different in kind. You need to know that on the second Tuesday the finance lead tried to import last year's ledger, got an error that said "invalid format," spent forty minutes on it, and decided to wait until the end of quarter. That is an event, with a cause, a timestamp, and a fix.

There is no likelihood-to-recommend score that recovers that Tuesday. The scale cannot carry it. And the person's eventual seven-out-of-ten is a blend of that Tuesday, the good demo they remember, the fact that the tool did eventually work for one of their three use cases, and their general mood when the email arrived.

It is worth knowing that NPS's core empirical claim has not held up well. Timothy Keiningham and colleagues published a longitudinal test in the Journal of Marketing in 2007, using data from the Norwegian Customer Satisfaction Barometer across 21 firms and more than 15,500 interviews. They found no support for the claims that NPS is the single most reliable predictor of a company's growth, or that it outperforms conventional satisfaction measures. In their analysis NPS was the best or second-best predictor in two of five industries.

None of that makes NPS useless. It is a serviceable, cheap trend line, and a sharp movement in it is worth investigating. What it is not is a diagnostic. Treating it as one is how a company ends up with a green arrow and a dying cohort.

Problem three: closed questions can only return answers you already imagined

This is the deepest of the three and the least discussed.

Every fixed-response question is a hypothesis you have already formed. "Rate the ease of setup, 1 to 5" assumes setup difficulty is the axis that matters. "Which of these features do you use most?" assumes the answer is on your list. A multiple-choice churn-reason picker assumes you know the reason categories, which is precisely the thing you were trying to find out.

You cannot discover an unknown blocker from a Likert scale. Structurally, you can only confirm or disconfirm blockers you thought of before you wrote the survey. The most expensive problems in onboarding are almost always the ones nobody at the vendor had considered — and by definition those never make it onto the form.

The free-text box does not rescue this. People type six words into free-text boxes. "Works fine, bit slow sometimes." Six words is a summary of a story, with all the causal detail — the specific step, the workaround they tried, the colleague who told them not to bother — stripped out. The compression is where the information lived.

What my friend's cohort actually turned out to be

She called the six. It took her most of a Thursday.

Two had lost their internal champion to a job change and nobody picked the project up. One had a single-sign-on requirement that came out of a security review three weeks after signature, which nobody had escalated because the account manager had answered "it's on the roadmap" and everyone treated that as closed. Two had a workflow mismatch — the product assumed approvals happened in a sequence their organisation ran in parallel, so every cycle required manual rework, and both had built a spreadsheet to route around it. The sixth had been sold on a use case the product genuinely did not support, which the sales engineer had known and had not flagged.

Look at what that list actually is. Two are a customer-success process gap. One is a product gap that was known and mis-handled. Two are the same product gap, discovered twice, and worth building. One is a sales-qualification failure that will recur every quarter until someone changes the qualification criteria.

Four distinct causes, four different owners, four different fixes. Every one of them had been rendered, on the dashboard, as the same thing: a stalled account with a declining health score.

And here is what makes the point about instruments: not one of those six causes could have been retrieved by any survey she could plausibly have written in advance. Two of them she did not know were possible.

What a conversation does that a form cannot

A conversation lets you ask the second question. That is the entire mechanism, and it is not a small thing.

Someone says setup was fine. You ask what they did first. They describe importing the ledger. You ask how that went. They mention, in passing and slightly apologetically, that they did it by hand because the import kept failing. You ask how long that took. Three hours. You ask whether they told anyone. No — they assumed it was something they had done wrong.

Nothing in that chain was available to the first question. The person did not withhold it; they did not think it was relevant, and a form has no way to signal interest and no way to follow a thread. The information was recoverable only by someone who could hear "fine" and ask what fine meant.

Conversations also capture the sequence. Churn is nearly always a chain of events, not a single cause, and the order matters — the SSO delay would have been survivable if the champion had not left in the same month. Surveys return unordered snapshots. Interviews return narratives, and narratives are what you can act on.

The counterweight to "this does not scale" is that it does not need to. Abbie Griffin and John Hauser's 1993 Marketing Science paper "The Voice of the Customer" found that twenty to thirty customer interviews surface ninety to ninety-five per cent of customer needs, with twenty interviews capturing over ninety per cent of what thirty produced. Thirty-three years on, that remains the best empirical answer to how many conversations are enough, and the answer is: fewer than you fear. If you want the concrete version, here are the nine questions worth asking.

Keep the survey. Stop asking it to do the other job.

None of this is an argument for deleting your NPS programme. It is cheap, it trends, and a five-point drop is a useful alarm.

It is an argument for being precise about which questions it can answer. A number tells you something changed. It cannot tell you what changed, for whom, or what to do — and no amount of survey redesign will make it able to, because the limitation is in the shape of the instrument rather than in the wording. The same limit applies to your funnel, for the same reason.

When the alarm goes off, the next step is not a longer survey. It is twenty phone calls.

---

Where Nosie fits

The second question is the entire mechanism, and it is also why this does not scale by hand: somebody has to be on the call, listening closely enough to notice that "fine" was doing a lot of work.

That is the part Nosie automates. It runs the cohort you choose as an outbound voice study — a conversation, not a form — where the next question depends on the last answer, so "setup was fine" gets followed down to the three hours somebody spent importing a ledger by hand and never mentioned. Answers come back transcribed, structured and counted: a ranked list of causes rather than a folder of call notes. Onboarding that actually listens.

Try it on yourself. Your first self-test interview is free — hear the follow-up questions before your customers do.