Data worth trusting: how to collect feedback that tells the truth

Part 4 of Trust Your Own Data, a four-part series on whether you can believe the numbers your business collects about itself.

If you have ever tried to rescue a photo where the sky went pure white, you already understand the deepest problem with the data your business collects about itself. The white part of the frame holds no detail. The editing software has nothing to work with, because the camera’s sensor was blown out the instant the shutter opened. You can drag the highlight slider all day, and the cloud you swear was up there never comes back. The camera simply never recorded it.

Business feedback works the same way, and this is the payoff the first three parts have been walking toward. Once an answer has been collected in a bad moment, no amount of clever analysis on the back end will recover the honest version that was never recorded. So the whole job moves forward, to the moment of collection. Part 1 named the steak-dinner problem: the score that measures how much someone likes you rather than how good the food actually was. In Part 2 the subject was consent, the quiet difference between a real yes and a cornered one. Then Part 3 walked through the everyday ways a business talks itself into trusting its own comfortable numbers. This part is the toolkit. Small, and usable next week.

Here is the one line to keep from Part 1, because every tool below is really just a way of honouring it: was this collected in a moment designed to get a real answer, or a kind one?

Design the moment before you collect anything

An honest answer costs the person giving it something. Sometimes it is the small social pain of disappointing a human who is standing right there, hopeful. Sometimes it is just effort, the work of writing a real sentence instead of ticking a box. Your one job, before you gather anything, is to make that cost as low as you can, and to make “no” and “this was bad” the easiest things in the world to say.

Most of that comes down to getting yourself out of the room. A waiter who stands at the table asking “how was everything?” will hear “wonderful” every time, because you are looking straight at the person who cooked it, more or less. The comment card you fill out after they have walked away tells a different, truer story. Same restaurant, same meal, same customer. The only thing that changed was whether the person being judged was watching you answer.

So when you design a feedback moment, ask who is in the room, and how much effort an honest answer really takes from the person giving it. Then get yourself out of the way and give people a cheap, safe way to say the disappointing thing.

Decide whether you want a name or the truth

Anonymous feedback and attributable feedback are different tools, and the mistake I see most is a business holding one while believing it has the other.

Anonymous feedback buys you honesty at the price of follow-up. People will tell you what they actually think, but you cannot go back to any one of them, ask a second question, or fix their specific problem. Attributable feedback is the reverse. You know exactly who said it, so you can close the loop with a real human, but everything they tell you is softened by the fact that their name is on it.

The worst version is the survey that promises confidentiality while quietly logging who submitted what. People can smell that. They answer as if their name is attached, because they suspect it is, and you have paid the cost of anonymity without getting anything back for it. Pick one on purpose. If you want the unvarnished truth, make it genuinely anonymous and mean it. If you want to help a specific customer, ask openly and accept that the answer will run a little kinder than the truth.

Watch what people do, not just what they say

Never trust a single touchpoint, and never trust a stated opinion over an observed action.

A survey can tell you customers love the new checkout. The abandonment rate on that same checkout will tell you whether they actually finished buying. When the two disagree, believe the behaviour every time. People will tell you what they think they should feel, and then go and do what they actually feel, and the doing is the honest record. Line up what people said against what people did, and pay close attention to the gap.

The other half of this is collecting the people who left. Most businesses only ever survey the folks still in the room, the loyal repeat buyers who stuck around long enough to answer. The people who cancelled or quietly never came back are carrying the most useful feedback you will ever get, and they are the hardest to reach precisely because they owe you nothing now. That is exactly what makes their answers worth chasing. A number built only from the survivors is a happy number, and a happy number is usually a lie of omission.

Write the question so it does not already know the answer

A question can hand you the answer before the person has thought for a single second. “How much did you enjoy today’s session?” has already decided that enjoyment happened, and most people will politely fill in the blank you left for them. “What did you think of today’s session?” leaves room for a real reply, including one you were hoping not to hear.

Read every question you ask out loud and listen for the answer it is fishing for. If you can hear the reply you want sitting inside the wording of the question, rewrite it until you cannot. This is the cheapest fix in the whole toolkit and the one businesses skip most often, because a leading question is a very comfortable thing to write.

Say your proxies out loud

Almost nothing you care about can be measured directly, so you measure a stand-in and hope it tracks the real thing. The danger is forgetting you did that.

I develop curriculum and teach at an international training facility, where the people in my classes range from finishers with decades on the tools to folks who have never held a spray gun. For a long time a course completion rate can look like a beautiful measure of success. Everyone finished, so everyone learned. Except completion was only ever standing in for learning, and those are not the same thing at all. The real test lands the same week, when a learner walks back onto a production floor and either can or cannot do the thing the course was about. Completion rate never measured that. It measured completion.

So say your proxies out loud, every time. “Our completion rate is a proxy for learning.” “Our review score is a proxy for satisfaction.” “Our open rate is a proxy for interest.” The sentence sounds almost silly, and that is the point of it. Naming the stand-in out loud is what stops you, six months later, from quietly mistaking it for the thing itself.

Ask the real question

Notice what none of this was. None of it was a smarter dashboard or a cleverer statistical test applied after the numbers were already in. Trustworthy data is a front-end act of honesty, performed in the moment you collect, the same way a good exposure is a decision made while the shutter is open rather than a rescue attempted afterward.

Which brings us back to the steak dinner. That glowing score was honest all along, and even useful. It answered a question you did not mean to ask, “do you like me?”, in place of the one you thought you were asking, “was the food any good?” You simply know now how to tell the two apart, and how to design a moment that asks the second one.

That is most of what I do when an organisation brings me in. I help them build measurement and training they can actually trust, starting at the moment the evidence is collected rather than the moment someone tries to explain it away. If that is the problem you are sitting with, here is how I can help.