Picture the call.
It is on your calendar for Thursday, thirty minutes, arranged by your vendor's account executive. On the other end is the HR director of a company roughly your size, in roughly your industry, who went live on the platform about a year ago. She is friendly, articulate, and well prepared. She tells you the implementation had "some bumps, like any project" but the team "worked through them together." She mentions a feature your CHRO happens to love. She says she would "absolutely choose them again."
You hang up reassured. The reference checked out. The box is ticked. The scorecard gets its green cell.
Here is what did not happen on that call. You did not learn what the system is like to live with. You did not learn what broke in month fourteen, because you asked about the implementation. You did not hear from anyone who actually operates the platform daily. And you did not speak to a single customer the vendor did not choose for you.
The reference check is supposed to be the one moment in a selection where you hear from someone with no stake in the sale. In practice, it is the most stage-managed step in the entire process — and the one organizations scrutinize least.
What a Vendor Reference Actually Is
A vendor-supplied reference is not a customer. It is a credential — selected, briefed, and maintained for exactly this purpose.
Vendors run reference programs the way they run demo environments: deliberately. The customers on the list have been screened for satisfaction, coached on their story, and matched to your profile so the conversation feels personal. Many participate through formal advocacy programs that come with real benefits — conference invitations, advisory board seats, early access to features, goodwill that matters at renewal time. None of this makes the reference dishonest. It makes them curated. Every customer who struggled, escalated, or quietly downgraded their scope has been filtered out before the list ever reached you.
There is a second filter that is easy to miss: time. Reference customers are disproportionately drawn from recent go-lives — live long enough to sound credible, not long enough to have hit the hard parts. A customer eight months post go-live has probably not yet run a year-end payroll cycle, survived their first open enrollment, absorbed their first major release upgrade, or sat through their first renewal negotiation. They are describing a honeymoon and calling it a marriage.
So treat the reference call for what it is: the last demo. It has been designed with the same care, toward the same outcome. That does not make it useless — a demo is not useless either. But nobody should mistake it for independent evidence.
Month One Is a Honeymoon. Ask About Month Fourteen.
The questions most evaluation teams ask a reference — "How was the implementation? Is the team responsive? Are you happy with the product?" — all share the same flaw. They invite a summary, and summaries default to politeness.
The useful information lives in specific moments, most of which happen after the first year. Ask about the first year-end close: how long did payroll reconciliation actually take, and how much of it happened in spreadsheets outside the system? Ask what the first major release upgrade broke, who found out, and how. Ask what happened to their support experience after go-live, when the implementation team rolled off and the account moved to the standard queue. Ask what the renewal conversation was like — whether the price moved, whether modules they thought were included turned out to be add-ons, whether the account executive who courted them was still in the picture.
A reference who has not reached these milestones yet cannot answer — which is itself an answer. If every reference the vendor offers is less than a year live, ask directly for a customer in year three. The hesitation you encounter will tell you something the call would not have.
Talk to the Person Who Runs Payroll, Not the Person Who Signed the Contract
Vendor reference calls are almost always staffed with the executive sponsor — the CHRO or HR director who championed the purchase. This is presented as a courtesy: you get a peer. It is also the single most effective filter on the call, because the executive sponsor has a personal stake in the project being remembered as a success. They chose this system. Their credibility is attached to it. When they tell you it went well, they are — honestly, and often unconsciously — also telling themselves.
The person who can actually tell you what the system is like is two levels down: the HR operations manager who runs payroll every cycle, the HRIS analyst who builds the reports, the administrator who manages the workflow exceptions. They know which screens take eleven clicks. They know what the team still does manually. They know which support tickets have been open for four months. And they have far less identity invested in the purchase decision, because they did not make it.
So when the reference call is being scheduled, make one request: "We'd love your executive's view — and could your payroll or HRIS lead join for the second half?" Some vendors will accommodate this. Some will deflect. Track which ones do which. It is one of the cleanest signals the entire reference process will give you.
The References the Vendor Didn't Give You
Everything above improves a process the vendor still controls. The real correction is to run part of the reference check outside their list entirely.
Off-list customers are not hard to find. LinkedIn will surface HRIS analysts and HR operations leads who list the platform on their profile — a polite message asking for twenty minutes of practitioner-to-practitioner candor gets answered more often than you would expect. User community forums and customer conference attendee lists tell you who actually lives in the product. Your implementation partner candidates have seen this platform inside dozens of organizations and will speak more freely than any reference. Your own network — and your team's — almost certainly touches a current or former customer within two introductions.
And the richest source of all: customers who left. A company that migrated off the platform two years ago has no relationship to protect and a complete story to tell — including the part where they explain what would have made them stay. One conversation with a churned customer is routinely worth five arranged reference calls. Former employees of the vendor itself — implementation consultants especially — occupy a similar position, and many will speak in general terms about where the product genuinely struggles.
None of this is exotic. It is two to three hours of work per finalist vendor. The reason almost nobody does it is not difficulty — it is that the arranged calls feel sufficient, and the scorecard has a row for "references checked," not for "references found."
The Three Questions References Will Answer Honestly
Even a vendor-arranged reference will give you real information — if the questions let them. The mistake is asking questions that force a loyalty test. "Are you happy with the system?" requires the reference to either endorse the vendor or betray them, and nobody betrays a vendor on a call the vendor arranged. The questions that work are the ones a person can answer honestly while still being kind.
First: "What do you know now that you wish you'd known before you signed?" This is not an attack on the vendor — it is an invitation to be generous to a peer. People love answering it, and the answers are almost always specific: a module that needed more configuration than anyone said, a data migration that took three times the estimate, an integration that exists on the price list but not really in practice.
Second: "What does your team still do manually that you expected the system to handle?" Every implementation has this list. The reference does not have to call anything a failure to share it — they are just describing their week. This question surfaces the gap between the demo and the operating reality more reliably than any other.
Third: "If you were negotiating the contract again today, what would you do differently?" This reframes criticism as wisdom. The answers go straight to the parts of the deal you can still change: which service tiers matter, which guarantees to put in writing, where the pricing moves at renewal, what to insist on in the implementation statement of work.
Ask all three of every reference, on or off the list, and write the answers down in the same place. Patterns across references are evidence. A single glowing call is not.
The Role This Keeps Pointing Back To
Issue 02 of this newsletter described the role nobody budgets for — the HR Tech expert who bridges business operations, HR processes and technology decisions. The reference check is another place where the absence of that role shows.
Without it, reference calls are scheduled by the vendor, attended by whoever is free, run without a shared question set, and summarized as a feeling — "the reference was positive." With it, the reference check becomes what it was always supposed to be: structured evidence-gathering. The same questions asked of every reference. Off-list conversations run in parallel with the arranged ones. Answers documented, compared across vendors, and weighted by who was actually speaking — a sponsor defending a decision, or an operator describing their Tuesday.
The vendor has a process for references. The only question is whether you have one too.
A Final Thought
There is nothing cynical about any of this. Vendors curate references for the same reason they rehearse demos — because it works, and because buyers let it. The reference customers themselves are mostly sincere; they are simply the survivors of a filter you never saw applied.
A disciplined reference check does not assume anyone is lying. It just refuses to outsource the evidence. It asks about month fourteen instead of month one, talks to the operator instead of the sponsor, finds the customers nobody offered, and asks questions people can answer honestly without being disloyal. None of it is complicated. All of it is rare.
The reference list is the last demo. Treat it like one.