Skip to Content

When a Customer Survey Changes, Keep the Comparison Honest

Intercom’s random-answer-order update gives clients and providers a reason to preserve survey changes alongside service reports.

Table of contents

No titles in the blog post

A client and an outsourcing provider can ask customers the same question and still change the way the answers are presented. When that happens, the service review needs to preserve the change alongside the results.

In a November 12 product update, Intercom announced random ordering for answers to multiple-choice and dropdown survey questions. Each respondent receives an order that stays consistent for that respondent. Intercom presents the feature as a way to reduce the influence of answer position and improve accuracy. That is the vendor’s stated rationale; we have not measured its effect on a client’s survey or verified a particular account’s configuration.

Record what changed in the question

The following recommendations are our original client–provider operating analysis. The example is hypothetical, and no change in customer opinion or service performance has been measured for this article.

Imagine an outsourced support team asking customers which part of a recent interaction needed most improvement. Its list of answers includes waiting time, clarity of explanation and follow-up. The client decides to change the order in which those choices appear.

At the next review, the team should be able to identify when that change took effect. Keep the question wording, answer labels and presentation arrangement with the reporting period. If wording and presentation both changed, record both instead of reducing the history to “survey updated.”

This need not become a large documentation exercise. A short version note gives the people preparing and reading the report a shared account of what customers were actually shown.

Agree what a comparison can support

Suppose the distribution of answers changes after the update. In our invented example, that result raises a question; it does not settle why the result changed. The team would need to examine the circumstances before calling it a service improvement or a deterioration.

The reporting discussion could ask whether the same kinds of customers were invited, whether the invitation was sent at a similar point in the interaction and whether the service handled a different mix of issues. These are questions to investigate, not claims that any of those factors caused the hypothetical result.

Keep the presentation change visible while that investigation happens. A chart that joins the periods without a note can invite a stronger interpretation than the report’s author intended. The provider and client should agree how they will describe the transition before using it to make a consequential staffing or training decision.

Divide the reporting work explicitly

The client may own the purpose of the survey while the provider manages invitations and prepares the monthly report. Decide who maintains the version note, who checks the actual survey and who explains the transition in the service review.

If another team edits the survey, make sure the reporting owner receives the change information. Otherwise, the provider may be asked to explain a movement without knowing that the customer experience of answering the question has changed.

A useful handover states what was approved, when it applied and which report will first include the revised survey. It should also identify any responses collected around the transition that need separate treatment. The arrangement should fit the platform and the survey; it should not depend on someone remembering a conversation weeks later.

Use comments to ask better follow-up questions

In the hypothetical review, customer comments might help the team understand what respondents meant by “follow-up.” Some could be describing a missing update, while others expected a later action that was never promised. Reading the comments could suggest a question for the service team to investigate.

Those examples should remain examples. Selecting a few vivid comments does not establish how common each experience was. Keep the distinction between the response distribution, individual accounts and the team’s proposed explanation.

A survey feature is an opportunity to improve how feedback is collected. For a client and provider working together, its introduction should also produce a clear reporting handover. Knowing which question was shown, under which arrangement and during which period makes the next conversation about service quality more useful.

AI-generated conceptual editorial photograph of fictional survey reviewers; not Intercom staff, an actual survey or measured results.