Why a 5% survey response rate is hiding your real problem

A 5% survey response rate is not proof that your results are biased. It is a warning that you cannot interpret them safely until you know who answered.

Suppose you invite 640 users to a four-question survey. Thirty-three complete it: a 5.2% completion rate. Twenty-one of those 33 select reporting as the area they most want improved.

It is tempting to write, “64% of users want better reporting.” The survey supports a narrower sentence: “21 of 33 respondents selected reporting.” Whether that result describes the other 607 invited users depends on why some people answered and others did not.

This distinction is the real problem hidden by a low survey response rate. You have two sources of uncertainty:

• Too few answers. Thirty-three responses give you limited precision, even if they came from users at random.
• The wrong mix of answers. If willingness to respond is related to engagement, satisfaction, role, tenure, or the topic itself, a larger count of similar respondents can still mislead you.

The first problem is about how many people answered. The second is nonresponse bias: how respondents differ from the people you hoped to understand.

Is a 5% survey response rate good or bad?

There is no universal survey response rate benchmark for SaaS.

Survicate’s current benchmark report found a median response rate of 7.74% across 905 SaaS surveys from 153 companies. That makes 5% plausible rather than extraordinary. But it is vendor data from one platform, and its definition matters: Survicate counts a response when someone starts a survey or clicks its welcome screen, then divides by survey views. A completed-responses-per-email-delivered rate is not directly comparable.

Before using any benchmark, check four things:

1. What counts as a response: a start, a partial response, or a completion?
2. What is the denominator: messages sent, messages delivered, prompt views, or eligible users?
3. Was the survey sent by email, shown in the product, or offered through another channel?
4. Does the benchmark cover a similar audience, relationship, and question length?

The American Association for Public Opinion Research distinguishes response, cooperation, contact, and completion measures because they answer different questions. It also cautions that response rate alone does not reliably separate accurate surveys from inaccurate ones.

So 5% is neither a clean bill of health nor a failed survey. It is a reason to inspect the sample before making a broad claim.

A low response rate cannot tell you why people stayed silent

In the opening example, you know that 33 people completed the survey. You do not know that all 607 others saw the request, rejected it, or considered your company unworthy of their time.

Some invitations may have bounced. Some may have landed in spam. An administrator may have received the message while the daily user did not. Someone may have intended to reply and forgotten. Another person may have felt too indifferent—or too frustrated—to bother.

Nonresponse is therefore not an answer in itself. It is missing data with several possible causes.

Topic interest is one cause worth testing. Robert Groves and colleagues ran experiments in which the stated topic of a survey affected who took part. In an earlier telephone study recapped in their paper, teachers agreed to a survey framed around education at 56%, compared with 39% for one framed around national issues. In a later mail experiment, members of the American Birding Association were 28.5 percentage points more likely to return a birding questionnaire than a shopping-centre questionnaire when no incentive was offered.

The same paper also reports experiments that did not produce the expected topic effect. Sponsor, mode, incentives, and the details revealed before participation all changed the result. The evidence does not justify assuming that every product survey attracts only champions or only critics. It does justify asking whether your subject line and topic made certain users more likely to answer.

A message titled “Help us improve reporting” might draw users for whom reporting is already important. A general product survey might over-represent people with strong feelings in either direction. A survey shown only after a completed task cannot include users who never reach that task.

That is why the response rate is a diagnostic clue, not a description of your customer relationship.

More responses help, but a higher rate does not guarantee less bias

You should still make a survey easy to notice and complete. A clear invitation, sensible timing, a short questionnaire, an appropriate incentive, and a reminder can all add useful responses. More observations usually improve statistical precision.

What they do to sample composition is an empirical question.

A Statistics Netherlands study illustrates both outcomes. Starting from the same labour-force survey response, an intensive call-back approach raised the response rate from 62.2% to 76.9% and improved its representativeness indicator from 0.80 to 0.85. A short basic-question follow-up raised the rate to 75.6% but lowered that indicator to 0.78. The extra responses improved the mix in one design and produced more of the same in another.

Pew Research Center found a related pattern in a 2012 telephone-survey study. Extra time, incentives, refusal letters, and experienced interviewers lifted response from 9% to 22%, yet 28 of 40 estimates changed by no more than two percentage points. That does not mean follow-up never helps. It means the rate itself cannot tell you whether it helped the estimate you care about.

When you send a reminder, ask two questions:

• How many additional people answered?
• Which previously under-represented groups did those answers add?

If the second answer is “none”, the response rate improved more than the evidence did.

How to check a SaaS survey for nonresponse bias

You often know more about invited users than a public-opinion researcher knows about sampled households. Product and account data can show whether the people who answered differ from those who did not on characteristics observed before the survey.

Use only data you are authorised to use, keep the analysis aggregated, and choose fields relevant to the decision. You do not need a personality profile. For an activation decision, useful fields might include activation status, time to first value, account role, company size, tenure, and recent use of the core workflow.

1. Define the population and the decision

Write down who the result is meant to describe. Is it all account owners, new users who attempted setup, active end users, or customers approaching renewal?

Then name the decision. “Improve the product” is too broad. “Choose which setup failure to address next” tells you which behaviours and segments matter.

2. Keep an invitation-level outcome

For every eligible invitee, record the states your channel can measure reliably: delivered, bounced, started, partially completed, and completed. Do not treat email opens as ground truth; privacy features and image blocking can make them unreliable.

Report the formula beside the result. For example:

Completion rate = 33 completed surveys / 640 delivered invitations = 5.2%.

That sentence is more useful than “response rate: 5.2%” because another team can reproduce it.

3. Join respondents to pre-survey data

Compare respondents and nonrespondents on variables that existed before the invitation. Do not use behaviour caused by the survey itself.

Here is an illustrative example, not Maren customer data:

In this example, 21 of 33 respondents selecting reporting would still be a valid result about the respondent group. But because recently active users supplied most responses, it would be unsafe to present the finding as the preference of all 640 invitees.

Run the same comparison for the characteristics tied to your decision. Useful cuts might include:

• activated versus not activated;
• account owner, administrator, and end user;
• new, established, and recently cancelled accounts;
• frequent, occasional, and inactive use;
• customers who used the feature in question versus those who did not.

Pew used commercial databases to compare responding and nonresponding households. Statistics Canada recommends the same general approach: use relevant auxiliary information, examine nonresponse by group, and target follow-up where bias risk is greatest.

4. Compare response waves—but treat them as a sensitivity check

People who answer only after a reminder can reveal whether your respondent mix is changing. Compare the first wave with the reminder wave on both known characteristics and survey answers.

If the second wave contains more inactive users and reports different problems, the first wave was incomplete in a way that matters. If both waves look alike, you have learned that this reminder added similar respondents. You have not proved that the remaining nonrespondents are the same.

5. Check the survey against behaviour

Survey answers are self-reported. Nielsen Norman Group notes that surveys cannot show what people actually did and usually cannot probe why an answer was given. Compare a claim with product data where the question permits it.

If respondents say reporting is their biggest problem, check report creation, failed exports, support contacts, and abandonment around the reporting workflow. A mismatch is not a reason to discard either source. It is a reason to investigate what each one is measuring.

How to hear from the users your survey missed

Once you find a gap, do not send the same message to the entire list again. Choose the missing group that matters to the decision and change the approach.

For example, if inactive users are scarcely represented:

1. Select invitees from several inactivity bands rather than from the full customer list.
2. Frame the request around their recent experience, not around praising or improving your product.
3. Try a suitable channel or moment instead of repeating an ignored email.
4. Offer a short conversation in which they can explain what happened in their own words.
5. Keep the refusal easy and respectful.

Timing matters. Nielsen Norman Group’s guidance is to ask after a relevant task rather than interrupting it, and to avoid adding another generic request to a crowded inbox. Different channels can also reach different people. The US Census Bureau’s American Community Survey, for example, combines internet and mail responses with personal-visit follow-up for selected nonrespondents; its current methodology weights the follow-up cases accordingly.

For qualitative interviews, do not promise a representative sample. Teresa Torres recommends interviewing for variation: include power users, first-time users, active users, disengaged users, and people who use different parts of the product. The goal is to uncover experiences the original respondent pool could not show, not to estimate how common each experience is.

This is also where a conversation adds something a form cannot. If a participant says, “Reporting”, you can ask:

• “Walk me through the last report you tried to create.”
• “Where did you expect the data to come from?”
• “What did you do when that did not work?”
• “Who needed the report, and what decision were they making?”

The answer may reveal an export bug, a permissions problem, a missing integration, or a reporting need. A checkbox cannot distinguish those stories.

For more on recruiting deliberately, see how to recruit participants for user interviews. For the wider set of survey failure modes, read survey response bias: why your survey results lie to you. If the survey is NPS, the interpretation problem deserves its own treatment: why NPS is a bad signal for product decisions.

How to report a low survey response rate honestly

Do not hide the denominator in a methodology note. Put it next to the conclusion.

Use this pattern:

We invited 640 eligible users and received 33 completed surveys (5.2%). Recently active users supplied 55% of completions while representing 22% of invitees. Of the 33 respondents, 21 selected reporting. We are treating this as evidence about the respondent group and are following up with less-active users before making a roadmap decision.

That wording does four jobs:

• reports the numerator and denominator;
• defines what counted as a response;
• names the observed imbalance;
• limits the conclusion to what the evidence supports.

Erika Hall’s warning is useful here: a flawed survey does not announce itself. The numbers still form a tidy chart. Honest reporting keeps the chart from claiming more than the method earned.

Where Maren fits

Maren can help with the follow-up conversation. You choose the participants, Maren can send email invitations and a reminder, and each participant opens a link without creating an account. The interview adapts to their answers, asks follow-up questions, and keeps the transcripts, summaries, and cross-interview themes together for review.

Maren does not decide which users are missing, make a self-selected group representative, or turn interview themes into prevalence estimates. You still define the population, select the segments, and check what you hear against product behaviour.

The practical response to a 5% survey response rate is not to ignore the 5% or invent an answer for the other 95%. It is to label the 5% accurately, inspect who is missing, and choose the next research method around the gap.