Decide what a quality review is for
A quality check should help a researcher decide whether a submission is reliable enough for a particular analysis. It should not turn every unusual response into an automatic deletion. A fast completion may be genuine for a short survey. Repeated ratings may reflect a consistent opinion. Several responses from one network may come from a workplace or shared household.
Before collection, write a brief review rule: which signals you will look at, who may make an exclusion decision, and how the decision will be reported. Keep the original submissions even when a response is excluded from charts. That gives you an audit trail and lets another analyst reproduce the reported base.
The survey pilot checklist is a good place to test these rules with controlled, fictional submissions before sending the real invitation.
Separate a signal from a decision
Start with signals that can be explained to another person. Useful examples include a repeated invitation token, unusually short server-measured completion time, very low answer coverage, the same choice across a long rating grid, or a burst of submissions from one network. Each signal has a different meaning and a different chance of a false alarm.
Review the survey itself before setting a time threshold. Someone who saw only three questions because of branching should finish faster than someone who saw twenty. A matrix with strongly worded items may legitimately produce similar answers. A duplicate token is more specific, but the second attempt may be an honest retry after a connection failure.
Record the observed fact, such as “completed in 42 seconds after viewing 18 questions,” rather than a verdict such as “bot.” If several flags apply to one record, count it once in the flagged-response total.
Use a small, repeatable review queue
For each flagged response, show the reason, the relevant timing or pattern, and enough context to judge it. Put the original answer record beside the flag. Give a reviewer three choices: retain, exclude from analysis, or leave pending while checking more evidence.
A reviewer should be able to enter a short reason and see who made the decision and when. “Retain: rapid completion was plausible because branching showed four questions” is more useful than “looks fine.” “Exclude: same unique invitation token submitted twice; the first completed record is retained” is more useful than “duplicate.” If the decision changes, preserve the earlier entry in a history rather than overwriting it.
Limit this queue to authorized researchers. Individual answers and timestamps can be sensitive, especially in employee surveys. The response-rate guide explains why contact tracking and answer access need separate controls.
Keep raw data and analysis data distinct
An exclusion should change the analysis view, not erase what was submitted. The raw export should still contain the original record with its flag explanations, current review decision and reason. The dashboard and reports should use the included set, and the report should say how many submitted records were excluded.
For a fictional example, suppose 300 submissions arrive. Eight are flagged for review; five are retained and three are excluded after documented checks. The analyzed base is 297 responses. A question answered by 264 of those 297 has 264 valid answers and 33 without a recorded answer. A percentage for that question should use the 264 valid answers when describing the distribution of its options, while the report should also show the 297-response analysis base.
Do not silently change a denominator when a filter or exclusion is applied. The reporting percentages guide explains how to label valid answers, filtered groups and multiple selections.
Check whether conclusions survive the review
When exclusions could materially affect a result, calculate the key figures twice: once with every submitted record and once with the reviewed included set. If a headline changes substantially, tell the reader. The difference may reveal a fragile finding or an overly aggressive review rule.
Check the excluded cases by group as well. If most excluded records come from one location, language or device type, investigate the collection process before drawing a substantive comparison. A slow page on one device or a translated survey with different reading time can make a blunt speed threshold unfair.
For a client report, document the fieldwork dates, submitted count, flags reviewed, exclusions, analyzed count and the main rules. Do not publish identifiable response examples merely to justify the exclusions. The cross-tabulation guide covers cautious interpretation of group differences.
Use challenges only where they help
A suspicious-traffic challenge can reduce automated submissions, but it also adds work for genuine participants. It is most useful when a concrete risk appears: repeated invalid attempts, a burst of very fast submissions, or unusual traffic to an open public link. Keep the challenge accessible and provide a clear retry path.
Do not treat passing a challenge as proof that every answer is good. A person can submit careless answers, and an automated actor may pass a simple challenge. Keep rate controls, unique-link checks and later response review as separate layers. For known respondents, verified invitations can give a stronger identity signal than an open link, but should still respect the privacy choice stated to participants.
Test the challenge on a phone and with keyboard controls before launch. If it prevents legitimate participants from answering, the resulting nonresponse can distort the study more than the traffic it stops.
Run a quality rehearsal before launch
Make a controlled test set: one ordinary response, one short but valid branched response, one duplicate invitation attempt, one incomplete response and one intentionally repetitive rating grid. Confirm that each flag explains the actual observation, the reviewer can retain or exclude it with a reason, and a later reversal remains visible in the history.
- Does the submitted count remain unchanged after an exclusion?
- Do charts, cross-tabs and exports show the intended included base?
- Does the raw export preserve the original record and decision trail?
- Can a reviewer tell why a response was flagged without guessing?
- Are small-group results and written answers protected?
- Can the final report explain the quality process in plain language?
Survey Studio's response-quality help describes its flag explanations, retain or exclude decisions and raw-data preservation. A defensible review is one that another researcher can understand and repeat.