Attention Checks in Online Surveys: What Still Works

Andrzej Szymkowiak · Last reviewed July 2026

TL;DR: Use two layers of attention checks, not one "select five" item. Experienced panel respondents recognize the classic check, and in one study about 61% of failures on it were deliberate refusal, not inattention (Silber, Roßmann, & Gummer, 2022). Add a bogus item hidden inside a scale and matched to its items ("A metro line has been built on the moon", correct answer strongly disagree). Fix exclusion rules before data collection. On Prolific, a study of five minutes or longer needs more than one failed check before you may reject.

You open the data file and 30 of your 400 respondents finished a 12-minute survey in under 4 minutes, all of them passing your single "please select 'agree'" item. Did your attention check work? This guide covers what careless responding is, which checks still detect it in 2026, why the standard ones are losing power, and the design I now use in my own studies: a camouflaged bogus item that cannot be passed without reading. It ends with the platform rules — what you may and may not reject on Prolific — and a worked set-up you can adapt.

What careless responding actually is

Careless (or "insufficient effort") responding means answering without reading — skimming, clicking patterns, or letting a bot do the work. It comes in two different shapes: random responding, which adds noise, and non-random responding such as straightlining, which can distort correlations and damage your factor structure (Meade & Craig, 2012). Even a modest share of careless respondents can change substantive conclusions, which is why screening is now standard practice; the current reference for the whole field is the Annual Review by Ward and Meade (2023).

Detection tools fall into two families:

Neither family is sufficient alone. Recent experimental work found that indirect indices — especially within-person consistency — often match or outperform the direct items (Goldammer et al., 2024), so a good screening plan combines both.

The problem: respondents have learned the classic checks

Online panel respondents are not naive. Frequent participants have seen the common paradigms and measures many times (Chandler, Mueller, & Paolacci, 2014), and platform samples routinely outperform university subject pools on attention checks — even on novel checks they could not have seen before, which reflects genuinely higher average attentiveness in these samples (Hauser & Schwarz, 2016). Familiarity with reused checks is the separate, real concern. The recruitment source itself sets a different starting risk before any check runs, from panel familiarity down to open bot traffic - see where to recruit survey participants and what goes wrong.

Two recent findings sharpen the point. First, when researchers examined failures on a classic instructed-response item, about 61% appeared to be deliberate noncompliance — respondents who recognized the check and chose not to follow the instruction — rather than genuine inattention (Silber, Roßmann, & Gummer, 2022). A trivial check that experienced respondents can spot from its shape measures compliance, not attention. Second, a large comparison of check types found their usefulness varies enormously, and recommended favoring memory and obvious-answer checks over formats like the classic instructional manipulation check (Perfecto & O'Donnell, 2026).

The conclusion is not "drop attention checks." It is: do not let all your checks look like checks.

A check that cannot be passed without reading

The design I use in my own studies adds a second layer next to the standard instructed item: a bogus item camouflaged inside a scale. The statement is obviously false for everyone — no prior knowledge, no ambiguity:

A metro line has been built on the moon.

The only defensible answer is strongly disagree (or definitely not). What makes the item work is not the statement alone — it is the camouflage:

• Match the format. The bogus item sits inside a Likert grid, uses the same response scale, and matches the surrounding items in length and grammatical construction. A one-line absurdity between two-line attitude items is easy to spot; a matched one is not.
• Match the position. Place it mid-survey, inside a block of real items — not first, not last, not alone on a page. One caveat: Prolific's guidance prefers checks near the start of a study when they are used as grounds for rejecting payment, so a mid-survey camouflaged check is best treated as an analysis-exclusion criterion (the respondent is paid, but removed from analysis) rather than a payment rejection.
• Make it universally false. The statement must require zero knowledge and zero judgment. "I have never used the internet" works in an online survey; "The capital of Australia is Sydney" does not (it is a knowledge question, and platforms will not accept it as an attention check).
• Mind the midpoint. If your scale offers a neutral middle option, a careless-but-cautious respondent can pick it to avoid being caught. Prolific's policy explicitly requires nonsensical items used for rejection to have no defensible neutral answer — so either use a response scale where the correct answer is unambiguous, or treat midpoint answers as a flag to review, not an automatic rejection.

Score the bogus item separately Score the bogus item on its own and drop it from the scale. It is a screening item, not an indicator of your construct — never let it enter the composite or load on the factor in a CFA or other measurement model. (This also stops it from being treated as a reverse-keyed item by accident.)

The honest evidence status: there is no dedicated head-to-head experiment yet comparing camouflaged and openly trivial checks. The recommendation is an inference — covert checks are advised over overt ones (Muszyński, 2023), deliberate noncompliance shows that recognizable checks stop measuring attention (Silber et al., 2022), and item format demonstrably matters for detection (Goldammer et al., 2024). In my own studies I use this as one layer of several: attention-check items plus timing and variance criteria, with the exclusions reported in the paper (Szymkowiak & Antoniak, 2024; Szaban, Szymkowiak, & Zdziennicka, 2025).

Set the rules before the data arrive

Exclusion rules defined after seeing the data are a "garden of forking paths" (making many small, defensible-looking choices until the result turns significant): each small choice looks innocent, but with enough discretion you can "clean" your way to significance. Three rules keep the screening honest:

1. Write the criteria down before data collection — which checks, how many failures, what time threshold, what happens to flagged borderline cases. Time thresholds should come from a pilot (for example, 40% of the pilot median), not from a guess.
2. Report the screening in the paper — how many were excluded, by which criterion, in which order. My own studies report exclusion counts per criterion (e.g., Szaban, Szymkowiak, & Chwialkowska, 2025), and reviewers increasingly expect this level of detail.
3. Know that exclusion is analysis, not housekeeping. Excluding participants can bias a treatment-effect estimate when attentiveness is related to condition or outcome (Mathur, 2025). At the same time, ignoring attention entirely can hide real effects (Roth & Yakobi, 2024). The defensible middle: pre-specified rules, sensitivity analyses with and without exclusions, and no inventing rules after seeing the results.

One older worry can be partly retired: two studies found no evidence that the presence of attention checks distorts scale means or measurement structure (Kung, Kwok, & Brown, 2018), though attention prompts can nudge socially desirable responding in some groups (Clifford & Jerit, 2015). Checks embedded quietly in scales do not introduce any prompt at all, which sidesteps this concern — one more argument for camouflage over confrontation. The same design-first logic applies to method bias: the remedies that matter most happen before data collection, not after — see common method bias: prevention beats detection →.

What you may reject on Prolific (and what you may not)

Platform rules are part of the method. Prolific's current policy (researcher help center, accessed July 2026) allows rejection of payment on exactly two kinds of attention checks: instructed-response items and nonsensical (bogus) items with an objectively correct answer. Comprehension checks, memory questions, and subjective items are not valid rejection grounds — and as a matter of fairness, you should not reject on criteria participants were never told about. For studies of five minutes or longer, a participant must fail more than one check before you may reject; under five minutes, one failure suffices. If you recruit through Prolific, build these rules into the study before launch — a rejection that violates the policy costs you the fee anyway, plus an appeal, plus your reputation score. Budget for exclusions too: if you expect to remove 5–10%, recruit above the sample size your analysis needs.

A worked set-up

A 12-minute consumer survey with two Likert scale blocks, targeted N = 400:

1. One instructed-response item early ("For quality control, please select 'somewhat disagree' for this statement.") — the standard layer.
2. One camouflaged bogus item mid-survey, inside the second scale block, matched in length and format ("A metro line has been built on the moon.").
3. Timing criterion set from the pilot: flag completions faster than, for example, 40% of the pilot median.
4. Variance criterion: flag zero-variance (straightlined) scale blocks.
5. Decision rule, fixed in advance: the default route — pay everyone, and exclude from analysis anyone who fails both embedded checks or fails one check plus trips a timing/variance flag. This is the cleaner option and sidesteps the placement caveat above: because the bogus item sits mid-survey rather than near the start, it is safer treated as an analysis-exclusion criterion than as grounds for rejecting payment. A stricter variant some teams use, consistent with Prolific policy: reject payment for respondents who fail both embedded checks (two failures, which is policy-compliant for a study this length). Use that variant with care and document it clearly, since it depends on meeting the platform's placement and multiple-failure requirements. All counts are reported in the paper either way.

A method-section sentence to adapt:

Data quality was screened with two embedded attention checks (one instructed-response item and one bogus item), a completion-time criterion set at 40% of the pilot median, and a zero-variance criterion for scale blocks; all rules were defined before data collection. Of 431 participants, 12 failed both embedded checks and a further 19 met at least one exclusion criterion; all 31 were paid and removed from analysis, leaving N = 400.

Common mistakes to avoid

• Relying on a single "select X" item — experienced respondents recognize it, and some fail it on purpose.
• Using trick, memory, or knowledge questions as checks — they measure the wrong thing and violate platform rejection policies.
• Writing the exclusion rules after seeing the data — that is a researcher degree of freedom, not quality control.
• Rejecting payment for failures your platform does not accept as grounds (comprehension checks, one failure in a long study).
• Giving a bogus item a defensible middle answer and then rejecting respondents who chose it.
• Stacking so many checks that attentive respondents feel watched rather than trusted — checks also communicate distrust.

Frequently asked questions

References

Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46(1), 112–130. https://doi.org/10.3758/s13428-013-0365-7

Clifford, S., & Jerit, J. (2015). Do attempts to improve respondent attention increase social desirability bias? Public Opinion Quarterly, 79(3), 790–802. https://doi.org/10.1093/poq/nfv027

Goldammer, P., Stöckli, P. L., Escher, Y. A., Annen, H., Jonas, K., & Antonakis, J. (2024). Careless responding detection revisited: Accuracy of direct and indirect measures. Behavior Research Methods, 56(8), 8422–8449. https://doi.org/10.3758/s13428-024-02484-3

Hauser, D. J., & Schwarz, N. (2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48(1), 400–407. https://doi.org/10.3758/s13428-015-0578-z

Kung, F. Y. H., Kwok, N., & Brown, D. J. (2018). Are attention check questions a threat to scale validity? Applied Psychology, 67(2), 264–283. https://doi.org/10.1111/apps.12108

Mathur, M. B. (2025). On the statistical analysis of studies with attention checks. Advances in Methods and Practices in Psychological Science, 8(2). https://doi.org/10.1177/25152459251338041

Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. https://doi.org/10.1037/a0028085

Muszyński, M. (2023). Attention checks and how to use them: Review and practical recommendations. Ask: Research and Methods, 32(1), 3–38. https://doi.org/10.18061/ask.v32i1.0001

Perfecto, H., & O'Donnell, M. (2026). The use and usefulness of attention checks in behavioral research. Journal of Consumer Research, 53(1), 182–200. https://doi.org/10.1093/jcr/ucaf058

Roth, Y., & Yakobi, O. (2024). Attention! Do we really need attention checks? Journal of Behavioral Decision Making, 37(2), e2377. https://doi.org/10.1002/bdm.2377

Silber, H., Roßmann, J., & Gummer, T. (2022). The issue of noncompliance in attention check questions: False positives in instructed response items. Field Methods, 34(4), 346–360. https://doi.org/10.1177/1525822X221115830

Szaban, M., Szymkowiak, A., & Chwialkowska, A. (2025). Breaking barriers: Navigating the stages of the green consumer journey. Environment, Development and Sustainability, 28(5), 13087–13124. https://doi.org/10.1007/s10668-025-06795-z

Szaban, M., Szymkowiak, A., & Zdziennicka, A. (2025). Unpacking consumer price perceptions: The role of sustainability and ethical labels in the cosmetics market. Journal of Cleaner Production, 490, 144730. https://doi.org/10.1016/j.jclepro.2025.144730

Szymkowiak, A., & Antoniak, M. A. (2024). Eco-endorsement dynamics: Unraveling the influence of celebrity attitudes on eco-product perceptions and purchase decisions. International Journal of Wine Business Research, 36(3), 453–472. https://doi.org/10.1108/IJWBR-11-2023-0068

Ward, M. K., & Meade, A. W. (2023). Dealing with careless responding in survey data: Prevention, identification, and recommended best practices. Annual Review of Psychology, 74, 577–596. https://doi.org/10.1146/annurev-psych-040422-045007

How Explonia does this for you

In Explonia, attention checks are ordinary items: you place an instructed-response or bogus item inside any scale block, and it automatically shares the block's response scale and layout — the camouflage this article recommends, with no special widget that gives the check away. Screen-outs, response limits, and per-respondent completion times are built in, so your pre-registered exclusion rules can be applied and reported transparently at analysis. If you write your own items, run them past the item-writing guide first, and see the screen-out demo in the gallery. The "Article description" feature drafts your results paragraph from the final, screened dataset — you stay the author and edit every word.

Ready to collect data you can defend? Build your survey with embedded attention checks and screen-outs in Explonia. Start free →