Can You Still Trust an Open Survey Link? What Four Studies Show About Fake Responses and What They Leave Out

Before a board adopts a position built on a member survey, someone has to decide whether the answers came from members. We read four documents on fake survey responses, and none of them studied an association, so what follows is what they show about other surveys and where they stop.

The check that no longer proves a person answered

The usual defense is a trap question: an attention check, a logic puzzle, a question only a machine could answer. Westwood’s article in PNAS, a journal article by a Dartmouth researcher, examines that defense in the author’s own tests. The abstract reports that the author built an autonomous synthetic respondent and that it passed 99.8 percent of 6,000 trials of standard attention checks. It also reports that the respondent got past logic puzzles and “reverse shibboleth” questions, which are tasks easy for a model and hard for a person, and that it could be told to alter polling outcomes. The author calls most current detection methods obsolete.

That is a demonstration of what is possible, not a count of fake answers in real surveys. The article’s discussion says its author does not contend that such respondents dominate online survey panels and expects the size of the problem to stay hard to measure. It did not study any association or any survey of members.

The closest number on how often workers use these tools comes from an EPFL preprint, a paper by three university researchers and a preprint its authors mark as work in progress. They estimate that 33 to 46 percent of crowd workers used a language model on their task. The task was summarizing medical research abstracts, and the estimate rests on 46 summaries from 44 workers collected around 1 June 2023. The authors say generalizing to other tasks is unclear. We read it as a sign that people paid per task will reach for these tools, not as a rate for any survey you run.

What a flooded link looked like

The most useful account of a survey under attack is a PLOS Global Public Health article by 13 authors at Johns Hopkins, Morgan State and the University of Maryland, funded by the National Institutes of Health. It describes a survey of Maryland parents that was promoted with social media ads, offered an incentive and began with a public screener. After the ads had run for one hour and reached about 125 people, the authors report, 2,578 screeners had been attempted and 950 survey responses completed, a volume they call highly suggestive of fraud.

Only two clicks on the ad were logged. The authors conclude that the responses likely came from the link spreading through other channels, and they hypothesize it was posted to online communities that share ways to manipulate paid surveys. The paper does not say whether people or software produced the fraudulent responses. It does not show an AI agent at work, so we do not read it as evidence about agents. A hidden question meant to catch auto-responding software did not appear to screen out fraudsters, since no completed survey answered it.

The paper recommends a two-stage design in which a short screener comes first and a personalized survey link goes by email only to people found eligible. The survey it describes already sent personalized links to people who passed the screener when the flood arrived, and the paper reports no test of what that design alone prevents. The authors also say even existing recommendations can be circumvented by fraudsters with adequate resources, and they conclude that fraudsters keep changing their approach, so monitoring has to be continuous rather than a single fix.

Sending the survey to people instead of posting it

The shortcut that fails is the open link in the newsletter and on social channels, with an attention check as the only gate. Westwood’s abstract says that gate can be passed, and the PLOS authors report how quickly a public link was flooded. The steps below are our reading of these four documents, not a tested method.

Start with the decision the results will feed. A survey that sets a lobbying position can tolerate less doubt than one that picks a webinar topic, so write the use down before the questions.

Next, send each member a link tied to that member, from your own member list, and keep a record of which links went out so invited answers can be told apart later. The PLOS authors recommend a personalized emailed link for people who qualify, and they advise keeping several copies of the survey at different web addresses, so that if one link is compromised that copy can be paused while the others keep running. Those findings come from a parent survey recruited with ads, not from a member survey. No source we read tested a survey that only logged-in members can open, so we make no claim either way about that setup, and a personal link is a reasonable step, not a proven one.

Keep any answers that arrive through a public link in a separate pile, and report them apart from the invited members’ answers or leave them out. If the survey offers a prize or a gift card, have a person review the responses before anything is paid, because the PLOS authors write that survey payments should never be automated and should always include a human review.

Then read the open-ended answers yourself. The PLOS paper suggests looking for nonsense and for repeated text, which fits the automatic completion it mentions. Westwood’s abstract reports open-ended answers shaped to a persona, which a repeated-text check would miss, so a clean read is not proof of a member.

Finally, describe the results by how they were collected. The AAPOR Code of Ethics, amended by members of a professional association of survey researchers in June 2026, says “survey” and similar terms imply that the primary source of data is human respondents, and that cases created by artificial intelligence and included in a purported study must be identified as such. The Code says it applies to survey researchers whether or not they belong to AAPOR. Whether an association’s staff count as that is for each association to decide, but its wording gives a plain test. If you cannot show a person answered, say what you can show.

A dues survey, handled this way

This is a teaching example, not a case study. A membership director wants to know how members feel about a proposed dues change and plans to post the survey link in the newsletter and on the association’s social accounts.

Instead, the director emails every member a personal link, records who received one, and sets aside any response that arrives without it. The report to the board carries one sentence of method: responses from members who were sent personal links, with public-link responses held out and not counted. Before any gift cards go out, a staff member reads the open-ended answers, compares the count of personal links sent with the count of responses received, and approves the list.

The board slide says “members who answered their invitation” rather than “what members think.” That wording is not excess caution. It is the only claim the collection method supports, and it leaves the board free to ask for more.

What this does not settle

No document we read says how many association survey answers are machine-written, and none tested ballots, event feedback forms or surveys behind a login. Treat each step above as lowering a risk we cannot size. Put the method next to the result every time, so a doubtful reader can see what was done.