I suspected AI in my survey results. I used AI to help me find out.

Oct 19

Written By Melissa Strompolis, Ph.D. - Director of Research

I was reading open-ended survey responses from a small group of community leaders – there were 10 open-ended items – too many, I know. The survey was invitation only and the questions asked for individual, organizational, and system level perspectives.

As I read through the responses, I noticed two distinct groups emerging. Neither group was good or bad, right or wrong. One was rough, short, and unpolished. The other was thorough, lengthy, and verbose. In this latter group, answers often restated the question, offered three or four well-ordered points, and closed with a graceful "at the same time." One respondent wrote nearly 150 words per question. Across 10 open-ended questions, I was left wondering if these two groups of respondents were giving me fundamentally different kinds of data.

This was my first survey where I strongly suspected some respondents had used AI to craft their answers. In full disclosure, I used Claude AI to help corroborate my suspicions (yes, the irony). (Sidenote: Claude noted three AI-assisted respondents, I suspect four).I then worked through the (de-identified) qualitative data with Claude to help spot patterns, test hunches, and build a check on whether polished responses were drowning out everyone else (sometimes yes, sometimes no).

That irony is part of the point. AI is now on both sides of the survey. Those of us who collect and interpret community voice need to talk about what that means.

So, I have three questions for my friends of data cause:

Have you run into this yet?
Do you know how your respondents are using AI?
How have you adapted your analysis and reporting?

For each, I will share what I found on the topic and what happened in my own work. I’d love to know what you know and your experiences.

Have you run into this yet?

This was my first experience on a community-based survey and in a recent study Zhang, Xu, and Alvero (2025) found that 34% of participants on a popular online research platform said they had used large language models (LLM) to help answer open-ended survey questions.

Moreover, in recent experiments by a University of Tulsa team, they compared crowdsourced surveys from before and after ChatGPT's release and found a rise in responses flagged as AI-generated after 2022. NORC at the University of Chicago now treats AI-generated responses as a data-quality threat, alongside falsification and falling response rates. NORC has built its own detector for survey data and presented the work at national survey research conferences.

Much of this work and research comes from online panels and paid crowdsourcing. In those situations, AI use might look like fraud or low effort, such as someone earning a few dollars by pasting questions into a chatbot (like when we have used gift cards as survey incentives). The usual fix is to screen those respondents out or spend an inordinate amount of time with data cleaning.

My experience

I’m guessing many of you have, or are about to face, survey situations that are like mine:

Small and invited. Leaders from community organizations, invited by name as they were part of the same initiative.
Unpaid and relational. No incentive. The survey was the first step in a relationship we were building with them, and we knew who they were.
Reflective by design. Alongside role questions and rating scales, the survey asked 10 open-ended questions. We estimated 15 minutes and encouraged "honesty and vulnerability."
For tailoring, not judging. The stated purpose was to shape the engagement around participants, not to evaluate or compare them individually or organizationally.

In that setting, excluding anyone was neither possible nor sensible. These were the people who would be in the room. My question was never "Is this a real respondent?" It was "Whose voice am I hearing in this answer?"

Do you know how your respondents are using AI?

The research suggests AI saves respondents time and changes what respondents say. Zhang and colleagues ran simulations comparing human answers with LLM-written ones. The LLM answers were more uniform and more positive than the human ones, and the gap was largest on sensitive questions. The danger is that AI flattens the real differences in what people think.

Spotting AI-written answers is also harder than it sounds. In a peer-reviewed study, Casal and Kessler (2023) asked 72 journal reviewers to tell AI-written research abstracts from human-written ones; they were right only 38.9% of the time. That was academic writing, not survey answers, but the lesson carries over. General-purpose detectors produce false positives too, and even NORC pairs its custom tool with human review.

What I think my respondents did

Again, three respondents showed strong signals of AI assistance, and I suspect that a fourth used AI for at least some responses. Here is what stood out:

Structure. Each answer restated the question stem, listed three or four points, and closed with a "at the same time."
Uniformity. Similar length and refinement on every question, including the most vulnerable ones.
Repetition. The same distinctive phrase appeared, nearly word for word, across several answers.
Vocabulary. Field-standard phrases that read like a report rather than a person talking (typing) their top-of-mind thoughts.
Timestamps. Long answers submitted in quick bursts, or every page stamped the same minute (i.e., pasting from somewhere else).
A shift within one person. One respondent wrote in a conversational voice on early questions, then became abruptly polished on the final, most abstract ones.

Length alone was not the signal, although the eye-test was jarring. One respondent wrote long answers full of typos and text carried over between questions, which signaled a human that was perhaps reaching for a length they thought we expected.

Taking all of that context, my guess is that this is how respondents in this situation used AI:

Full drafting. Feeding each question, perhaps with background on the organization and engagement, into a chatbot and pasting the answer back.
Help with the hard questions only. Writing the easy ones and turning to AI for the most abstract or the most vulnerable ones.
Polishing. Their own thoughts and ideas and asking AI to refine, smooth, and/or add details.

Decades of survey research has taught us that our beliefs, motivations, and environments do matter – both for the researcher and evaluator and for the respondent. We are all busy. Not everyone feels confident writing. Even with a promise of confidentiality and anonymity, a survey can feel like an exercise in judgment rather than reflection, especially for leaders facing strategic and funding pressures. The questions were genuinely hard. And for many people, this is the new normal for how to generate ideas or how to write.

How have you adapted your analysis and reporting?

In panel research, the usual answer is to detect AI responses and exclude, but this approach has its downsides. Pepin and Wasserman (2024) argue that leaning on detection tools removes good participants and introduces bias, and they recommend judging respondent quality on several criteria at the same time. NORC treats its detector as one input among several, with human oversight.

I found little published guidance for small, invited groups, where excluding people is not the goal. Recent evaluator guides, such as Itad's 2026 guide to generative AI, focus on how evaluators use AI, not on how participants do. In this specific situation, here is what I did:

Flagged, didn't exclude. An AI-assisted answer is not automatically a fake one. The respondent chose what it said and submitted it. I noted "likely AI-assisted" as a context variable.
Counted people, not words. I coded themes at the respondent level, counting each person once per theme however often they repeated it. This mattered more than I expected. In one set of three related questions, the three (sometimes four) flagged respondents were 25-30% of respondents but wrote 50% or more of the words. Each of their answers touched six to 10 sub-themes compared to human-voiced answers with one to three sub-themes.
Ran a sensitivity check. For specific questions and themes, I examined whether the theme and sub-themes held without the AI-assisted responses.
Quoted human-voiced responses, and label themes in their words. Short, specific human phrases were selected to add depth to qualitative data or voice to qualitative themes.

Some great side conversations

Like many of you (okay, could just be one!) reading this, you could just be a shop of one or a data “team” of one (here’s looking at you solo community psychologists and AEA Independent Consultants!). When I suspected the survey had some AI-assisted responses, and after some initial analysis, I chatted with a member of my team and a colleague of mine to get their thoughts, experiences, and insights. These conversations were so helpful for processing what I was seeing, challenging assumptions, ensuring quality data presentations, thinking about how to share our experience, and what the future will look like.

Most of our time and energy has been spent on the analysis and reporting space, but the impact of AI doesn’t start or end there. The next time we get into survey mode, we’ll be thinking about AI from design to deployment to reporting. For example, in the design we might ask more about specific moments rather than abstract situations and set shorter response limits. In the invitation we might also include language that gives permission for “rough and short” while also acknowledging AI (e.g., “if you use AI to help, that's okay; just tell us how”) or asking about the amount of AI use (e.g., not at all, a little, a lot, completely). The analysis would likely include the steps noted above and reporting would note the steps taken with human vs. AI voice.

The point here is not to catch people or stop AI. It's how to keep hearing people and meet them where they are. Community voice is the point of this work, and polished prose or overly optimistic tones can quietly crowd out the rough, specific, sometimes uncomfortable truths we most need.

If you've run into this in your own surveys, I want to hear what you saw and what you tried. We're all working this out in real time. Let me know: