Hundreds of Written Member Comments and a Board Date: Walking One Survey Through AI Sorting

A membership director has several hundred written answers from the annual member survey and a board meeting on the calendar, and the decision is whether to give the comments to a commercial language model to sort. We found no source here that studied an association: the data are consultation and survey responses to UK government departments, one focus group about health technology, student answers to one question about a skin condition, and public submissions to an Australian parliamentary inquiry. The survey in this walk-through is invented for teaching and is not a case study, and each stop borrows from those sources while marking where they are thin.

The blank page needs a theme list first

The model has to sort comments into something, so the first job is a list of themes. The Department for Transport’s evaluation of its consultation analysis tool is a government report written by the department’s AI team with The Alan Turing Institute, who built the tool. It says the tool found about 75 percent of the themes people had generated when no person oversaw it (a recall figure for finding themes), and 90 percent of the themes staff validated in live pilots, where they reviewed and adjusted the tool’s first list. The report concludes that human review stays important so that main themes are not missed.

A University of Southampton study in PLOS Digital Health, marked peer reviewed on its page, ran two general-purpose language models, one coding application and blinded human analysts on a single focus-group transcript of seven participants. Its abstract reports that only one model reached the authors’ non-inferiority bar for finding themes with no list to start from.

Your call: a person should write or approve the list after reading a first batch of comments, and the model should propose additions to it, not the reverse. We read the evidence that way because both documents point to a person reviewing the themes, and open theme-finding was the weaker result in the Southampton test.

Sorting against a fixed list

Once the list is fixed, the evidence is friendlier. In the Southampton test, coding against a ten-code, six-theme codebook gave the language-model tools a mean agreement of 93.5 percent with an expert panel’s adjudication, and the blinded human coders 92.7 percent, according to the abstract. The authors add that a second statistic, Cohen’s kappa, was only 0.34 for both groups because the codes were rare in the transcript.

A BMC Medical Informatics and Decision Making article from Mohammed VI University of Sciences and Health compared nine language models with manual expert analysis of 448 answers from students to one open-ended question, and its abstract reports that advanced models showed strong congruence with the reference standards, some with perfect concordance on one index. The authors name the single geographic and cultural setting as a limit.

Your call: fix the list, then sort. Agreement this high was measured on one transcript and one question, not on a survey with your members’ vocabulary, so the sort is a first pass that someone checks.

Why the agreement figure runs softer than it looks

The Incubator for AI report on a Department for Work and Pensions consultation, a government report from the team that developed the tool, covers 87,738 answers to consultation questions reviewed by 125 people. Reviewers left the tool’s themes unchanged for 73 percent of answers, while two reviewers who each judged the tool’s mapping for 17,000 answers gave identical themes for 65 percent. The report says reviewers verified and amended the tool’s mappings, and that nothing says a reviewer’s label is correct. We read that to mean 73 percent counts how often a person accepted an answer already in front of them, which differs from matching an independent reading.

The Department for Transport report gives two figures that differ by test design. Its blind test, on 11 survey questions, gave an F1 score of 0.75 for assigning comments to a fixed theme list, a different measure from the roughly 75 percent recall in theme-finding quoted earlier, while its live pilots gave 0.93 on seven consultation questions with 720 answers, compared with a human-adjusted version of the tool’s own output, and the report names anchoring bias as a risk in that second design. Its authors also call ground truth “slippery” in qualitative analysis. In the Work and Pensions test, reviewers added almost three times as many theme mappings as they removed, and the report’s authors read that as reviewers leaning further toward including themes than the tool does.

Your call: have the checker label a batch of comments from the raw text before seeing the model’s labels, then compare the two sets. That step borrows the design of the Department for Transport’s blind test, scaled down by us, and neither report tested it on association comments.

The comments that fit no theme

In the Work and Pensions evaluation, reviewers used the “none of the above” option for 4.5 percent of answers, and the report treats that as a sign the list was thorough. The Southampton authors found the models struggled to infer peer support and connection beyond what participants stated outright. In open theme-finding they also report a strict hallucination rate of 1.2 percent against a comprehensive error rate of 12.4 percent, which they say arose largely from the models coding the facilitator’s own lines in the transcript.

The Australian Securities and Investments Commission, in an agency submission to a parliamentary committee, describes an exploratory trial in which a language model summarized a sample of public submissions to an inquiry. The agency says the summaries scored lower than human ones on all criteria, with limited ability to pick up nuance or context, and that the results should not be extrapolated more widely. That trial summarized and did not sort, so it is a caution about nuance and not a measure of sorting.

Your call: read in full everything the model puts under “other”, every comment that is long or angry, and any answer that changes a decision, such as a lapsed member’s reason for leaving. The facilitator finding also suggests taking any staff replies out of the file first.

What the board summary can claim

The Department for Transport report says its theme-prevalence statistics should be read as estimates, and the board summary can borrow that framing: theme counts as approximate, with a line saying who read how many comments and how often the checker changed the model’s label. The Southampton authors say they could not quantify efficiency or cost-effectiveness, and the Department for Transport’s savings figures are its own projections for its own consultation portfolio, so a claim that the work was faster needs your own timing of the check.

Screen the file for responses that may not come from real members before anything is sorted, using the survey-screen tutorial. What member information may go into a tool at all is covered in the member data check.

The two large evaluations were written by the teams that built the tools, and the reviewers in the Work and Pensions test and the live pilots saw the tool’s answer first. The research studies are small and cover health topics. Tools also change, so test the sort on your own comments before trusting any figure above.