Your Customers Already Told You

Thematic Analysis for the Tickets Nobody Reads

Carmenta was a Roman goddess of prophecy whose name comes from carmen, meaning song, spell or oracle. She delivered her prophecies out loud, as song, and spoken prophecy has a built-in problem: it exists only while somebody is listening. You cannot set two prophecies side by side, count how often the same warning came up, or hand one to somebody who was not in the room.

Romans credited Carmenta with fixing that. She adapted Greek letters into the alphabet her son Evander carried into Latium, and Rome got a way to hold what had been said. Her gift was not knowing more. It was keeping what was known in a form somebody could go back and read.

Product Managers (PMs) in life science tools have the opposite problem. Our record is already written down, in hundreds of technical support cases, service tickets and voice of customer (VOC) transcripts. What we have never had is the second half of Carmenta’s gift: a way to read them as one body rather than one at a time.

You are not short of data, you are short of a way to read it

There is a specific discomfort in knowing the answer already exists inside your own company and having no practical way to get at it. Somewhere in eighteen months of support cases is the reason your reorder rate slipped. So you commission a survey, and ask customers a question they already answered, for free, in writing, while annoyed enough to be honest.

This bites harder in our sector than most. Scientists stay quiet in public for real reasons, confidentiality and the fear of being scooped (I covered why in the social listening playbook), so consumer social channels carry little of them. Your support queue is the opposite. It is dense, unprompted, and written at the moment the friction happened.

Start here, before you automate anything

Pull thirty technical support or service tickets at random from the last quarter and group them into themes by hand. If you cannot name three recurring themes, the problem is not your data volume, it is that nobody has read them as a body. If you can, you have a human baseline, and everything below is about scaling it without losing the judgment you just used.

Three passes: tag wide, check it, rank

Thematic analysis means reading a body of qualitative material for recurring patterns and organizing them into themes. Braun and Clarke’s 2006 paper set out a six-phase process that became standard across the social sciences, and the phase order is why it works on support tickets: read everything first, tag each case with a one or two word summary, then group the tags into themes, rather than starting with the theme you expect. What has changed is throughput. A large language model (LLM) can tag 500 cases in a few minutes.

What has not changed is that the LLM is a fast assistant, not an analyst. It works in concert with your judgment, and the three passes set out which part is which.

What you send it

Three things:

1. The ticket text and metadata in batches, with a case ID on every row.

2. A glossary of your technical vocabulary, meaning instrument names, error codes, kit names and the abbreviations customers actually type, which is the step most people skip.

3. And an open instruction asking for recurring themes in the customers’ own language. Write “look for reagent handling issues” and you will get reagent handling issues, and miss the themes you were not looking for.

What you ask for in return

Specify the output structure up front, and require four things of every theme:

• The case IDs behind it, so you can open three and read them yourself.
• Prevalence, as a count out of the total, not a percentage. Nine out of 500 is honest. Two percent sounds like a finding.
• Exceptions and uncertainty. A theme with no stated counter-evidence has not been tested, it has been asserted.
• A likely owner, the function that would act on it. This turns a theme into a decision.

Then require a limitations section: sample size, what the data cannot tell you, and whether the themes reflect stated preference or observed behaviour. If the LLM will not write its own caveats, you are reading marketing, not analysis.

Where your judgment goes

Pass two runs once. You are calibrating, not co-working. If the LLM’s themes match yours on twenty cases you read carefully, you have earned the right to trust it across five hundred. If they do not, you have learned that before you took anything to a VP. Group your twenty by hand before you look at the output, or you will simply agree with it (anchoring bias).

Pass three is yours for a different reason. Ask the LLM to rank, because its order is a useful first sort and naming a likely owner per theme saves you an argument later. What it cannot know is that engineering is frozen until March, or that you can get a setup guide rewritten in six weeks without asking anyone. Prevalence is visible in the data. Authority is not.

What this looks like

This example is illustrative. A PM owns a benchtop qPCR instrument program with 500 technical support cases and 150 service tickets over eighteen months. The business believes this is a hardware reliability problem, and the evidence for that belief is the budget. Field service visits and replacement parts show up as a cost line that grows every quarter, with a name attached to it. Nobody has a line item called “customers cannot follow the setup guide,” so that cost stays invisible even though it generates most of the visits.

Pass one returns four candidate themes. The largest, at roughly a third of cases, is not hardware at all. It is reagent handling during setup, clustered around new users in their first six weeks. Hardware faults are real but sit at about an eighth. That is a training and documentation problem, and it changes who owns the fix.

Pass two earns its keep. Grouping twenty cases by hand first, the PM finds the LLM has merged two things: a thermal block that genuinely fails to hold temperature, and a user who sees the resulting error message and does not know what it means. One is an engineering defect, the other a labelling gap. In a ticket queue they can look identical, and collapsing them would have sent the wrong team to fix it.

This is a known property of the tool. When researchers ran LLMs against expert human analysts, the experts often rated the model’s tags as good or better, and still found it splitting one idea into several and drawing theme boundaries where a human would not. It is good at noticing patterns and unreliable about where one ends and the next begins. That is the call deciding whether you send engineering or technical writing.

Pass three sorts what survived. The reagent handling theme wins, not because it is largest, but because a revised setup guide and a two-minute video sit inside the PM’s authority and can be shared in six weeks.

One caution across all three passes: a frequency count is not market prevalence. Tickets over-represent people who hit a problem and contacted you, and say nothing about those who never did.

Write it down before you try to see forward

Carmenta’s gift to Rome was not foresight. It was letters, and the foresight followed, because a city that can read its own record can see what is coming.

Tag wide. Check it by hand. Rank by the decision you own. Your customers have been writing to you for years. The only question is whether anyone in your organisation has read it as a body of evidence rather than a queue to be closed.