Using Regression Testing to Find and Fill Article Gaps in ServiceAI

What Regression Testing Does in ServiceAI

ServiceAI's Regression Test feature replays recent real tickets through your assistant and evaluates the responses against what your knowledge base actually contains. It's the fastest way to answer the question every administrator eventually asks: "If a ticket like this came in tomorrow, would the assistant handle it well?"

A regression run picks up your most recent batch of tickets (the latest 20 by default), feeds the first comment of each one to the assistant as if it were a fresh question, and scores the response. The result is a per-ticket pass/fail picture of how your current knowledge base, rules, and prompts are performing on real-world questions — not synthetic tests.

Regression tests are rate-limited to once per hour per tenant so they don't compete with live traffic.

How a Regression Run Surfaces Article Gaps

When the assistant gives a weak or incomplete answer to one of the test tickets, that almost always means one of two things:

1. An article exists on the topic, but the assistant didn't retrieve or didn't trust it. (A rules or prompt issue.)
2. An article doesn't exist on the topic at all. (A documentation gap.)

ServiceAI looks at every regression run that processes at least three tickets and asks an evaluator model to identify the second case — the documentation gaps. When it finds one, it writes a new Recommendation directly into your Recommendations tab.

These auto-generated recommendations are tagged with the Documentation category and the New status. The recommendation text describes the topic, why the assistant struggled, and what an article on that topic should cover. Lower evaluation scores on the underlying tickets are used to decide which gap is the most valuable to fill first.

The pattern to internalize: regression failures are how ServiceAI tells you which article is missing. You don't have to guess at what your knowledge base is short on — the recommendations show up with the topic and the rationale already written.

Filling a Gap with the Generate Article Prompt

Once a Documentation recommendation surfaces, the next step is to draft the article. ServiceAI's Articles tab includes a Generate Article prompt that takes a plain-English description of the article you want and returns a full HTML draft you can edit and save into your knowledge base.

To use it:

1. Open the Articles tab and choose Generate Article.
2. In the Prompt field, paste or paraphrase the recommendation text from the Recommendations tab. Be specific about the topic, the audience, and the kind of content you want. For example: "Create a comprehensive troubleshooting guide for network connectivity issues in Windows 11."
3. Click Generate. The Content tab fills in with a generated title and body.
4. Edit the title and HTML content as needed.
5. Click Add to KB to save the article into your connected knowledge base, or Save as File to keep an offline draft for review.

The generated draft is a starting point, not a finished article. Treat it the way you would an outline from a subject-matter expert who's never used your product: review it for accuracy, add the specific UI labels and steps that only you know, and remove anything that doesn't match how your team actually does the work.

Rewriting an Existing Article Instead of Creating a New One

Sometimes the regression recommendation points to an article that does exist but is incomplete or out of date. The Generate Article prompt can rewrite an existing article in place. Use this prompt format:

Rewrite article [ID]/[Source] [additional instructions]

For example: Rewrite article 123/0 add a section explaining how to handle credentials with special characters. The article ID and source come from the article's URL or detail page. The additional instructions are anything you want changed or added.

Putting the Loop Together

Regression testing and the Generate Article prompt are designed to be used together as a continuous improvement loop:

1. Run a regression test against your latest tickets.
2. Review the new Documentation recommendations on the Recommendations tab.
3. For each recommendation, decide whether the gap is best filled with a new article or an update to an existing one.
4. Use the Generate Article prompt to draft the change.
5. Review, edit, and save the article into your knowledge base.
6. Run another regression test and confirm the previously failing tickets now get a strong response.

Running this loop on a regular cadence — weekly is typical — keeps your knowledge base aligned with the questions your team is actually being asked, instead of the questions someone thought to write articles about months or years ago. It also gives you a measurable view of knowledge-base coverage over time: as the gap recommendations decrease, you can see your assistant's coverage improving against real tickets.

Tips for Getting the Most Out of Regression Testing

• Run regression tests after every meaningful KB change. If you publish a batch of new articles, run a test to confirm they're being picked up and used.
• Don't ignore low-volume gaps. A single failing ticket on a topic can still represent a category of questions you'll see again. The recommendation will tell you whether the topic is general or one-off.
• Use the recommendation text as your prompt. The text the regression evaluator generates is already structured to describe what an article should cover. You can often paste it directly into the Generate Article prompt with minor edits.
• Keep articles short and named. Articles that name UI elements explicitly — buttons, tabs, fields — perform better in regression tests than articles that rely on screenshots or vague references. Specific names give the assistant something concrete to retrieve and quote.