Configuring RPS Scoring in Settings

The RPS Scoring page in Settings lets you customize the AI prompts that calculate Relative Performance Scores (RPS) for your tickets, agents, and users. RPS values are AI-generated quality and performance scores on a 1–10 scale (a score of 0 means the item was not scored). This article explains where the page is, what you can configure, how your prompts are used, and what problems to watch for if a prompt is written poorly.

This guide covers configuring RPS scoring. For an explanation of what the resulting Ticket, Agent, and User scores mean and how to interpret them, see Understanding Your RPS Values in ServiceAI.

Where to find it

Go to Settings → RPS Scoring. The page is organized into three tabs — Ticket, Agent, and User — and includes a Recalculate button in the top-right corner.

What you can configure

Each tab contains one or more prompt editors. Every editor has an Edit Prompt button to change the guidance and a Reset to Defaults button to restore the built-in prompt. Leaving a prompt blank uses the built-in default.

Ticket tab

• Ticket Quality Score — Scores how useful a ticket is as context for the AI to resolve similar issues (1–10).

Agent tab

• Agent Score — Scores how well the agent handled each ticket (1–10). Per-ticket scores are averaged into each agent's overall RPS.
• Agent Analysis — The on-demand narrative analysis shown on an agent's detail page. Each editor sets the guidance for one part of the result:
  • Performance Factors
  • Performance Summary
  • Recommendations
  • Coaching

User tab

• User Score — Scores the user's satisfaction with how each ticket was handled (1–10). Per-ticket scores are averaged into each user's overall RPS.
• User Analysis — The on-demand narrative analysis shown on a user's detail page. Each editor sets the guidance for one part of the result:
  • Satisfaction Rating (0–10)
  • Rating Factors
  • Account Perspective
  • Relationship Impression
  • Recommendations

How to edit a prompt

1. Open Settings → RPS Scoring and select the tab you want to edit (Ticket, Agent, or User).
2. Click Edit Prompt on the editor you want to change.
3. Write or adjust the guidance, then save.
4. To discard your customization and return to the built-in prompt, click Reset to Defaults and confirm. (Saving an empty prompt has the same effect.)

How your prompt is used

For the three scoring rubrics (Ticket Quality Score, Agent Score, User Score), the AI receives the ticket's conversation history. Conversations are separated by ||||, and the fields within each conversation are separated by ||:

• the date of the interaction,
• the role — Agent (support) or Customer (the user),
• the name of the person who wrote it, and
• the content of the message.

Your prompt is the rubric the AI follows to choose a score. The result must be a single number from 1 to 10.

Only the rubric text is editable. The field names, item ordering, and the output format the AI must return are handled automatically by ServiceAI, so your edits cannot add, remove, or reorder fields, and cannot change the structure of the result. This means a prompt change can adjust how a score is chosen, but it cannot break the scoring feature itself.

When changes take effect

• Ticket scoring rubrics apply to tickets processed after the change. Existing tickets keep their current scores.
• Agent and User analysis prompts apply the next time an analysis is run.
• To re-score existing tickets after editing a rubric, use Recalculate.

Recalculating existing scores

Click Recalculate to open the Recalculate RPS Scores panel. Choose a range under Re-score tickets opened in (tickets are selected by their open date; choosing All tickets re-scores your entire history). Recalculation re-runs AI scoring on each ticket in the range and then updates the rolling ticket, agent, user, and company averages.

Note: Recalculation re-runs AI analysis on every ticket in the selected range and consumes credits. Larger ranges take longer and cost more. You will receive an email when the recalculation completes.

Tips for writing effective prompts

• Keep the rubric focused on how to choose a single 1–10 score. Describe what a high score looks like versus a low score.
• Be specific and consistent. Vague or contradictory instructions produce inconsistent scores.
• Do not try to change the output format or ask for anything other than a score — the format is handled for you.
• If you are unsure, start from the built-in default (shown in the editor when the prompt is blank) and make small, targeted changes.

What happens if a prompt is entered incorrectly

Because only the rubric text is editable and the output format is owned by ServiceAI, a poorly written prompt cannot break parsing or change the set of scores produced. The problems you might observe are therefore about score quality, not about the feature failing structurally.

Problems you might observe

Zeros that are not caused by your prompt

Some 0 scores are expected and are not a sign of a bad prompt:

• Agent RPS is 0 when no agent ever worked the ticket (no agent replies).
• User RPS is 0 when the requester never commented on the ticket.

A 0 in these cases simply means “not scored,” and those tickets are left out of the averages rather than dragging them down.

How to fix skewed scores

1. Review and revise the rubric on the relevant tab, or click Reset to Defaults to return to the built-in prompt.
2. Use Recalculate to re-score existing tickets so they reflect the corrected rubric. Remember that recalculation consumes credits.
3. Spot-check a few tickets to confirm the new scores look reasonable before recalculating a large range.