Skip to main content

Conversation audits

Conversation Audits configures the hourly root-cause analysis (RCA) job that samples your conversations, works out what went wrong, and files each finding under a category and tag. Use this page to control how much the audit reviews, which conversations it picks up, the root-cause categories it sorts findings into, and the tag vocabulary it uses. The audit's results are read in Insights → Conversation Audits.

Edits are staged. A sticky Save / Discard bar appears at the bottom whenever the current configuration differs from what's saved. Two buttons at the top act immediately, without going through Save:

  • Run once — queues an audit run right now instead of waiting for the hourly schedule.
  • Reset to defaults — restores the platform-default triggers and analysis groups (asks for confirmation first).

General​

The name and per-run limit for the audit job. Audits always cover the previous business day (see How it works).

  • Config name — a label for this audit configuration.
  • Max interactions per run — the maximum number of conversations audited in a single run (minimum 1). Caps cost and runtime.

An Enabled / Disabled tag next to this section shows whether the audit configuration is currently active. (Whether audits run at all for the account is a super-admin toggle — see Super Admin Tools.)

Triggers​

Triggers are the conditions that cause a conversation to be picked up for audit. Each trigger has:

  • An enable toggle — turn the trigger on or off.
  • A label and description — what the trigger looks for.
  • Condition tags — a read-only summary of what the trigger matches: channels, customer moods, and/or CSAT ratings (shown as stars).
  • Focus instructions — free text telling the auditor what to pay attention to for conversations caught by this trigger.

A conversation matching any enabled trigger becomes eligible for the audit.

Analysis groups​

Analysis groups are the root-cause categories the audit sorts findings into, each with its own tag taxonomy. Groups are shown in a collapsible list; each header shows the group's name, an Enabled / Disabled tag, and its tag count. Inside each group:

  • Enabled — a toggle for whether this category is used.
  • Default stream — where findings in this group are routed by default: Platform fix, Business insight, AI quality (internal), or Let planner decide.
  • Instructions — free text guidance for how the auditor should categorise into this group.
  • Tags — the closed vocabulary of tags available in this group. Remove a tag with its close (×) button, or type a label and press Add (or Enter) to create one. New tag labels are automatically converted to a normalised key.

Suggested tags​

A read-only feed of tags the audit job has proposed from recent findings that aren't yet in your taxonomy (looking back 30 days). Each row shows the suggested tag, the group it belongs to, and how many times it was seen. Click Add tag to fold a suggestion into that group's tag list (it then appears under Save like any other edit). Suggestions already present in your taxonomy are hidden, and the section shows "No new suggested tags right now" when there's nothing to review.

How it works​

On its hourly schedule (or when you click Run once), the audit selects up to Max interactions per run conversations from yesterday (midnight to midnight in the business timezone) that match any enabled trigger. It never audits a conversation from an earlier day: a run on September 8 audits only September 7, including late-night conversations and tickets that are still open. An email thread belongs to the day of its latest message. For each one it performs a root-cause analysis, assigns it to an enabled analysis group, and tags the finding from that group's tag list — routing it to the group's default stream. Over time the job proposes new tags it keeps encountering, which surface under Suggested tags for you to accept into the taxonomy. The findings themselves are reviewed in Insights → Conversation Audits.

A ticket's chat sessions, email threads and calls are audited separately, but an issue is recorded only once per ticket. Problems with the ticket as a whole count once per tag, however many conversations show them. These are the Agent handoff, Operations issue, Product or service issue, Policy or commercial issue, Resolution issue and Data or missing information issue groups. A later conversation that shows the same issue updates its last-confirmed time instead of adding a finding. Problems with a reply count once per reply, so three replies with spelling mistakes are three findings. These are the AI response quality, Agent response quality, AI commitment or unauthorized action and SOP adherence groups, any custom group, and the Promised follow-up missing tag. A reply that is quoted again in a later conversation is not counted twice, including when the customer's next email opens a new ticket: a reply found on one of the customer's tickets is not recorded again on their other tickets. The same goes for a ticket problem when the customer's complaint is quoted again on a new ticket: if both tickets cite the same customer message, the problem is recorded once. When one problem is reported under several tags of the same category (for example damaged and missing item), it's one finding that carries all of them.

Operations issue and Product or service issue findings, such as Delay issue or a damaged item, are recorded only when the customer raises the problem in their own message. A delay mentioned only in an AI or agent reply, in order data, or in a ticket task change is not enough, so a customer who only shares an order number or date gets no delay finding. In the conversation view, these findings are marked on the customer's message, not on the replies about it.

When several tickets share one conversation, such as two tickets on the same email thread or one customer's chat on the same day, the conversation is audited once. The other tickets' audits are skipped as already covered while the conversation is unchanged. This includes email threads joined by Merge split email threads: the merged ticket's audit is re-run over the whole conversation instead of a second audit being added.

Unhappy customers are audited first. Each run starts with yesterday's conversations that are not yet audited and match an enabled trigger, and whose ticket has a low CSAT rating (the ratings your enabled triggers look for) or a frustrated, aggressive, potential-escalation or disputed-outcome mood. Only then does the run spend what is left of its limits on other matching conversations, so a busy day doesn't leave unhappy customers unaudited. The default triggers include Potential escalation and Disputed outcome.

When Unhappy Customer Reasons is turned on in Super Admin Tools, each run also gives unhappy tickets one main reason and who can act on it, once their conversation from yesterday has been audited. Tickets whose audited conversation is from an earlier day are not judged. A ticket is unhappy when it has a low CSAT rating (the ratings your enabled triggers look for) or the customer was frustrated, aggressive, threatened to escalate or disputed an outcome. The reasons appear under Why customers are unhappy in Conversation Audits → Unhappy customers.

Audit all conversations​

When Audit All Conversations is enabled in Super Admin Tools, each scheduled or ordinary Run once audit selects conversations from yesterday only, from midnight inclusive to today's midnight exclusive in the business timezone (default Asia/Kolkata). For example, a run on September 8 audits September 7. This includes yesterday's late-night conversations even if they are less than 24 hours old or their tickets are still open.

The previous day's window is fixed when the run starts. Conversations within it are processed oldest first. Previously audited conversations are normally excluded, so repeated runs on the same day continue through that day's remaining conversations. Re-audit rules, such as new email messages, still apply inside the window. Older backlog is never pulled into these runs; when the calendar day changes, the window moves forward even if older work remains. Unhappy customers from yesterday are audited first, ahead of yesterday's other conversations.

With Full coverage, every conversation from yesterday is audited: the daily audit limit (Max/day) does not apply, and runs repeat until the day is done. A day left unfinished when the next day starts is not carried over.

Each account gets its own Max processed/run budget (default 500, configurable from 1 to 2,000). Skipped or failed conversations do not consume it. The job fetches additional pages within yesterday until it completes the configured number of AI analyses or exhausts eligible conversations. The operational CONVERSATION_AUDIT_MAX_INTERACTIONS_PER_RUN ceiling (default 2,000) applies per account. Explicit manual-run limits bound processed conversations for broad audits and selected conversations for trigger-only audits.

Historical passive-email recovery remains available only through an explicit account-scoped Run once API request with passiveEmailBackfillDays (1–30). This intentionally uses the requested historical range instead of yesterday and does not move the scheduled cursor.

Passive email accounts​

When email Passive analysis is enabled, the audit reviews the external email exchange. It reconstructs replies from supported quoted and forwarded email formats, including conversations contained in a single stored email. Internal ticket statuses, assignments, and AI task events are excluded because they do not establish what the external support team did.

Enable Agent response audit to check contradictions, unnecessary repeated requests, and material spelling or grammar errors, including errors in support templates. When a support reply has a clear personal signature, the audit extracts the name and attributes that reply's findings to the signing agent. The Agents view shows each name's finding counts, severity and issue-type distribution, and findings per 100 audited conversations. Replies without a clear signature appear as External support — agent unknown. Names come from signatures, so people using identical signing names are grouped together; initials and full names are not automatically combined. Quoted sender identities and dates are unverified; findings should be reviewed against the original email. Email threads keep their quoted history as context, but a quoted reply sent more than 7 days before the thread's first email reached the platform is not coached again, because it belongs to an earlier conversation.

The audit can only assess messages received by the platform, directly or in quoted history. It cannot establish that an external reply or action never happened just because it is absent. Within the scan window and run limits, previously processed or skipped email conversations become eligible again when a new email arrives or reconstruction changes.

SOP adherence audit​

The SOP adherence audit checks whether a human agent followed the ticket's task after taking over from the AI. Turn on SOP Adherence Audit in Super Admin Tools; it requires Agent Response Audit.

A task counts as the agent's procedure only when the AI could have kept running it:

  • the customer asked for a person;
  • the customer was upset, or disputed an outcome the AI gave;
  • the AI could not finish the task, or an agent turned the AI off and took over;
  • an agent created the ticket or handled the call from the start.

With passive email analysis, your team answers every email from its own mailbox, so each email ticket is checked against the task the AI picked from the customer's emails. Your team's actions happen in your own systems, so the audit judges the replies only. If the task does not fit the customer's request, the ticket is marked not applicable. Tasks that only acknowledge the request, with no questions, actions or decision rules, are not checked.

Handoffs the task itself plans are not checked, because the task does not say what the agent should do next. These are tasks that complete without auto-resolve and so go to an agent, and human-agent objectives that create a child ticket. A ticket is checked once it is resolved, when the audit reviews the conversation that closed it.

The guideline is the task's summary, as it was when the agent took over. If that version is no longer kept, the audit uses the oldest version still kept, and if that version has no summary, the current one. Tasks without a summary are not checked. A good summary covers:

  • when the task is triggered;
  • the details it collects;
  • how it routes and completes, with the conditions and where each one leads;
  • what happens for the customer at the end.

The audit judges broadly whether the agent's handling served what the summary is meant to achieve:

  • Wording, order and format don't matter, and the AI's prompts and completion messages are not used. An outcome reached for a different stated reason still follows the guideline.
  • The summary applies to whoever runs the task. After a handoff the agent is expected to do what remains, even steps the AI would have done, and anything the AI or the customer already did counts as done.
  • Values the AI collected before the handoff are listed, so asking for them again counts as a deviation. So does asking for something the customer already gave in the conversation.

Each ticket is marked followed, deviated, not applicable, done before the handoff, or can't verify. Entering a value or running an action on the ticket counts the same as doing it in the conversation.

The audit gives agents the benefit of the doubt:

  • Work that could have happened outside the conversation and ticket record, such as on a phone call or in another system, is marked can't verify, not deviated.
  • A deviation needs a quote from the conversation or the ticket record.
  • A deviation based only on something being missing is capped at medium severity.

Each deviation becomes a finding in the SOP adherence category of the Agent coaching stream, attributed to the agent. Its tag is one of: Skipped required step, Wrong decision, Action not run, Asked again, Wrong escalation, or Wrong closure. The Agents view adds an SOP adherence column showing the share of judged tickets where each agent followed the guideline; hover over it for the counts. Agents see their own findings on the Agent Findings page, where the Category filter set to SOP adherence shows only these. In Insights → Conversation Audits → Findings, use the Groups filter. When many agents deviate on the same task, its summary is usually unclear or out of date, so review it before coaching agents.