# Automated QA for every call and chat your agents handle.

Write rules in plain English and an LLM judge grades every finished call and chat against them. Failures raise alerts, cluster into issues, and become prompt fixes that are validated by replay before they ship.

## What it does

### Monitor rules written in plain English

Describe what should never happen in plain English: unauthorized discounts, missed escalations, compliance slips. Each rule carries a category, a severity, and a channel scope, and applies to one assistant or the whole workspace. The card shows how often it fired over the last 7 days.

- Five categories: compliance, quality, safety, escalation, custom
- Scope a rule to one assistant or the whole workspace
- Rules target calls, chats, or both
- Disabled rules keep their history but stop judging

### An LLM judge on every finished conversation

When a call ends or a chat reaches its end node, the judge grades the transcript against your enabled rules and returns a verdict with a cited reason and confidence. Grading is idempotent and deduped before spend, so webhook replays and retries never produce duplicate verdicts or bills.

- Verdicts cite the decisive moment in the transcript
- One verdict per rule per conversation, database-enforced
- Conversations are treated as untrusted data while grading
- Inconclusive judge runs are retried later, never marked cleared

### Alerts with the transcript attached

Triggered rules raise alerts you work through in a drawer: the judge's verdict, its confidence, and the full transcript snapshot side by side. Resolve it, dismiss it as a false positive, or hand it to the AI to draft a fix. Cleared evaluations are stored too, so silence is provable.

- Resolve, dismiss as false positive, or reopen in one click
- Transcript snapshots stored up to 50,000 characters
- Dismissals feed the rule's health state
- Judge confidence shown on every verdict

### Fixes drafted by AI, proven by replay

Repeat alerts cluster into issues by assistant and rule. The AI drafts a minimal edit to the flow's global prompt, shows a line diff, and then proves it: the failing conversation is regenerated through your published flow three times and re-judged. Confidence is the measured pass rate.

- Confidence is measured by replay, never self-reported
- Three trials re-judge the regenerated conversation
- Replays never call tools or fire webhooks
- Line-by-line prompt diff before you approve

### Auto-apply behind hard guardrails

Auto-apply ships validated fixes without a human in the loop, and it is off until you opt in. It requires passed validation, replayed regression cases, measured confidence that clears your threshold, and severity within your cap. Applied fixes bank their conversation as a regression case.

- Auto-apply is off by default
- Default bar: 90% measured confidence, low severity only
- Applied fixes bank a regression case
- Failed applies roll back to an actionable state

### Health checks on the monitors themselves

Rules are graded too. Fire rate, drift between time windows, silence, and how often humans overturn a rule's alerts roll into a per-rule health state. A one-click consistency audit re-grades recent transcripts three times each to catch ambiguous prompts and flaky judging.

- Seven health states, from noisy to silent
- Drift detection splits the window and compares fire rates
- Consistency audits re-grade each sample three times
- Human overturns count against the rule

## Under the hood

- **JUDGE:** Every finished call and chat is graded by an LLM judge against your enabled rules
- **RULES:** Plain-English detection prompts with five categories, four severities, per-assistant scope
- **VALIDATION:** Three replay trials per fix; confidence equals resolved over conclusive trials
- **GUARDRAILS:** Auto-apply is off by default; the default bar is 90% confidence and low severity
- **REGRESSIONS:** Applied fixes bank the failing conversation as a regression case that guards future fixes
- **HEALTH:** Per-rule fire rate, drift, dismiss rate, and three-trial consistency audits
- **METERING:** Judge and replay spend is metered to usage records; BYOK runs are not billed

## FAQ

**How does the grading work technically?**

When a call ends or a chat reaches its end node, the transcript goes to an LLM judge with your rule's detection prompt. The judge returns triggered, a one-sentence reason citing the decisive moment, confidence, and severity. Unique database indexes enforce one verdict per rule per conversation, so retries never double-grade.

**Can the AI change my agent without approval?**

Not unless you turn it on. Auto-apply is off by default. When enabled, a fix ships only if validation passed, regression cases were actually replayed, measured confidence clears your threshold (90% by default), and the rule's severity is within your cap. Applied fixes record whether a human or the system approved them.

**What are the limits?**

Each conversation is judged against at most 10 rules per run to keep spend bounded. Transcript snapshots cap at 50,000 characters, fix validation runs 3 replay trials plus the 3 most recent regression cases, and platform-key judge spend is metered to your account like any other LLM usage.

**How do I know the judge itself is accurate?**

The Analytics tab scores every rule: fire rate, drift between window halves, and how often humans dismiss its alerts as false positives. A consistency audit re-grades recent transcripts three times each and reports self-agreement, so ambiguous rules surface instead of misfiring quietly.

**Does it cover voice and chat?**

Both. Rules target calls, chats, or both channels. Voice calls are judged when the call ends; chat and SMS conversations are judged when the conversation ends and in on-demand sweeps. The same alert, issue, and fix pipeline applies across channels.

---
Source: https://www.assistable.ai/platform/observability