Platform · Observability
Automated QA for every call and chat your agents handle.
Write rules in plain English and an LLM judge grades every finished call and chat against them. Failures raise alerts, cluster into issues, and become prompt fixes that are validated by replay before they ship.
Get started for free. Pay as you go. Or talk to sales
What it does
Monitor rules written in plain English
Describe what should never happen in plain English: unauthorized discounts, missed escalations, compliance slips. Each rule carries a category, a severity, and a channel scope, and applies to one assistant or the whole workspace. The card shows how often it fired over the last 7 days.
- Five categories: compliance, quality, safety, escalation, custom
- Scope a rule to one assistant or the whole workspace
- Rules target calls, chats, or both
- Disabled rules keep their history but stop judging
An LLM judge on every finished conversation
When a call ends or a chat reaches its end node, the judge grades the transcript against your enabled rules and returns a verdict with a cited reason and confidence. Grading is idempotent and deduped before spend, so webhook replays and retries never produce duplicate verdicts or bills.
- Verdicts cite the decisive moment in the transcript
- One verdict per rule per conversation, database-enforced
- Conversations are treated as untrusted data while grading
- Inconclusive judge runs are retried later, never marked cleared
Alerts with the transcript attached
Triggered rules raise alerts you work through in a drawer: the judge's verdict, its confidence, and the full transcript snapshot side by side. Resolve it, dismiss it as a false positive, or hand it to the AI to draft a fix. Cleared evaluations are stored too, so silence is provable.
- Resolve, dismiss as false positive, or reopen in one click
- Transcript snapshots stored up to 50,000 characters
- Dismissals feed the rule's health state
- Judge confidence shown on every verdict
Fixes drafted by AI, proven by replay
Repeat alerts cluster into issues by assistant and rule. The AI drafts a minimal edit to the flow's global prompt, shows a line diff, and then proves it: the failing conversation is regenerated through your published flow three times and re-judged. Confidence is the measured pass rate.
- Confidence is measured by replay, never self-reported
- Three trials re-judge the regenerated conversation
- Replays never call tools or fire webhooks
- Line-by-line prompt diff before you approve
Auto-apply behind hard guardrails
Auto-apply ships validated fixes without a human in the loop, and it is off until you opt in. It requires passed validation, replayed regression cases, measured confidence that clears your threshold, and severity within your cap. Applied fixes bank their conversation as a regression case.
- Auto-apply is off by default
- Default bar: 90% measured confidence, low severity only
- Applied fixes bank a regression case
- Failed applies roll back to an actionable state
Health checks on the monitors themselves
Rules are graded too. Fire rate, drift between time windows, silence, and how often humans overturn a rule's alerts roll into a per-rule health state. A one-click consistency audit re-grades recent transcripts three times each to catch ambiguous prompts and flaky judging.
- Seven health states, from noisy to silent
- Drift detection splits the window and compares fire rates
- Consistency audits re-grade each sample three times
- Human overturns count against the rule
At scale
Built for the agent lifecycle.
01 · Deploy
No logging pipeline or SDK to wire up. Judging hooks into the call-end webhook and chat end events that already run, so rules you write on day one grade the first production conversation.
02 · Test
Run now sweeps recent unevaluated calls and chats on demand, so test conversations get verdicts before launch. Candidate fixes are validated against real failing transcripts, not synthetic ones.
03 · Scale
Evaluation is capped at 10 rules per run and deduped in the database, so webhook replays never double-bill. Repeat failures collapse into one issue with an occurrence count, not an alert flood.
04 · Observe
Every verdict is stored with its transcript snapshot, alerts carry the judge's cited reason and confidence, and per-rule health flags noisy, drifting, silent, or human-overturned monitors.
Under the hood
- JUDGE
Every finished call and chat is graded by an LLM judge against your enabled rules
- RULES
Plain-English detection prompts with five categories, four severities, per-assistant scope
- VALIDATION
Three replay trials per fix; confidence equals resolved over conclusive trials
- GUARDRAILS
Auto-apply is off by default; the default bar is 90% confidence and low severity
- REGRESSIONS
Applied fixes bank the failing conversation as a regression case that guards future fixes
- HEALTH
Per-rule fire rate, drift, dismiss rate, and three-trial consistency audits
- METERING
Judge and replay spend is metered to usage records; BYOK runs are not billed
FAQ
How does the grading work technically?
When a call ends or a chat reaches its end node, the transcript goes to an LLM judge with your rule's detection prompt. The judge returns triggered, a one-sentence reason citing the decisive moment, confidence, and severity. Unique database indexes enforce one verdict per rule per conversation, so retries never double-grade.
Can the AI change my agent without approval?
Not unless you turn it on. Auto-apply is off by default. When enabled, a fix ships only if validation passed, regression cases were actually replayed, measured confidence clears your threshold (90% by default), and the rule's severity is within your cap. Applied fixes record whether a human or the system approved them.
What are the limits?
Each conversation is judged against at most 10 rules per run to keep spend bounded. Transcript snapshots cap at 50,000 characters, fix validation runs 3 replay trials plus the 3 most recent regression cases, and platform-key judge spend is metered to your account like any other LLM usage.
How do I know the judge itself is accurate?
The Analytics tab scores every rule: fire rate, drift between window halves, and how often humans dismiss its alerts as false positives. A consistency audit re-grades recent transcripts three times each and reports self-agreement, so ambiguous rules surface instead of misfiring quietly.
Does it cover voice and chat?
Both. Rules target calls, chats, or both channels. Voice calls are judged when the call ends; chat and SMS conversations are judged when the conversation ends and in on-demand sweeps. The same alert, issue, and fix pipeline applies across channels.
