Support ticket confidence score: how Watari decides which tickets become bugs

Watari qualifies a support ticket as a Mapped Bug only when two independent confidence gates both clear: the extraction step must produce a high-confidence structured bug from the raw ticket, and at least one code location in your GitHub or Azure DevOps repository must match with high confidence. If either gate fails, the ticket is routed for manual review — no billing event fires, and no draft PR is opened.

Why two gates instead of one

A single confidence score collapses two fundamentally different questions into one number. The extraction gate asks: is this ticket actually describing a software defect? The code-location gate asks: can we pin that defect to a specific file and function? Answering both with one threshold means you either let noisy tickets through to the code-mapping step (wasting compute and generating irrelevant PRs) or you drop legitimate bugs at extraction because their code-location signal happened to be weak.

Watari separates the two questions deliberately. A ticket from Zendesk, Intercom, or Freshdesk that clearly describes a payment-flow crash with full repro steps passes extraction easily — but if the crash happens in a third-party library with no owned code in your repository, the code-location gate will correctly withhold a PR rather than hallucinate a fix. Conversely, a vague complaint like "the app feels slow" may fail extraction entirely, never reaching the mapping step.

This architecture is documented in detail in the Mapped Bug meter and core concepts pages.

Gate 1: extraction confidence

Extraction confidence measures how well the extraction model can construct a complete, actionable bug report from the raw ticket text.

When a ticket arrives via webhook from Zendesk, Intercom, or Freshdesk, Watari reads the full conversation thread — subject, body, attachments, screenshots, PDFs, HAR captures, and voice transcripts — and attempts to extract a structured record with consistent fields:

• Severity
• Reproduction steps
• Expected vs. actual behavior
• Customer impact
• The customer's original words

The extraction model assigns a confidence score to this output. Tickets that fail to produce a high-confidence structured bug at this stage are not promoted to the mapping pipeline. The practical effect:

• Feature requests lack repro steps and expected/actual contrast — they consistently fail extraction.
• Billing questions describe a business problem, not a software defect — they fail.
• Vague complaints without reproduction steps ("checkout is broken sometimes") produce low-confidence extractions and are held for manual triage.
• Clear bug reports with stack traces, repro steps, and observable failure modes score high and advance to mapping.

This is not a keyword filter. The extraction model reads the semantic content of the full ticket, including attachments. A ticket with a screenshot showing a JavaScript console error and a one-sentence description can still pass — the visual content contributes to confidence.

See the full field list and what gets read in Ticket to bug.

Gate 2: code-location confidence

Code-location confidence measures how precisely a structured bug can be pinned to files and functions in your repository.

Once a ticket passes extraction, the mapping pipeline runs. Watari has already indexed your repository by cloning it, parsing every supported source file with tree-sitter at function-level granularity, and storing vector embeddings of each chunk in a pgvector HNSW index. (Supported languages include TypeScript, JavaScript, Python, Ruby, Go, and Rust — see the Bug to code docs for the current list.)

Mapping works in three steps:

1. The structured bug description is embedded and used to retrieve the top candidate chunks from the pgvector index.
2. A reranking step scores each candidate against the full bug context.
3. Each surviving code_location entry — file path, function name, and a confidence score — is persisted alongside the bug.

The billing meter fires, and the draft PR pipeline begins, only when at least one code_location clears the high-confidence threshold. If the reranker returns candidates but none score high enough, the bug is recorded as mapped at low confidence and routed for manual review. The meter does not fire.

This gate is what prevents Watari from opening draft PRs against wrong files. A hallucinated fix in the wrong module is worse than no fix — the code-location gate is the mechanism that enforces that guarantee.

What "Mapped Bug" means for billing and SLAs

A Mapped Bug is the unit Watari bills on, and it is also the trigger for every downstream action: draft PR generation, PR iteration, RCA drafting, and sync to Linear, Jira, or Slack. All of that is bundled — the only thing the meter charges for is the Mapped Bug qualification event itself.

For support-ops leads setting SLA expectations, this has a direct implication: only tickets that survive both gates generate engineering work. A support queue receiving 1,000 tickets in a month might yield 40 Mapped Bugs. The other 960 are either silently noise-filtered (feature requests, billing questions, "works on my machine" reports) or held in manual review because they described a real bug that couldn't be mapped to owned code with sufficient confidence.

This is not a failure mode — it is the intended behavior. Your engineering team's sprint capacity is not consumed by tickets that Watari cannot responsibly route. See Plans and pricing for how per-period Mapped Bug quotas map to plan tiers.

Tuning the threshold balance in Settings

Watari exposes threshold preferences in Settings → AI Behavior that let you shift the precision/recall tradeoff without touching code.

The two directions:

• Higher precision — fewer draft PRs, more tickets routed to manual review. Choose this when your engineering team has limited review bandwidth or when the cost of reviewing a wrong PR is high.
• Higher recall — more draft PRs, more review work. Choose this when your support queue is high-volume and you want Watari to surface more candidates, accepting that a larger fraction will need human correction.

Changes apply to future processing only — they do not retroactively re-promote or re-demote bugs already in the dashboard. This means you can run a cohort experiment: tighten the threshold for a week, measure how many Mapped Bugs the manual-review queue absorbs, then loosen it and compare.

For support-ops leads, the practical workflow at setup is:

1. Connect your support tool (Zendesk, Intercom, or Freshdesk) and install the GitHub App per the quickstart.
2. Let the first 48–72 hours of tickets flow through at the default threshold.
3. Review the manual-review queue. If legitimate bugs are consistently sitting there, the threshold is too tight — loosen it in Settings → AI Behavior.
4. Review the draft PRs. If they're consistently pointing at wrong files, the threshold is too loose — tighten it.

Full controls are documented in AI behavior controls.

What happens to tickets that fail a gate

Neither gate failure silently discards a ticket. Watari's behavior for each outcome:

• Fails extraction confidence — the ticket is flagged as not qualifying, with a reason visible in the dashboard. No billing event. No PR. The ticket remains in your support tool (Zendesk, Intercom, or Freshdesk) without any mutation.
• Passes extraction, fails code-location confidence — the structured bug is recorded with its extracted fields, but is held in low-confidence status. The bug detail page shows the candidate code locations and their scores. A human reviewer can confirm a location manually, which advances the bug to PR drafting without waiting for the threshold to be cleared automatically.
• Passes both gates — the billing meter fires, the draft PR pipeline starts, and routing to Linear, Jira, or Slack fires if configured.

This means the two-gate model also functions as a triage tool. Bugs that make it through extraction but not code-location mapping are real, structured, actionable bug reports — they're just ones where Watari isn't confident enough to act autonomously. A support-ops lead or engineering manager reviewing that queue is looking at pre-structured bug reports with severity, repro steps, and candidate file locations already filled in, not raw ticket text.

FAQs

Does a ticket that fails both gates cost anything?

No. The billing meter fires only when a bug qualifies as a Mapped Bug — that is, when both the extraction confidence gate and the code-location confidence gate clear. A ticket that fails either gate does not increment your Mapped Bug count. See The Mapped Bug meter for the full qualification logic.

Can I see why a specific ticket was dropped?

Yes. The bug detail page shows the gate that stopped the ticket and the reason. For extraction failures, the page shows which required fields were missing or low-confidence. For code-location failures, it shows the candidate locations and their scores so you can confirm one manually if appropriate.

Does threshold tuning affect tickets already in the dashboard?

No. Changes to threshold preferences in Settings → AI Behavior apply to future processing only. Bugs already in the dashboard — whether Mapped Bugs or held in manual review — are not retroactively re-scored.

What happens if a Mapped Bug turns out to be wrong?

Watari offers a mismapped credit window of 7 days from the billing event. If a bug was incorrectly promoted — for example, it mapped to the wrong module — you can flag it within that window and receive a credit applied to your next invoice. See Invoices, credits, and caps.