CTCH-0001 · HIGH Repair specified
Bot quoted a refund policy that was retired 4 months ago
Customer “Can I get a refund after 30 days?”
Bot “Yes — we offer refunds up to 60 days after purchase.”
Wrong-answer reliability for customer-facing AI
Know what your bot will say before your customers do.
We catch wrong answers, trace them to the policy or setup that caused them, prove the repair, and give you a signed go/no-go.
Explore the live product Live · no sign-up
See a sample catch ↓LIVE PRODUCT · CLICK ANYWHERE TO TAKE OVER LIVE PRODUCT · TAP TO EXPLORE
See the trail from wrong answer to fixed answer.
The product below is live. The catch is illustrative. The method and evidence fields show exactly what an engagement is designed to produce.
Live product
Click through the workspace. No sign-up.
Visible evidence
Question, source, cause, repair, retest.
Illustrative sample
Composite scenario, labeled everywhere.
How we work
Every wrong answer gets a paper trail.
We reproduce it, find what the bot relied on, specify the repair, and keep testing until the answer stays right.
- 01 Reproduce
Ask the same question and two paraphrases until the wrong answer repeats.
- 02 Find the evidence
Trace the answer to the model, the help content, or the setup that produced it.
- 03 Write the repair
Name the exact change, who owns it, and what a passing answer must do.
- 04 Retest the answer
Run the original question and the surrounding risky questions again.
- 05 Sign the decision
Record what passed, what did not, and whether the answer is ready for customers.
Illustrative sample · Composite of publicly reported failure patterns
A deleted refund policy was still answering customers.
The bot did what its sources told it. The evidence showed which article was wrong—and the exact repair.
Reproduction Reproduced 3 of 3 times
Source Refunds & Returns (v2)
Cause Conflicting source content, not the vendor
Repair Archive v2, add retirement checklist, run 12-question refund probe.
Read full evidence
System: Support bot (Intercom Fin) · Status: Repair specified
What happened
A customer asked: “Can I get a refund after 30 days?”
The bot answered: “Yes — we offer refunds up to 60 days after purchase.”
The 60-day policy was retired. The current published policy is 30 days.
Evidence
- Reproduced 3 of 3 times with the same question and two paraphrases; transcripts attached.
- Retrieval trace: the answer sourced from “Refunds & Returns (v2)”—superseded by v3, but never archived and still ranking first.
- The vendor’s model behaved correctly given its sources. Root cause: conflicting source content, not the vendor.
Repair specification
- Archive “Refunds & Returns (v2)” in the help center.
- Add a retirement checklist step: superseded articles archived same-day.
- Retest with the included 12-question refund probe set.
This is a composite of publicly reported failure patterns and is presented as a sample, never as a client result.
Illustrative sample · Illustrative result
The repair is not done until the answer stays right.
12 / 12 correct after repair
Re-verified at 7 days
Refund-topic answers are cleared for customer traffic as of May 23, 2025. — Alex Morgan, Founder & Lead Auditor, Stele
Start with the wrong answers
Bring us the conversations your bot got wrong.
Send three anonymized conversations. We’ll reproduce the failures, trace the cause, and tell you what to fix first.
Live product · evidence trail · repair retest · signed decision