When your agent is consistently underperforming, you have two options: a partial retrain (updating specific parts of the knowledge base) or a full reset (clearing everything and starting fresh). Here's how to decide which you need — and how to do each one.
When to reset vs retrain
Most of the time, you need a partial retrain — not a full reset. A full reset makes sense only when:
- The business has fundamentally changed (new services, rebrand, completely new audience)
- The knowledge base is so polluted with outdated, contradictory content that targeted fixes aren't working
- The agent was built on a client's old website and they've had a complete redesign
For most underperforming agents, targeted updates to the knowledge base and system prompt will fix the issue without starting over.
Partial retrain — how to do it
A partial retrain means updating specific content while keeping the rest intact:
- Identify the specific topics/questions the agent is getting wrong
- Find the relevant KB entries and update or delete them
- Add new entries with correct, specific information
- If website content has changed significantly: re-scrape the affected pages
- Update the system prompt if tone or scope has shifted
- Re-test the specific questions that were failing
This process takes 20–30 minutes and fixes the vast majority of performance issues.
Full rebuild — how to do it
If a fresh start is genuinely needed:
- Export or note down the existing system prompt (you'll reuse and improve it)
- Delete all current KB sources (websites, documents, manual entries)
- Re-scrape the website from scratch
- Re-upload any relevant documents
- Rewrite manual Q&A entries based on the current list of customer questions
- Update the system prompt with any improvements you've learned
- Run the full 20-question test before going live
Don't lose good content Before doing a full rebuild, export your list of manual Q&A entries. These took time to write and many will still be useful — copy them into a doc before clearing the KB so you can re-import the ones that still apply.
Retrain checklist
- ☐ Identified the specific failing questions
- ☐ Removed outdated/incorrect KB entries
- ☐ Added correct entries with specific detail
- ☐ Re-scraped if website content changed
- ☐ Updated system prompt if needed
- ☐ Ran 20-question test — 80%+ passing
- ☐ Tested the specific questions that were failing before