What breaks when you put a RAG chatbot in front of customers

AgentStack is a multi-tenant platform our team runs: businesses build an AI agent trained on their own website, files and videos, then embed it on their site as a chat widget with streaming text answers, spoken replies and real-time voice. The same platform powers several branded deployments for different clients. This year I added real-time voice end to end, spoken replies and faster website scraping.

Retrieval-augmented generation sounds like a retrieval problem. In production, a lot of what goes wrong happens before retrieval or after it. These are the places I'd look first.

Ingestion decides answer quality

If the right text never makes it into the knowledge base, no model can answer from it. For agents trained on websites, that starts with scraping.

AgentStack collects page URLs from a site's sitemap, fetches them and keeps text from content elements: paragraphs, headings, articles and tables. That keeps menus and footers out. It also means text that lives only in other elements can be skipped, and a page that renders its content with JavaScript can come back nearly empty from a plain HTML fetch.

When I made scraping faster, much of the work was really failure handling. The old code requested every page at once and waited for all of them, so one slow or broken page could sink the whole import. Now pages are fetched in batches of ten, each with a 15-second timeout, and whatever succeeds is kept.

That fix raises its own question: a partial import is better than a failed one, but only if someone can see what's missing. Whatever your stack:

• Record which pages failed or came back near-empty, and why.
• Show customers what the agent actually learned, not just a success tick.
• Don't trust sitemaps blindly; they can miss pages or list dead ones.
• For files and videos, read a sample of the extracted text. An agent trained on a video is only as good as its transcript.

Stale content

An agent's stored text is a snapshot from when it was trained. When the business changes its prices or opening hours, the website is right, the chatbot is wrong, and it sounds equally sure of itself.

Re-scraping on a schedule helps, as does using the sitemap's lastmod dates to re-fetch only changed pages. So does showing the training date in the dashboard, so customers know when to retrain. For facts that change daily, such as stock, availability or prices, don't put them in the knowledge base at all: give the agent a tool that looks them up live.

Chunking, and why our vector database is switched off

The standard recipe splits documents into chunks, embeds them, and retrieves the closest few for each question. Chunks bring their own bugs: a line saying "plans start at 49" with no product name, a table split in two, an FAQ answer separated from its question. Anthropic's write-up on contextual retrieval describes this loss of context and points out that a knowledge base under about 200,000 tokens can simply go into the prompt whole.

That's the trade-off we made. A Pinecone path exists in AgentStack's code, with a namespace per agent, but it's switched off. Each agent's source text is stored in MySQL, and the answer pipeline puts that agent's full text into the prompt. When the text fits comfortably in the context window, this gives us:

• No chunking bugs, because there are no chunks.
• No embeddings to maintain. The vector path needed a separate index per embedding provider; the text path doesn't care which model answers.
• One less service to run, pay for and keep in sync.

The costs are real. Every question sends the full text, so input tokens scale with the size of the site. There's a ceiling on how much an agent can know. And long inputs aren't read evenly: the 2023 Lost in the Middle study found models used information in the middle of long contexts less reliably than at the start or end.

Two things soften this. Put the stable source text first and the question last, so the provider's prompt caching can reuse the prefix; caching needs an exact prefix match. And set a size threshold above which an agent switches to retrieval, rather than discovering the limit when answers degrade.

A vector database earns its place when the corpus won't fit in the prompt, when different users may see different documents, or when you need to cite the exact passage.

Grounding and refusing

A customer-facing bot that answers outside its material is worse than one that says it doesn't know. AgentStack grounds answers in two places.

The answer prompt tells the model to answer only from the provided context, keep it short, make no assumptions, and include only links that already appear in the source text. An invented URL is an especially embarrassing failure, because customers click it.

The pipeline also has a no-data branch: if retrieval returns nothing, it sends a fixed reply without calling the model. With full text in the prompt there is always context, so that branch effectively never fires and the prompt does the work. Test it with questions the site doesn't cover.

Treat scraped content as untrusted input. OWASP's top LLM risk, prompt injection, includes indirect injection through websites and files, and OWASP notes that RAG doesn't fully mitigate it. A customer's own pages are usually low risk, but reviews, comments and forum posts carry other people's text into your prompt.

Multi-tenant isolation

On a platform where many businesses each have an agent, the worst bug is one agent answering from another's content. Isolation should be structural, not a filter you hope was applied.

In AgentStack, the answer path reads source text with a query keyed to a single agent, so there's no shared pool to filter. With a shared vector index, the equivalent is a namespace or index per tenant rather than a metadata filter someone can forget.

The same rule covers everything around retrieval: chat history, leads, uploaded files, caches and logs. Voice is one more path to the same data. When I added real-time voice, the voice model didn't get its own copy of the knowledge; it calls a rag_search tool that goes through our API. One path to the data means one place to enforce isolation.

Evaluation

Trying a few questions after a change isn't evaluation. Build a small question set per agent, drawn from real chats where you can, and re-run it after every scraper, prompt or model change.

Score retrieval and generation separately. Ragas, for example, defines faithfulness as the share of an answer's claims that the retrieved context supports. Without a vector database, the "retrieval" question becomes an ingestion question: did the fact get scraped at all? That's a cheap string search over the stored text, and it catches scraping regressions before a model is involved.

Cost

Cost comes from three places: ingestion (scraping, extraction and transcription), answers (input tokens per question), and voice, which has its own audio pricing. With full text in the prompt, answer cost scales with site size times traffic, so one tenant with a large site and a busy widget can outweigh dozens of small ones. Track cost per agent, not just per platform, and use the size threshold as a cost control as well as a quality one.

Conclusion

A better model won't fix missing pages, stale facts, confident answers outside the source or leaks between tenants. Those are pipeline problems, and they're where customer trust is won or lost. Start with the simplest retrieval that fits your data, which is sometimes no retrieval at all, and add machinery only when an evaluation shows you need it.