Production RAG and LLM Agent Engineering for Funded Startups
Multi-tenant RAG with namespace isolation, agent tool-call loops that recover from failure, prompt-cache-aware cost engineering, and per-request token logging — built so your AI feature ships once and stays shipped.
The problem
The demo worked. The investor video looked great. Then a real customer signed in, asked a question their team had asked the chatbot in their last company's tenant, and got back a snippet from a competitor's pitch deck — because the vector store was a single namespace and "tenant_id" was a metadata filter that one of the retrieval helpers forgot to apply. Or the agent that confidently orchestrated three tool calls in the demo started timing out on the second call in production, returned a partial answer, and a customer noticed before the on-call engineer did. Or the OpenAI bill that was $400 in February came in at $11,000 in April with no clear explanation of which feature, which tenant, or which prompt was responsible.
Retrieval that bleeds across tenants
The first failure mode is retrieval bleed. A single-namespace vector store with tenant_id as a metadata filter puts the whole isolation guarantee on the caller. One retrieval helper forgets the filter and one customer's chunks surface in another customer's response. The bug never appears in the prototype because there's only one tenant; it appears the week after launch.
Agent loops that fall apart on the first tool failure
The second failure mode is agent fragility. The demo orchestrated three tool calls cleanly because the tools all worked. In production, the second tool times out, the model returns a partial answer, and the agent run crashes instead of recovering — because retry, timeout, and error-shape handling is the calling application's responsibility, not the model's, and nobody built that layer.
AI cost curve growing faster than revenue
The third failure mode is the cost surprise. The OpenAI bill was $400 in February and $11,000 in April with no clear explanation of which feature, which tenant, or which prompt was responsible. There's no per-request token logging, no prompt-cache-aware structure on the long stable head of every prompt, and no per-tenant rollup the CFO can defend.
Infrastructure layer, not the prompt
None of these appear in the prototype. All of them appear the week after launch. And the gap between "the LangChain example works" and "this survives 200 concurrent customer sessions" is where most AI-feature engineering teams spend an unplanned quarter. For a Series A AI startup with a public roadmap that promised three agent-powered features in Q3, the cost of getting this wrong is not just engineering time. It is the support ticket from the customer whose data appeared in another tenant's chat history, the renewal conversation where the buyer cites "the chatbot was unreliable", and the board meeting where AI cost-of-goods is suddenly a margin question. The prompt-engineering tweet that promised "we cut hallucinations 80%" does not fix any of these. What fixes them is the boring infrastructure layer underneath the prompt.
What changes for your business
A production RAG and agent stack treats the LLM as one component of a system, not the system itself. The boundary is clear: the model handles language, your application owns retrieval, identity, retry, and cost. That means every retrieval call carries the tenant identifier as a typed argument (not a string concatenated into a filter), every tool-call loop has a max-depth and a typed error contract, every request logs token counts against a tenant and feature label, and every prompt is structured so the long, stable parts — system instructions, tool schemas, retrieved context — hit the provider's prompt cache on the second request rather than getting billed at full input rate.
Tenant isolation enforced at the store
Retrieval becomes safe because tenant isolation is enforced at the vector store layer (a dedicated Pinecone namespace per customer, or a per-row RLS policy on a pgvector table) rather than at the application layer where one missed filter turns into a data incident. A typed retrieve(tenantId, query) signature makes a cross-tenant call a compile error rather than a runtime bug.
Agent reliability as a typed retry contract
Agent reliability becomes measurable because tool failures are caught, logged, and retried with the model in the loop — using Claude's interleaved thinking or OpenAI's parallel function calls — rather than crashing the whole turn. Transient network errors, schema violations, and semantic failures each feed a different tool_result shape back to the model so it can recover instead of giving up.
Prompt-cache-aware cost engineering
AI cost becomes predictable because the cacheable portion of every prompt is structured for cache hits (Anthropic charges cache reads at one-tenth the base input price, and OpenAI's automatic prompt caching applies similar discounts), and because every request writes a row to an llm_usage table that finance can slice by customer, feature, and model.
Eval suite, not the demo, gates a ship
What changes for the reader's business: the AI feature stops being the part of the product that the team is nervous about. New tenants onboard without a code change. New tools attach to the agent without rewriting the retry loop. Cost per active tenant becomes a number the team can show in a board deck instead of a line item nobody wants to explain. And the eval suite — not the demo — becomes the thing that decides whether a prompt change ships.
What gets shipped
An engagement leaves your repository with the layer between the LLM SDK and your product code. Concretely:
- A retrieval module with tenant isolation enforced at the vector store — dedicated namespaces on Pinecone or per-row Row-Level Security on pgvector, plus a typed
retrieve(tenantId, query)interface that makes a cross-tenant call a compile error rather than a runtime bug. - An agent orchestration layer that wraps the model's tool-use loop with max-depth, per-tool timeouts, typed error contracts, and the prompt-caching headers that Anthropic and OpenAI both support — so the same conversation costs less on the second turn.
- A tool-call retry layer that distinguishes the three error classes agents fail on (transient network, schema violation, semantic failure) and feeds the right shape of
tool_resultback to the model so the agent can recover instead of giving up — the area Anthropic's tool-use guide describes as the calling application's responsibility, not the model's. - A
llm_usagetable with per-request rows for input tokens, output tokens, cached tokens, tenant, feature, model, latency, and cost — populated by a thin SDK wrapper, exposed to your finance and product teams via a dashboard, with budget alerts when a tenant or feature crosses a threshold. - An eval harness that runs a frozen set of representative queries against every prompt or model change, blocks the deploy if quality drops or cross-tenant leakage appears, and reports the cost delta of the change before it ships.
- A prompt structure convention that puts stable system instructions and tool schemas first, dynamic context second, and the user turn last — so the long, stable head of every request hits the prompt cache on every call after the first.
- Runbooks for the failure modes that actually happen: a tool that starts returning malformed JSON after an upstream version bump, a vector index that drifts after a re-embedding, a tenant whose token usage suddenly 10x's, a model deprecation that needs a swap with no behavior regression.
What buyers ask first
Technical founders evaluating an AI engineering engagement tend to ask the same five questions. "Will the agent stop hallucinating?" — No, but the eval harness will catch regressions before customers do, and grounded retrieval is the structural answer to the failure mode you actually have. "How do I keep tenant data isolated?" — Enforce it at the store, not the application; namespaces or RLS, with an eval that fails the build on a cross-tenant retrieval. "How do I keep AI cost from blowing up?" — Per-request logging plus prompt-cache-aware prompt structure, both shipped on day one rather than after the first finance review. "What happens when OpenAI deprecates the model we're on?" — The wrapper isolates provider and model so the swap is a config change; the eval harness confirms behavior parity. "Can my team maintain this after you leave?" — The code lives in your repo, follows your existing conventions, and the runbooks cover the failure modes that page an engineer at 2am. The FAQ below covers the longer answers.
How BoostFrame approaches this
BoostFrame Engineering AI (BFEAI) runs a six-engine LLM orchestration in production today — ChatGPT, Claude, Gemini, Perplexity, AI Overview, and AI Mode — across the production apps that have generated 200K+ AI-assisted keywords, run 1,500+ AI scans, and automated work for 7,000+ customer sites. The retrieval, agent, and cost-logging stack described above is the same one those production apps run on. The engagement is sized to your stage: a RAG and agent engagement for a seed or Series A startup is typically a 3–6 week build, scoped to one or two flagship AI features and the shared infrastructure underneath them, with the goal of leaving your team confident enough to ship new AI features on top of it without bringing an outside engineer back in.
The deliverable is working code in your repository, with tests, with a per-request cost log finance can audit, with the eval harness wired to your CI, and with the runbooks required to keep the AI feature working at 2am on a Sunday when a model deprecation, a vector index drift, or a tool schema change would otherwise wake somebody up.
FAQ
The questions below come up in nearly every first call. Short answers here; longer conversations on a 15-minute architecture read.
Outcomes you should expect
What this delivers
- Ship AI features that survive real customer load — retrieval scoped per tenant, tool-call loops that retry instead of crash, and a logged failure mode you can actually debug.
- Cut LLM cost per request by 60–90% on the cacheable portion of your prompts, by structuring system prompts and tool schemas to hit Anthropic and OpenAI prompt caches on every call.
- Stop the silent retrieval bug where one tenant sees another tenant's documents — namespace isolation, per-request tenant assertions, and an eval suite that fails the build if a cross-tenant leak appears.
- Get an AI cost line your CFO can defend — per-request token logging, per-tenant cost rollups, and budget alerts wired before the OpenAI bill triples in a month.
Industry data
By the numbers
Anthropic's prompt caching charges cache reads at 0.1x the base input token price and 5-minute cache writes at 1.25x, which means a stable system prompt and tool schema reused across requests can cut input-token cost on that portion by roughly 90% versus an uncached call.
Claude tool use follows an explicit agentic loop: the model returns stop_reason 'tool_use' with one or more tool_use blocks, your application executes the call, and you send back a tool_result — making the retry, timeout, and error-shape handling the calling application's responsibility, not the model's.
OpenAI's function-calling guide recommends keeping fewer than 20 functions available at the start of a turn and suggests deferring rarely-used tools via tool search for large tool ecosystems, because every function definition counts as input tokens on every request.
Pinecone supports multitenancy by giving each customer a dedicated namespace within a single index, with all upserts and queries targeting one namespace at a time so retrieval cannot cross tenant boundaries by accident.
Supabase's pgvector guidance recommends HNSW over IVFFlat for production workloads because of its performance and robustness against changing data, with pgvector 0.7.0+ supporting up to 2,000 dimensions for the vector type and up to 4,000 for halfvec.
Live in production today
The same engineering, shipped in production at BFEAI.
I'm co-founder & CTO of Be Found Everywhere (BFEAI), a 7-app AI SaaS platform running today. The work I deliver for clients is the work I do every week on my own platform.
7
Production apps
200K+
Keywords generated
1,500+
AI scans run
7,000+
Sites automated
Common questions
What buyers ask before reaching out
How do you stop one tenant's data from showing up in another tenant's retrieval results?
Tenant isolation is enforced at the vector store, not the application code. On Pinecone, each tenant gets a dedicated namespace and every upsert and query targets exactly one namespace. On pgvector, the tenant column has a Row-Level Security policy that the database enforces regardless of which application path called the query. The eval harness includes a cross-tenant leakage test that fails the build if a query for tenant A surfaces a chunk owned by tenant B.
What's the realistic cost reduction from prompt caching?
Anthropic charges cache reads at one-tenth the base input token price and 5-minute cache writes at 1.25x. If 60% of your prompt is stable system instructions and tool schemas that hit the cache, the effective input-token cost on that 60% drops by roughly 90% after the first request in a 5-minute window. OpenAI's automatic prompt caching applies similar discounts. The savings depend on how you structure prompts — long stable head, short variable tail — which is part of what the engagement ships.
How do you keep an agent from falling apart when one tool call fails?
The agent loop wraps the model's tool-use response with a typed error contract. Network and timeout errors get retried with backoff. Schema violations get caught and fed back to the model as a `tool_result` with the error shape, so the model can correct and retry. Semantic failures (the tool returned valid data but the wrong answer) get surfaced to the model with context so it can choose a different tool or ask the user. Anthropic's tool-use guide is explicit that this retry and error-shape handling is the calling application's responsibility — the engagement builds that layer once for all your agents.
How do I keep my OpenAI or Anthropic bill from surprising me?
Every request flows through a thin SDK wrapper that writes a row to a `llm_usage` table with input tokens, output tokens, cached tokens, tenant ID, feature name, model, latency, and computed cost. That table powers a dashboard your finance and product teams can slice by customer or feature, plus budget alerts that fire before a tenant or feature crosses a configurable threshold. The wrapper is the same one the agent and retrieval modules use, so coverage is complete by construction.
Can my team maintain this after the engagement ends?
Yes — that's the design constraint. The code lives in your repository, in your stack (TypeScript or Python), following your existing conventions. The retrieval, agent, and cost-logging modules each have a small, typed interface that the rest of your code calls. New AI features attach to those interfaces without rewriting them. The runbooks cover the specific failure modes that page an engineer at 2am — model deprecation swap, vector index drift, malformed tool output, sudden tenant cost spike — with the exact commands to run for each.
What if OpenAI or Anthropic deprecates the model we're built on?
The LLM SDK calls go through a wrapper that isolates the provider and model behind your application's tool and prompt interfaces, so a model swap is a configuration change rather than a code rewrite. The eval harness then runs the same frozen query set against the new model and reports any behavior or cost delta before the change ships, so the swap isn't a leap of faith.
Do you work with LangChain, LlamaIndex, or roll-your-own?
Whichever fits your stage and team. LangChain and LlamaIndex are fine starting points but the production-critical pieces — tenant isolation, retry contracts, cost logging, eval harness — are the same regardless of which framework you use, and most of them live above or beside the framework rather than inside it. The decision is driven by your existing codebase, not by a preference.
Ready to see if this is a fit?
A 15-minute call. No deck, no slides. We talk about what you're shipping and where engineering is the bottleneck. Either way, you walk away with a senior engineer's read on your situation.