RAG & LLM Agents for AI SaaS — Outlives Outages & Deprecations

One provider is enough until it isn't. Anthropic publishes a 99.5% uptime target on its Priority Tier, OpenAI has had multi-hour incidents that ripple through dependent products, and both providers retire models on a published schedule that forces migration. An AI product whose flagship feature stops working when one provider has a bad afternoon is a product whose churn rate is partially controlled by someone else's status page. A fallback chain — a primary model with a typed degradation path to a second provider or a cheaper tier — turns a provider outage from a customer-visible incident into a logged event with a degraded-mode notice.

A single SDK wrapper accepts the model and provider as a parameter resolved per request from a routing rule. The rule can key on tenant ID, feature, plan tier, or a quality-vs-cost flag set per customer. Enterprise tenants might route to Claude Opus on Priority Tier; self-serve tenants might route to a Haiku or a 4o-mini tier; a regulated tenant might pin to a specific provider for data-residency reasons. Every request logs the routed model alongside tenant, tokens, and cost, so the per-tenant economics of a routing rule are visible in the same dashboard as the rest of your AI usage.

Cache invalidation follows a strict tools → system → messages hierarchy. Changing tool definitions invalidates the tools, system, and messages caches in one shot, and changing the system prompt invalidates the system and messages caches. That means a casual prompt edit shipped to all tenants at once is also a cost event — the next call on every tenant pays the cache-write price. The fix is to version prompts as deployable artifacts, canary the new version on a slice of traffic, measure cost and quality delta against the eval set, and then promote — instead of treating prompts as inline strings in application code.

A frozen set of representative queries with expected behavioral properties — not necessarily exact-match expected outputs. Each entry has a tenant context, a query, and one or more graders: a string or schema check for structured outputs, a model-graded check for open-ended outputs, a retrieval-quality check that asserts the right document chunk was in context, and a cross-tenant leakage check that fails the build if a chunk from a different tenant surfaces. The harness runs on every prompt change, every model swap, and on a schedule against production prompts so silent drift is caught.

The wrapper isolates provider and model behind a typed interface, so the swap is a configuration change. The eval harness runs the frozen query set against the candidate model and reports the behavior and cost delta before the change ships to any tenant. If a regression appears, the migration plan is to ship a routing rule that keeps high-value tenants on the deprecated model until its retirement date and moves lower-stakes traffic first, buying time to tune prompts for the replacement. The deprecation email becomes a planning artifact instead of an emergency.

Prompts are versioned artifacts in the repo, each with an identifier the request layer includes when calling the model. A new prompt version ships first to a canary slice — internal tenants, then a small percentage of external traffic — while the eval harness runs against the candidate on representative production queries. If the canary's success rate or cost falls outside the configured tolerance, the deploy halts. Only after the canary passes does the new version promote to all tenants. The cost of doing this is small; the cost of skipping it is a product-wide regression the support team finds out about from customers.

Retrieval quality is treated as a first-class signal alongside latency and cost. Each retrieval call logs the query, the retrieved chunk IDs, the similarity scores, and — when the downstream answer is graded — whether the right context was in the top-K. A drift detector watches the rolling distribution of top-K scores per tenant and per feature; if scores degrade after a re-embedding, an index resize, or a new document corpus lands, the alert fires before the answer-quality eval would catch it indirectly. The metric is owned by your team because vendor RAG products don't expose it at this resolution.

For a Series A or A-prime AI startup with a working LLM feature already in production, the fallback chain, per-customer routing, prompt versioning, and eval harness layer is typically a 4–6 week build, depending on how many providers and features need to be wrapped. Most of the time goes into the eval suite — picking the queries that actually exercise the failure modes — not into the SDK wrapper. The shorter the existing surface area, the faster the build.