Architecture Audit for AI SaaS Products — Pre-Launch and Pre-Series A

Six AI-specific layers get read on top of the standard SaaS audit. Per-customer LLM cost attribution — can you tell which tenant is burning the budget. Eval harness coverage — do you have the regression tests that catch a prompt change shipping a quality drop. Prompt versioning — can you roll back to the prompt that worked yesterday. Model fallback chains — what happens when the primary provider returns a 529 or a 429. Tenant isolation in RAG — the namespace pattern Pinecone documents versus the metadata-filter anti-pattern that bleeds documents across tenants. PII handling in prompts and logs — what your application is actually sending to the provider and what's getting captured by your observability stack. A generic SaaS audit will catch the billing webhook bug; it won't catch the cost-leak bug that comes from a tool-call retry loop firing 14 times against GPT-4 before giving up.

By reading the call site. Every LLM call in your codebase should pass through a thin wrapper that captures tenant ID, model, input tokens, output tokens, cache_creation_input_tokens, cache_read_input_tokens, and the dollar conversion before the response gets returned. If the wrapper doesn't exist — or if half the call sites bypass it for streaming or for a one-off agent path — the cost report at the end of the month is a single line item with no way to break it back down. The audit finds the bypassed call sites, the missing fields, and the places where the wrapper is being called but the result isn't being persisted with a stable tenant key. This is the finding that most often turns into the $50K/month bug a few months later — usually one runaway customer or one agent loop that nobody can pin down because the data isn't in the warehouse.

First, the storage pattern. Pinecone's documented multi-tenancy approach is one namespace per tenant inside a serverless index — physical separation, not a metadata filter. If the RAG layer is using a single shared namespace with a tenant_id filter applied at query time, that's the namespace-bleed anti-pattern: any code path that forgets to apply the filter — a debug endpoint, an admin tool, an embedding refresh job — returns documents across tenants. Second, the embedding pipeline. The audit reads how documents get embedded and indexed, whether the tenant ID is captured at ingest time, and whether deletes actually purge the vectors when a customer offboards. Third, the query path. The audit reads every call to the vector store and confirms the namespace or filter is set from authenticated context, not from a request parameter the client could spoof.

Because prompt regressions are invisible at deploy time and only show up as quality drops in production weeks later. A small change to a system prompt that helps 95% of cases and breaks 5% will not surface in your unit tests, will not surface in your manual QA, and will not surface in your error monitoring — it will surface as a slow drift in customer satisfaction scores or a cluster of support tickets nobody can trace back to the deploy that caused it. An eval harness is a stored set of representative inputs that get run against the new prompt and graded — usually by a stronger model or by hand for the hardest cases — before the prompt ships. The audit finds whether the harness exists, whether it covers the cases your customers actually hit, and whether the deploy pipeline blocks on it.

OpenAI rate-limits across five metrics simultaneously (RPM, TPM, RPD, TPD, IPM) and whichever one trips first throws a 429. Anthropic returns 529s on overload. Both providers also retire models with shutdown dates from three months to a year out. A fallback chain is the code that catches the provider error, switches to a secondary model or provider, and continues the request — usually with a small quality tradeoff but no user-visible outage. The audit looks for whether the chain exists, what triggers the fallback (just 429s, or also 5xx, or also latency over a threshold), and whether the fallback writes to your cost meter as a separate line item so you can see how often it's firing. An AI product with no fallback chain is one OpenAI incident away from a customer-visible outage, and one deprecation email away from a code change that has to ship faster than your normal release cycle.

By tracing what your application actually sends to the LLM provider and what gets captured downstream. The two questions that matter: does customer PII end up in the prompt body, and does it end up in your observability stack. The audit reads how you construct prompts — whether user inputs get scrubbed or pseudonymized before they leave your boundary, whether system context loads from authenticated sources or from request parameters — and reads your logging pipeline to see what fields get captured. Common findings include prompt bodies logged in full to Datadog or Sentry without redaction, agent transcripts written to a database with no retention policy, and tool-call arguments containing names, emails, or account IDs that didn't need to be in the prompt in the first place. If the product touches HIPAA or any regulated regime, the audit also reads which provider endpoints your code is actually hitting versus what your BAA covers.

The audit reads which provider endpoints you're calling and cross-checks them against your BAA coverage in writing. Anthropic's BAA for first-party API customers covers the Messages API plus prompt caching, structured outputs, memory, web search, bash, and text editor — but explicitly excludes the Batch API, Files API, Skills API, Code Execution, Computer Use, and Web Fetch. If your agent path ever calls a non-covered endpoint with PHI in the payload, that call is outside your BAA. The audit catches that drift, and catches the related case where the right BAA exists with the right vendor but a sibling integration (a vector store, an embedding provider, a logging vendor) doesn't have one. The finding gets written in plain language so your compliance lead can act on it without needing a translator.

Same fixed range — $5-8K, 1-2 weeks. The lower end fits a single AI app with one LLM provider, one vector store, and a clean tenant boundary. The upper end fits a multi-app AI platform, multiple providers, multiple model versions in play, or a RAG layer with several embedding pipelines. The AI-specific layers add depth to the read, not separate billable scope — the audit is structured as one engagement with one deliverable, not a base audit plus AI add-ons. The deliverable is the same: severity-tagged written report, recorded screen-share walkthrough, prioritized next-6-weeks list.

Two windows. Pre-launch — the codebase works in the founder's dev environment, the first paying customers are about to come online, and the team wants a senior read before a cost-leak bug or a tenant-bleed bug surfaces in front of users. Pre-Series A diligence — the round is being prepared, the technical due-diligence partner is going to ask about model strategy, cost attribution, eval coverage, and tenant isolation, and the founder wants those answers in a written report instead of in a 45-minute live grilling. The audit is also useful post-launch when costs have started drifting upward without an obvious cause; that one usually surfaces a missing per-tenant cost meter inside the first three days.