Private AI — running language models on the client's own hardware, with no third-party API in the data path — is a legitimate architectural choice with legitimate trade-offs. We have built private AI deployments for clients whose threat model required them. We have also told other clients, in writing, that the private AI architecture they were asking for was the wrong tool for the work they actually had.
This is the honest version of when private AI is the right call, and when it is not.
When private AI is the right call
The clean signal is a regulatory or contractual constraint that prevents sending the data anywhere off the network boundary. Three categories show up most often.
The first is legal privilege. Law firms operating on matters where the worst case is privileged material in the wrong hands face a threat model that the major model APIs do not address. Even with strong contractual protections — no training on submitted data, EU-based processing, no human review — a privileged document sent to a third-party model has crossed the privilege boundary in a way that a partner cannot defend on cross-examination. Tulgra exists for exactly this reason.
The second is HIPAA and similar regulated health data. The major model providers offer Business Associate Agreements, and for many use cases a BAA is sufficient. But organisations running clinical trials, building diagnostic tools, or processing identifiable patient material at scale increasingly conclude that the simplest defensible architecture is one where the data physically does not leave the boundary. The BAA is a contract; the architecture is a property of the system.
The third is trade-secret IP. Pre-IPO companies, biotech research operations, and hardware firms with patentable engineering depth often have material that they will not send to a third-party model on principle. A prompt that includes a fragment of an unfiled patent specification, sent to an API the company does not control, is a category of decision the legal team will not approve. Private AI is the architecture that removes the decision.
In each of these cases the constraint is genuine and the trade-offs are worth paying. Private AI deployments are slower to deploy, harder to operate, more expensive in compute, and they lag the frontier of model capability by twelve to eighteen months. Organisations under one of these three constraints accept those trade-offs because the alternative — the constraint failing — is unacceptable.
When private AI is the wrong call
The wrong-call pattern is usually one of three things.
The first is a constraint that does not actually exist. We have seen organisations spend three months scoping a private AI deployment to satisfy a compliance team that, on closer reading, only required a BAA and a documented data-handling agreement. The major model providers offer both. The right answer was to negotiate the BAA and move on, not to build a private deployment that would lag the frontier and require dedicated operational support.
The honest test: can the legal team articulate, in writing, the specific clause of a contract or regulation that prevents the data from leaving the network boundary? If they can, private AI may be the right architecture. If they cannot — if the answer is "we are uncomfortable" or "we want to be safe" — the answer is not private AI. The answer is a written data-handling policy and a contracted API with audit logging.
The second is the wrong scale. Private AI deployment economics work above a certain usage threshold. Below that threshold, the operational cost of running the infrastructure exceeds the cost of the API calls that infrastructure replaces, by an order of magnitude. A team of fifteen knowledge workers using a model for occasional tasks does not need private AI. A team of two hundred using a model as a core workflow component might.
The break-even depends on hardware, model selection, and usage pattern, but the heuristic we use is: if the organisation is not generating at least twenty thousand model completions per month against the use case being considered, private AI is hard to justify on cost alone, and the operational complexity is rarely worth it.
The third is operational immaturity. Private AI deployments require operational ownership — model versions, hardware health, index freshness, audit logging, incident response when the model misbehaves. Organisations without operational maturity in their existing systems struggle to add that operational surface. We have seen private AI deployments that ran reliably for three months after handoff and then drifted into degraded states because the in-house team did not have the bandwidth to operate them.
If an organisation does not run its existing infrastructure with the discipline private AI requires, deploying private AI usually makes the operational problem worse, not better. The right answer is to fix the operational discipline first, or to stay on a managed API where the operational concerns are someone else's.
The questions that decide
Before we recommend private AI to a client, we put four questions in writing.
Can you name the specific clause of contract or regulation that requires the data to stay inside the boundary? If yes, private AI is on the table. If no, it is not yet.
What is the projected monthly usage volume? If it is below the operational break-even, the architecture is not justified by cost alone.
Do you have an in-house operator who can run the system unattended after we leave? If not, the engagement scope has to include training that person, or the architecture is not going to be sustainable.
What is the cost of being one model generation behind? If the answer is acceptable — the use case is not the kind of frontier reasoning that depends on the newest model — private AI is sustainable. If the answer is unacceptable, the constraint that required private AI in the first place is in tension with the operational reality of running it.
Four honest answers lead to the right architecture. The clients we have served well were the ones who got the four right. The deployments we have built that are still running, two and three years after handoff, are the ones where the constraint was real, the scale was sufficient, the operational ownership was clear, and the model-generation lag was acceptable. The ones that did not survive in the field were the ones where one of those four was missing.
Private AI is real. It is not for everyone. The work we do on the Software & Private AI practice starts with that honest assessment — and sometimes it ends with us writing down, in the scoping document, that the right tool for the work is a managed API with a strong contract, not the private deployment the client originally asked for.
