Home / Blog / On-premise LLM vs cloud AI

October 11, 2026 · Deployment · ~12 min read

On-premise LLM vs cloud AI — when it really pays off

An on-premise LLM is a language model running on hardware you own, instead of on someone else's server billed by usage. This article is mostly not about privacy — we cover GDPR briefly further down. It's about something companies often discover only after a year: you don't control the tool you built a process on. The price list, the limits, the terms and the model itself can change without you, and all you get is a deadline.

Below: what exactly slips out of your hands, how to calculate the cost of both options without guessing, what you can't audit in someone else's API — and an honest section on when the cloud is simply the better choice. Facts and dates were checked on October 11, 2026 against the sources listed at the end.

Self-hosted LLM vs cloud API: what's the difference

A language model is a file with parameters, called weights, plus a program that runs them. The whole difference comes down to where that file lives and whose processor does the math.

With a cloud model, the weights belong to the provider and you never see them. Your application sends the content of each request to their server through an API, you pay for the number of tokens processed, and you get an answer back. A self-hosted LLM works the other way round: you download the weights once — for models with open licenses, legally and at no cost — and they stay on your drive. An employee's prompt never leaves the company network, and the answer is generated on site, on a machine in a server room or under a desk.

The distinction is sharper than it looks. In the cloud you buy access to a service, which lasts as long as the provider wants to offer it on the current terms. On premises you have a file. A file doesn't change its price list, doesn't introduce limits and doesn't disappear on its release anniversary. That's the whole argument of this article; the rest is its consequences.

Five things you don't control at a cloud AI provider

This isn't a list of hypotheticals. Each of these five points has actually happened to AI providers' customers, and each is documented in public sources.

1. The price list changes without you

Token prices do sometimes go down, and that's worth remembering — competition works in the buyer's favor. The problem is something else: you have no control over it and no guaranteed timeline. An API agreement isn't a contract with a price frozen for three years; it's a price list published on a website and changed at the provider's discretion. If you build a department's budget for the year, you're basing it on a number nobody guaranteed you. And if your company doesn't work in US dollars, there's the exchange rate on top — billing is usually in dollars, and the currency risk is yours.

2. The model your process depends on gets retired

This is the most underrated point on the list, because it sounds abstract until the email with a date arrives. The big providers keep public retirement calendars, and they stick to them.

Anthropic describes four stages of a model's life: Active, Legacy, Deprecated and Retired — and after retirement, as its documentation puts it plainly, requests to the model fail. Anthropic commits to at least 60 days' notice, and in practice that's what it gives: Claude Sonnet 4 and Claude Opus 4 were announced for retirement on April 14, 2026 and switched off on June 15, 2026; Claude Opus 4.1 was announced on June 5 and switched off on August 5, 2026. The next one is already scheduled: on September 30, 2026 developers using Claude Sonnet 4.5 were told it retires on November 30, 2026. OpenAI works the same way, with longer notice: on June 11, 2026 it told developers that older GPT-5 and o3 snapshots will be removed from the API on December 11, 2026, and on October 1, 2026 it gave six months' notice for GPT-5.1, GPT-5.3-Codex and GPT-5.4-Nano, to be removed on April 1, 2027.

Two takeaways. First, the model you use today already has a date — on Anthropic's page, even active models have a tentative retirement date, “not sooner than”. Second, if you use a model through another company's cloud platform, that platform's schedule applies, not the model maker's — Anthropic states explicitly that Amazon Bedrock and Google Cloud set their own retirement schedules.

The operational consequence is simple: a process that worked fine for a year stops working on a set day, and you have a migration on your hands that you neither planned nor budgeted. With an open-weight model on your own drive, this point doesn't exist.

3. Terms and limits are a one-sided document

The terms of use cover permitted use cases, requirements for your end users and content moderation rules. They change more often than price lists, and you accept the new version by continuing to use the service. For a company, that means checking the deployment against the terms regularly, not once when signing up.

4. Throughput is allocated, not bought

Providers cap requests and tokens per minute and per day, and the height of those caps depends on your account tier, which you earn through cumulative spend and account age. Above the limit you get an HTTP 429 error and have to retry. For one employee's chat, that's an inconvenience. For overnight processing of two thousand documents, it's the difference between a job that finishes before morning and one that doesn't. Add outages on the provider's side — then you have a status page and nothing else.

5. The provider disappears or changes hands

The language model market is consolidating, and that includes companies that recently looked like a safe choice. In March 2024 Microsoft hired Inflection AI's co-founders Mustafa Suleyman and Karén Simonyan, along with most of the team, and paid the company about $650 million, mainly to license its models. Inflection didn't go bankrupt — but it abandoned its consumer product and pivoted to business customers. A customer who had built a process on it didn't lose access overnight; they lost something subtler: the product they chose and the team developing it.

With your own weights, a change of ownership at the model maker is industry news, not your operational problem. The model you downloaded keeps working regardless of what happens to the company that released it.

The bill grows with your success

This is an asymmetry that's easy to miss during a pilot. In the cloud you pay per use, so the better the deployment works, the more it costs. A successful pilot that a whole department starts using is both a project win and a step change in the bill. On premises it's the other way round: the cost is paid up front, and each additional request is practically free — you pay for electricity, not a license.

We won't quote cloud prices here, and that's deliberate. Providers' price lists change often enough that a number written in October may be out of date by spring, and an article that misleads is worse than no article. Instead, here's a method you can run on current rates in fifteen minutes.

How to calculate the cloud cost

  1. Count the volume. How many documents or requests go through the process each month? Take the number from a real month, not from the pilot.
  2. Measure tokens per request. Input and output separately, because they're priced differently. OpenAI's rule of thumb for English is about 4 characters, or ¾ of a word, per token — so a 500-word page is roughly 670 tokens. Other languages usually need more tokens for the same content, so measure a sample with the provider's tokenizer. If you attach context from your own documents, the input can be ten or more times bigger than the question itself — and that's usually what drives the bill.
  3. Open the provider's official price list and take the current rate per million tokens, separately for input and output, for that specific model.
  4. Multiply and add a margin for tests, corrections and retries. Realistically: 20–30 percent on top of production traffic.
  5. Multiply by the decision horizon — 36 months, the typical service life of the hardware.

How to calculate the on-premise cost

On the on-premise side the components are different, but the list is shorter: hardware (one-off), setup and integration (one-off), electricity — the device's power in watts times hours of operation times your price per kWh; a GB10 machine such as DGX Spark or the ASUS GX10 has a 240 W power supply — and maintenance: updates, backups and someone's time. You get the break-even point by dividing the one-off costs by the monthly cloud cost from the previous step. Compare the result in months with the 36-month horizon — if it comes out clearly lower, the decision is calculated, not guessed. Current hardware prices, with sources and dates, are in our ASUS GX10 vs DGX Spark comparison.

An honest caveat: this calculation doesn't include the value of the time you won't spend on migrations forced by model retirements. It's hard to price, but it isn't zero.

What you can't audit in someone else's API

There's a class of questions you can't answer with a cloud model — not because the provider hides something, but because you have no access to the layer where the answer lives.

Reproducibility. This is the most serious issue, and it has a documented, technical form. When you call a model, you choose one of two paths. You can name a specific version with a date in its name — then behavior is stable, but that version has a finite life and one day gets switched off; that's exactly what happens to the GPT-5 and o3 snapshots being removed in December 2026. Or you can use a general alias pointing to the “currently recommended” model — then nothing breaks on retirement day, but the model behind that name can change, and with it the behavior of your process, without any signal on your side. It's a choice between a failure with a date and a silent change without one, and there's no third option. With local weights there is: the same file gives the same behavior until you replace it yourself.

Logs and evidence for an auditor. You have your own records of what you sent and what came back, but no visibility into what happened in between. In an investigation or an industry audit, the difference between “our provider states” and “we have a full log on our side” can be the difference between a closed case and an open one.

Data handling. One thing here needs honesty, because articles about local AI often stretch it: business plans and API access usually don't provide for training models on customer content — the real risk is mostly free consumer versions that employees use on their own. If you want to use cloud models but without sending personal data outside the company, there's a middle path: a local model replaces sensitive data before the request leaves the company, and the mapping of replacements stays with you.

GDPR and AI: what changes when the model runs on your server

If your prompts contain personal data — names in emails, client records, CVs — then sending them to a cloud AI provider means that provider processes personal data on your behalf. Under the GDPR, that requires a data processing agreement (Article 28), and if the data leaves the European Economic Area, a legal basis for the transfer (Chapter V). Both are things you may have to explain to a data protection authority.

With on-premise AI, the inference step has no outside processor and no transfer: prompts and documents stay on hardware you control. That doesn't take you out of the GDPR — you still need a lawful basis for the processing, security measures (Article 32) and access control, including the permission mapping we describe in our DGX Spark setup guide. But it removes one processor and one transfer from the list you have to justify. This is a general description, not legal advice; for your specific case, ask your data protection officer or lawyer.

Air-gapped AI: does on-premise mean offline?

It can, but it doesn't have to. A fully air-gapped AI setup — no connection to the internet at all — is used where regulations or professional confidentiality require it. Technically it's straightforward: once the models are downloaded, they run without a network connection, and system updates are brought in during a controlled window. Our DGX OS page has more on updating DGX Spark safely.

In a typical company, the server sits on the local network and has internet access for updates, but the content of prompts and documents is never sent to an outside model provider. A hybrid is also common, which we come back to below.

When the cloud IS the right choice

If on-premise AI always won, this article wouldn't be needed. There are four situations in which we honestly advise against leaving the cloud.

SituationWhy the cloud wins
Prototype and explorationYou don't know yet whether the process is suitable at all. An API key costs about as much as a few hours of work and lets you find out in a week. Buying hardware before testing is the most expensive way to learn.
Irregular, light useA few dozen requests a month, in bursts. The machine would sit idle, and the break-even point would never arrive within its service life.
Nobody in ITA server needs updates, backups and someone who reacts when it stops responding. Without that person — in-house or at an outside provider with a support agreement — on-premise AI becomes one more piece of idle hardware in a cupboard.
You need the very strongest modelThe hardest research and programming tasks are still handled better by the largest commercial models, running on hardware no single company would buy. If your process needs exactly that, a local model will be a compromise.

The most common sensible setup isn't “either/or” anyway, but a split: confidential data and client documents are processed by a local model, and tasks without protected data go to the cloud. That gives you control where you need it, without paying for it in convenience for the other 80 percent of the work.

What you actually need to move off the cloud

Three things — and only one of them is hardware.

A self-hosted LLM server: hardware sized to the model

Counter-intuitively, what matters most isn't compute but the amount of memory available to the model: the weights have to fit in it entirely, or the model either won't start or will run too slowly to work with. That's why the order of decisions is always the same — first the task, then a model that does it well enough, and only then a device that can hold that model. Buying hardware before testing the model is the most common cause of failed deployments we see.

Roughly, the options from cheapest up:

A private LLM: open weights and a checked license

Building a private LLM doesn't mean training a model from scratch — that takes budgets in the hundreds of millions, and nobody sensible does it inside their own company. It means taking an open-weight model, running it on your own hardware and adapting it to your documents — usually by connecting a company knowledge base, not by retraining the model itself. One thing needs checking before you decide: the license. “Open weights” doesn't always mean “commercial use allowed” — some models restrict company size or use cases. Read it in the model card; don't assume it.

Someone to maintain it

A server with a model is a normal production system: updates, monitoring, backups, access control and someone responsible when it stops responding. That can be your own IT team or an outside provider with a support agreement. What you can't do is install the machine and forget about it. What the setup involves, step by step, and where do-it-yourself deployments usually stall, is in our DGX Spark setup guide.

Frequently asked questions

What is an on-premise LLM?

An on-premise LLM is a language model running on hardware you own, in your office or in a rented server rack. The model's weights — the file with its parameters — are downloaded once and stay on your drive. An employee's prompt never leaves the company network, and the answer is generated on site. That's the difference from a cloud model, where you pay for access to someone else's server and send it the content of every request.

Is a self-hosted LLM cheaper than the cloud?

It depends on scale, and you can calculate it instead of guessing. In the cloud you pay per use, so the bill grows with the number of requests. On premises you pay up front — hardware, setup and maintenance — and afterwards mostly electricity. The break-even point is your one-off costs divided by your monthly cloud bill calculated from the provider's current price list. With irregular, light use the cloud usually wins; with steady, growing load the balance flips.

Can a local model match cloud models on quality?

Not on every task. The largest commercial models run on hardware no single company would buy, and they still lead on the hardest tasks. But the gap depends on the task: in typical office work — summarizing documents, finding information in your own files, sorting correspondence, drafting a first version of a letter — open-weight models that fit on one machine are hard to tell apart. The right test isn't a leaderboard, it's a sample of your own documents.

How do I build a private LLM for my company, and where do I start?

Start with the process, not the hardware. Pick one task that's done regularly and work out what it costs today in hours. Then collect 20 to 50 real documents from that process and test an open-weight model on them. The result of that test tells you how big the model needs to be, and the model size tells you how much memory the machine needs. Doing it the other way round — buying hardware before the test — is the most common cause of failed deployments.

Does on-premise AI have to be air-gapped?

No, although it can be. A fully air-gapped setup is used where regulations or professional confidentiality require it: once the models are downloaded, they run without any internet connection, and updates are brought in during a controlled window. In a typical company the server sits on the local network and has internet access for updates, but prompts and documents are never sent to an outside model provider. A hybrid setup is also possible: a local model handles sensitive data and tasks without protected data go to a cloud model.

Does running AI on premises make it GDPR compliant?

Not by itself, but it simplifies things. When prompts with personal data go to a cloud AI provider, that provider processes the data on your behalf, which under the GDPR requires a processing agreement (Article 28) and, if the data leaves the European Economic Area, a legal basis for the transfer (Chapter V). With a self-hosted model the inference step has no outside processor and no transfer. You still need a lawful basis, security measures and access control. This is a general description, not legal advice.

Sources

Dates and facts in this article were checked on October 11, 2026 against the sources above. Model retirement schedules change — check the current status on the provider's page before you decide. Bonsai 2 27B results come from our own test described on its page.