In brief
- On-device, on-premises, self-hosted cloud and a managed model API are distinct deployment paths.
- Open weights do not automatically mean local operation; open-weight models can also run in the cloud.
- Local is not automatically secure and cloud is not automatically insecure—the complete data and control path matters.
- Model quality, functions and freshness can differ substantially by deployment path.
- Compare total cost and operational responsibility, not merely API price or hardware purchase.
Which deployment models actually exist?
There are several options between a model on one laptop and a public model API. A defensible decision identifies who operates inference infrastructure, where inputs are processed and who owns model versions, capacity and incidents.
- Workstation or edge device
- A smaller model runs directly on a PC, mobile device or machine. This can work offline with low latency, but hardware, power, memory and model size constrain it.
- Own data centre
- The organisation operates model servers and usually GPU infrastructure inside its own network. It gains stronger environmental control but owns capacity, hardening, patches, monitoring and outages.
- Self-hosted cloud or private VPC
- The model runs on rented cloud infrastructure but is operated by the organisation or a partner. Cloud hardware and owned software operations are combined; region and network boundaries still need precise assessment.
- Managed model service
- A provider exposes models through an API or platform and handles much of the infrastructure, scaling and model maintenance. Contract, product configuration, regions and data processing remain material.
Where does the data really flow?
The model's location alone does not answer privacy or security questions. A locally installed application may send telemetry, searches or error logs to external services. Conversely, an assessed cloud service may offer explicit regions, short retention and contractually limited processing. The complete application's data path is decisive.
Do not inspect only the prompt. Retrieved documents, images, audio, embeddings, tool results, outputs, feedback and evaluation data may create additional processing or storage steps.
- Which data classes are permitted on each deployment path?
- Where are inputs, context, outputs and logs processed and stored?
- Which providers, subprocessors, regions and support access paths are involved?
- Which information leaves the internal network through updates, telemetry or connected tools?
- How are access, retention, deletion and provider exit evidenced?
Which responsibility shifts?
Self-hosting creates control but also work. A managed service removes infrastructure effort, while the organisation still owns purpose, permitted data, application security, quality and organisational adoption.
| Decision area | Managed service | Self-hosted |
|---|---|---|
| Hardware and capacity | Provider scales within purchased limits | Plan GPUs, storage, headroom and utilisation |
| Models and functions | Provider catalogue and product lifecycle | Models compatible with the owned runtime and hardware |
| Updates and security | Base service at provider; application remains your task | Maintain serving stack, models, containers and infrastructure |
| Availability | Service levels, regions, quotas and external dependency | Own redundancy, on-call cover, spares and recovery |
| Data and evidence | Review contracts, configuration and provider evidence | Evidence your own technical and organisational controls |
How should quality, performance and cost be compared?
A smaller self-hosted model may fit stable extraction very well while performing poorly on complex instructions or multiple languages. A capable cloud service may deliver better results without its request price showing the full integration and operating cost.
Compare the same real cases on model versions that are actually available. Include hardware, power, cloud resources, platform charges, engineering, security, observability, on-call cover and idle capacity.
- Quality and critical failures on a shared test set
- Latency, throughput and behaviour under peak load
- Required languages, context length, image processing and tool use
- Cost per correctly completed business case
- Internal work for updates, evaluation, incidents and support
When does a hybrid architecture make sense?
Not every task and data class needs the same deployment path. An organisation can execute bounded or offline tasks locally and send demanding, approved cases to a managed service. The model must not choose the path where that could expose sensitive data to an unapproved environment.
- Route outside the model using task, data class and approval
- Use the same domain quality thresholds for every model
- Make the deployment path used for each case visible
- Define behaviour when a model or network connection is unavailable
- Reassess whether additional complexity still creates evidenced value
How do you make an evidence-based decision?
Start with the use case rather than a general position for or against cloud. Document mandatory requirements, compare a few realistic options and test them under the same conditions.
- Step 1
Set the boundary
Define data classes, offline need, regions, latency, availability and prohibited failures.
- Step 2
Design viable options
Include a managed API, self-hosted cloud or local operation only where it can meet the minimum requirements.
- Step 3
Test with the same cases
Compare quality, latency, throughput, failure behaviour and usability using representative data.
- Step 4
Assess full cost and ownership
Include technology, people, security, support, headroom, contracts and exit across the lifecycle.
- Step 5
Keep the decision reviewable
Document assumptions, accepted disadvantages and a date for reassessment.
Example: maintenance assistance in a production facility
A manufacturer needs fast classification of machine alerts even when the internet connection is unavailable. A small, evaluated model therefore runs at the site and receives only approved sensor data and manuals. Complex questions about public supplier documentation may pass to a managed service after a technical data check. A fixed rule outside the models selects the deployment path; confidential production data is not forwarded automatically. The team measures quality and cost for both paths and keeps a manual process for outages.
What to remember
Choose the deployment path that delivers the required quality and control with sustainable ownership. Assess the complete application, compare fully loaded cost and preserve the technical and organisational ability to change later.
Sources and further reading
These primary sources provide further detail on definitions, technical foundations or responsible use.
Content reviewed