Start from required behavior
Runtime selection is a dependency check, not a popularity poll. Complete the fields below before comparing commands. Record the exact release or immutable build, since documentation labeled “stable” and repository head may describe different states. Use must-have, nice-to-have, rejected, or unknown for each requirement; unknown is a real decision state.
| Requirement | Capture precisely | Gate or evidence |
|---|---|---|
| Host OS and architecture | Distribution/version, x86_64 or ARM, kernel/container boundary, CPU features, NUMA where relevant | Exact runtime release installation/build documentation and owner |
| Accelerator | CPU-only or named GPU, device visibility, driver/toolkit and backend | Published compatibility for the chosen release; a device label alone is insufficient |
| Artifact | Repository and revision, exact files/shards, format, quantization, adapters/projector, license | Runtime format plus model architecture/backend support; test the selected files |
| API consumer | Caller and base URL, required endpoints, request fields, streaming, embeddings or tools | Versioned endpoint and behavior comparison with the actual client |
| Load and concurrency | Peak simultaneous requests, context, input/output size and workload window | Acceptance thresholds selected by the workload owner, not a generic benchmark |
| Lifecycle and signals | Load/unload expectations, process manager, metrics/logs, alert owner and cadence | Documented control and an observation plan; distinguish model unload from host stop |
| Auth and exposure | Caller topology, trust boundary, required controls, who reviews network path | Review endpoint-by-endpoint using Article 8’s exposure guide |
| Deployment owner | Who pins versions, builds, patches, stores artifacts, and handles rollback | Named person and evidence record; otherwise pause the deployment decision |
Keep artifact bytes separate from host RAM, VRAM, process allocation, activations, KV cache, download/build cache, and persistent data. A quantized file size does not predict total serving memory. For the distinct memory worksheet, use Article 4; for CPU/GPU procurement requirements, use Article 3. This page does not supply universal memory multipliers.
Also distinguish a feature that exists in a project from the operational path you can actually maintain. A backend may be present while the required package, driver, kernel, model architecture, or client parameter is absent in your selected combination. Put each such dependency in the ledger instead of compressing it into “supports GPU” or “supports OpenAI.” Name who will verify upstream documentation and who will own the resulting build and updates.
Record artifact identity at file level when the runtime accepts a local file or repository reference. Include hashes or an equivalent immutable revision where available, shard count, sidecars, tokenizer/config files, and any adapter that the serving path needs. Keep the license and permitted use as a separate review field. A model name alone can resolve to a changed tag or a different package, and a format name alone does not establish architecture support. If files are obtained from a registry, note the source and the exact snapshot retained for rollback; do not imply that a later pull will reproduce the same bytes.
Compare platform and build paths
Packaging convenience and backend breadth are different selection dimensions. For each candidate, pin the release, host platform, artifact, and accelerator combination. A demonstration using a different model or machine is not compatibility evidence for yours.
Ollama
Ollama presents a model-management interface and a local serving API, with documented Linux and Docker paths and hardware support guidance. Its documentation describes pulling and running named models; a Docker volume can persist model data. That packaging does not establish that a particular provider’s GPU is passed through correctly, that every GGUF works, or that a container should be exposed to the internet. Verify the selected Ollama release, model tag or imported artifact, host OS, GPU support path, and persistent model directory together. GGUF support in llama.cpp or one Ollama backend does not establish identical support across runtimes.
Ollama context guidance has a version-sensitive documentation discrepancy: the context-length page gives VRAM-banded defaults (<24 GiB 4k, 24–48 GiB 32k, and ≥48 GiB 256k), while the FAQ says the default is 4096 tokens. Treat neither as a universal effective setting. Record the installed release and explicitly inspect or set the selected context only after sizing its memory effect. The documented environment control is OLLAMA_CONTEXT_LENGTH; the exact service-manager method depends on the platform. Do not paste a public bind address as a routine setup step. The documented ollama ps observation can help record loaded model and processor allocation, but it is not a performance test.
vLLM
vLLM has Python/package and hardware-specific installation paths, including GPU builds and a basic x86 CPU inference/serving path. CPU support does not mean speed parity with a GPU or equivalent feature/backend coverage. Capture Python, package or image version, operating system, architecture, accelerator, driver/toolkit, PyTorch combination, and model support from the selected release’s installation and supported-model documentation. Recheck wheel and backend constraints before procurement; do not generalize from “vLLM runs on CPU.”
For GPU operation, gpu_memory_utilization participates in profiling available GPU memory for weights, activations, and KV cache. It is a version-specific control, not a promise that this percentage is free capacity available to another process. In documented configurations, explicitly setting kv_cache_memory_bytes changes how KV cache memory is specified and can supersede the ratio-based cache calculation. Model, configuration, framework, hardware, and release affect the result. Record the chosen values and resulting startup/profile observations; do not copy a flag from a different release or treat it as a host-wide memory budget.
vLLM documents model quantization and quantized KV cache as separate capabilities with backend and model constraints. Verify each independently. A weight quantization choice does not establish KV cache format support, accuracy, or memory fit. Likewise, API compatibility is a subset of request behavior, not proof that every OpenAI client feature is identical.
llama.cpp
llama.cpp places more emphasis on selecting or building the executable for a backend. Capture the repository revision, compiler/build options, CPU instruction support, GPU backend, device/layer split, threading and NUMA plan, memory mapping choice, and format/model requirements. Build guidance distinguishes CPU and accelerator backends; availability depends on the artifact and exact build. GGUF files can contain mixed tensor encodings, so a filename label is not a complete memory or compatibility specification. Check metadata and the selected backend’s support.
One captured upstream source head, commit 6c01da6278ab6c69ed67d9124806b50d2b299ba7, documents llama-server defaults including loopback binding and port 9931, optional metrics and API-key authentication being off, and a public /health path. These are observations about that immutable source snapshot, not promises for all releases, packages, builds, UI, MCP or tools configurations. Inspect the exact executable’s help and versioned documentation before relying on any default or endpoint. AI46 records head lineage only; it is not evidence that this commit shipped as a stable release.
Compare API and lifecycle fit
“OpenAI-compatible” is useful shorthand for particular routes and request shapes. It does not promise feature parity, identical defaults, secure exposure, or identical model behavior. Maintain a versioned contract for each caller.
Endpoints and consumer compatibility
| Candidate and evidence scope | What to check with the consumer | Security and uncertainty |
|---|---|---|
| Ollama documented API; selected release required | Exact API route, payload/options, streaming, model naming, embeddings or tool needs | Confirm listener and caller path separately; do not assume auth or network isolation |
| vLLM OpenAI-compatible server; selected release required | Chat/completions route, template, parameters, streaming, supported model task and error handling | Captured docs state API-key authentication covers selected path prefixes, not every endpoint; /invocations is an exception exposing inference capability |
| llama-server observed source head; release must be checked | Required route and schema, streaming, model-specific behavior, optional UI/tooling | Do not inherit source-head defaults; inspect all enabled endpoints and controls in the actual build |
Run consumer checks from an authorized, non-public test path with a harmless request. Compare status handling, required fields, response shape, stream termination, timeouts, and error behavior to the client’s needs. Use redacted test data. Record route, runtime version, model revision, caller, and result. A successful sample proves only that tested combination worked for that check; it does not prove comprehensive compatibility or security.
Load, unload, and process controls
Document who starts the server, how a model becomes resident, what action releases model memory, and what action stops the process. Ollama has model-level stop and keep-alive behavior that should be checked against the chosen release. In other runtimes, a process manager or operator may own model and process lifetime. Do not conflate model unload with stopping the service, stopping the VPS, deleting model files, or cancelling an attached storage subscription.
Write the intended transitions before deployment: idle service, model loaded, model unloaded, process stopped, host stopped, and data retained or removed. Assign an operator and record the observed control and expected state for each. For deletion, residual files, backups, or billable resources, use Article 9’s lifecycle worksheet; this comparison does not prescribe destructive cleanup commands.
Metrics and operational signals
Choose signals tied to an acceptance question: process health, request errors, running and waiting requests, preemptions, cache use, memory pressure, disk growth, and response duration. vLLM documents a Prometheus metrics endpoint with signals such as running/waiting requests, preemption, KV cache, and latency. Availability, names, labels, collection overhead, and semantics can change by release. Treat these as candidate observations, not results. llama.cpp metrics are optional in the cited source head; verify the selected build. For Ollama, identify the supported logs and process observations for the release rather than assuming a comparable metric set.
For each signal record the source, collection method, interval, workload window, version, owner, retention, and threshold that triggers a pause. Collect only the fields needed for operations and protect metrics endpoints like other operational interfaces. Monitoring can explain what occurred during a bounded test; it cannot establish a universal capacity figure.
Choose a small observation set before adding a monitoring stack. For example, a host operator may already have process, memory, disk, and service logs; a runtime may add request queue or cache metrics. Confirm that each metric is emitted by the selected build and can be collected from the intended network. Record units and labels so an operator does not mistake a per-process gauge for a host total, or one replica’s observations for the entire service. Avoid retaining prompts or responses as a substitute for operational metrics; define data handling separately if application logs contain user content.
Decision matrix and bounded validation plan
Use this matrix to make a conditional choice. Fill every unknown with a person and a next evidence step. Select one candidate path at a time; do not treat the column as a score.
| Requirement | Ollama candidate | vLLM candidate | llama.cpp candidate | Evidence, unknown, validation and owner |
|---|---|---|---|---|
| Release and host | Pin release; verify Linux/container and device path | Pin package/image and Python, OS, backend combination | Pin release/commit and build options | Link versioned docs; record unresolved compatibility; deployment owner |
| Artifact and format | Record exact model/tag or import path and files | Check architecture/task and quantization/backend matrix | Check GGUF metadata and backend/build support | Model owner confirms revision, shards, sidecars, rights; run selected artifact gate |
| API contract | Map required routes and options | Check routes, chat template and endpoint coverage | Check server schema and enabled optional interfaces | Consumer owner tests required request/response and streaming behavior |
| Lifecycle | Define model unload and service lifetime | Define process/service and model lifetime | Define server process and model lifetime | Operator records transitions; destruction/deletion handled separately |
| Signals and auth | Identify available logs/observations and boundary | Review metrics plus endpoint-specific auth behavior | Verify build-specific metrics, auth and optional UI/tools | Network owner reviews exposure under Article 8; metrics owner defines stop threshold |
| Placement | List resident model service and other processes | List accelerator, memory and disk co-residents | List build/runtime caches and other residents | Placement owner checks Article 10; host requirements return to Article 3 and Article 4 |
Bounded acceptance sequence
- Freeze inputs. Record host architecture, runtime release/build, artifact revision and files, caller, and deployment owner. Stop if these are not known.
- Review the path. Confirm official versioned documentation for the exact platform, accelerator, artifact format, API, lifecycle, and metrics. Treat unsupported combinations as unresolved, not implied support.
- Build or select a candidate in a controlled environment. The operator follows the selected runtime’s official instructions; this article does not claim that any install, build, or model load was executed.
- Check the consumer contract. From the authorized caller, submit a bounded harmless request and record endpoint, response, streaming/error behavior, and redacted evidence. Stop if an unexpected route is reachable or access controls differ from the reviewed design.
- Observe a defined workload window. Capture startup/profile state, resource signals, queue/cache indicators where available, and errors at a stated interval. Do not expand traffic or concurrency when a stop threshold is reached.
- Check lifecycle and recovery ownership. Confirm model unload, process stop, retained artifacts, logs, and restart expectations without deleting data. Record who can restore or repull and what cost inputs return to the cost worksheet.
- Accept, revise, or pause. Accept only when build and artifact, API consumer, lifecycle, observation plan, and endpoint exposure gates pass for this selected version. Otherwise write the missing evidence, owner, and next review date.
There are no speed scores here. If latency or throughput is a business requirement, define a separate workload-specific measurement plan and compare only candidates tested with the same artifact, host, prompt mix, concurrency, context, and measurement method. Keep results and limitations with that test record; do not generalize them to other releases or hosts.
A useful decision record is short but reproducible: candidate and exact release; host and backend; artifact identifiers; caller and required API operations; lifecycle transitions; proposed metrics; security reviewer; open questions; acceptance owner; and date. If the candidate changes after an update, rebuild, model revision, or host move, identify which gates need to be repeated. This avoids treating an earlier successful check as permanent approval for a materially different stack.
Keep placements mutually exclusive in the comparison. If one option requires a GPU host and another is being considered on CPU, compare their own host, disk, build, and operator inputs; do not add both machines into a pretend combined architecture. Likewise, model downloads, build caches, retained prior versions, and backups are distinct persistent or temporary quantities. Send selected values to the dedicated storage and egress cost worksheet rather than duplicating a price calculation here. Unknown billing or support fields stay open until verified for the chosen product and region.
Evidence and updates
Source captures checked 2026-11-06. Documentation and captured HTML are evidence for bounded statements, not proof of a deployment or current behavior after a release change. Before selecting a runtime, recheck its official documentation against the exact release, OS, architecture, backend, model revision, and deployment method.
- Ollama context length documentation — VRAM-banded context guidance; version and FAQ conflict must be resolved for the selected release.
- Ollama FAQ — captured 4096-token default statement, API and service controls.
- Ollama Linux and Ollama Docker — native/container platform and model persistence references.
- vLLM GPU installation, CPU installation, cache configuration, metrics, and OpenAI-compatible server — captured version-sensitive platform, profiling, observation, and endpoint behavior.
- llama.cpp observed source commit, including its server README and build guide — immutable source-head evidence, not a stable release claim.
- GGUF format documentation — format metadata and tensor encoding context; verify the actual file.
For the broader decision path, return to the Local AI selection hub. This article compares runtime requirements and does not set memory totals, exposure procedure, or lifecycle billing rules.