Model quantization, VRAM, and memory: fill in the fields

Identify the exact artifact

“A 7B Q4 model” is a search description, not an artifact manifest. Repositories can contain multiple revisions, files, tensor mixes, conversion variants, shards, adapters, and supporting configuration. Runtime compatibility can also change by format, architecture, build options, and accelerator. Before estimating any host resource, identify the bytes that the selected runtime will actually read.

Repository and revision fields

Record the canonical source or repository, an immutable commit or dated release identifier, and the retrieval date. A moving branch or tag can point to different contents later; if you use one, preserve the resolved revision alongside the tag. Record who selected and retrieved it, and where the local copy is staged. Do not include credentials, private tokens, or signed download URLs in a public worksheet.

Keep rights and technical compatibility as separate gates. Capture the repository’s license file or model card reference and any gating or use conditions that apply to the selected weights, conversion, tokenizer, and sidecars. A repository’s metadata label is a starting reference, not a complete legal review. A permissively named code license does not automatically settle rights for model weights, training data, a derived checkpoint, or your intended use. If a file was converted, record the source model revision, conversion tool and version, conversion settings, and the person or process that produced it.

Then name the intended executor and exact release/build. “Supports GGUF” or “supports quantization” is too broad to establish compatibility with every architecture or tensor type. Record the relevant platform, accelerator model and driver/toolkit path where applicable, runtime commit/tag, build options, and the documentation page checked. For a llama.cpp source-head observation, pinning commit 6c01da6278ab6c69ed67d9124806b50d2b299ba7 identifies a specific source snapshot; it does not make that snapshot a generally released or current default. A new runtime version is a new compatibility gate.

Files, tensors, and format

Make one row per file, including every shard and auxiliary file. Record the exact filename, byte count from the local inventory or trusted source manifest, shard index and total shard count, format, checksum, source, and whether it is required at serving time. List a projector, vision encoder, adapter, tokenizer, configuration, chat template, vocabulary, or other sidecar separately. A file can be technically optional for one runtime path and required for another; have the selected runtime documentation settle that question.

GGUF metadata can describe architecture, tensor names and types, tokenizer data, and other model properties. The format specification also allows metadata and tensor representation details to vary. A GGUF SizeLabel is not a universal parseable size or memory field. Do not infer exact bytes, one encoding for every tensor, or runtime allocation from a repository label such as “Q4.” Inspect the file manifest and tensor type map, then retain the actual files’ byte counts as the disk source of truth. Mixed tensor encodings can make a single quantization label incomplete.

When local inspection is permitted, use a bounded, read-only inventory of a directory the operator has already selected. These are discovery templates: first confirm the platform and path, and consult the local command’s help if its output or option differs. They only list files and sizes; they do not download, alter, or execute a model.

# Operator discovery template: set MODEL_DIR to the already-selected local directory.
# Confirm the path and platform before using; do not paste secrets into the report.
command -v find
find "$MODEL_DIR" -maxdepth 2 -type f -printf '%p\t%s bytes\n'

find -printf is not portable to every system. On platforms without it, consult the installed file-listing tool’s help or use a file manager and record the byte count shown for each file. Do not install a tool just to run this example. If a directory may include unrelated files, narrow the selected path before inventory and avoid publishing private path names.

Quantization scope

Write down exactly what is quantized, the target data type or encoding, the method, the source, and any quality or compatibility qualification. Weight quantization changes the representation of model weights. KV-cache quantization changes the representation of attention key/value state for a serving context. Vector quantization changes a representation used in a vector database. These are different objects, storage locations, controls, and quality tradeoffs.

Do not move a ratio from one scope to another. Ollama’s FAQ describes q8_0 as using about half the memory of f16 and q4_0 about a quarter for KV cache, with a possible quality impact. Those ratios are only about the KV cache compared with f16 in that documentation context; they do not mean the model weights or total VRAM will use one-half or one-quarter. Verify that the chosen release supports the selected cache format and confirm the applicable quality criterion with the model owner.

Keep disk and memory ledgers separate

Use distinct rows, units, time windows, and evidence for each resource. Disk measures retained or temporary bytes. RAM and VRAM are capacity pools used by processes and the operating system or device. Runtime allocation, activations, buffers, and cache explain what occupies those pools under a particular build and workload. A model file can be mapped, copied, staged, or partly offloaded differently by different runtimes; its file size does not predict those behaviors with a universal multiplier.

Permanent artifact disk

For selected files, add exact bytes once for each retained copy that will actually exist on that filesystem. Include each selected shard and needed sidecar. If you retain two revisions for rollback, count both. Do not count an optional alternative that will not coexist. Convert units consistently: decimal GB is 1,000,000,000 bytes; GiB is 1,073,741,824 bytes. Preserve raw byte counts so a later reviewer can recalculate using the provider’s billing unit.

Permanent model files are only one disk category. Keep pull staging, partial downloads, extraction or conversion output, build artifacts, runtime image layers, logs, and temporary migration space in separate rows. Whether a temporary copy is removed after a successful operation depends on the selected workflow and operator decision; do not assume automatic cleanup or count a temporary peak as permanent retention. For the billable disk, egress, and recovery roll-up, carry these itemized values to the storage and egress worksheet.

System RAM

Host RAM has its own baseline and peak. Record the operating system and unrelated resident services at an idle or defined baseline, then observe the selected runtime process and other selected services during a named workload. The process may use host memory for model placement, CPU offload, staging, buffers, allocator overhead, or other implementation-specific work. Do not calculate this from artifact bytes or from parameter count alone.

Reserve an explicit operating margin chosen by the operator for the host’s real services and failure tolerance. Name the acceptance threshold and explain its basis rather than calling a fixed percentage universal. Capture which process or host metric reports the value, its unit, sampling interval, and observation window. A peak from one prompt or one quiet period is not a general capacity guarantee.

GPU VRAM and runtime allocation

Separate the device’s physical memory from what is available to this process at the time of the run. Record the GPU model, device count, any partitioning or other occupants, and the selected driver/runtime path. Then capture the runtime’s model placement or offload report, its allocations, activations, scratch/work buffers, and caches if the chosen runtime exposes those fields. The name “VRAM” does not imply that all model weights or all intermediate state are placed there.

Runtime controls and profiling behavior are version scoped. For example, vLLM documents gpu_memory_utilization as a per-instance profile control and documents kv_cache_memory_bytes for explicitly sizing KV cache; their interaction and availability must be checked in the exact selected release. A configured fraction is not a measurement of free device memory and does not prove that two serving processes can safely share the device. The runtime comparison owns runtime-specific settings; the CPU/GPU procurement checklist owns the host and accelerator comparison.

KV cache and context

Record context length per request, prompt and generation assumptions, maximum parallel requests, cache format, runtime setting, and release. Context is not just a model card maximum: the selected runtime’s configured context and concurrent active sequences determine the workload to observe. More context or parallel work can require more KV state, but the amount depends on the architecture, runtime, tensor representation, and allocation strategy. Use the runtime’s own documented calculation or observation method for that exact version instead of a generic tokens-to-VRAM table.

The FAQ’s q8_0 and q4_0 comparisons apply to KV cache relative to f16, with a quality qualification. Keep that ratio beside the cache-format field, not beside the artifact-size field. A context change, runtime update, or concurrency change invalidates an earlier observation unless you show that the relevant configuration and workload remain equivalent.

Co-resident services

Add only processes selected for this design: inference runtime, UI, embedding or reranking service, vector database, and orchestration service where applicable. Record each process’s baseline and observed peak separately, the host or device where it runs, and whether it shares the same memory pool as the model. Do not add mutually exclusive placements as though all run together. If service boundaries or placement are still undecided, leave the row open.

Source documents, vector index files, database snapshots, model download caches, and recovery copies are disk objects, not memory rows. A service may cache or map some of them, but record that only when the selected system exposes evidence for it. Article 6 covers vector data structures and their own memory/disk treatment; this worksheet does not derive Qdrant sizing from LLM quantization. Separate runtime/model build cache and temporary downloads from permanent artifacts in the disk ledger and cost handoff.

Worksheet and acceptance fields

Copy this ledger once per candidate artifact and runtime. Use bytes for files, MiB/GiB for observed memory with the reporting tool identified, tokens for context, requests for concurrency, and a named time interval for peaks. Store raw values and units alongside any conversion. A blank or unavailable figure is UNKNOWN, not zero. If alternatives cannot coexist, mark the selected one and keep the others out of the total.

Accept a candidate for the next planning step only when the artifact and runtime/build are exact, user-selected context and concurrency are filled, co-resident services are enumerated, and every unknown has an owner and resolution step. “Accept” here means that the inputs are ready for a bounded validation plan; it is not a claim that the model fits or meets a response-time target.

Hand off the resulting file inventory and selected copy count to Article 2 for storage and transfer costs. Hand the CPU/GPU and device requirements to Article 3 for procurement checks. Use Article 5 for runtime-specific allocation behavior, and Article 6 only for separate vector data. The Local AI hub returns the overall workload to the selection decision.

Inspect and observe safely

Turn the worksheet into a proposal for a future, bounded observation. Before any run, the operator should confirm that the artifact may be used, its provenance and license record are understood, the selected machine and accelerator are authorized, and the test workload contains no data that should not be used. Define a stop threshold and responsible operator in advance. This article has not installed a model, launched a server, tested a device, or measured a fit.

1. Freeze the candidate. Record file names, byte counts, checksums where available, revision, runtime release/build, hardware and driver path. If any changes, start a new observation record.
2. Choose the actual workload. Name the request shape, context length, maximum planned concurrency, selected services, and observation duration. Avoid collecting real sensitive prompts in logs; use an approved representative input.
3. Capture a baseline. Record host RAM and device memory before loading the model, plus the processes and services already using them. Note the metric source, units, and sampling interval.
4. Use the runtime’s documented inspection method. For Ollama, its documentation identifies ollama ps as a way to see loaded models and the CPU/GPU processor split; record the installed version and relevant output fields, redacting names or paths as needed. Other runtimes need their own exact-version documentation. Do not infer a universal command from this example.
5. Observe the bounded run. During the named workload window, capture process and device peaks, placement/offload, KV setting, context, concurrency, errors, and whether other services changed. Preserve timestamps and configuration references, not credentials or full user data.
6. Stop and review. Stop the planned run if memory exhaustion occurs, an unexpected OOM or restart appears, an agreed threshold is sustained, or the artifact/runtime differs from the record. Preserve relevant redacted logs and configuration for diagnosis. Do not respond by changing production settings or disabling protections as part of this worksheet.

A future observation record should include: operator and date; candidate ID; host/device; artifact revision and file checksums; runtime/build; workload and observation window; baseline and peak values with units and sampling source; selected settings; errors and restarts; threshold outcome; and next decision. Mark each value documented, observed, or unknown. One run supports only the recorded configuration and interval. A different context, concurrency, service mix, runtime build, driver, or artifact requires a fresh review.

If a report appears inconsistent with the worksheet, first compare exact files, release, settings, workload, measurement source, and co-resident processes. An error string or a single issue report does not establish a general product defect. If an expected metric is unavailable, record it as unavailable and select a documented alternative with the runtime owner; do not invent an endpoint or flag. The resulting evidence can support a capacity decision, but latency, throughput, and user acceptance need their own measured criteria.

Evidence and update gate

These source captures were checked on 2026-11-06. They establish documentation statements at capture time, not a promise about current software behavior or a measured model fit. Before procurement or publication, recheck the exact model repository and revision, license and gated-access conditions, selected runtime release/build, accelerator support, and documentation for context, cache, and allocation settings. Keep a dated URL and the exact version beside any behavior that can change.

• Ollama context length documentation — captured context guidance by VRAM and version-sensitive setting notes.
• Ollama FAQ — captured 4096-token default statement, override options, GPU placement inspection, and KV-cache q8/q4 qualification.
• vLLM GPU installation documentation — check the selected release’s platform and hardware requirements.
• vLLM quantization documentation and KV-cache quantization — format, backend, and quality conditions are release and hardware scoped.
• vLLM optimization documentation and cache configuration reference — verify controls against the selected version.
• llama.cpp README at the recorded commit — a specific source snapshot, not a stable-release claim.
• Hugging Face GGUF documentation and GGUF format specification — metadata and tensor representation references; inspect exact files rather than reading a size label as a memory estimate.
• Hugging Face repository license documentation — explains repository license metadata; check the selected repository’s actual license and applicable terms.

The supplied issue material includes one reporter’s Ollama context and parallelism memory-pressure account associated with v0.15.5. It lacks enough hardware and configuration detail to generalize, and is used here only as a prompt to record context, concurrency, and process-level evidence. It is not an incidence estimate or proof of root cause. For broader evidence-method notes, see Methodology. This article records source review and editorial self-check only; no server, model, runtime, or endpoint was tested.