Harshit Joshi

Quantization trades numerical precision for memory and speed. The weights get packed into fewer bits, so the model shrinks and the arithmetic gets cheaper. The question is always how much quality you are willing to give up for that.

Q4_K_M is the default I reach for: the K-quant mixed scheme keeps the sensitive layers at higher precision while squeezing the rest, so you get most of the size win without the blunt quality drop of a flat 4-bit. Q8_0 is for when output fidelity is non-negotiable and you have the memory to spare - evaluation runs, structured extraction, anything where a subtle drift in logits actually matters.

The -t flag sets how many CPU threads handle token generation. More threads is not always faster: past the number of physical cores you hit memory-bandwidth limits and hyperthreading contention, and throughput plateaus or drops.

# match physical cores, not logical threads
./llama-cli -m model.Q4_K_M.gguf \
  -t 8        # generation threads (≈ physical cores)
  -tb 16      # batch threads for prompt ingest

I set -t to the physical core count and let -tb (batch threads, used during prompt processing) run a little higher, since the prefill stage parallelizes better than autoregressive decode. Leave a core free for the OS so the UI does not stutter under load.

These two shape the sampling distribution. Temperature flattens or sharpens the probabilities; top-p (nucleus sampling) clips the long tail by keeping only the smallest set of tokens whose cumulative probability reaches p.

Tune one at a time. For anything that has to be correct - code, extraction, tool calls - I pull temperature toward zero and let top-p do the trimming. For ideation I raise temperature first and only widen top-p if the output feels too safe.

The -ngl flag offloads N transformer layers to the GPU. Each layer you move off the CPU is a chunk of matrix math running on far more parallel hardware - the single biggest speedup available, as long as the layers fit in VRAM.

# offload all layers if they fit in VRAM
./llama-cli -m model.Q4_K_M.gguf -ngl 99

# Metal (Apple Silicon) builds: unified memory, no copy
# CUDA (NVIDIA) builds: watch VRAM headroom for the KV cache

• Metal (Apple Silicon): unified memory means the GPU and CPU share one pool, so there is no PCIe copy tax. Offload aggressively; the ceiling is total system RAM.
• CUDA (NVIDIA): dedicated VRAM is faster per layer but finite. Size -ngl so weights and the KV cache fit - overshoot and you spill back to host memory over PCIe, which is slower than just keeping those layers on the CPU.