Inference — LLM Toolkit

GENERATION STRATEGIES

Always pick the token with highest probability. Deterministic, fastest, but often repetitive and boring output.

Use when: You need deterministic output, or evaluating model capability.

Divide logits by temperature before softmax. Low temp (<0.5) = focused/deterministic. High temp (>1.0) = creative/random.

Starting points (not recipes): factual QA and code 0–0.2, general chat 0.7, brainstorming 0.8–0.9. Calibrate per task — no universal setting exists.

Only sample from the K most likely tokens. Prevents low-probability nonsense while maintaining diversity.

Typical range: K=40-100. Combine with temperature for best results.

Sample from the smallest set of tokens whose cumulative probability exceeds P. Adaptive to distribution shape.

Typical range: P=0.9-0.95. Often preferred over top-K for variable-quality text.

KV CACHE

The problem: Without KV cache, each new token requires recomputing attention over all previous tokens. Complexity: O(n²) per step.

The solution: Store computed K,V pairs. Each new token only computes attention against cached keys/values. Complexity: O(n) state, O(1)-ish work per step.

Memory cost (per request): 2 × seq_len × num_layers × num_kv_heads × head_dim × bytes × batch.

Example: LLaMA-3-8B-like (32 layers, 8 KV heads, head_dim 128, BF16): 131 KB/token — 4K context ≈ 0.5 GB/request, 32K ≈ 4 GB. GQA (fewer KV heads) cuts this proportionally.

Watch out: weights-only budgeting OOMs on the first long-context batch. Budget weights + cache × concurrency + overhead.

INFERENCE ENGINES

vLLM

PagedAttention, continuous batching, highest throughput. The default for GPU serving. Use when throughput per GPU matters.

TGI (Text Generation Inference)

Hugging Face's production server. Good integration with Hugging Face ecosystem. Use when staying inside HF tooling.

Ollama

Local inference with GGUF quantized models. Simplest way to run models locally. Use for development and personal use.

llama.cpp

CPU + GPU hybrid inference. Best for running large models on limited hardware. Use when no datacenter GPU is available.

Next actions

Estimate inference memory →Learn batching →Master sampling →Deploy it →