Inference — LLM Toolkit

GENERATION STRATEGIES

Always pick the token with highest probability. Deterministic, fastest, but often repetitive and boring output.

Use when: You need deterministic output, or evaluating model capability.

Divide logits by temperature before softmax. Low temp (<0.5) = focused/deterministic. High temp (>1.0) = creative/random.

Typical range: 0.7-1.0 for most applications. Below 0.5 risks repetitive output.

Only sample from the K most likely tokens. Prevents low-probability nonsense while maintaining diversity.

Typical range: K=40-100. Combine with temperature for best results.

Sample from the smallest set of tokens whose cumulative probability exceeds P. Adaptive to distribution shape.

Typical range: P=0.9-0.95. Often preferred over top-K for variable-quality text.

KV CACHE

The problem: Without KV cache, each new token requires recomputing attention over all previous tokens. Complexity: O(n²).

The solution: Store computed K,V pairs. Each new token only computes attention against cached keys/values. Complexity: O(n).

Memory cost: 2 × seq_len × num_layers × num_kv_heads × head_dim × 2 bytes (BF16).

Example: 7B model, 4096 context, BF16 = ~1.3GB KV cache. At 32K context = ~10GB.

INFERENCE ENGINES

PagedAttention, continuous batching, highest throughput. The industry standard for serving.

Hugging Face's production server. Good integration with Hugging Face ecosystem.

Local inference with GGUF quantized models. Simplest way to run models locally.

CPU + GPU hybrid inference. Best for running large models on limited hardware.