Home/Deployment

Deployment

From training run to production API. Serving, monitoring, and operational concerns.

DEPLOYMENT PATTERNS

Local Inference

Run on your own machine. Use Ollama, llama.cpp, or LM Studio. Best for development and personal use.

Workstation Server

Dedicated GPU machine serving via vLLM or TGI. Good for team access and moderate traffic.

Cloud API

Containerized deployment on cloud GPU instances. Auto-scaling, load balancing, monitoring.

QUANTIZATION FOR DEPLOYMENT

Halves model size with minimal quality loss. Supported natively on most modern GPUs. Good first step for deployment.

When: the default first try — verify on your evals, especially math, code, and non-English slices, before going to 4-bit.

Post-training quantization to 4-bit. 4x size reduction. GPTQ and AWQ are the most common methods. Minor quality impact on well-calibrated models.

When 4-bit is good enough: chat, summarization, and RAG usually qualify — but verify per task. Precise math, code generation, and low-resource languages degrade first: test 8-bit/BF16 there before accepting 4-bit. Calibrate on data resembling production, and never quantize an already-quantized artifact.

Universal quantization format supporting CPU + GPU offloading. Q4_K_M, Q5_K_M, Q6_K are popular middle-ground quantizations.

OPERATIONAL CONSIDERATIONS
  • Monitoring: Track latency (TTFT, inter-token), throughput, error rate, queue depth, KV pressure — plus sampled output quality (LLMs fail softly with 200 OK and worse answers)
  • Autoscaling: Scale GPU instances based on queue depth or request rate. Utilization above ~70% without admission control means latency explosion under bursts
  • Model versioning: Keep old versions running while testing new ones. Artefacts immutable and tagged; rollback is one config flip, drilled before you need it
  • Deploy gates: Promote quantized or new checkpoints only after eval-gated comparison (task score + general suite + contamination status)
  • Rate limiting: Prevent abuse and manage GPU costs
  • Caching: Cache frequent prompts and shared prefixes to reduce redundant computation
  • Load testing: Size from measured per-GPU throughput on your p95 prompt mix: GPUs = ceil(peak load ÷ per-GPU throughput) + 1 spare
Next actions
Size the deployment →Serving observability →Failure modes →