Abstract: DeepSeek-R1 Distill Llama-70B delivers SOTA reasoning at lower costs than massive MoE models. This guide details production deployment via vLLM, hardware sizing, and memory tuning.1. Hardware BenchmarksHardwareEngineQuantizationContextSpeed2x RTX 4090 (48GB)vLLMAWQ (INT4)16k30-45 tok/s4x RTX 4090 (96GB)vLLMFP832k65-85 tok/s2x A100 (160GB)vLLMBF16 Native64k+90-120 tok/sUnquantized FP16 requires at least 160GB VRAM to leave headroom for KV cache allocations.2. vLLM Production SetupLaunch containerized vLLM across 4 GPUs with FP8 quantization:docker run --gpus ‘“device=0,1,2,3”’ -d -p 8000:8000
–ipc=host vllm/vllm-openai:latest
–model deepseek-ai/DeepSeek-R1-Distill-Llama-70B
–tensor-parallel-size 4
–max-model-len 32768
–gpu-memory-utilization 0.95
–quantization fp8
–enable-chunked-prefill
–tensor-parallel-size 4: Distributes model weights across 4 GPUs to reduce latency.–enable-chunked-prefill: Prevents memory lockups caused by long reasoning prompts.3. Troubleshooting & FAQOOM During Long Chains: Decrease --max-model-len or set --gpu-memory-utilization to 0.95. Ensure flash-attn is active.Slow Multi-GPU Setup: Verify NVLink status with nvidia-smi topo -m. PCIe bottlenecking degrades Tensor Parallel performance.API Timeout: Reasoning outputs generate lengthy token streams. Increase client HTTP timeout settings accordingly.