The NLP Pipeline Checklist: 14 Things That Break Between Notebook and Production
Every data scientist has shipped a model that worked perfectly in a notebook and failed catastrophically in production. Here's the checklist I use before deploying any NLP pipeline.
Data Layer
1. Encoding assumptions — Your training data was UTF-8. Your production data has Windows-1252, Latin-1, and raw bytes from a 2008 MySQL dump. Always detect and normalize encoding before tokenization.
2. Text length distribution — If you trained on 50-200 word documents and production feeds you 10,000-word contracts, your model will silently truncate and return garbage. Log input lengths from day one.
3. Language detection — "English-only" datasets always contain French, Spanish, and Chinese. Add a langdetect gate or your model outputs become unpredictable on non-English input.
Preprocessing Layer
4. Tokenizer mismatch — The tokenizer you trained with must be the tokenizer you serve with. Version-pinning isn't optional. A BPE vocab update can shift token IDs and destroy model performance without any visible error.
5. Normalization order — Lowercasing before NER destroys proper noun detection. Stemming before sentiment analysis merges "great" and "grate". The order of your preprocessing steps is a hyperparameter.
6. Missing spaCy models — en_core_web_sm and en_core_web_trf produce different tokenizations, POS tags, and entity boundaries. Your Docker image must pin the exact spaCy model version.
Model Layer
7. GPU memory fragmentation — PyTorch doesn't return GPU memory to the OS after freeing tensors. Long-running servers accumulate fragmented memory until OOM. Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and monitor torch.cuda.memory_allocated().
8. Non-deterministic batching — Padding sequences to max length within a batch means predictions depend on what other texts are in the same batch. This makes debugging nearly impossible. Use fixed-length padding or sort by length.
9. Softmax overconfidence — A model returning 0.99 confidence doesn't mean it's 99% correct. Uncalibrated neural networks are notoriously overconfident. Add temperature scaling or Platt scaling before trusting confidence scores.
Serving Layer
10. Cold start latency — Loading a transformer model takes 5-30 seconds. Your health check must wait for model loading to complete. A load balancer routing traffic to an unloaded model returns 500s.
11. Quadratic attention scaling — BERT's attention is O(n²) in sequence length. A 512-token input is 4x slower than a 256-token input. Set hard max-length limits and reject inputs that exceed them.
12. No dynamic batching — Processing one request at a time wastes 90% of your GPU capacity. Implement dynamic batching (collect requests for 50ms, process as a batch) for 5-10x throughput improvement.
Monitoring Layer
13. Data drift goes undetected — Your model's confidence distribution is a canary. If average confidence drops 10% over a week, your input distribution has shifted. Set up automated drift detection with KS tests.
14. No prediction logging — If you can't replay last Tuesday's predictions, you can't debug last Tuesday's bug. Log every prediction with input, output, confidence, latency, and model version.
---
I cover all 14 of these (with production Python code for each) in my NLP Pipeline Engineering course. Every lesson is built from patterns I've used shipping NLP systems to production.