A Practical Guide to Choosing a Local LLM — A PDF Translation Pipeline Based on NLLB-200 (600M)
Contents
When preparing for product security certifications (RED, CRA, etc.), we often read English standards and guidance documents. They are long and dense with security terminology, and general-purpose translators always fell short: they render those terms awkwardly, or require sending internal documents to external servers. So we started this work with the idea that “it would be great to have a translator that understands the security domain and runs in-house.”
1. Local LLM Selection Basics — Hardware and Quantization
The first constraint you face when adopting a local LLM in practice is always GPU memory (VRAM).
Loading the original model into memory as-is very often runs out of VRAM, causing the process to stall or crash (OOM). So before adopting a model, quantization and FP16 / 4-bit compression strategies should be reviewed first.
LLMFIT is a tool that finds, among hundreds of LLM models and providers, the models that can actually run on your system (RAM, CPU, GPU), with a single command.
It helps greatly in the first step of choosing a model that fits your hardware. Installation and usage are simple.
brew install llmfit
llmfitIt shows each model’s expected throughput (tokens/s), quantization format, memory usage, context length, and fit rating (Perfect/Good/Marginal/Too Tight) at a glance.
Notes
FP16 (half precision): The best and safest in precision and quality, but VRAM requirements are high.
GGUF / AWQ (4-bit/8-bit): Compresses model size dramatically to about one third, extending the limits so models can actually run even in low-VRAM environments such as laptops.
1.1 Comparison of Model Compression Options
FP16 (16-bit): preserves high precision and model quality / high VRAM requirements
GGUF / AWQ (4/8-bit): greatly reduces VRAM usage / runs smoothly, but some performance loss must be accepted
1.2 Recommended Model Families by Task
General conversation and instructions:
Llama-3,Mistral— strong general-purpose and instruction-following abilityCode generation and analysis:
CodeLlama,DeepSeek— specialized through pretraining on programming-language corporaDocument translation (PDF, etc.):
NLLB-200,SeamlessM4T— extremely lightweight and optimized for fast batch sentence translation
2. Our Choice — Goals and NLLB-200 (600M)
The goal was clear: build a PDF translation web service that preserves the original layout, so anyone on the team could translate foreign-language documents quickly and accurately. The constraints were tough, though. External APIs like DeepL were out because documents would leave the company and usage-based costs add up, while our existing Ollama-based general-purpose LLMs could not deliver the translation speed and hardware efficiency we needed.
Our answer was to run Meta’s multilingual model built solely for translation, NLLB-200 (No Language Left Behind, 200 languages), fully on-premise. It was a choice that addressed both data security and processing speed.
2.1 Why NLLB-200 (600M)
Fit for purpose: Unlike general-purpose LLMs (Llama, etc.), it is trained solely for text translation and delivers excellent translation quality with few parameters.
Hardware cost-efficiency: The 600M model is very light, using under 1 GB of VRAM, so even on a laptop (RTX 5060, 8 GB VRAM) we could raise translation concurrency to 4–16 without trouble and achieve strong parallel throughput. (We also kept it extensible so it can be swapped for the 1.3B or 3.3B models if needed.)
Tip. On very new GPUs (Blackwell, such as the RTX 5060,
sm_120), the default PyTorch build may refuse to run — updating the NVIDIA driver and PyTorch to the latest versions and loading the model not as.binbut assafetensorsresolves it in most cases.
3. Implementation — The PDF Translation Pipeline
3.1 PDF Layout Preservation Engine (PyMuPDF)
Rather than simply scraping text, it extracts content while tracking coordinates per Block (paragraph) or Line (e.g., table headers) in the PDF. Using this position data, the translated text is placed back at the same coordinates as the original, with the font size calculated automatically so text does not overflow its box. This keeps the original layout intact after translation.
3.2 Asynchronous Parallel Translation Engine (FastAPI + asyncio)
The key was to translate many sentences at once while running only as much as the GPU can handle.
A semaphore dynamically limits the number of sentences translated at once within the VRAM budget, preventing out-of-memory (OOM) crashes at the source.
Another pitfall was the tokenizer. HuggingFace’s Rust-based fast tokenizer raises an Already borrowed error when multiple threads access it at once; we used
threading.Lock() to serialize only the model inference section (model.generate()), preventing the conflict and completing a stable pipeline.
3.3 Security Domain Translation Glossary
The biggest weakness of general-purpose translators is that they mistranslate security terms. So we implemented a dedicated security-domain glossary so that designated terms are always replaced with the same expression during translation. For example, terms such as attack surface, SBOM, threat model are translated consistently regardless of context, greatly improving terminology consistency and accuracy across the whole document.
4. Conclusion
Even in the era of general-purpose LLMs, we confirmed that a small local model (sLLM) specialized for a specific task (translation), when well chosen and paired with a well-designed pipeline, can deliver performance sufficient for production without API costs. As next steps, we are considering OCR integration (Tesseract, etc.) for scanned PDFs and enhancing the security glossary.
What we did was less about finding the right answer to “which model is best” and more about finding “which combination actually runs in our environment and produces meaningful results.” Small experiments and automated exploration help make realistic decisions.
If you are considering adopting a local LLM in a similar environment, we hope the selection criteria and implementation notes in this post serve as a small starting point.
