I built my first TTS serving platform during a company hackathon. The requirements seemed straightforward: accept text input, return audio in various formats, and let admins add voice samples for cloning (where the model learns to mimic someone’s voice from a short audio clip).

But building a framework that actually handles production traffic taught me that audio APIs need different patterns than text-based systems.

Working with audio changes your API design because you’re dealing with binary data instead of text. File sizes directly affect your bandwidth costs. Users notice delays with audio differently than with text. A 500ms delay for text generation feels fine, but the same delay before audio plays feels broken. You’ll need to think about streaming, caching, and format conversions in ways you probably haven’t if you’re coming from text APIs.

This post walks through building a complete TTS serving system from scratch. I’ll cover audio encoding, HTTP streaming for progressive playback, and the engineering patterns for serving compute-intensive models. These patterns apply beyond TTS. I’ve used similar approaches for image generation APIs.

What You Should Know Before Starting

You should be comfortable with Python, REST APIs, and making HTTP requests. Basic knowledge of neural networks helps but isn’t required. I’ll explain the audio fundamentals and binary data handling you need as we go. I promise it’s not as intimidating as it seems 😉.

Why TTS Feels Different Than Text Generation

I’ve worked with both LLMs and TTS models over the past year, and they feel completely different to work with. With LLMs, you generate discrete tokens that you can count and measure with metrics. The output is text - you can print it, analyze it with regex, or store it in a database easily.

TTS generates continuous audio waveforms. You can’t just run BLEU scores and call it done. The output is binary data that needs special handling.

Quality is subjective too, which makes evaluation tricky. What sounds natural to me might sound robotic to you 🎤🤖.

File size becomes a concern too. Ten seconds of WAV audio is 1.6MB. The same audio as MP3 might be only 80KB. Your choice of audio format directly impacts bandwidth costs and user experience.

Audio Fundamentals 🔉

Before we build anything, you need to understand how audio works. I know it sounds dry, but trust me, understanding these basics will save you hours of debugging weird audio issues later.

Sample Rate

Sample rate measures how many samples per second your audio contains. Think of it like the frame rate of a video - more samples mean more detail, but also larger files.

Common sample rates you’ll encounter:

• 44.1kHz (CD quality) - Standard for music, but overkill for speech

• 22.05kHz (good for speech) - Sweet spot for TTS quality vs file size

• 16kHz (acceptable for speech) - Minimum for decent speech quality

• 8kHz (phone quality) - You’ll recognize this as “that phone voice quality”

The Nyquist theorem says your sample rate must be at least twice the highest frequency you want to capture. Human speech tops out around 8kHz, which is why 16kHz sampling captures everything you need for speech. Music goes much higher (up to 20kHz for human hearing), which is why music uses 44.1kHz.

For TTS specifically, I recommend 22.05kHz. Going 44.1kHz doesn’t improve quality because speech just doesn’t have those higher frequencies. You’re literally wasting storage and bandwidth.

Bit Depth

Bit depth determines how many bits represent each sample. This is like color depth in images - more bits mean more precision.

Common bit depths:

• 8-bit - Sound terrible, only use for prototyping

• 16-bit - Standard for speech, gives you 96dB of dynamic range

• 24-bit - Used for professional audio recording, overkill for TTS

16-bit is standard for speech. You get 96dB of dynamic range, which is way more than enough for human speech (normal conversation is around 60dB). Going to 24-bit doesn’t help TTS at all.

Channels

Audio can be mono (one channel) or stereo (two channels). TTS is almost always mono because speech doesn’t need stereo imaging. Think about it - when someone talks to you, you don’t need them in surround sound.

Stereo doubles your file size for zero benefit in TTS applications. The only exception is if you’re doing something creative like placing different speakers in different channels, but that’s pretty niche.

Audio Formats

This is where things get practical. You need to understand these formats because you’ll use different ones at different stages of your pipeline.

WAV (Uncompressed)

WAV stores raw PCM (pulse-code modulation) data with RIFF header. It’s universal, maintains perfect quality, but the files are massive! One minute of 16-bit, 44.1kHz mono audio is 5.3MB. For a typical audiobook chapter (20 minutes), that’s over 100MB uncompressed.

I use WAV for:

• Intermediate processing in my pipeline

• Debugging audio quality issues

• Archival storage when quality matters most

When you’re manipulating audio (resampling, normalizing, etc.), always work with WAV. The last thing you want is to lose quality at each processing step through multiple lossy compressions.

MP3 (Lossy Compression)

MP3 uses “lossy” compression, meaning it permanently throws away data to shrink files. The codec (the algorithm that encodes and decodes audio) uses psychoacoustic models to figure out what to remove. It analyzes which frequencies humans can’t hear or won’t notice because they’re masked by other sounds, then discards them. Pretty clever.

MP3 compression is dramatic. A one-minute audio clip that’s 5MB as WAV becomes 500KB as MP3. When you’re serving thousands of audio clips, that 10x reduction in file size directly impacts your bandwidth costs.

The tradeoff is quality degradation. Each encode/decode cycle loses some information. This is why you should never edit an MP3 directly - always decompress to WAV first, edit, then re-encode. But for delivering audio to user over the internet, MP3 works well.

Opus (Modern Lossy)

Opus is the newer codec and it’s legitimately better than MP3 at the same bitrate. It’s a hybrid codec optimized for both speech and music. Lower latency (important for real-time), better quality at low bitrates, and it’s an open standard (no licensing fees).

The catch? Slightly less universal browser support than MP3. Safari only added full support relatively recently. For WebRTC or real-time streaming applications, Opus is excellent. For broader compatibility across all browsers and devices, MP3 still wins.

My rule of thumb: Use MP3 for maximum compatibility, use Opus if you’re targeting modern browsers or need real-time capabilities.

Working With Audio in Python

Here’s how you load and manipulate audio in Python. I’ll walk through each line.

Audio is just a Numpy array! Each element is a sample value (usually a float between -1.0 and 1.0), and the sample rate determines how those samples map to time. It’s actually pretty straightforward once you realize it’s just an array of numbers.

The sr=None parameter in librosa.load() is important. It preserves the original sample rate. If you omit it, librosa defaults to 22050Hz and resamples your audio automatically, which might not be what you want.

Choosing Your TTS Model

Model selection is crucial and involves trade-offs between quality, speed, voice variety, and infrastructure requirements. High-quality models are generally slower. More voices mean larger model downloads. Voice cloning adds significant complexity.

I’ve tested most of the popular open-source options. Here’s my assessment of each:

Coqui XTTS v2: This is what I’m using in production. Multi-lingual support (17+ languages), supports voice cloning, model size is around 2GB. You need at least 4GB+ VRAM. The quality is solid, not quite ElevenLabs level, but close enough for most applications. Generation speed is about 1-2 seconds for a typical sentence on a T4 GPU.

Bark: Highly expressive and can generate music and sound effects, which is pretty cool. But it’s significantly slower than XTTS (sometimes 5-10 seconds or a single sentence). Great for creative applications where you need those extra sound effects, but not practical for real-time or high-throughput scenarios.

Piper: Fast CPU inference, multiple voices out of the box, but limited customization options. This is excellent if you don’t have GPU infrastructure or need something lightweight. Quality is decent but not amazing. I use this for dev environments where I don’t want to spin up GPU instances.

StyleTTS2: High quality with voice cloning capability, moderate speed. It requires more ML knowledge to tune properly though. If you’re comfortable with model training and fine-tuning, this gives you more control. But the learning curve is steeper.

For this tutorial, I’m using Coqui XTTS v2. It hits the sweet spot of quality, speed, and voice cloning capability. The model needs 4GB+ VRAM and supports voice cloning with just 6-30 second samples (which is impressively short). XTTS is licensed under Mozilla Public License 2.0, so you can use it commercially without licensing headaches.

Our Project Structure

Here’s how I organize TTS projects. Don’t come for me 🫣.

tts-api/
├── server/
│   ├── __init__.py
│   ├── main.py          # FastAPI application
│   ├── model.py         # TTS model loading and inference
│   ├── audio.py         # Audio processing utilities
│   ├── storage.py       # File storage management
│   ├── schemas.py       # Pydantic validation models
│   └── config.py        # Configuration
├── client/
│   └── client.py        # Python client library
├── storage/
│   ├── voices/          # Voice sample storage
│   └── generated/       # Generated audio cache
├── tests/
│   ├── test_audio.py
│   ├── test_api.py
│   └── test_integration.py
├── requirements.txt
├── .env
└── README.md

This structure separates concerns cleanly. Audio processing lives in audio.py, model inference in model.py, storage management in storage.py. FastAPI handles HTTP in main.py, Pydantic validates inputs in schemas.py, and the client library in client/ provides a clean interface for users.

Building Audio Processing Utilities

Audio processing needs careful handling. I centralize these operations to prevent bugs from inconsistent processing across different parts of the codebase.

Normalization: Making Audio Levels Consistent

User-uploaded voice samples have wildly different volumes. Some people record super quiet, others are basically YELLING into their microphone. Training on inconsistent volumes confuses the model, and generation produces unpredictable loudness that makes your API feel buggy.

Here’s the normalization function with detailed explanation 👇🏾

This ensures consistent loudness across all your audio. The math converts between linear amplitude (what the actual samples represent) and logarithmic decibels (how humans perceive loudness). Standard practice for speech is -20dB LUFS (Loudness Units relative to Full Scale).

Why -20dB specifically? It’s loud enough to be clearly heard, but leaves enough headroom that you won’t clip (distort) when processing.

Resample: Getting the Right Sample Rate

Models expect specific sample rates, typically 22.05kHz or 24kHz for XTTS. User uploads might be 44.1kHz (CD quality), 48kHz (video standard), or even 8kHz (phone quality). Mismatched sample rates cause pitch and speed issues.

Trust me, you don’t want your API accidentally generating chipmunk voices because someone uploaded 44.1kHz audio and you didn’t resample 🐿️.

The implementation looks simple, but librosa does the heavy lifting behind the scenes. It uses polyphase filtering, which is a sophisticated resampling algorithm that avoids aliasing.

Aliasing is the audio distortion you get from poor resampling (think of it like jagged edges in a low-resolution image, but for sound). Librosa handles this complexity so you don’t have to worry about it.

Format Conversion: Moving Between Formats

Internal processing uses WAV for quality. User delivery uses MP3 for size. Different clients might prefer different formats based on their use case.

The -q:a 2 parameter for MP3 sets the quality level (0-9, lower is better). I use 2 which provides excellent quality while keeping file sizes reasonable. This is a good balance for most TTS applications.

Building the Model Server

Now for the core of our system, the model server that actually generates speech.

Why Use a Singleton Pattern?

The TTS model is 2GB+ in memory. Loading takes 10-20 seconds (mostly downloading model weights on the first run). Loading multiple copies would quickly exhaust GPU memory. One model instance can efficiently serve all requests.

Here’s the singleton implementation.

The singleton pattern uses Python’s __new__ method to control instance creation. Every time you call TTSModelServer(), you get the same instance. The model is loaded once on first access and reused for all subsequent requests.

This is a classic creational pattern and works perfectly for resource-heavy models. Plus, the logging helps you debug issues; you’ll know immediately if the model failed to load or if you’re accidentally trying to use it before loading.

How XTTS Actually Works - A Brief Technical Deep-Dive

I think it’s worth understanding what’s happening under the hood, even if you’re not planning to modify the model.

XTTS uses a Transformer-based architecture with a GPT-style autoregressive decoder. The generation process flows like this:

1. Text → Phonemes: Text is converted to phonetic representation (like “hello” becomes “h eh l ow”)

2. Phonemes → Acoustic Features: The transformer predicts mel spectrogram frames. A mel spectrogram is a visual representation of sound that shows which frequencies are present over time. Imagine a heat map showing which frequencies appear at what times. It’s basically the blueprint the model uses to understand what the speech should sound like.

3. Acoustic Features → Audio: A vocoder (the algorithm that synthesizes the actual sound waves) converts mel spectrograms back into audio you can hear. The vocoder XTTS uses is called HiFi-GAN, which is good at producing natural-sounding speech.

Voice cloning works by encoding a 6-30 second sample into a speaker embedding vector (basically a numerical fingerprint of that voice). Generation then conditions on both the text input and this speaker embedding. The temperature parameter controls randomness in token sampling, just like with GPT models (higher temperature = more variation).

This architecture is why XTTS can clone voices with such short samples compared to older methods that needed hours of training data!

Generating Speech

Here’s the core generation function with comprehensive error handling.

The RTF (Real-Time Factor) metric I log is useful. It tells you if your system can keep up with real-time audio. RTF of 1.0 means it takes 1 second to generate 1 second of audio. RTF of 0.5 means you’re generating 2x faster than real-time (good!). RTF of 2.0 means you’re generating 2x slower than real-time (might be a problem for real-time applications).

Voice Cloning Implementation

Voice cloning is where XTTS really shines. You can clone any voice with just a short sample.

⚠️ Important: Voice samples must be 6-30 seconds of clear speech with minimal background noise. One speaker only. If you have multiple speakers talking in the sample, the model will get confused and generate weird blended voices that sound like neither person. I learned this by accidentally using a podcast clip with two hosts talking over each other. The result was…interesting 😅.

Also, background music or noise degrades quality significantly. I recommend recording in a quiet room or using a sample from a professional podcast with good audio quality.

Storage Management and Caching Strategy

You need to properly organize storage for voice samples and generated audio. File paths, metadata, and cleanup all need careful management. More importantly, caching frequently requested generations can save you tons of compute time and money.

Why Caching Matters So Much

Caching matters more than you might expect for TTS APIs. After analyzing production logs, about 40% of requests were for the same 20 phrases. Things like “Welcome to our service”, “Your order has shipped”, common notification messages that apps send repeatedly.

Each generation takes 1-2 seconds of GPU time. Cache lookup? 10-20ms. That difference adds up fast. Implementing caching dropped GPU utilization by 30%, which meant handling more concurrent users on the same hardware.

Here’s the caching implementation:

This caching system saved me a fortune in compute costs. The LRU (Least Recently Used) eviction strategy means popular phrases stay cached while rarely-used ones get cleaned up automatically.

Pop Quiz: Why use content-based keys (hashing the input) instead of just incrementing counters? 🤔

Answer: Content-based keys are deterministic! The same input always produces the same key, so we can look up cache hits without maintaining a separate mapping. Plus, it naturally handles duplicate requests without complex deduplication logic.

Validating Requests with Pydantic

Pydantic is amazing for API input validation. It catches bad inputs at the API boundary before they hit your expensive model operations or corrupt your storage layer.

This validation layer has saved me countless debugging hours. Users sometimes send the wildest inputs. Empty strings, 50,000 character novels, random binary data misidentified as text. Pydantic catches all of it before it hits my model.

Building the FastAPI Server

Now for the exciting part, actually serving our TTS model over HTTP! This is where everything comes together.

Streaming Audio Responses

Now we’re cooking 🔥. Audio files can be several megabytes, and users can hear audio while it’s still downloading (progressive playback). This dramatically lowers perceived latency and improves the user experience.

Here’s the implementation.

The browser receives the first chunk and begins playback immediately. Audio codecs can decode partial files, so progressive download enables playback during transfer. This is absolutely essential for good UX on slow connections.

Why Async Matters Here

Here’s something I wish someone had explained to me when I first learned async programming:

# Bad: Blocks event loop during I/O
with open(path, ‘rb’) as f:
    data = f.read()  # Blocks entire server - other requests wait!

# Good: Allows other requests to process
async with aiofiles.open(path, ‘rb’) as f:
    data = await f.read()  # Other requests handled during I/O

With blocking I/O, your entire server pauses while reading the file. If the file read takes 100ms, no other requests can be processed during that time. With async I/O, Python’s event loop can handle other requests while waiting for the disk.

This made a massive difference in my server’s throughput. Before async, I could handle maybe 10 concurrent requests. After implementing async properly, I could handle 50+ concurrent requests on the same hardware.

Voice Upload Endpoint

Users need a way to upload their voice samples for cloning. Here’s how I handle that with proper validation:

The finally block is critical here. Even if something fails during processing, we clean up the temporary file. Otherwise, failed uploads would fill up your disk over time. Ask me how I know 😏.

Building the Python Client

The client library provides a clean interface for users. It handles all the HTTP details, streaming, retry logic, and provides a Pythonic API that feels natural.

The retry logic with exponential backoff smooths out transient errors. Sometimes servers get temporarily overwhelmed. Instead of failing immediately, the client automatically retries with increasing delays.

Testing Your System

Testing audio systems is different from testing text APIs. String comparison doesn’t work when you’re dealing with binary audio files. You need to verify formats, check durations, validate file sizes, and sometimes manually listen to outputs. Here’s how I structure tests.

Unit Tests for Audio Processing

Integration Tests

Integration tests verify the whole system works together.

What We’ve Built

Let’s take a step back and bask in the glory of our beautiful TTS creation!

We have:

• Model server with singleton pattern - Efficient resource usage, handles model lifecycle

• Audio processing pipeline - Normalization, resampling, format conversion with quality preservation

• File storage and caching - Content-based caching with LRU eviction, saving compute costs

• Voice sample management - Upload, validation, preprocessing for voice cloning

• RESTful API with streaming - Progressive playback, proper error handling, async I/O

• Python client library - Clean API, automatic retries, comprehensive error messages

• Comprehensive testing - Unit tests, integration tests, load tests

The fundamental lessons were efficient resource management, smart caching, streaming delivery, and clean separation of concerns.

Next Steps

In Part 2, I’ll cover production deployment.

We’ll tackle:

• Dockerization: Creating efficient Docker images, multi-stage builds for smaller sizes

• Orchestration: Kubernetes deployment, scaling strategies, load balancing

• CDN integration: Serving cached audio from edge locations for better global performance

• Cost optimization: Reducing GPU costs, efficient batching, spot instances

• Monitoring and alerting: Metrics that matter, detecting quality degradation, performance tracking

• Scaling considerations: Horizontal vs vertical scaling, when to scale up

For now, I encourage you to run this code locally and experiment!

Homework

I recommend you:

1. Upload your own voice - Try cloning your voice with a 15-second sample

2. Measure cache performance - Generate the same phrase 10 times, compare first vs subsequent requests

3. Test different audio formats - Compare MP3 at 128k vs 192k, see if you can hear the difference

4. Experiment with text lengths - How does generation time scale with text length?

5. Monitor GPU utilization - Run nvidia-smi while generating, watch memory and utilization

6. Try different voices - Upload multiple voice samples, see how well cloning works

You can also extend this system in many directions.

• Emotion control: Add parameters for controlling emotional tone of speech

• SSML support: Implement Speech Synthesis Markup Language for prosody control

• Multi-speaker conversations: Generate dialogues with different voices

• Real-time streaming: Build WebSocket endpoint for real-time generation as you type

• Audio effects pipeline: Add reverb, echo, pitch shifting for creative applications

• Batch processing: API endpoint that handles hundreds of texts at once efficiently

I could probably create branching posts that go into each of these different avenues.

Regardless, the patterns here transfer to whatever you build next. Whether you’re serving image generation models, video processing pipelines, or any compute-intensive API, you’ll use these same principles: caching for cost savings, streaming for better UX, singleton pattern for resource management, comprehensive testing, and async I/O for scalability.

I hope this guide gave you both the technical knowledge and practical insights to build your own production TTS system! Feel free to reach me on LinkedIn or at cynthia@cynscode.com.

Happy coding 👍🏾

You could try being more descriptive: “the red car on the left side near the building.” Now the model has to understand relative positions, colors, spatial relationships, and landmarks all at once. That’s asking a lot, and the results are hit or miss.

Microsoft’s Magma8B solves this with Set-of-Mark (SoM) prompting. You draw numbered boxes directly on the image. Instead of vague spatial descriptions, you mark the car as #1 and ask “What color is object #1?” The model understands these visual markers because it was specifically trained on them. No ambiguity, no confusion.

I’m walking you through building a complete client-server API using Magma8B. You’ll learn how to serve an 8-billion parameter vision model, handle image encoding over HTTP, implement proper error handling, and structure code that doesn’t fall apart when things go wrong. The software engineering patterns here apply whether you’re serving ML models, building microservices, or designing any distributed system.

This post will separate you from the other machine learning engineers or data scientists who can only get a model working in a Jupyter notebook. By the end of this article, you’ll walk away knowing how to build a system that handles real requests.

What You Need to Know

You should be comfortable with Python classes and functions. REST APIs should make sense at a basic level. If you’ve used requests.get() and understand what HTTP status codes mean, you’re fine. Some understanding of neural networks helps, but you don’t need to derive backpropagation or understand gradient descent in depth.

Docker knowledge isn’t required for this part. We’ll cover containerization and deployment in Part 2.

Understanding Set-of-Mark Prompting

Traditional visual question answering (VQA) systems struggle with spatial references. They can tell you what’s in an image, but when you need to point at something specific and ask about that exact thing, they don’t have a reliable mechanism to understand your reference.

Set-of-Mark prompting changes this by making spatial references explicit. Here’s how it works:

1. Add visual markers to your image (numbered bounding boxes, highlights, or tags)

2. The model processes both the marked image and your text prompt

3. Reference specific markers in your text: “What is object #3 doing?”

4. The model grounds its response in the marked regions

The model learned to associate text references like “#3” with visual regions during training. When you ask about “#3”, the model knows exactly which part of the image you mean.

This isn’t just a clever prompt engineering trick. Magma8B was trained on millions of images with Set-of-Mark annotations. The model’s weights encode the relationship between numeric references in text and visual markers in images. You can’t just take any vision-language model and expect it to understand marked images.

Real-World Applications

Set-of-Mark prompting unlocks some practical use cases.

Document Analysis: When working with scanned documents, you can mark specific paragraphs or sections. “Summarize paragraph #3” becomes unambiguous. Legal document review, contract analysis, and research paper summarization all benefit from this precision.

UI Testing: Automated testing frameworks can mark specific UI elements. “Click button #2” gives you maintainable tests that don’t break when the UI layout changes slightly.

Robotics: Pick-and-place tasks become more reliable. “Grasp object #1” tells the robot exactly which object to target, even in cluttered environments.

Medical Imaging: Radiologists can mark regions of interest and ask specific questions. “Analyze the tissue marked as #1” focuses the model’s attention on the relevant area.

The key advantage is precision. Spatial references are explicit rather than implicit, reducing errors and making the system’s behavior predictable.

Understanding Magma8B 🤖

Magma8B is Microsoft’s vision-language-action model released in February 2025. It has 8 billion parameters and combines a CLIP-ConvNeXt-XXLarge vision encoder with Meta’s Llama-3 language model.

What makes Magma significant is that it’s the first foundation model designed specifically for multimodal AI agents!

Most vision-language models stop at understanding. They describe images or answer questions about them. Magma goes further. It generates goal-driven visual plans and actions. The same model that answers ‘What color is object #1?’ can tell a browser to click the search button or generate gripper coordinates for a robot arm.

In Microsoft’s benchmarks, Magma is the only model that performs across the full task spectrum: visual QA, UI navigation, and robotics manipulation. GPT-4V can’t control a robot arm. OpenVLA can’t answer questions about charts. Magma does both.

The architecture has three main components:

Vision Encoder: CLIP-ConvNeXt-XXLarge, trained by the LAION team on billions of image-text pairs. It processes images into visual tokens that capture colors, shapes, textures, spatial relationships, and the visual markers you add.

Language Model: Meta’s Llama-3-8B-Instruct serves as the backbone. It processes both text tokens and visual tokens together. This is where reasoning happens.

Training Data: Microsoft trained Magma on a mix of generic image and video data (LLaVA-Next, ShareGPT4Video), instructional videos (Ego4D, Epic-Kitchen), robotics manipulation data (Open-X-Embodiment), and UI grounding data (SeeClick). The diversity matters; it’s why the model generalizes across such different tasks.

Set-of-Mark isn’t something you can apply to just any vision model. It’s specific to how Magma was trained. Microsoft introduced two techniques during training:

Set-of-Mark (SoM): For static images, objects are labeled with visual markers. The model learned to ground its responses to specific marked regions.

Trace-of-Mark (ToM): For videos, object movements are tracked over time. This helps with action planning and understanding temporal dynamics.

We’re focusing on Set-of-Mark for static images. Trace-of-Mark is powerful for video and robotics data, but that’s beyond scope here.

⚠️ Important Research Context: Magma8B is explicitly designed for research purposes. Microsoft’s model card states it should not be deployed in production settings or used in high-risk scenarios like military, financial services, or critical infrastructure.

This doesn’t mean you can’t learn from it or build prototypes. It means you need to understand its limitations:

• The model was trained primarily on English text

• It can perpetuate stereotypes or produce inappropriate content

• It has common multimodal model limitations around hallucination

• It hasn’t been extensively safety-tested for production use

For learning, experimentation, and non-critical applications, Magma8B is excellent. For production systems handling sensitive data or critical decisions, you’d want a model with more extensive safety testing and production support.

Why Client-Server Architecture Matters

Running this model directly in your application isn’t practical for several reasons.

The model checkpoint is 16GB. You’re not shipping that to mobile devices or embedding it in web applications. Inference needs a GPU with at least 24GB of VRAM. Most consumer laptops don’t have that. Loading the model takes several seconds on a good machine, longer on slower hardware. You don’t want users waiting that long on first launch.

Model updates become a nightmare. If you need to fix a bug, you’ll need to push to every client. If you need to upgrade to a newer version, you’ll need to update every installation. How about using a different model? You’ll need to rebuild and redistribute your entire application.

Client-server architecture solves these problems.

1. Load the model once on a server with proper GPU resources

2. Clients send images and prompts over HTTP

3. Server runs inference and returns results

4. Clients are lightweight and model-agnostic

This separation means you can upgrade the model, switch to an entirely different architecture, scale to multiple GPUs, or add caching without changing a single line of client code.

The architecture looks like this

Client (Python/JS/Mobile) → API Server (FastAPI) → Model Server (Magma8B on GPU)

The API server handles HTTP requests, validates inputs, manages authentication (if needed), and orchestrates inference. The model server focuses solely on loading the model, running inference, and managing GPU memory. Each component has one job and does it well.

Separation of Concerns

This follows a fundamental software engineering principle. Your API layer handles authentication, rate limiting, and logging without touching model code. Your model code focuses exclusively on inference. Want to swap models? Change one file. Want to add API key validation? Modify only the API layer.

You see this pattern everywhere in well-designed systems.

• Web applications separate frontend, backend, and database layers

• Microservices split functionality across independent services

• Operating systems separate kernel space from user space

Network Boundaries Force Good Design

Building across network boundaries actually makes you write better code because it forces constraints.

You must serialize data: Can’t pass Python objects over HTTP. This forces you to think carefully about your data structures and interfaces.

You must handle errors explicitly: Networks fail. Requests time out. Servers crash. You can’t pretend these won’t happen.

You must define clear interfaces: Ambiguous interfaces break quickly over network boundaries. Clear contracts become essential.

You must think about performance: Network latency matters. You optimize data transfer and response times.

These constraints push you toward maintainable, robust code. This is why microservices (despite their complexity) can lead to better system design. The boundaries force discipline.

Building the Model Server

Let’s start with proper project structure. Good organization makes debugging easier and helps others (including future you) understand your code.

magma-api/
├── server/
│   ├── __init__.py
│   ├── main.py          # FastAPI application
│   ├── model.py         # Model loading and inference
│   ├── schemas.py       # Pydantic models for validation
│   └── config.py        # Configuration management
├── client/
│   └── client.py        # Python client library
├── requirements.txt
├── .env                 # Local environment variables (add to .gitignore!)
└── README.md

Each file has a single responsibility. The API code lives in main.py, model logic in model.py, validation schemas in schemas.py, and configuration in config.py. This separation makes testing easier and changes more localized.

Create requirements.txt:

fastapi==0.104.1
uvicorn[standard]==0.24.0
torch==2.1.0
transformers==4.35.0
Pillow==10.1.0
pydantic==2.5.0
python-multipart==0.0.6

Quick breakdown of dependencies 🕸️

• FastAPI and Uvicorn: Web framework and ASGI server. FastAPI gives you automatic validation, documentation, and excellent async support.

• PyTorch and Transformers: Model inference. Transformers make loading Hugging Face models straightforward.

• Pillow: Image processing for loading and manipulating images.

• Pydantic: Data validation. This will save you from so many bugs.

• python-multipart: File upload support for when we extend the API.

Model Loading and Management

Loading a 16GB model takes time, consumes significant memory, and you definitely don’t want to reload it for every request. That would be painfully slow and waste resources.

The solution is the Singleton pattern. Load the model once, keep it in memory, reuse it for every request. This is standard practice for serving ML models.

Create server/model.py:

Understanding the Singleton Pattern

The __new__ method controls instance creation in Python. Most classes use the default implementation, but we override it to ensure only one instance ever exists.

First time you create ModelServer(), Python creates a new instance and stores it in cls._instance. Every subsequent call checks if _instance exists and returns it if so. One model in memory, shared across all requests.

Why does this matter? If you loaded a new model for every request:

• You’d wait several seconds per request (terrible user experience)

• You’d run out of GPU memory immediately (each model copy takes 16GB)

• You’d waste compute resources loading the same weights repeatedly

The Singleton pattern is common in production systems for expensive resources:

• Database connection pools (one pool, many connections)

• Configuration managers (one config object, accessed everywhere)

• Logger instances (one logger per module)

• Cache managers (one cache instance, shared state)

Memory Optimization with bfloat16

torch_dtype=torch.bfloat16

This cuts memory usage in half compared to float32. Instead of 32-bit floating point numbers, we use 16-bit Brain Float format.

“Wait, doesn’t that hurt accuracy?” Not really for inference. Training needs float32 (or even float64 sometimes) for stable gradients across many iterations. Inference runs once through the network, and bfloat16 precision is usually sufficient.

The memory savings are massive. With bfloat16, Magma8B fits comfortably in 24GB of VRAM. With float32, you’d need closer to 32GB. This is standard practice in production ML serving.

Device Mapping

device_map="auto"

This tells Transformers to automatically distribute the model across available devices. If you have multiple GPUs, it can split layers across them. If your model barely fits in GPU memory, it can offload some layers to CPU memory (slower, but better than crashing).

For most cases with a single 24GB GPU, this just loads everything onto CUDA device 0.

Property Decorator for Clean Access

@property
def model(self):
    if self._model is None:
        raise RuntimeError(”Model not loaded. Call load_model first.”)
    return self._model

This creates a clean access pattern with built-in validation. Instead of accessing _model directly, you use model_server.model. If someone tries to use the model before loading it, they get a clear error message:

RuntimeError: Model not loaded. Call load_model first.

Much better than:

AttributeError: ‘NoneType’ object has no attribute ‘generate’

Good error messages save hours of debugging. When something goes wrong, you want to know exactly what and where.

Implementing Set-of-Mark Inference

Now let’s add the actual inference logic. This is where Set-of-Mark prompting happens.

Add to server/model.py:

Drawing Visual Marks

The marks get drawn directly on the image. This is crucial: the model needs to see them to understand references like “#1.” We’re modifying the visual input, not just adding metadata.

draw.rectangle([x1, y1, x2, y2], outline=”red”, width=3)
draw.text((x1, y1 - 30), label, fill=”white”, font=font)

We draw red bounding boxes with 3-pixel width for visibility. The label goes above the box with a red background, making it stand out even on complex images.

During training, Magma saw similar visual markers. It learned to associate these visual cues with text references. When you ask “What is object #1?”, the model looks for the visual marker labeled “#1” and reasons about that specific region!

Chat Template Format

Magma expects inputs in a specific chat format:

convs = [
    {”role”: “system”, “content”: “You are agent that can see, talk and act.”},
    {”role”: “user”, “content”: “<image_start><image><image_end>\nWhat is #1?”}
]

The system message sets the context. The user message includes special tokens <image_start> and <image_end> that tell the model where the image embedding goes.

The apply_chat_template method formats these messages according to Llama-3’s conversational structure. Different models have different templates, and Transformers handle this automatically.

Tensor Processing

inputs = self.processor(images=[image], texts=text, return_tensors=”pt”)
inputs = inputs.to(self.model.device)

The processor handles several transformations:

1. Resizes the image to the expected dimensions (typically 224x224 or 384x384)

2. Normalizes pixel values to the range the model expects

3. Converts everything to PyTorch tensors

4. Tokenizes the text into token IDs

Moving to the model’s device is essential. The model lives on GPU, and mixing CPU and GPU tensors causes errors fast.

Inference Mode

with torch.inference_mode():
    outputs = self.model.generate(...)

This is better than torch.no_grad() for inference. It disables gradient computation and enables inference-specific optimizations. Always use this for production serving.

Without this context manager, PyTorch tracks gradients for every operation (in case you want to do backpropagation). This uses significant memory and slows things down. For inference, you never need gradients.

Deterministic Generation

do_sample=False

This makes generation deterministic. The model always picks the highest probability token at each step. Same input gives same output.

For analytical queries like “What color is object #1?”, you want consistent results. The answer shouldn’t randomly vary between “red”, “crimson”, and “burgundy” depending on sampling randomness.

If you wanted creative or varied outputs (like for creative writing), you’d set do_sample=True and adjust temperature. But for analytical tasks, deterministic output makes sense.

Autoregressive Generation

The generate method works autoregressively:

1. Model generates one token

2. Appends it to the input

3. Generates the next token based on all previous tokens

4. Repeats until reaching max_new_tokens or an end-of-sequence token

This is why generation can be slow for long responses. Each token depends on all previous tokens, so you can’t parallelize across the sequence length dimension.

For a 100-token response, the model runs 100 forward passes.

Error Handling and Resource Management

Things will go wrong. GPUs run out of memory. Images are corrupted. Networks time out. Users send invalid inputs. Let’s handle these gracefully.

Add to server/model.py:

Custom Exception Hierarchy

We define custom exceptions for different error types:

• ModelLoadError: Something went wrong loading the model (missing files, incompatible versions)

• InferenceError: Inference failed (OOM, CUDA error, unexpected exceptions)

• InvalidImageError: Input validation failed (corrupted image, invalid marks)

This lets you (and your API) distinguish between error types:

• ModelLoadError: Don’t retry, something is fundamentally wrong

• InferenceError from OOM: Retry with smaller inputs

• InvalidImageError: Client error, return 400 Bad Request

• InferenceError from other causes: Might be transient, could retry

Your API can map these to appropriate HTTP status codes. Your logs become more useful when you can filter by error type and identify patterns.

Input Validation

if x2 <= x1 or y2 <= y1:
    raise InvalidImageError(”Mark has invalid dimensions”)

We validate marks before attempting inference. A bounding box where x2 ≤ x1 makes no geometric sense. It would have zero or negative width. Catch this early with a clear error message.

This is way better than letting bad data propagate into the model and causing weird errors deep in the inference stack. “Mark has invalid dimensions” is much clearer than some cryptic error from PIL or PyTorch.

Memory Management After OOM

except torch.cuda.OutOfMemoryError:
    torch.cuda.empty_cache()
    raise InferenceError(”GPU out of memory...”)

After an OOM error, we call torch.cuda.empty_cache() to release unused GPU memory. This doesn’t free memory actively in use, but it helps prevent fragmentation.

Use this sparingly because it has overhead. But after an OOM error, it’s worth trying to clean up before the next request.

⚠️ Important: Catching CUDA OOM errors is tricky. By the time the exception is raised, your GPU might be in a weird state. In production, you might want to restart the process after multiple consecutive OOM errors. It’s not elegant, but it’s reliable.

What I usually do: after three OOM errors in a row, log a critical alert and exit gracefully. A process manager (like systemd or Kubernetes) restarts the service. Brief downtime beats serving incorrect results or crashing unexpectedly.

Building the API Layer

Now let’s wrap the model server in a REST API. This is the interface clients actually use.

Defining Request and Response Schemas

Pydantic models define your API’s contract. They validate input, generate documentation automatically, and make your life much easier.

Create server/schemas.py:

What Pydantic Gives You

Automatic Validation: Invalid data gets rejected before hitting your code. If someone sends a string where you expect an integer, Pydantic returns a clear error message. You don’t write validation code, Pydantic handles it.

Type Safety That’s Enforced: Python’s type hints are suggestions. mypy can check them statically, but at runtime, Python doesn’t enforce them. Pydantic enforces type constraints at runtime. This catches bugs early.

Automatic Documentation: FastAPI uses these schemas to generate interactive API documentation. Your users see exactly what fields are required, what types they accept, what constraints apply, and example requests. All from your Pydantic models. Zero extra work.

Serialization: Pydantic handles conversion between Python objects and JSON automatically. You work with nice Python dataclasses, Pydantic handles the wire format.

Field Validators

@validator(’x2’)
def x2_greater_than_x1(cls, v, values):
    if ‘x1’ in values and v <= values[’x1’]:
        raise ValueError(’x2 must be greater than x1’)
    return v

Validators let you encode business logic beyond basic type checking. The @validator decorator runs custom validation after Pydantic parses the field. Here we ensure bounding boxes have positive dimensions before the request ever reaches the model layer.

This works together with the validation in ModelServer.generate(). Bad coordinates get caught at the API boundary with a user-friendly error, not deep in the inference stack.

Why Base64 for Images?

JSON doesn’t support binary data natively. Base64 encodes binary as ASCII text, making it JSON-safe. The tradeoff is about a 33% size increase (every 3 bytes becomes 4 characters), but you get simplicity and compatibility.

Alternative approaches exist:

multipart/form-data: Send images as file uploads. More efficient (no encoding overhead), but more complex to implement and test.

Binary protocols: Use Protocol Buffers or similar. Fastest option, but way more complex and less debuggable.

For most use cases, Base64 in JSON is fine. If you’re sending thousands of images per second, consider alternatives. But at that scale, you have bigger architectural concerns anyway.

Configuration Management

Settings should live in environment variables, not hardcoded in your source. You should be able to change settings between environments without touching code.

Create server/config.py:

Why External Configuration

Different environments need different settings.

• Development: Might use CPU (no GPU), have verbose logging, and permissive CORS

• Staging: Might use GPU, have moderate logging, and restricted CORS

• Production: Uses GPU, has minimal logging, strict CORS, and monitoring enabled

You change environment variables, not code. Secrets live in environment variables or secret management systems, never in Git!

This principle comes from the Twelve-Factor App methodology: store config in the environment. It seems obvious once you know it, but violating it causes countless problems.

Creating a .env File

For local development, create .env in your project root (and add it to .gitignore):

MODEL_PATH=microsoft/Magma-8B
DEVICE=cuda
API_HOST=0.0.0.0
API_PORT=8000
MAX_IMAGE_SIZE=10485760
LOG_LEVEL=INFO
CORS_ORIGINS=*

Pydantic loads this automatically. In production, you’d set these as actual environment variables through Docker, Kubernetes, or your deployment platform.

FastAPI Implementation

Now let’s build the actual API. This is where everything comes together.

Create server/main.py:

Understanding the API Components

Lifespan Events

@asynccontextmanager
async def lifespan(app: FastAPI):
    logger.info(”Loading model...”)
    model_server.load_model(settings.MODEL_PATH, settings.DEVICE)
    yield
    logger.info(”Shutting down...”)

This replaced the deprecated @app.on_event(“startup”) decorator. Code before yield runs at startup, code after runs at shutdown.

This ensures:

• A model loads before accepting requests (no 500 errors from an uninitialized model)

• Resources clean up gracefully on shutdown (important for Docker containers)

• You can run initialization code that might fail (and handle failures appropriately)

Health vs Readiness Checks

@app.get(”/health”)
async def health_check():
    return {”status”: “healthy”}

@app.get(”/ready”)
async def readiness_check():
    _ = model_server.model  # Raises if not loaded
    return {”status”: “ready”, “model_loaded”: True}

These serve different purposes and are essential for production deployments.

/health: Is the service running? Returns 200 if the process is alive. Load balancers use this for liveness probes.

/ready: Is the service ready to handle requests? Returns 200 only if the model is loaded. Load balancers use this for readiness probes.

Kubernetes and other orchestrators use these endpoints:

• Liveness probe: If this fails repeatedly, restart the container (it’s dead)

• Readiness probe: If this fails, stop sending traffic (it’s not ready yet)

During deployment, traffic only routes to pods that pass readiness checks. This prevents users from hitting a server that’s still loading its 16GB model, which is essential for zero-downtime deployments.

CORS Middleware

app.add_middleware(
    CORSMiddleware,
    allow_origins=cors_origins,
    allow_credentials=True,
    allow_methods=[”*”],
    allow_headers=[”*”],
)

This allows web browsers to call your API from different domains. Without CORS, browsers block cross-origin requests for security.

The allow_origins=[“*”] setting allows requests from any origin, which is fine for development. In production, restrict this to your actual domains:

allow_origins=[”https://myapp.com”, “https://www.myapp.com”]

If you’re only doing server-to-server communication, you don’t need CORS. Browsers enforce CORS policies, servers don’t care.

Base64 Decoding with Validation

This handles several cases.

Data URLs: Browsers often send data:image/png;base64,.... We strip the prefix.

Size validation: Reject images larger than MAX_IMAGE_SIZE. Prevents memory exhaustion attacks.

Image verification: image.verify() checks if the file is actually a valid image. Catches corrupted files early.

Reload after verify: verify() closes the file handle, so we need to reload.

Color mode normalization: Convert to RGB. Some images use RGBA (transparency), L (grayscale), or other modes. The model expects consistent RGB input.

HTTP Status Codes

Different status codes communicate different things.

400 Bad Request: Client’s fault (invalid data, malformed input). Client should fix their request. Don’t retry the prior request.

500 Internal Server Error: Server’s fault (inference failed, unexpected error). Client can retry, it may or may not work.

503 Service Unavailable: Service exists but isn’t ready (model hasn’t loaded). Client should wait and retry.

This distinction matters. Well-behaved clients know:

• Don’t retry 400 errors (the problem is in the request)

• Do retry 500/503 errors with exponential backoff (it’s a temporary issue, might resolve)

HTTP status codes are a universal language. Your API speaks the same language as every other HTTP service. Use them correctly.

Building the Client

Your client’s job is simple: send images and prompts, and receive responses. Let’s build a clean Python client.

Create client/client.py:

Key Design Decisions

Session Reuse

self.session = requests.Session()

TCP connections are expensive to create. A Session reuses connections through connection pooling. For multiple requests to the same host, this is significantly faster.

This is the same principle as database connection pooling. Create expensive resources once, reuse them many times.

Timeouts Are Essential

timeout=30

Without a timeout, requests can hang indefinitely. Your program just waits forever, consuming resources.

30 seconds is reasonable for ML inference. Always set timeout in production. Always!

Retry Logic with Exponential Backoff

for attempt in range(self.max_retries):
    try:
        # ... make request ...
    except requests.exceptions.Timeout:
        if attempt == self.max_retries - 1:
            raise
        time.sleep(2 ** attempt)  # 1s, 2s, 4s, ...

Transient errors (like network blips or temporary server overload) often resolve if you wait and retry. Exponential backoff prevents overwhelming the server with retries.

We retry server errors (5xx) but not client errors (4xx). If you sent invalid data, retrying won’t help.

Context Manager Support

with VisionAPIClient() as client:
    result = client.analyze(”image.jpg”, “What is this?”)

The __enter__ and __exit__ methods let you use the client as a context manager. This ensures the session closes properly even if an exception occurs.

Testing the System

Time to see everything work together.

Running the Server

You should see:

First time running, the model downloads from Hugging Face. This can take a while (remember it’s 16GB). However, subsequent runs load from cache.

Testing with the Client

Create test_api.py:

Expected Behavior

Test 1: Model should describe both shapes (a red circle and a blue square).

Test 2: Model should focus specifically on the red circle, mentioning it’s red.

Test 3: Model should compare both marked objects, discussing colors and shapes of each.

Test 4: Model should reason about spatial relationships between the marked objects.

If you see reasonable responses to all tests, congratulations 🎉! Your Set-of-Mark vision API is working.

Common Issues and Solutions

“Model not found” Error

Check logs during startup for the actual error. Verify MODEL_PATH points to a valid model. First download can be slow. Check your Hugging Face cache at ~/.cache/huggingface/.

Try explicitly downloading the model first:

CUDA Out of Memory

You need a GPU with at least 24GB VRAM. Check with nvidia-smi.

If you don’t have enough VRAM:

• Try DEVICE=”cpu” (it’s much slower, but works)

• Close other GPU-using programs

• Reduce image size before sending

• Lower max_tokens

Timeout Errors

First request is often slower, so you should increase timeout in client:

client = VisionAPIClient(timeout=60)

Reduce max_tokens if responses are very long. Check GPU utilization during requests:

watch -n 0.5 nvidia-smi

You should see GPU usage spike during inference.

Image Encoding Errors

Verify the image file exists and is valid. Check format is supported (JPEG or PNG work, while WebP might cause issues). Try resaving the image in a different format.

Model Responses Seem Wrong

Check if model is in eval mode (we do this in load_model). Verify model actually loaded by checking the /ready endpoint. Try simpler prompts first to verify basic functionality. Save marked images and visually inspect them.

Performance Issues

Measure request latency:

This snippet wraps a request with timing so you can see how long inference takes. First requests typically run slower because of JIT compilation and cache warming. After that, expect 0.5-2 seconds per request on GPU. If you’re seeing 10+ seconds, check that the GPU is actually being used with nvidia-smi while a request is running.

What You Built 🧰

You have a complete production-grade vision API that handles Set-of-Mark prompting.

The model server uses a Singleton pattern to load Magma8B once and serve requests efficiently. The FastAPI layer validates inputs, handles errors gracefully, and generates documentation automatically. The Python client includes retry logic and connection pooling for reliable communication.

The patterns here transfer beyond ML serving:

• Singleton pattern for expensive resource management

• Pipeline pattern for data transformation (image → marks → inference → response)

• Separation of concerns across API, model, and validation layers

• Fail-fast validation at system boundaries

• Custom exception hierarchy for meaningful error handling

• Configuration via environment variables

These show up everywhere. Database connection pools use Singleton. ETL pipelines follow the same transformation patterns. Microservices separate concerns across network boundaries. Every production system validates at boundaries and handles errors explicitly.

What separates this from a simple script or a Jupyter notebook is the infrastructure around the model. There’s proper error handling so failures don’t crash the server, health checks so load balancers know when to route traffic, logging so you can debug issues in production, and configuration management so you can change settings without redeploying code. These details compound.

Looking Ahead

In Part 2, we’ll tackle production deployment:

• Dockerization: Package everything for consistent deployment

• Scaling strategies: Handle concurrent requests, load balancing

• Monitoring: Metrics, logging, alerting that actually helps

• Performance optimization: Caching, batching, query optimization

• Security hardening: Authentication, rate limiting, input sanitization

These tools and techniques will allow you to run your application on any machine!

Try It Yourself

Before moving on, I encourage you to:

1. Run the code locally and verify everything works

2. Test with your own images and prompts

3. Experiment with different mark configurations

4. Try various prompts and see how responses change

5. Measure performance with different settings

6. Intentionally break something and see how error handling works

The best way to learn is by doing. Play around with the code. Change things. See what breaks. That’s how you develop intuition for what works and why.

You’re not just serving models anymore. You’re building systems.

And hey, if you made it this far, you’re doing great! Building production ML systems is hard. There’s a lot to learn. But you’re on the right path 🌟.

See you in Part 2 where we’ll make this production-ready!

Happy coding 💻

That question reveals a significant gap. Running docker-compose up on your machine is one thing. Serving production traffic to users who expect reliability is entirely different. The distance between these two stages is larger than most engineers expect.

At 5 million requests per month, downtime has real consequences. That’s roughly 7 requests per second. When vLLM goes down, revenue drops. Users get frustrated. Your phone starts buzzing with alerts. You can’t experiment with production traffic without risking your reputation and your users’ trust.

This post bridges the gap. We’re taking everything from the previous parts of this series and building a production system that handles real user traffic reliably.

What We’ve Built So Far

The previous posts in this series established the foundation for a robust system. Part 1 identified problems with notebook code: no type safety, no tests, just pure chaos. Part 2 introduced BAML to add structure and type safety to LLM interactions. Part 3 covered comprehensive testing strategies to catch bugs before production. Part 4 walked through vLLM self-hosting, monitoring, dashboards, and fallback strategies to OpenAI.

These pieces work well independently, but they’re still isolated components. Now we need to connect them into a cohesive production system. This means automated deployments that don’t require manual SSH sessions. Proper deployment strategies that minimize risk. Rollback plans for when things inevitably go wrong. Runbooks so your team knows exactly what to do at 3AM. Operational patterns that prevent most of those 3 AM emergencies in the first place.

The Complete Architecture

Your production system needs several components working together seamlessly.

Start with a load balancer that distributes traffic across API instances. This load balancer handles SSL termination so your API services don’t worry about certificates. It kicks out unhealthy instances automatically so users never hit broken services. It implements rate limiting to prevent abuse and ensures no single user consumes all your resources. You can use AWS Application Load Balancer, nginx, or HAProxy depending on your infrastructure preferences and existing expertise.

Behind the load balancer sit multiple API service instances. Having multiple instances is critical for reliability. If one instance crashes because of a memory leak or unexpected bug, traffic continues flowing to the healthy instances. Each instance runs your BAML code from Part 2, handles HTTP requests from users, calls vLLM for inference, and falls back to OpenAI when needed. Running three to five instances provides good redundancy without excessive costs.

The API services connect to vLLM instances for model inference. These are GPU-backed containers running vLLM from Part 4. Each instance operates independently with its own GPU allocation. The load balancer distributes inference requests across them using algorithms like round-robin or least connections. This distribution prevents any single GPU from becoming a bottleneck while others sit idle.

When vLLM fails, requests automatically fall back to OpenAI. This safety net catches you when self-hosted infrastructure has problems. Part 4 established this fallback pattern using circuit breakers that detect failures and route traffic appropriately. The circuit breaker prevents cascading failures where every request times out waiting for a broken LLM instance.

Monitoring and logging sit beneath everything, collecting data about system health. Prometheus tracks metrics like request rates, error rates, latency percentiles, and GPU utilization. Grafana visualizes these metrics in dashboards that make system health obvious at a glance. Structured logs capture detailed information about what’s happening and why. This observability layer is absolutely essential for maintaining sanity in production. You can’t fix problems you can’t see.

Directory Organization

Organization matters more than you might expect when working on production systems. A well-structured project helps team members find what they need quickly. New engineers understand the system layout without extensive explanation. Here’s how to structure your production application.

The api directory contains your main application code, tests from Part 3, and BAML definitions from Part 2. Everything related to the API service lives here. The vllm directory holds vLLM-specific Docker configuration and startup scripts. Keeping this separate from the API makes it easier to update vLLM independently.

The infrastructure directory stores deployment configurations. Terraform files define your cloud resources declaratively. Kubernetes manifests describe how to deploy your services. Docker Compose production configurations show how pieces fit together. Having all infrastructure-as-code in one place makes environments reproducible and prevents configuration drift.

The monitoring directory contains Prometheus scraping configurations and Grafana dashboard definitions. Storing dashboards as code means you can version control them and restore them if someone accidentally deletes a dashboard.

The .github/workflows directory holds CI/CD pipeline definitions. These YAML files automate testing, building, and deploying your application. The scripts directory contains bash scripts for common operations like deploying, rolling back, and health checking. These scripts get used by CI/CD pipelines and can also be run manually when needed.

The docs directory stores operational documentation. The runbook explains how to handle common issues. The architecture document describes how components interact. Future team members will thank you for thorough documentation.

This structure separates concerns clearly while keeping related files together. Engineers can navigate the codebase efficiently, which becomes increasingly important as your team grows.

Implementing CI/CD

Manual deployments create problems that compound over time. Before implementing CI/CD, deployments required SSH access to production servers. You’d manually stop services, pull new Docker images, restart everything, and hope it worked correctly. Each engineer performed these steps slightly differently. One engineer might restart services in a different order than another. Someone might skip health checks in a rush. There’s no automated record of what changed or when it changed.

This approach fails at scale. With five deployments per week, manual processes consume significant engineering time. Each deployment takes 30-45 minutes including coordination, execution, and verification. That’s 2.5-3.75 hours per week just on deployment mechanics. Documentation for manual processes goes stale quickly because maintaining deployment docs requires discipline most teams lack. New team members take weeks to learn the deployment process through shadowing and trial by error. When something breaks during deployment, debugging is difficult because you don’t know exactly what changed or in what order.

CI/CD solves these problems through automation and consistency. Every code change triggers tests automatically. Tests must pass before any deployment proceeds. This catches bugs early before they reach production. Deployments happen exactly the same way every time, eliminating inconsistencies from human variation. You get a complete audit trail showing what changed, when it changed, and who approved it. Most importantly, bugs get caught in CI rather than production. The testing happens before code merges, preventing broken code from ever reaching users.

The investment in CI/CD pays dividends quickly. After spending a few days setting up pipelines, deployments become routine operations instead of anxiety-inducing events. You can safely deploy ten times per day if needed. New engineers deploy confidently during their first week because the process is automated and documented. When problems occur, the audit trail helps you identify exactly what changed. The consistency means fewer deployment-related incidents because you’re not introducing human error.

The Test Pipeline

This pipeline runs on every pull request and prevents broken code from reaching the main branch. Every PR must pass these tests before merging.

Create this file at .github/workflows/test.yml:

Unit tests run on every pull request because they’re fast (usually completing in under 2 minutes) and free. They validate your code’s logic without making external API calls. These tests catch type errors, logic bugs, and regression issues early in the development cycle. Catching bugs during code review is much cheaper than catching them in production.

Integration tests only run on the main branch because they’re slower and cost money through OpenAI API calls. Running them on every PR would waste money testing code that might not even merge. Running them on main provides a safety net after merge, confirming the integrated system still works correctly. If integration tests fail on main, you know immediately and can investigate before deployment.

⚠️ Never commit API keys directly to your repository. Use GitHub Secrets to store sensitive credentials. Navigate to Settings, then Secrets and Variables, then Actions to add secrets. GitHub encrypts these secrets and only makes them available to workflows. This prevents credentials from leaking into version control history, which would require key rotation and potential security incident response.

The coverage report uploads to Codecov, giving you visibility into which code paths your tests exercise. Aim for 80% coverage as a minimum. Lower coverage means untested code paths that might contain bugs. Higher coverage gives you confidence that your tests actually validate the code.

The Build Pipeline

This pipeline creates Docker images and pushes them to a container registry. Every successful merge to main triggers a build.

Create this file at .github/workflows/build.yml:

Image tagging follows a clear pattern that enables precise version tracking. The main branch gets the latest tag, which always points to the most recent build. Each commit receives a SHA tag like sha-abc123, allowing you to deploy or rollback to any specific commit. Git tags like v1.2.3 get semantic version tags following semver conventions. This tagging strategy enables easy rollbacks to specific versions when you need to quickly revert a problematic change.

The cache configuration dramatically improves build times by reusing layers from previous builds. The first build might take 10 minutes while Docker downloads base images, installs dependencies, and builds your application. Subsequent builds with caching take around 2 minutes because most layers are reused. The cache-from directive tells Docker where to find cached layers. The cache-to directive tells Docker where to store new cached layers. The mode=max option caches as many layers as possible for maximum speed.

Without caching, your CI/CD pipeline would waste significant time on every build. With caching, builds complete quickly enough that engineers don’t context switch while waiting. This keeps development velocity high.

The Deploy Pipeline

This pipeline automatically deploys built images to production after successful builds. It waits for the build to succeed, deploys to your infrastructure, verifies the deployment succeeded, runs smoke tests, and notifies your team.

Create this file at .github/workflows/deploy.yml:

The workflow_run trigger waits for the build to complete successfully before starting deployment. The if condition checks that the build actually succeeded before deploying. This prevents deploying broken images to production.

The AWS ECS deployment forces a new deployment of service, which pulls the latest image and performs a rolling update. The wait services-stable command blocks until ECS confirms the new tasks are healthy and receiving traffic. This prevents the pipeline from proceeding if the deployment fails.

Smoke tests verify basic functionality after deployment. These tests should be fast and check critical paths like authentication, database connectivity, and basic API responses. Smoke tests catch obvious problems immediately after deployment while you still have context about what changed.

The notification step uses if: always() to send Slack messages regardless of success or failure. Success notifications confirm deployments completed successfully. Failure notifications alert the team immediately when problems occur. Having this visibility prevents situations where deployments fail silently and nobody notices until users complain.

Pre-commit Hooks

Finding problems on your local machine saves time compared to waiting for CI pipelines to run. Pre-commit hooks run formatters, linters, and quick checks locally before you even create a commit.

Create this file at .pre-commit-config.yaml:

Install pre-commit with pip install pre-commit then run pre-commit install in your repository. Now every commit triggers these checks automatically before the commit completes. If checks fail, the commit gets blocked until you fix the issues. This catches formatting inconsistencies, linting violations, and type errors immediately while the code is fresh in your mind.

Black formats your Python code consistently. This eliminates arguments about code style in pull requests. Everyone’s code looks the same regardless of personal preferences. Flake8 checks for common Python programming errors and enforces PEP 8 style guidelines. It catches issues like unused imports, undefined variables, and overly complex functions. Mypy performs static type checking, catching type-related bugs before runtime. These tools work together to maintain code quality without you being in the loop.

Deployment Strategies

Stopping all services, deploying the new version, and bringing everything back up at once is not the best idea. Production systems need sophisticated deployment strategies that minimize risk and enable quick rollbacks. These strategies determine how new versions roll out to users.

Blue-Green Deployment

Blue-green deployment maintains two complete production environments running at the same time.

The blue environment runs your current version and serves all production traffic. The green environment sits idle or runs your new version with no traffic. When you deploy, you update green with the new version while blue continues handling users with no interruption. You run your full test suite against green, verifying everything works correctly. You can even have a small number of internal users test green manually. The load balancer still points at blue during all of this testing so production users never see the new version until you’re ready.

Once you’ve confirmed green is healthy and behaving correctly, you update the load balancer configuration to route traffic to green instead of blue. This switch happens in seconds at the load balancer level. Users never experience downtime because one environment is always serving traffic. The cutover is nearly instantaneous from a user perspective.

If anything goes wrong after the switch (maybe you found a bug that tests didn’t catch), you point the load balancer back to blue. The rollback takes seconds because blue is still running with the old version You haven’t destroyed anything.

After running green in production successfully for a few hours or days, you can update blue with the next version and repeat the process. This alternating pattern continues: deploy to the idle environment, test it thoroughly, switch traffic, repeat. Each environment alternates between serving production and waiting as the deployment target.

Here’s an example script implementing blue-green deployment 👇🏾

Blue-green deployment provides several advantages.

You get zero downtime deployments because one environment always serves traffic. You can test the new version thoroughly before users see it, catching issues in a production-like environment. Rollback happens instantly by switching the load balancer back.

The strategy is conceptually simple and easy to understand! Making it beginner friendly to teams new to advanced deployment strategies.

However, the main disadvantage is cost. You’re running double the infrastructure during deployments and even between deployments if you keep both environments running. For a system using three API instances and three vLLM instances, blue-green means running twelve instances instead of six. That doubles your compute costs.

Database migrations become complex because both environments must handle the same schema during the transition period. You can’t make breaking schema changes without coordination between versions. You also can’t test at real production scale until after the switch, so some issues only surface when all users hit the new version.

Use blue-green deployment for critical services where downtime is completely unacceptable.

Financial systems processing payments, healthcare applications managing patient data, and other high stakes services benefit from this approach. The instant rollback capability makes it worth the expense when every minute of downtime has significant business impact.

Canary Deployment

Canary deployment gradually shifts traffic to the new version by starting with a small percentage of users and increasing over time. Think of it like testing the water before jumping in. You start with 5% of users seeing the new version. If their experience looks good based on metrics, you increase to 25%, then 50%, then 100%. If anything looks wrong at any stage, you immediately route all traffic back to the old version.

The name comes from a coal mining practice back in the 20th century. Miners brought canary birds into mines because canaries are highly sensitive to toxic gases like carbon monoxide. If the canary becomes sick or dies, miners knew to evacuate immediately (thank goodness we have sensors now).

In deployment terminology, that small percentage of users acts as the canary. You monitor their experience closely to detect problems before affecting everyone else.

Here’s how the deployment process works. You deploy the new version alongside the old version. Initially, 95% of users continue hitting the old version while only 5% hit the new version. You monitor error rates, latency percentiles, and business metrics for both versions. If the canary group shows similar or better metrics than the control group, you increase the percentage. If the canary group shows worse metrics (higher error rates, slower response times, fewer successful transactions), you abort the rollout and route everyone back to the old version.

This gradual rollout continues in stages. At each stage, you wait a predetermined time (perhaps 10-15 minutes) while monitoring metrics. The waiting period lets issues surface before you expose more users. After confirming metrics look healthy, you proceed to the next stage. The final stage routes 100% of traffic to the new version and removes the old version.

Here’s an example bash script that implements a canary deployment 👇🏾

You can automate canary monitoring with Python.

Canary deployment provides several benefits. It carries lower risk than immediate full rollout because only a small percentage of users see the new version initially. It tests at real production scale with real user behavior patterns, catching issues that staging environments miss. Problems surface early when only 5-10% of users are affected rather than impacting everyone. You don’t need double infrastructure like blue-green deployment, keeping costs manageable.

The approach has some disadvantages. The complexity is higher than blue-green because you’re managing gradual rollout percentages and monitoring at each stage. Some users in the canary group might get a worse experience if the new version has issues, though this affects fewer people than a full rollout would. Good monitoring is absolutely essential because you can’t detect problems without it. And the deployment process takes longer than a simple cutover.

Overall the canary deployment provides a good balance between safety and practicality.

Rolling Deployment

Rolling deployment updates instances one at a time. With four API instances, you update the first instance while the other three continue serving traffic normally. Once the first instance is healthy and passing health checks, you update the second instance. You continue this pattern until all instances run the new version.

This strategy is the simplest to implement and understand. You start with instance one, update it with the new version, wait for health checks to pass, then move to instance two. The process continues sequentially through all instances. The load balancer automatically routes traffic away from instances during their update and back to them once they’re healthy.

You don’t need additional infrastructure. The deployment happens gradually, giving you opportunities to pause if problems appear. During the rollout, most of your capacity remains on the old version until near the end. If you notice issues during deployment, you can stop the rollout and investigate. Kubernetes actually implements rolling updates by default, making this approach accessible to teams already using Kubernetes.

The main issue with rolling deployments is version mixing. Both versions run simultaneously during deployment, which can cause problems. Imagine your new version expects a database field that doesn’t exist yet. Or maybe the new version changes an API response format that the old version can’t handle. These version compatibility issues require careful consideration.

Database migrations become tricky when different code versions expect different schemas. You need backward-compatible database changes.

The deployment is slower than blue-green cutover but faster than canary.

You can use rolling deployment as your default strategy when starting out. It’s simple to implement and understand. The gradual rollout provides some safety without requiring complex infrastructure or sophisticated monitoring. Many teams start here and slowly move up to canary deployment after establishing good monitoring.

Comparing and Choosing Strategies

Your requirements determine which strategy fits best. Let me walk through a decision framework.

If zero downtime is absolutely required, use blue-green or canary deployment. Rolling deployment might cause brief traffic disruptions during instance restarts, though this depends on your load balancer configuration.

If you need instant rollback capability, use blue-green deployment. The idle environment provides a perfect rollback target. Canary and rolling deployments require rolling back through your CI/CD pipeline, which takes longer.

If budget is limited and you want to avoid double infrastructure, use rolling or canary deployment. Blue-green requires maintaining two complete environments, doubling your infrastructure costs during and between deployments.

If you want to test at real scale before full rollout, use canary deployment. Blue-green tests in a production environment but not at production scale until after the switch. Rolling deployment provides no isolated testing.

If you want something simple to start with and plan to evolve later, use rolling deployment. It requires minimal infrastructure changes and no sophisticated monitoring initially.

Production Readiness Requirements

Before launching to production, your system must meet minimum requirements. These items are non-negotiable for production deployment. Skipping them can lead to major problems.

Your infrastructure needs at least two instances for redundancy. Single instances create a single point of failure. When that instance crashes due to an unexpected bug or infrastructure issue, your entire service goes down. Users get errors. Revenue stops. Metrics spike. Two instances mean one can fail while the other continues serving traffic. Three to five instances provide even better redundancy and handle traffic distribution more smoothly.

The load balancer must distribute traffic across these instances properly. Configure health check endpoints that the load balancer queries every few seconds. When an instance fails health checks, the load balancer stops routing traffic to it automatically. This prevents users from hitting broken instances. The load balancer should also handle SSL/TLS termination so your application doesn’t worry about certificate management.

Configure SSL/TLS certificates because serving unencrypted traffic in production is unacceptable in modern systems. Users expect HTTPS. Browsers show scary warnings for HTTP sites. Also, search engines penalize sites without HTTPS. Let’s Encrypt provides free certificates that auto-renew, eliminating cost as an excuse. AWS Certificate Manager also provides free certificates for AWS infrastructure. There’s no reason to skip this.

Configure DNS to give your service a domain name instead of requiring users to remember IP addresses. Configure A records pointing to your load balancer. Set up CNAME records for any subdomains. Test DNS resolution from multiple locations. DNS problems are frustrating to debug because of caching at multiple levels.

Security requires several configurations beyond just HTTPS. Remove all secrets from code completely! Even comments containing old API keys are dangerous. Use environment variables or a secrets management service like AWS Secrets Manager or HashiCorp Vault. These services encrypt secrets at rest and provide audit logs of secret access.

Configure rate limiting to prevent any single user from consuming all your resources. A misconfigured client making thousands of requests per second shouldn’t take down your entire service. Rate limiting protects you from both malicious attacks and accidental misuse.

Track GPU utilization to see whether you’re using GPU resources efficiently. If GPU utilization sits at 30%, you’re wasting money. If it constantly hits 100%, you need more GPUs. Request rates show traffic patterns and help you understand usage trends. Error rates indicate problems before users complain. A sudden spike in errors signals something broke. Track these metrics continuously and set up basic alerts 📈.

Create alerts for critical failure conditions. Alert when vLLM crashes completely so you know immediately. Alert when error rates exceed 5% so you can investigate before too many users are affected. Alert when latency exceeds acceptable thresholds so you can scale before users complain about slowness. Start with these basic alerts and add more as you learn what matters for your system.

Automate your deployment process with a CI/CD pipeline. Manual deployments don’t scale and introduce human error. The pipeline should run tests, build images, deploy to production, and verify success automatically. Document each step even though it’s automated. Documentation helps when the pipeline breaks or when you need to run steps manually during debugging.

Document the rollback procedure in detail and actually test it before you need it in an emergency. Rolling back under pressure is stressful. Having clear documentation and practiced procedures reduces that stress. Test rollbacks in a staging environment regularly to verify they work and to keep the team familiar with the process.

Implement post-deployment smoke tests that verify basic functionality after each deployment. These tests should be fast and check critical functionality. Can users authenticate? Can the service connect to the database? Do the main API endpoints respond correctly? Smoke tests catch obvious deployment problems immediately while you still have context about what changed.

Within the first month after launch, add more sophistication to your system. Configure auto-scaling so your infrastructure grows automatically with demand. This prevents manual scaling operations at 3 PM when traffic suddenly spikes. Auto-scaling based on CPU utilization, request rates, or custom metrics keeps your service responsive without constant manual intervention.

Set up proper network security with firewalls and security groups. Restrict access to internal services. Only expose what needs to be public. Close unnecessary ports. Use private subnets for database and internal services. These security layers provide defense in depth against attacks.

Track latency at different percentiles to understand user experience. The median (p50) shows typical performance. The 95th percentile (p95) shows what slower users experience. The 99th percentile (p99) shows the worst-case experience. A service with good p50 latency but terrible p99 latency frustrates some users even if most users are happy ☝🏾.

Monitor costs so you don’t get surprised by your cloud bill at month end. Set up billing alerts that notify you when spending exceeds expected amounts. Tag resources appropriately so you can attribute costs to specific services or teams. Review the bill monthly to identify optimization opportunities.

Create dashboards for all key metrics so anyone can see system health at a glance. The dashboard should show request rates, error rates, latency percentiles, GPU utilization, and fallback rates. Grafana dashboards work well for this. Share dashboard links with your team so everyone knows where to look when investigating issues.

Add circuit breakers for external dependencies like OpenAI so cascading failures don’t take down your entire system. When OpenAI is slow or returning errors, the circuit breaker opens and stops sending requests. This prevents your service from waiting indefinitely for responses that will never come.

Verify the OpenAI fallback actually works under production conditions by deliberately testing failure scenarios. Turn off vLLM in staging and verify traffic routes to OpenAI correctly. Check that the circuit breaker opens appropriately. Test that recovery works when vLLM comes back online.

Achieve 80% or higher test coverage to catch regression bugs early. Lower coverage means untested code paths that might contain bugs. Higher coverage gives you confidence that your tests actually validate the code. Focus coverage on critical paths first, then expand to edge cases.

Run load tests at expected production scale to find performance bottlenecks before real users hit them. For example, simulate 7 requests per second (your expected production load) against your staging environment. Monitor response times, error rates, and resource utilization. Fix any problems you discover before launching.

Operations need documentation for when things inevitably break. Write a runbook covering common issues and their solutions. The runbook should include symptoms, diagnostic steps, and remediation steps for each issue. Update the runbook after every incident with lessons learned.

Eventually you might add features that provide extra value but aren’t critical for launch. Multi-region deployment improves latency for global users and provides geographic redundancy. Deploying in US East, US West, and Europe regions means users in each geography get fast response times.

Distributed tracing helps debug complex issues that span multiple services. Tracing shows how a single request flows through your system, identifying where time is spent and where errors occur. This becomes invaluable as your system grows more complex.

Don’t try to implement everything on day one. That leads to scope creep, burnout, and delays to launch. Focus on the essentials first. Add sophistication as you learn what your system actually needs in production. Real usage patterns reveal requirements better than upfront speculation.

Creating Effective Runbooks

A runbook documents how to operate your system when things break. It’s like a detailed instruction manual for fixing problems.

Start with quick links to essential resources at the top of the runbook. Include the Grafana dashboard URL so you can immediately see what metrics look wrong. Include the log aggregation interface (Kibana, CloudWatch, or whatever you use) so you can search for error messages. Include the on-call schedule so you know who to contact for help.

Document common issues with clear symptoms, diagnostic steps, and remediation steps. Each issue should follow a standard template that makes information easy to find under pressure.

Example 1: vLLM Outages

For complete vLLM outages, the symptoms include 100% error rate on the monitoring dashboard, all API requests failing with timeout errors, and constant alert notifications flooding your phone. The diagnostic process involves checking the vLLM health endpoint at /health to see if it responds. Run nvidia-smi on the GPU host to verify GPU availability and look for GPU errors. Check that the vLLM container is actually running with docker ps or kubectl get pods. Look at vLLM logs for error messages that explain what went wrong.

The remediation steps start with restarting vLLM using your container orchestration tool. For Docker Compose, run docker-compose restart vllm. For Kubernetes, run kubectl rollout restart deployment/vllm. Verify GPUs are available and accessible after restart. Check that the circuit breaker routes traffic to OpenAI while vLLM is down so users aren’t completely blocked. If the OpenAI fallback also fails, escalate to senior engineers or management because you have a bigger problem than just vLLM.

Example 2: Slow Performance

For slow performance where the system is working but users complain about speed, symptoms include p99 latency exceeding 3 seconds on dashboards, user complaints in support channels, and requests timing out occasionally. Diagnose by checking GPU utilization with nvidia-smi. If GPU utilization is constantly at 100%, you don’t have enough GPU capacity. Examine queue depth in metrics to see how many requests are waiting. If queue depth keeps growing, requests are arriving faster than you can process them. Look for memory leaks where memory usage creeps up over time on host metrics.

Fix high GPU utilization by scaling horizontally with additional instances. For Kubernetes, run kubectl scale deployment/vllm --replicas=5 to add more capacity. For AWS ECS, update the service’s desired count. Address large queue depths the same way because more instances provide more processing capacity. If you suspect a memory leak (memory usage growing steadily without returning), restart vLLM to clear memory temporarily, then file a bug report and monitor for recurrence.

Example 3: OOM Errors

Out of memory errors cause vLLM to crash with “CUDA out of memory” messages in logs. This usually happens when requests are too large for available GPU memory. The immediate fix involves lowering the max-model-len configuration parameter in your vLLM startup script. Reduce gpu-memory-utilization to leave more buffer for large requests. Restart vLLM to apply configuration changes. If problems persist, consider upgrading to a larger GPU instance type like AWS p4d instances that offer more memory.

Example 4: Unreliable Service

High OpenAI costs indicate your fallback is being used too frequently, which means vLLM is unreliable. Check the fallback rate metric. Investigate why vLLM is unstable by looking at instance health, error logs, and recent deployments. Examine the circuit breaker state to see if it opened due to detected problems. Fix the underlying vLLM issues causing the high fallback rate. Reset the circuit breaker if needed after fixing the root cause.

Looking Forward: Next Steps

After implementing everything in this series, you have a complete production system. You’ve built type-safe LLM applications using BAML. The test coverage from Part 3 catches regression bugs before they reach users. The self-hosted vLLM infrastructure from Part 4 reduces costs when the economics make sense. Monitoring and alerts tell you what’s happening. CI/CD pipelines deploy consistently without manual steps. Runbooks guide your team through incidents. You can deploy confidently because the infrastructure and processes support production traffic reliably.

At 5 million requests per month, the economics clearly favor self-hosting. Using OpenAI API exclusively costs $13,750 per month or $165,000 annually. Self-hosting costs $727 per month for infrastructure ($8,724 annually) plus roughly $4,000-5,000 monthly in engineering time for maintenance and incidents. Net savings remain around $150,000 per year even after accounting for engineering overhead.

However, self-hosting doesn’t make sense at all volumes. Below 200,000 requests monthly, just use the API. The engineering overhead isn’t justified by the cost savings. Between 200,000 and 500,000 requests, the API is probably still the right choice depending on your team’s capabilities and available engineering time. Above 500,000 requests monthly, self-hosting starts making financial sense. Above 5 million requests, self-hosting is definitely worth the operational complexity!

As your system grows, you’ll need more advanced infrastructure. Deploy vLLM in multiple AWS regions (US East, US West, Europe) and route users to the closest one. This cuts latency for global users but requires more complex traffic routing and data synchronization.

Kubernetes handles more complex orchestration than Docker Compose. You get horizontal pod autoscalers, native rolling updates, and granular resource management. The tradeoff is operational complexity. Learning and maintaining Kubernetes takes time, but it’s worth it once your system reaches a certain scale.

Model fine-tuning becomes relevant when open source models aren’t quite good enough for your specific use case. Fine-tuning requires ML expertise to design training approaches, GPU time for training runs, and data preparation infrastructure. However, fine-tuned models can dramatically improve quality for domain-specific applications. The investment pays off when model quality directly impacts user satisfaction or business metrics.

The LLM space moves fast. New models drop frequently. vLLM releases updates every few weeks. Stay current by following the vLLM GitHub for releases, checking HuggingFace for new models, and joining communities like r/LocalLLaMA where people share what actually works in production.

Upgrade thoughtfully. Test new vLLM versions in staging every 2-3 months before rolling them to production. Try new models when the quality improvement justifies the migration risk. Upgrade infrastructure when the benefits clearly outweigh the cost and complexity of migration

Conclusion

This series covered everything you need to run LLMs in production. Type-safe code with BAML. Tests that catch bugs before users do. Monitoring that tells you what’s happening. CI/CD that deploys consistently. Deployment strategies that limit risk.

Building production systems is hard work. It requires infrastructure knowledge, software skills, and operational discipline. But there’s real satisfaction in building something that works reliably at scale.

Happy building 🔨

Quick math: at $0.00275 per request (gpt-4o), that’s $5.5K/month by Q2, scaling to $13.75K/month by Q3. $165K annually on API calls alone.

My manager asked: “Can we run this ourselves?”

My first thought: “Self-hosting sounds expensive and complicated.”

Turns out I was wrong about the expensive part. At our scale, self-hosting would cost about $700/month. That’s over $156K in annual savings at our projected scale.

When Self-Hosting Makes Sense (And When It Doesn’t)

Let’s do the math first.

The Cost Breakdown

OpenAI API costs (GPT-4o):

Assuming ~700 input tokens + 100 output tokens per request.

Cost per request calculation:

• Input 700 tokens × $0.0025/1K = $0.00175

• Output: 100 tokens × $0.010/1K = $0.001

• Total: ~$0.00275 per request

📝 Note: Your token counts will vary based on your specific use case.

• Cost per request: ~$0.00275

• 100K requests/month: $275

• 500K requests/month: $1,375

• 1M requests/month: $2,750

• 5M requests/month: $13,750

Self-hosted vLLM costs (1-Year Reserved Instance):

Assuming 24/7 usage.

• AWS g5.xlarge (1-Year RI): $0.634/hour × 720 hours = $456.48/month

• storage: $50/month

• Network transfer: $20/month

• Infrastructure Total: ~$527/month

📝 Note: On-Demand pricing is $1.006/hour ($724/month), so the 1-year RI saves you $267/month compared to on-demand.

Add maybe $200/month for engineering time (monitoring, maintenance). You’re looking at $727/month total.

Break-even point is ~264K requests/month.

Below that? Stick with the API. Above that? Self-hosting saves you money.

When to Self-Host

High Request Volume

With self-hosting, the costs don’t scale linearly like API costs do. With OpenAI, if you double your requests, you double your bill. But with self-hosting, your infrastructure costs stay roughly the same whether you’re processing 300K requests or 3M requests per month (until you need to scale horizontally).

Let’s walk through what this actually looks like at different volumes.

At 500K requests/month

OpenAI API Cost

• 500,000 requests × $0.00275 per request = $1,375/month

• Annual cost: $1,375 × 12 = $16,500/year

Self-Hosted Cost

• Infrastructure: $527/month (this stays constant)

• Engineering time: $200/month (this stays constant)

• Total: $727/month

• Annual cost: $727 × 12 = $8,724/year

Annual savings: $16,500 - $8,724 = $7,776/year

Not bad, but is it worth the operational complexity? Maybe, maybe not.

At 2M requests/month (projected Q2 volume)

OpenAI API Cost

• 2,000,000 requests × $0.00275 per request = $5,500/month

• Annual cost: $5,500 × 12 = $66,000/year

Self-Hosted Cost

• Infrastructure: $527/month (still the same!)

• Engineering time: $200/month (still the same!)

• Total: $727/month

• Annual cost: $727 × 12 = $8,724/year

Annual savings: $66,000 - $8,724 = $57,276/year

Now we’re talking. That’s a junior engineer’s salary you’re saving!

At 5M requests/month (projected Q3 volume)

OpenAI API Cost

• 5,000,000 requests × $0.00275 per request = $13,750/month

• Annual cost: $13,750 × 12 = $165,000/year

Self-Hosted Cost

• Infrastructure: $527/month (yep, still the same)

• Engineering time: $200/month (still the same)

• Total: $727/month

• Annual cost: $727 × 12 = $8,724/year

Annual savings: $165,000 - $8,724 = $156,276/year

That’s not a typo. You’re saving $156K per year at that scale.

Why Does this Matter?

Notice how our self-hosted costs stayed at $727/month across all three scenarios. That’s important! A single g5.xlarge instance can handle way more than 5M requests per month (with proper batching and optimization). Your costs are fixed until you need to scale horizontally.

Meanwhile, OpenAI’s costs scale perfectly with your usage. Great when you’re small, painful when you’re big.

This is why I say self-hosting becomes economical around 500K+ requests monthly. Below that, the savings aren’t dramatic enough to justify the operational overhead. Above that, the savings accelerate fast.

At our current trajectory (200K requests now, 5M by Q3), we’d be leaving $156K on the table by staying with the API. That pays for a lot of engineering time.

Rule of thumb: If your monthly request volume varies by more than 50%, you’ll want to either use on-demand instances (which costs more but scales with usage) or calculate your average monthly volume across the year.

Data Privacy Requirements and Custom Models

Let’s say you work at a healthcare company building an AI assistant that helps doctors write patient notes. Your training data includes thousands of real patient records with medical jargon, treatment patterns, and clinical language specific to your hospital system.

Unfortunately, you can’t upload that data to OpenAI’s servers to fine-tune GPT-4. Why? Because it’s protected health information under HIPAA. Sending patient data to a third party (even encrypted) violates compliance requirements. Your legal team would have a meltdown.

So what do you do? You need to fine-tune a model on your proprietary data, but that data legally cannot leave your infrastructure. The only solution is to download an open source model (like Llama-2 or Mistral), fine-tune it on your own servers with your own data, and run it entirely within your infrastructure. Self-hosting isn’t optional here. It’s the only way to do this legally.

Scenarios Where This Matters

Healthcare: You need a model that understands your hospital’s specific terminology, drug protocols, and documentation style. Training data includes patient records that must stay within your HIPAA-compliant infrastructure.

Financial Services: You’re building a model to detect fraudulent transactions based on your bank’s historical transaction patterns. That data includes customer financial information that can’t be sent to external APIs under PCI DSS regulations.

Legal Firms: You want a model trained on your firm’s past case documents, legal strategies, and client communications. Attorney-client privilege means this data absolutely cannot leave your servers.

Manufacturing: You’re creating a model to predict equipment failures based on proprietary sensor data from your factory floor. Your competitors would love to see that data. It’s a trade secret that stays internal.

Specialized Models Not Available via API: Sometimes the model you need just doesn’t exist as an API. In these cases, you’ll often find an open source base model and fine-tune it yourself. But once you’ve done that work, there’s no API to call. The model only exists on your infrastructure. You have to self-host it.

Latency-Sensitive Applications

When you call the OpenAI API, your request travels across the internet to OpenAI’s servers, waits in the queue, gets processed, and then the response travels back to you. The entire journey (what we call the “roundtrip”) takes 1-2 seconds on average.

With local inference using vLLM, your request stays inside your own infrastructure. No network hops, no external queues. The model responds in 200-500ms (that’s 0.2 to 0.5 seconds).

That might not sound like a huge difference, but think about typing in a chatbot. If every response takes 2 seconds, the conversation feels sluggish. Users notice. They get impatient. They leave.

But when responses come back in 300ms? The conversation flows naturally. It feels responsive. Users stay engaged.

This matters most for:

• Real-time chat applications where users expect instant responses

• Code autocomplete (imagine waiting 2 seconds every time you hit tab)

• Interactive demos or customer support bots

• Any feature where latency directly impacts user experience

If you’re building a batch processing system that analyzes resumes overnight, that extra 1-2 seconds per request doesn’t matter. But if you’re building a feature where users are actively waiting for responses? Every millisecond counts.

Latency test: Before committing to self-hosting for latency reasons, measure your current p95 and p99 latencies. Then ask: what would a 3-5x improvement actually change for your users? If the answer is “not much,” stick with the API.

When NOT to Self-Host

Low Volume

At 200K requests/month, you’re spending $550/month with the OpenAI API. Self-hosting costs $727/month. You’d be losing $177/month by self-hosting.

The break-even point is around 264K requests/month. Below that, the API is literally cheaper.

No Upfront Investment:

With an API, you create an account, get an API key, and start making requests in under 10 minutes.

With self-hosting, you need to provision a GPU instance, download models, write Docker configs, set up monitoring. That’s days of engineering work before your first request. And you’re paying for that GPU the entire time.

Pay-as-you-go:

With the API, the provider handles model updates, infrastructure scaling, security patches, and monitoring. While your team focuses on building features.

With self-hosting, your team handles GPU monitoring, service restarts, out-of-memory errors, version updates, security patches, and on-call rotations when things break.

Start with the API then switch to self-hosting when you’ve hit your break-even point.

Early Prototyping

When you’re building a new feature, you don’t know if it’ll work and you need to test fast.

With the OpenAI API, you’re testing your idea in 10 minutes:

import openai

response = openai.ChatCompletion.create(
    model=”gpt-4”,
    messages=[{”role”: “user”, “content”: “Your prompt here”}]
)

Done. No infrastructure setup, no model downloads, no GPU provisioning. Just an API key is needed.

Need Cutting-Edge Performance

Open source models lag 6-12 months behind frontier models. When GPT-5 launched, the best open source models are still catching up to GPT-4’s capabilities from 2023. If you need the absolute best performance, stick with APIs.

What is vLLM?

Before vLLM, people used tools like Ollama for local LLM development. Ollama is great for tinkering on your laptop. Easy setup, good for learning. But Ollama has limitations for production: memory inefficient, static batching, throughput plateaus under load.

vLLM rethought the architecture from scratch. It’s a high-performance inference engine built by researchers at UC Berkeley. Used in production by Databricks, Anthropic, and others.

The Core Problem vLLM Solves

Most LLM servers waste 60-80% of available GPU memory through pre-allocation.

Traditional servers look at the maximum possible memory a request could need (let’s say 8GB) and reserve the entire 8GB upfront, even though most requests only use 2-3GB. The other 5-6GB sits there unused and locked. With a 24GB GPU, you can only handle 3 concurrent requests because you’ve pre-allocated everything.

vLLM allocates memory dynamically in small blocks (similar to virtual memory in operating systems). It gives each request exactly what it needs, when it needs it. That same 24GB GPU now handles 12 concurrent requests instead of 3.

You can now serve 2-4x more requests on the same hardware.

vLLM’s Key Innovation

PagedAttention

Inspired by virtual memory in operating systems. Instead of allocating huge chunks of memory upfront, it breaks attention computation into small pages and blocks. This lets vLLM allocate memory dynamically as requests actually need it.

Continuous Batching

Traditional batching waits for a batch to fill up (say, 10 requests), processes them all together, then waits for the slowest request to finish before starting the next batch. Think of it like standard college admissions, where the next cycle doesn’t start until every request from the current cycle has been processed.

Continuous batching processes multiple requests together, but as soon as one request finishes generating its response, vLLM immediately adds a new request to fill that spot (think of it like rolling admissions). The batch size stays constant, but requests flow in and out continuously.

The result is higher throughput and lower latency because you’re not wasting GPU cycles waiting for every request to finish.

Optimized CUDA Kernels

When you run PyTorch on a GPU, it uses generic operations that work for any neural network. These operations are flexible, but flexibility comes at a cost. Remember that saying: jack of all trades, master of none.

vLLM’s team wrote custom GPU code (CUDA kernels) specifically optimized for running LLM inference. Instead of using PyTorch’s general-purpose matrix multiplications, they wrote code that does exactly what LLMs need and nothing more.

As a result, those custom kernels run 2-3x faster than PyTorch’s generic operations.

What does this mean for you? Let’s say you’re running a 7B model on a g5.xlarge instance. With generic PyTorch operations, you might process 20 tokens per second. With vLLM’s optimized kernels, you’re processing 40-60 tokens per second on the same hardware.

What’s nice is that you don’t need to understand CUDA or GPU programming to benefit from this. vLLM handles all the optimization automatically. You just call the API and get faster inference.

Getting Started with vLLM

Let’s get vLLM running locally first. Then we’ll connect it to the BAML code from Part 2.

Installation

System Requirements

• Linux (Ubuntu recommended)

• Python 3.8+

• NVIDIA GPU with CUDA support (or CPU for testing, but it’ll be slow)

• At least 16GB RAM

📝 Note: You can run this on a laptop with a decent GPU, but for production you’ll want a cloud GPU instance.

# Install vLLM
pip install vllm

# Verify installation
python -c “import vllm; print(vllm.__version__)”

Running Your First Model

Start with something small for testing. I’ll use Mistral-7B-Instruct, which is a popular open source model that works well for most tasks.

You can find models on the HuggingFace model hub at huggingface.co/models. HuggingFace has become the de facto repository for open source LLMs. HuggingFace is kind of like GitHub but for AI models.

When you see a model name like mistralai/Mistral-7B-Instruct-v0.3, here’s what the naming convention means:

• mistralai is the organization that created the model

• Mistral-7B-Instruct is the name of the model (7B refers to the number of parameters)

• v0.3 is the version number (newer versions usually fix bugs or improve performance)

One thing to keep in mind: a 7B parameter model takes about 14GB of disk space. Make sure you have enough storage before downloading. The first time you run vLLM with a model, it’ll download automatically, which can take 5-10 minutes depending on your internet connection.

Understanding the Code

• LLM() loads the model into GPU memory

• SamplingParams() controls generation (temperature, max length, etc.)

• .generate() runs inference

Running as an API Server

vLLM can run as an OpenAI-compatible API server. This is a game changer.

If you’re currently using OpenAI’s API, your code probably looks something like this:

With vLLM running as an API server, you change one line:

That’s it. Your application code stays the same. You’re not rewriting your entire codebase. You’re not learning a new API. You just point your existing OpenAI client to a different URL.

This means you can:

• Test vLLM locally before committing to infrastructure

• Switch between OpenAI and self-hosted models by changing one environment variable

• Use the same codebase for both your local dev environment (vLLM) and production (OpenAI or vLLM)

• Gradually migrate from OpenAI to self-hosted without a big rewrite

The OpenAI SDK has become the de facto standard for LLM APIs. By being compatible with it, vLLM lets you experiment with self-hosting without throwing away all your existing code.

Connecting BAML to vLLM

This is where all that abstraction work from Part 2 pays off. Remember when we set up BAML to use different LLM providers without changing our application logic? This is exactly why we did that.

We’re going to swap out OpenAI for vLLM by changing exactly one thing: the client configuration. Your application code, your prompts, your type definitions - all of that stays exactly the same. Zero changes to your business logic!

Update Your BAML Client

Open your BAML configuration file (usually baml_src/clients.baml) and add a new client for vLLM.

A few things to note here.

The provider is still openai because vLLM implements the OpenAI API spec. This is what makes the whole thing work - vLLM speaks the same language as OpenAI’s API.

The base_url points to your local vLLM server. If you’re running vLLM on a different machine or port, change this URL accordingly.

The api_key can be anything. vLLM doesn’t check API keys when running locally. I just put not-needed-for-local to make it explicit.

The model parameter needs to match exactly what you passed to vLLM when you started the server. If you started vLLM with --model mistralai/Mistral-7B-Instruct-v0.3, use that exact string here.

Update Your Function to Use the New Client

Now find the function you want to switch over. In Part 2, we created a resume analyzer. Here’s how you change it.

Everything else remains the same except for that one line.

Test It Works

Now let’s verify that vLLM is actually being used and returning valid results by creating a test file.

Run the script:

python test_vllm.py

You should see output like:

Name: John Doe
Experience: 5
Skills: [’Python’, ‘JavaScript’, ‘Docker’]

If you get an error like Connection refused or Failed to connect, check that your vLLM server is actually running on port 8000. Go back to the terminal where you started vLLM and make sure it says “Running on http://0.0.0.0:8000”

This is the Payoff

Let’s take a step back and appreciate what just happened. You switched from a $0.00275/request API (GPT-4) to a self-hosted model that costs $0.000088/request (assuming our $727/month infrastructure costs and 2M requests). That’s a 97% cost reduction.

And how much code did you change? One line.

This is why we built that abstraction in Part 2. Without BAML, you’d be rewriting your entire application - updating every API call, changing response parsing logic, handling different error formats, rewriting tests. Instead, you changed client GPT4 to client LocalVLLM.

You can now:

• Test locally with vLLM during development

• Switch to OpenAI for staging/testing if you want higher quality

• Use different models for different functions (cheap models for simple tasks, expensive models for complex ones)

• Implement fallback logic (try vLLM first, fall back to OpenAI if it fails)

All without touching your application code. Just configuration changes.

That’s the power of abstraction!

Production Setup

Running vLLM locally on your laptop is great for testing and development. But when you’re ready to deploy to production and serve real user traffic, you need a more robust setup. Production environments require three things that local testing doesn’t: reproducibility (it should work the same way on any machine), proper configuration (tuned for your specific workload), and monitoring (so you know when things break).

Let’s set up vLLM for production using Docker.

Docker Setup

I know some people groan when they hear “Docker,” but hear me out. Docker solves real problems that you’ll encounter in production.

Without Docker, you’re installing vLLM directly on your production server. That means you’re managing Python versions, CUDA drivers, system dependencies, and all the little quirks of that specific machine. If your server crashes and you need to spin up a replacement, you’re manually recreating that environment. If it works on your machine but fails in production, you’re spending hours debugging environment differences.

Docker packages everything your application needs into a container. The same container that works on your laptop will work on AWS, Google Cloud, your company’s on-prem servers, or anywhere else. You deploy once, and it runs everywhere identically.

Here’s what we’re going to build: a Docker container that includes Python, vLLM, and all dependencies. We’ll use Docker Compose to manage the container and configure GPU access. Then we’ll add environment variables so you can adjust settings without rebuilding the container.

Create Your Dockerfile

Let me walk through what each section does.

We’re starting from nvidia/cuda:12.1.0-runtime-ubuntu22.04, which is NVIDIA’s official image that includes CUDA drivers. CUDA is what lets vLLM talk to your GPU. Without this base image, vLLM can’t access the GPU and will fall back to CPU inference (which is painfully slow for LLMs).

The first RUN command installs Python 3.10 and pip. We’re also running rm -rf /var/lib/apt/lists/* at the end to delete the package manager’s cache, which reduces the final image size by a few hundred megabytes.

The second RUN command installs vLLM via pip. This will take a while the first time you build the image (vLLM and its dependencies are several gigabytes), but Docker caches this layer so subsequent builds are faster.

We’re copying a startup script (which we’ll create next) into the container. The chmod +x command makes the script executable.

EXPOSE 8000 documents that this container listens on port 8000. This doesn’t actually open the port (Docker Compose handles that), but it’s good documentation for anyone reading your Dockerfile.

Finally, CMD tells Docker what command to run when the container starts. We’re running our startup script.

Create Your Startup Script

Create a file called start_vllm.sh in the same directory.

#!/bin/bash
python3 -m vllm.entrypoints.openai.api_server \
    --model ${MODEL_NAME:-mistralai/Mistral-7B-Instruct-v0.3} \
    --host 0.0.0.0 \
    --port 8000 \
    --gpu-memory-utilization ${GPU_MEMORY:-0.9} \
    --max-model-len ${MAX_LENGTH:-4096}

This script starts vLLM with configuration passed via environment variables. The ${MODEL_NAME:-mistralai/Mistral-7B-Instruct-v0.3} syntax means “use the MODEL_NAME environment variable if it exists, otherwise default to mistralai/Mistral-7B-Instruct-v0.3.” This lets you change the model without rebuilding the Docker image.

The --host 0.0.0.0 flag tells vLLM to listen on all network interfaces. Inside a Docker container, this is necessary for external connections to reach vLLM. If you used 127.0.0.1 (localhost), only processes inside the container could connect, which defeats the purpose.

Create Your Docker Compose File

Create a file called docker-compose.yml.

Docker Compose lets you define multiple services (containers) and how they interact. We’re defining two services: vllm (the inference server) and app (your application that calls vLLM).

The ports section maps port 8000 inside the container to port 8000 on your host machine. This lets you access vLLM at http://localhost:8000 from outside the container.

Note 📝: You don’t have to use port 8000. In fact, you have over 65,000 network ports to use on your machine!

The deploy.resources.reservations.devices section gives the container access to your GPU. Without this, Docker won’t let the container see the GPU, and vLLM will fail to start.

The volumes section mounts a local directory (./models) into the container at /root/.cache/huggingface. This is where HuggingFace stores downloaded models. By mounting a local directory, the model persists between container restarts. Without this, you’d re-download 14GB every time you restart the container.

The restart: unless-stopped policy automatically restarts the container if it crashes. In production, you want this. If vLLM hits an out-of-memory error and crashes, the container will automatically restart and try again.

The app service is your application code. The depends_on: vllm line tells Docker Compose to start vLLM before starting your app. The VLLM_URL environment variable tells your app where to find vLLM. Since both services are in the same Docker Compose network, your app can access vLLM at http://vllm:8000 (using the service name as the hostname).

Build and Run Everything

docker-compose up -d

The -d flag runs containers in detached mode (in the background). Without it, Docker Compose would print logs to your terminal and you’d need to keep that terminal window open.

On the first run, Docker will:

1. Download the base NVIDIA CUDA image (~4GB)

2. Build your vLLM image (~6-12GB depending on dependencies)

3. Start the containers

4. Download the Mistral-7B model (~14GB)

This takes 10-30 minutes depending on your internet connection. Subsequent starts are much faster because everything is cached.

You can check if vLLM is running:

curl http://localhost:8000/health

If you get a 200 response, vLLM is up and ready to serve requests.

Configuration Options

Now that vLLM is running in Docker, let’s talk about the configuration flags we’re using and what they actually do.

GPU Memory Utilization

The --gpu-memory-utilization flag controls how much of your GPU’s memory vLLM will use. It’s a float between 0.0 and 1.0, representing a percentage.

On a g5.xlarge with 24GB of GPU memory, setting this to 0.9 means vLLM will use up to 21.6GB (24GB × 0.9). The remaining 2.4GB is reserved for CUDA operations, PyTorch overhead, and a safety buffer to prevent out-of-memory errors.

You might think “why not use 100% of the GPU?” Because GPUs need some memory for internal operations. If you set this to 1.0, you’ll see random OOM crashes when vLLM tries to allocate just a bit more memory than available. Setting it to 0.9 gives you a buffer.

If you’re getting OOM errors even at 0.9, lower this to 0.85 or 0.8. You’ll handle fewer concurrent requests, but the server will be more stable.

If your GPU is barely utilized (you’re seeing 40-50% memory usage in nvidia-smi), you can try increasing this to 0.95. More memory means more concurrent requests, which means higher throughput.

Start with 0.9. Only adjust if you’re seeing problems.

Max Model Length

The --max-model-len flag sets the maximum number of tokens (input + output) that a single request can use.

Mistral-7B technically supports up to 8192 tokens (about 6000 words). But longer contexts use more memory. If you set --max-model-len 8192, each request reserves enough memory for 8192 tokens, even if most requests only use 500 tokens. You’re wasting memory.

Think about your actual use case. If you’re analyzing resumes (like our example), most resumes are 500-1000 words. That’s roughly 700-1400 tokens. Setting --max-model-len 2048 gives you plenty of headroom and doesn’t waste memory on unused capacity.

If you set this too low, requests with longer inputs will get truncated or rejected. If you set it too high, you’re using memory inefficiently and handling fewer concurrent requests.

Here’s a rule of thumb based on your use case:

💡 For resume analysis or short document processing, use 2048. Most business documents are under 1000 words.

💡 For customer support chat where context from previous messages matters, use 4096. You need room for conversation history.

💡 For long document summarization where you’re feeding in entire articles, use 8192. But be aware this significantly reduces your concurrent request capacity.

You can see the tradeoff: on a 24GB GPU with max-model-len=2048, you might handle 12 concurrent requests. With max-model-len=8192, you might only handle 4 concurrent requests. Choose based on your actual needs, not the maximum possible value.

Tensor Parallelism

The --tensor-parallel-size flag splits a model across multiple GPUs. This is only relevant if you’re running models that don’t fit on a single GPU.

For 7B and 13B models on a 24GB or 40GB GPU, you don’t need this. The model fits comfortably on one GPU.

For 70B models, you’d need something like --tensor-parallel-size 4 to split the model across 4 GPUs. Each GPU holds 1/4 of the model weights.

Unless you’re running very large models (70B+), ignore this flag entirely. Setting it when you don’t need it just adds complexity and slightly reduces performance due to inter-GPU communication overhead.

Configuration Examples for Different Use Cases

Let me show you hypothetical configurations for different scenarios.

Resume Analysis (our example)

• Model: Mistral-7B

• GPU: 24GB A10

• Max Model Length: 2048

• GPU Memory Utilization: 0.9

• Expected Throughput: 20-50 requests/sec

This configuration handles typical resumes lengths (1-2 pages) efficiently. You’re not wasting memory on unnecessarily long context windows.

Customer Support Chatbot

• Model: Llama-13B (better quality for complex queries)

• GPU: 40GB A100

• Max Model Length: 4096 (need room for conversation history)

• GPU Memory Utilization: 0.85 (being conservative with a larger model)

• Expected Throughput: 12-15 requests/sec

Customer support needs higher quality responses than resume parsing, so we’re using a 13B model. The longer context window accommodates conversation history (10-15 messages back and forth).

Document Summarization

• Model: Mistral-7B

• GPU: 24GB A10

• Max Model Length: 8192 (processing long articles)

• GPU Memory Utilization: 0.8 (lower because of long contexts)

• Expected Throughput: 10-15 requests/sec

Long contexts eat memory. We’re being conservative with GPU memory utilization and accepting lower throughput. If you need to summarize entire research papers or long-form articles, you need the full 8192 token context.

Notice the pattern? Smaller models, shorter contexts, and higher memory utilization give you better throughput. Larger models, longer contexts, and conservative memory settings give you better quality and stability. There’s always a tradeoff.

Configure based on what your application actually needs, not what’s theoretically possible.

Model Selection

Choosing the right model is probably the most important decision you’ll make when self-hosting. Get this wrong and you’re either wasting money on GPU power you don’t need, or delivering poor quality results that frustrate your users.

The good news is that for most applications, you don’t need the biggest, fanciest model. Let me walk you through your options and help you figure out what makes sense for your use case.

Starting with 7B Models

If this is your first time self-hosting an LLM, start with a 7B parameter model like Mistral-7B-Instruct or Gemma-7B. These models are the sweet spot for most applications.

A 7B model means the model has 7 billion parameters. Parameters are the weights and connections in the neural network that the model learned during training. More parameters generally means better quality, but also means more GPU memory, slower inference, and higher costs.

7B models fit comfortably on a 24GB GPU like the g5.xlarge we’ve been using. You’ll need at least 16GB of VRAM to run them, but 24GB gives you room for batching multiple requests and handling longer contexts. With a 7B model on a g5.xlarge, you’re looking at processing 50-100 tokens per second. That’s fast enough that users won’t notice any lag.

The quality is good for most tasks. Tasks like resume parsing, email classification, simple Q&A, content generation, basic code comments, and data extraction. If you’re building something like “analyze this customer support ticket and categorize it,” a 7B model will nail it.

About 60-70% of LLM applications work perfectly fine with 7B models.

When To Move Up to 13B Models

Sometimes you need better reasoning or more nuanced understanding. That’s when you consider a 13B model like Llama-2-13B.

The jump from 7B to 13B isn’t just about size. The quality difference is noticeable. A 13B model can handle more complex instructions, follow multi-step reasoning better, and produce more coherent long-form content.

This is what changes when you move to 13B.

You need more GPU memory. A 13B model technically works on a 24GB GPU but you’ll be pushing the limits. For production, you want a 40GB GPU like the A100. That gives you breathing room for batching and longer contexts without hitting out-of-memory errors.

Your speed drops to about 30-50 tokens per second. That’s still pretty fast for most applications, but you’re processing roughly half as many requests per second compared to a 7B model.

When should you use a 13B model? Complex customer support where you need to understand nuance and tone. Code generation beyond simple snippets. Content that requires maintaining context over multiple paragraphs. Analysis tasks where you need the model to make inferences beyond what’s explicitly stated.

The 70B Question

Llama-3-70B and similar models are the best open source models available. They’re approaching GPT-3.5 quality. However, they’re rarely worth it for self-hosting.

A 70B model requires multiple GPUs. You’re looking at 4 GPUs with 40GB each, using tensor parallelism to split the model across them. That’s roughly $10,000-$23,000 per month in cloud infrastructure costs if you’re using AWS p4d instances or Google Cloud A2 instances. On-premises is even worse upfront (4x A100 40GB GPUs cost $40,000-60,000 just for the hardware, plus servers, cooling, and power). You’re processing maybe 10-20 tokens per second.

At that point, just use GPT-4 via the API. You get better quality, faster inference, no infrastructure management, and probably lower costs unless you’re doing truly massive volume (we’re talking 50+ million requests per month to break even on those infrastructure costs).

I’ve only seen 70B models make sense for self-hosting in two scenarios. First, you have extremely strict data privacy requirements and absolutely cannot send data to OpenAI, even encrypted. Second, you’re processing tens of millions of requests per month and the cost savings of self-hosting actually pencil out even with multiple A100 GPUs.

For everyone else? If you need 70B-level quality, use the API. Self-hosting is about saving money at scale, not about having the biggest model.

A Critical Note About Instruction-Tuned Models

Always use instruction-tuned models for production. These are models with “-Instruct” or “-Chat” in the name, like “Mistral-7B-Instruct-v0.3” or “Llama-2-7B-Chat.”

Base models (like “Mistral-7B-v0.1” without the “Instruct” suffix) are not trained to follow instructions. They’re trained to predict the next token, which means they’ll continue your prompt rather than answer your question. If you prompt a base model with “Write a professional email to schedule a meeting,” it might just generate “Write a professional email to schedule a doctor’s appointment. Write a professional email to...” and keep going.

Instruction-tuned models are fine-tuned specifically to understand and follow instructions. They know when to stop. They format their responses appropriately. They understand conversational context.

💡 Base models are for researchers who want to fine-tune their own models. Instruction-tuned models are for everyone actually building applications. Use the instruction-tuned versions.

Benchmarking Performance

Once you’ve got vLLM running, you need to know if it’s actually performing well.

I’ll show you how to benchmark your vLLM setup properly so you know what kind of performance to expect.

Create a Simple Benchmark Script

Here’s a Python script that sends 1000 concurrent requests to your vLLM server and measures throughput and latency.

This script does something important: it tests concurrent requests, not sequential ones. In production, you’ll have multiple users hitting your API simultaneously. Sequential testing (sending one request, waiting for a response, sending another request) doesn’t reflect real-world usage.

The script creates 100 tasks and uses asyncio.gather() to send them all at once. This simulates 100 users hitting your endpoint simultaneously. That’s what actually stresses vLLM and shows you real performance characteristics.

Save this as benchmark.py and run it.

python benchmark.py

You’ll see output like:

Processed 100 requests in 2.47 seconds
Throughput: 40.49 requests/second
Average latency: 24.7 ms/request

Troubleshooting Performance

Performance is Worse than Expected

If you’re seeing less than ideal numbers, something is up.

First, check your GPU utilization. Run nvidia-smi in another terminal while your benchmark is running. You should see GPU utilization at 80-95%. If it’s lower, you’re not batching effectively or your requests are too small to keep the GPU busy. If it’s at 100% constantly, you’re GPU-bound and might benefit from a larger GPU or smaller model.

Second, check your memory usage. Also visible in nvidia-smi. If you’re close to your memory limit, you might be hitting out-of-memory errors intermittently. Try reducing --max-model-len to free up memory. Each reduction in max length lets you handle more concurrent requests.

Third, if you have headroom on memory (you’re only using 60-70% of available VRAM), try increasing --gpu-memory-utilization from 0.9 to 0.95. More available memory means more concurrent requests in the batch.

Fourth, consider whether your GPU is actually the bottleneck or if it’s something else.

What if You Need Better Performance

You have a few options. The easiest is to scale horizontally. Run multiple vLLM instances behind a load balancer. Each instance handles its own batch of requests. This works well because LLM inference is embarrassingly parallel; one request doesn’t depend on another.

You can also try a smaller model. Mistral-7B-Instruct might perform almost as well as a 13B model for your specific use case. Run quality tests, but don’t assume you need the bigger model.

Or upgrade your GPU. A g5.2xlarge has the same GPU as a g5.xlarge (A10) but with more system RAM and CPU, which can help with pre/post-processing. For significantly better performance, consider the A100 (40GB or 80GB versions).

The important thing is to benchmark early and often. Don’t wait until production to discover your setup can’t handle the load. Run benchmarks during development. Run them after every configuration change. Know your numbers!

Monitoring and Maintenance

Running vLLM in production isn’t a “set it and forget it” situation. Production systems fail. GPUs run out of memory. Network connections drop. Models get stuck in weird states. You need to know when these things happen and why they’re happening.

Basic Health Checks

The simplest form of monitoring is a health check endpoint. vLLM exposes one at /health that returns a 200 status code if everything is working.

You can test it manually:

curl http://localhost:8000/health

If you get back a 200 status code, vLLM is running and accepting requests. If you get a connection error or timeout, something is wrong.

Automated Health Checks in Python

Manual curl commands are fine for testing, but in production you want automated health checks that run continuously. Here’s a Python script that checks vLLM’s health and alerts you if something goes wrong.

This script checks vLLM’s health every 30 seconds. If the health check fails, you’d want to send yourself an alert (via email, Slack, PagerDuty, or whatever alerting system your team uses). I’m just printing to the console here, but in production you’d integrate with your monitoring infrastructure.

The timeout=5 parameter is important. Without a timeout, the request could hang forever if vLLM is stuck. Five seconds is reasonable for a health check - if vLLM can’t respond in five seconds, something is seriously wrong.

Key Metrics to Watch

Health checks tell you if vLLM is running, but they don’t tell you if it’s running well. You need to monitor actual performance metrics to catch problems before they become outages.

GPU Utilization

This is the single most important metric to watch. GPU utilization tells you what percentage of your GPU’s compute capacity is being used.

You can check GPU utilization with nvidia-smi.

nvidia-smi

This displays a table showing GPU memory usage and utilization percentage. You’re looking for the line that says something like “GPU-Util: 87%”.

During active request processing, your GPU utilization should be 80-95%. If it’s consistently lower (say, 40-60%), you’re not batching requests effectively or your requests are too small to keep the GPU busy. You’re paying for GPU compute that you’re not using.

If your GPU utilization is at 100% constantly, you’re GPU-bound. Every request is waiting for GPU time. This isn’t necessarily bad if your latency is acceptable, but it means you can’t handle any more load without adding more GPUs.

Here’s what I typically see in production:

During normal business hours (moderate load), GPU utilization bounces between 70-90%. The GPU is busy but not maxed out. There’s headroom for traffic spikes.

During traffic spikes (like when a batch job runs and sends 1000 requests at once), GPU utilization hits 95-100% temporarily. That’s fine for short bursts.

During off-hours (minimal load), GPU utilization drops to 10-30%. The GPU is mostly idle, which is expected when there’s no traffic.

If your GPU utilization is consistently low even during peak hours, something is misconfigured. Check your batching settings and make sure requests are actually reaching vLLM.

Memory Usage

GPU memory usage tells you how much of your GPU’s VRAM is being used. This is different from utilization (which is compute capacity).

With nvidia-smi, you’ll see something like “15234MiB / 24576MiB” which means 15GB used out of 24GB available.

You want your memory usage to be high but not at the limit. If you set --gpu-memory-utilization 0.9 and you’re seeing 21-22GB used on a 24GB GPU, that’s perfect. You’re using the memory you allocated.

If you’re consistently hitting the memory limit (23.9GB used on a 24GB GPU with no headroom), you’re at risk of out-of-memory errors. Any request that needs slightly more memory than usual will crash vLLM. Lower your --max-model-len or --gpu-memory-utilization setting to create a buffer.

If you’re only using 12GB on a 24GB GPU, you’re being too conservative. You could handle more concurrent requests or longer contexts. Try increasing --gpu-memory-utilization from 0.9 to 0.95.

The worst scenario is seeing memory usage gradually climb over time. Let’s say it starts at 15GB, then after an hour it’s 18GB, then after two hours it’s 21GB, and eventually it hits 24GB and crashes. That’s a memory leak. Restart vLLM and check if there’s a known bug in your version. Memory leaks are rare in vLLM but not impossible.

Request Queue Depth

vLLM maintains an internal queue of requests waiting to be processed. When your GPU is busy processing a batch, new incoming requests sit in this queue.

vLLM logs the queue depth periodically. You’ll see log messages like:

INFO: Queue depth: 3
INFO: Queue depth: 7
INFO: Queue depth: 12

A small queue (1-5 requests) is normal and healthy. It means requests are arriving slightly faster than you can process them, but you’re keeping up.

A growing queue is a red flag. If you see the queue depth climbing from 5 to 10 to 20 to 50 over the course of a few minutes, you’re not keeping up with incoming traffic. Requests are piling up faster than you can process them.

What happens when the queue gets too large? Latency explodes. If there are 50 requests in the queue and you process 5 requests per second, the 50th request waits 10 seconds just to start processing. Users don’t wait 10 seconds - they timeout and retry, which adds even more requests to the queue. Death spiral 😵‍💫.

If you see your queue growing consistently, you need more capacity. Either scale horizontally (add more vLLM instances behind a load balancer) or vertically (move to a larger GPU that can process requests faster).

Latency Percentiles

Average latency is useful, but it doesn’t tell the whole story. You need to track latency percentiles to understand the full distribution.

Here’s what each percentile means.

p50 (median) is the latency that half your requests experience. If p50 is 200ms, half your requests complete in under 200ms.

p95 means 95% of requests complete under this threshold. If p95 is 500ms, 5% of requests take longer than 500ms.

p99 means 99% of requests complete under this threshold. If p99 is 1000ms, 1% of requests take longer than 1 second.

Why track percentiles instead of just average? Because outliers matter. You could have an average latency of 300ms but a p99 latency of 5 seconds. That means 1 in 100 users is having a terrible experience, even though your average looks great.

If your p99 is over 2 seconds, users are definitely noticing. Some are probably timing out. You need to investigate why the slowest 1% of requests are so slow. Possible causes: occasional long inputs that hit your max-model-len limit, memory pressure causing occasional slowdowns, or queue backups during traffic spikes.

To track percentiles in Python, you can use a library like prometheus_client or just log request latencies and calculate percentiles periodically.

Common Failure Modes

Here are some production failures you could come across in production.

Out-of-memory Errors

This is the most common failure mode. You’ll see an error message like:

RuntimeError: CUDA out of memory. Tried to allocate 2.00 GiB 
(GPU 0; 23.70 GiB total capacity; 21.80 GiB already allocated)

This means a request tried to allocate more memory than available. The GPU had 21.8GB already in use, tried to allocate another 2GB, but only had 1.9GB free.

There’s usually three reasons why this happens.

First, you set --max-model-len too high. You told vLLM that requests can use up to 8192 tokens, so it reserves enough memory to handle 8192-token requests in each batch slot. But your GPU doesn’t have enough memory for multiple 8192-token requests at once. Solution: lower --max-model-len based on your actual use case.

Second, you set --gpu-memory-utilization too high. You told vLLM to use 95% of GPU memory, which doesn’t leave enough buffer for CUDA operations and PyTorch overhead. Solution: lower it to 0.85 or 0.9.

Third, you got unlucky. Most requests are short, but occasionally someone sends a 5000-token input. That one request needs way more memory than usual and causes an OOM. Solution: set a maximum input length in your application layer before requests reach vLLM. Reject requests over your limit with a clear error message.

When vLLM hits an OOM error, it usually crashes completely. Your Docker container’s restart: unless-stopped policy will automatically restart it, but you’ve lost all in-flight requests. Users get errors. Not great.

The best solution is to prevent OOM errors in the first place by tuning your memory settings conservatively.

Slow Startup and Cold Starts

The first request to vLLM after startup takes 30-60 seconds to complete. This is because vLLM needs to load the model weights into GPU memory, initialize CUDA kernels, and warm up the inference pipeline.

Subsequent requests are fast, but that first request is painfully slow. If you’re running health checks, that first health check might fail because it times out before vLLM finishes initializing.

Solution: send a warmup request after vLLM starts up. In your startup script or Docker entrypoint, add a curl command that waits for vLLM to be ready.

This script starts vLLM in the background, waits for the health endpoint to respond, sends a warmup request to load everything into memory, and then declares vLLM ready.

Performance Degrading Over Time

Sometimes vLLM starts out fast but gets slower over hours or days. Your p50 latency starts at 200ms, but after 12 hours it’s 400ms. After 24 hours it’s 600ms.

This usually indicates memory fragmentation or a memory leak. As vLLM processes millions of requests, the GPU memory gets fragmented (lots of small free chunks instead of large contiguous blocks). This makes it harder to allocate memory for new requests and slows things down.

Solution: restart vLLM periodically.

You can automate this with a cron job or your orchestration platform (Kubernetes can do rolling restarts automatically). Just make sure you’re doing rolling restarts (restart one instance at a time) so you don’t lose all capacity at once.

The GPU is There But vLLM Can’t See It

Sometimes Docker doesn’t properly pass GPU access to the container. vLLM starts up but immediately errors out with:

RuntimeError: No CUDA GPUs are available

This happens when:

• You forgot the deploy.resources.reservations.devices section in docker-compose.yml

• NVIDIA Docker runtime isn’t installed on the host

• The NVIDIA drivers on the host are outdated or broken

Check that NVIDIA Docker is working on your host:

docker run --rm --gpus all nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smi

If this command fails, Docker can’t access your GPU. You need to install the NVIDIA Container Toolkit. If it works, the problem is with your docker-compose configuration.

These are some issues I’ve hit in production. Monitor for them proactively instead of discovering them when users are complaining. Set up alerts for OOM errors, high queue depth, and latency spikes. Check your GPU utilization daily. Restart instances during off-hours to prevent performance degradation.

Production isn’t glamorous, but it’s where your self-hosted LLM either proves its worth or becomes a maintenance nightmare. Put in the monitoring work upfront and you’ll sleep better at night.

Fallback Strategies

Never depend on 100% uptime of any single service. It doesn’t matter how reliable that service is. It will go down at the worst possible time.

Multi-Tier Fallback Architecture

The idea behind multi-tier fallback is simple: have multiple ways to get an answer, ordered by preference. Try the cheapest, fastest option first. If that fails, try the next option. Keep going until you get a result or run out of options.

Here’s an example.

Tier 1 is your self-hosted vLLM instance. This is your primary service. It’s fast, cheap, and handles 95% of requests under normal conditions. When everything is working correctly, all your traffic goes here. You’re paying $727/month for infrastructure and processing requests at $0.000088 each.

Tier 2 is OpenAI’s API. This is your fallback when vLLM is down, overloaded, or timing out. It’s more expensive ($0.00275 per request for GPT-4o), but it’s incredibly reliable. Under normal conditions, only 4-5% of your requests should hit this tier. That’s okay. You’re paying a small premium for reliability insurance.

Tier 3 is cached responses. For common queries that don’t change often, you can return cached results from previous requests. This is your emergency fallback. If both vLLM and OpenAI are unreachable (which is extremely rare), you can at least return something for common cases. Maybe 1% of requests use this tier.

Implementing Multi-Tier Fallback

Here’s an example on how to code this pattern.

The function first tries vLLM with a 5-second timeout. If vLLM responds within 5 seconds, great. You return the result and you’re done. That’s the happy path that happens 95% of the time.

If vLLM takes longer than 5 seconds (timeout) or throws an exception (crash, out of memory, network error), you immediately fall back to OpenAI. You’re not making the user wait while vLLM struggles. Five seconds is already too long from a user experience perspective.

If OpenAI also fails (which is rare but possible during major outages), you check your cache for a similar request. Maybe someone analyzed this exact resume yesterday and you cached the result. Better to return a slightly stale result than nothing at all.

The cache is your emergency parachute. It won’t work for unique inputs, but for common queries (like “analyze this standard software engineer resume”), you might have a cached response. It’s better than showing users an error page.

The Circuit Breaker Pattern

Fallback tiers handle individual request failures, but what happens when vLLM is completely dead? You don’t want to keep trying it for every request, waiting 5 seconds for a timeout each time, then falling back to OpenAI. That’s 5 seconds of unnecessary delay per request.

This is where the circuit breaker pattern comes in. It’s like the circuit breaker in your house’s electrical panel. When there’s a problem (too much current), the breaker trips and stops trying to send electricity until you manually reset it. In software, a circuit breaker does the same thing: it stops trying a failing service and automatically recovers when the service is healthy again.

Here’s how it works:

When vLLM is healthy, the circuit breaker is “closed” (electricity flows). Requests go to vLLM normally.

When vLLM starts failing repeatedly (say, 5 failures in a row), the circuit breaker “opens” (electricity stops). Now all requests immediately skip vLLM and go straight to OpenAI. You’re not wasting time trying a service you know is down.

After a timeout period (say, 60 seconds), the circuit breaker goes into “half-open” state. It tries one request to vLLM to see if it’s recovered. If that request succeeds, the circuit closes and traffic resumes normally. If it fails, the circuit stays open for another 60 seconds.

This prevents cascading failures. Without a circuit breaker, you’d keep hammering vLLM with requests while it’s struggling, making the problem worse. With a circuit breaker, you give vLLM time to recover while your users get fast responses from OpenAI.

Implementing a Circuit Breaker

The circuit breaker prevents you from wasting resources on a service that’s clearly broken. If vLLM fails 5 times in a row, something is seriously wrong. Stop trying. Give it a minute to recover. Your users get fast responses from OpenAI instead of waiting 5 seconds per request for vLLM to timeout.

Monitoring Fallback Usage

Once you’ve implemented fallbacks, you need to monitor which tier is serving requests. If 50% of your traffic is hitting OpenAI instead of vLLM, something is wrong with your vLLM setup. You’re paying way more than you budgeted for.

Track fallback metrics using Prometheus (or whatever monitoring system you use).

Set Up Alerts for Abnormal Fallback Rates

If more than 10% of requests are using OpenAI fallback, something is wrong with vLLM. Maybe it’s hitting OOM errors, maybe your GPU is overloaded, maybe there’s a network issue. Investigate immediately before your costs spiral.

If any requests hit cache fallback, notify someone immediately. Cache fallback means both vLLM and OpenAI failed. That’s a critical outage. Users are getting degraded service (stale cached responses or errors).

I monitor these metrics in Grafana with alert rules.

The first alert fires if more than 10% of requests hit OpenAI for more than 5 minutes. That’s abnormal and indicates a problem with vLLM.

The second alert fires immediately if any request hits cache in the last 5 minutes. That’s critical because it means both vLLM and OpenAI failed.

Understanding Your Fallback Costs

Let’s do some math to understand the cost impact of fallbacks.

Assume you’re processing 1 million requests per month. Under normal conditions:

• 950,000 requests go to vLLM at $0.000088 each = $83.60

• 50,000 requests fall back to OpenAI at $0.00275 each = $137.50

• Total: $221.10 per month

That’s still way cheaper than running everything through OpenAI (which would cost $2,750 for 1 million requests).

But what if vLLM is down 10% of the time?

• 900,000 requests go to vLLM at $0.000088 each = $79.20

• 100,000 requests fall back to OpenAI at $0.00275 each = $275

• Total: $354.20 per month

Your costs went up by 60%, but you’re still operational. Without fallbacks, you’d have 10% downtime and angry users. The extra $133 per month is cheap insurance.

This is why monitoring fallback rates matters. If your fallback rate is creeping up, you’re paying more than you expected. Fix vLLM before costs become a problem.

Final Thoughts on Resilience

Building resilient systems isn’t glamorous. It’s extra work upfront to handle edge cases that might never happen. But when they do happen, you’ll be glad you invested the time.

Your users don’t care if you’re saving 97% on infrastructure costs by self-hosting. They care that your application works when they need it. Multi-tier fallbacks, circuit breakers, and monitoring give you both: cost savings during normal operation and reliability during failures.

That’s the whole point of self-hosting with proper fallbacks. You get the economics of running your own infrastructure with the reliability of managed services.

Wrapping Up

We’ve covered a lot of ground in this four-part series. If you’ve made it this far, you now know more about self-hosting LLMs than most engineers working with AI today. That’s not hyperbole. Most teams jump straight to OpenAI’s API without ever questioning whether there’s a better way.

Let me recap what we’ve built and why it matters.

The Economics Make sense at Scale

In Part 1, we did the math. Self-hosting becomes economical above roughly 264,000 requests per month. Below that threshold, stick with the API. Above it, you’re potentially wasting thousands of dollars per month by not self-hosting.

However, the exact breakeven point doesn’t matter as much as understanding your usage patterns. If you’re processing 200,000 requests per month now but growing 20% month over month, you’ll hit the breakeven point in two months. Start planning your self-hosting infrastructure now, not when you’re already bleeding money on API costs.

vLLM is Genuinely Impressive Technology

The efficiency gains from vLLM aren’t marketing hype. PagedAttention, continuous batching, and optimized CUDA kernels combine to give you 2-4x better throughput compared to naive PyTorch implementations.

The technology works. Trust the benchmarks, but verify them with your own workload.

Production is Where Theory Meets Reality

Everything we covered in the Monitoring and Maintenance section about Docker, monitoring, and fallbacks might seem like overkill when you’re just testing vLLM locally. It’s not. Production is where all your assumptions get stress-tested by real users with real deadlines.

Build resilience from day one. Set up monitoring before you launch. Implement fallbacks before you need them.

BAML Abstractions Are the Secret Weapon

The ability to swap between self-hosted vLLM and OpenAI’s API with just a configuration change is not just convenient. It’s strategically important.

Without BAML, migrating from OpenAI to vLLM would mean rewriting your entire application. Changing every API call, updating response parsing logic, handling different error formats, rewriting tests. That’s weeks of engineering work and substantial risk.

With BAML, it’s a one-line change: client GPT4 becomes client LocalVLLM. Everything else stays the same. Your prompts don’t change. Your type definitions don’t change. Your application logic doesn’t change.

This gives you flexibility. You can test vLLM during development, use OpenAI for staging, and run different models for different functions in production. You can implement fallback logic where cheap requests go to vLLM and complex requests go to GPT-4. You can experiment without committing to a complete rewrite.

That abstraction work we did in Part 2 pays for itself many times over. It’s the difference between being locked into a vendor and having options.

Key Principles Worth Remembering

Throughout this series, a few themes kept coming up. These are worth internalizing:

Always do the cost analysis for your specific situation. The numbers I provided are examples, not gospel. Your infrastructure costs might be different. Your request volume might follow different patterns. Your quality requirements might justify different model choices. Plug in your actual numbers and make decisions based on your reality, not my examples.

Build fallback strategies before you need them. Every production system fails eventually. Your self-hosted infrastructure will go down. OpenAI’s API will have an outage. Your circuit breaker will trip. When these things happen, having fallbacks means your users never notice. Not having fallbacks means you’re scrambling to fix things while your application is broken.

Monitor everything you care about. You can’t fix what you can’t see. Track your GPU utilization, memory usage, request queue depth, and latency percentiles. Set up alerts before problems become outages. I’ve spent too many hours debugging issues that would have been obvious if I’d just looked at the right metrics earlier.

Start simple and graduate to complexity. Begin with OpenAI’s API. When you hit the economic threshold, move to self-hosting. Start with a 7B model before jumping to 70B. Run one vLLM instance before implementing multi-instance load balancing. Every layer of complexity you add creates new failure modes. Add complexity only when you have a specific problem to solve.

Where to Go From Here

If you’re just starting your LLM journey, stick with OpenAI’s API for now. Focus on building your application and understanding your users’ needs. Once you’re processing hundreds of thousands of requests per month, come back to this series and start planning your self-hosting migration.

If you’re already at scale and paying too much for API calls, start with a cost analysis. Figure out your breakeven point. Then work through Part 2 to set up BAML abstractions. Finally, use the Monitoring and Maintenance section as a deployment guide. Take it step by step.

If you’ve been running self-hosted LLMs but struggling with reliability, focus on monitoring and fallback strategies. Resilience engineering isn’t glamorous, but it’s what separates hobby projects from production systems.

My Final Thoughts

Writing this series taught me that self-hosting LLMs is no longer a bleeding-edge experiment for companies with unlimited engineering resources. The tooling has matured. vLLM is production-ready. The economics make sense at reasonable scale.

But it’s also not a silver bullet. Self-hosting introduces operational complexity. You’re trading API costs for infrastructure management, monitoring, and on-call responsibilities. That tradeoff makes sense for some teams and not for others.

The key is knowing when you’re on which side of that line. If you’re processing 50,000 requests per month, stay with the API. If you’re processing 5 million requests per month and haven’t considered self-hosting, you’re probably overpaying by tens of thousands of dollars.

Do the math. Build the abstractions. Plan for failure. Monitor everything. And don’t be afraid to experiment. The worst that happens is you learn something new about your system.

Good luck with your self-hosting journey.

Happy savings 💰

The function called OpenAI’s API. Which costs money 🤑. And returns different outputs each time.

My first attempt? I just called the API directly in my tests. Wrote five test cases, ran pytest, watched my terminal fill with API calls. Tests took 45 seconds. Then I checked my OpenAI dashboard: $0.50 gone.

That adds up fast. Run tests 20 times a day during development? $10/day. $50/week. $200/month. That’s just me. Scale to a team of five? $1,000/month on test runs.

There’s a better way.

What We’re Fixing

Quick recap from Part 1: one of our problems was that we couldn’t test LLM code without literally calling the API. That made our tests expensive, slow, and flaky.

Now that we’ve got structured prompts and type safety from Part 2, we can build a proper testing strategy. Today we’ll:

• Learn how to test LLM code without calling APIs (mocking and test doubles).

• Understand the difference between unit tests and integration tests (and when you need each).

• Test that your prompts truly work (evaluation suites).

• Set up test infrastructure that doesn’t suck.

Most of what I’m covering works whether you use BAML or not. These are software engineering principles that apply to any LLM application. BAML just makes some of it easier.

Why Testing LLMs is Different

Traditional software is deterministic. Same input, (hopefully) same output, every time.

def add(a, b):
    return a + b

# This passes every single time
assert add(2, 3) == 5

LLMs? Non-deterministic by design. Same input, different output. Even with temperature=0, you get variations.

And it gets weirder: with traditional software, you test your code. With LLMs, you’re also testing your prompts. Is the bug in your code or your prompt? Sometimes you can’t tell.

I’ve seen tests that pass 8 times out of 10. The other 2? The LLM formats the response slightly differently and the parsing breaks. That’s not a code bug. That’s a prompt engineering problem disguised as a testing problem.

So we need a testing strategy that handles both: testing our code works correctly AND testing our prompts produce useful outputs.

Universal Testing Principles

I’ll show you the core testing principles first, using plain Python. This works whether you’re using BAML, LangChain, raw OpenAI calls, or building your own abstractions.

Don’t Test the LLM, Test Your Code

You don’t need to test the LLM itself. OpenAI already tested GPT-4. You need to test YOUR code that wraps the LLM.

Here’s our original resume analyzer (simplified) 👇🏾

The problem: every time you test this, you hit the OpenAI API. Expensive and slow.

The solution: dependency injection. Make the LLM client something you can swap out.

Now in your tests, pass in a fake client.

No API calls. No cost. Runs in milliseconds.

This is the fundamental pattern: separate the “call the LLM” part from the “process the result” part.

Build Different Mocks for Different Scenarios

One mock isn’t enough. You need mocks for different scenarios.

Now you can test all the edge cases.

This builds confidence that your error handling actually works, without spending money on API calls.

📝 Note: Testing error handling is especially important for LLM applications because LLMs can return unexpected formats. If your error handling isn’t tested, you won’t discover the bugs until production (been there, learned that lesson the hard way).

Property-based Testing

Sometimes you don’t care about the exact output. You care about properties of the output.

This type of testing is about invariants: things that should ALWAYS be true. It’s less brittle than testing exact outputs, which is perfect for LLM applications where exact outputs vary.

Integration Tests vs Unit Tests

Mocks are great. But at some point, you need to test that your code works with the real API.

Unit tests: Fast, cheap, test your code in isolation using mocks. Run these on every commit.

Integration tests: Slow, expensive, test that your code actually works with real APIs. Run these less frequently (once a day or before releases).

Here’s how I’d structure these tests 👇🏾

Running Tests Locally

During normal development, you want fast feedback. Run just the unit tests:

pytest

This runs in seconds, costs nothing, and catches most bugs. You’ll run this dozens of times a day as you write code.

When you’re about to commit or merge a feature, verify the integration works.

RUN_INTEGRATION_TESTS=1 pytest

This hits the OpenAI API. Takes a minute or two. With 5-10 integration tests, you’re looking at $0.10-0.50 per run depending on your prompt complexity and which model you’re using. You run this maybe once or twice before pushing your code.

You can also run just the integration tests.

RUN_INTEGRATION_TESTS=1 pytest -m integration

This is useful when you’ve changed your prompt and want to verify it works with the real API, but don’t want to wait for all the unit tests to run again.

Setting up CI/CD

In CI/CD, run unit tests on every PR, but integration tests only on the main branch or before releases.

This way you’re not burning money on API calls during development, but you still have confidence that the integration works.

Evaluation Suites for Prompt Quality

You need to test your prompts separately from your code.

A prompt evaluation suite is a collection of real examples with known correct outputs. Like a test dataset, but for your prompts.

Test your prompt against all of them.

When you change your prompt, run this evaluation suite. If your pass rate goes down, you made the prompt worse. If it goes up, you made it better. Iterate on prompts scientifically instead of guessing.

💡 Pro Tip: Start your evaluation suite small (5-10 cases) and add to it over time. Don’t try to create a comprehensive eval suite on day one.

Regression Testing

Every time you find a bug in production, add it to your test suite.

These tests document what went wrong and ensure it doesn’t happen again. Plus, they’re documentation for your team about quirks in the LLM’s behavior.

Setting Up Test Infrastructure

Directory Structure

my-llm-project/
├── baml_src/
│   └── resume.baml
├── src/
│   └── resume_analyzer.py
├── tests/
│   ├── unit/
│   │   ├── test_resume_analyzer.py
│   │   └── mocks.py
│   ├── integration/
│   │   └── test_resume_analyzer_integration.py
│   └── evaluation/
│       ├── test_prompt_quality.py
│       └── evaluation_suite.py
├── pytest.ini
└── requirements-dev.txt

pytest.ini

Running Different Test Suites

# Default: only unit tests
pytest

# Include integration tests
pytest -m “unit or integration”

# Run full evaluation suite
pytest -m evaluation

# Run everything
pytest -m “”

In CI/CD (GitHub Actions)

This setup gives you fast feedback during development (unit tests) while still having confidence that integration works (integration tests), without burning through your API budget. (No more wasting $5-10 every time someone pushes a typo fix 🔥💰).

What Good Looks Like

My rule of thumb for a well-tested LLM application is:

• 80%+ unit test coverage - Test your code logic with mocks.

• 5-10 integration tests - Verify real API calls work for critical paths.

• 10-20 evaluation cases - Test prompt quality against real examples.

• Regression tests for every production bug - Ensure bugs stay fixed.

Unit tests run in seconds and cost nothing. Integration tests run in minutes and cost a few cents. Evaluation tests cost a bit more but give you confidence in your prompts.

The mock-based approach is both cheaper AND more thorough than running all tests against the real API.

Your Homework

Pick that function you converted to BAML in Part 2 (or any LLM function you have). Write:

1. One happy path unit test - Mock a perfect response, verify your code handles it.

2. One error handling unit test - Mock a bad response, verify your error handling works.

3. One property-based unit test - Test that certain properties are always true.

4. One integration test - Call the API with a real example.

If you don’t have a function yet, use this example.

Write those four tests. I mean it! The muscle memory matters.

⭐️ Bonus: Set up pytest markers so you can run your unit tests separately from integration tests. This will save you time and money as your test suite grows.

Next post: Part 4 on self-hosting LLMs with vLLM. Because once you’ve got type-safe prompts (Part 2) and comprehensive tests (Part 3), you might start thinking “what if I didn’t have to pay OpenAI for every single request?”

See you there 🥾

It crashed.

My downstream processing code expected years_of_experience to be an integer. The LLM returned “5-7 years” for someone with a range on their resume. My code that did if years_of_experience > 3 threw a TypeError and took down the whole pipeline.

This wasn’t an LLM problem. My code had no way to enforce what the LLM should return. I was hoping it would always give me an integer. When it didn’t, I had no safety net.

That search for a better approach led me to BAML.

The Two Problems We’re Fixing

From Part 1, we’re tackling two specific issues 👇🏾.

No type safety. We don’t know what our LLM functions return. Is it a dict? What keys? What types? Your IDE can’t help you, and you won’t find out until runtime.

String manipulation chaos. Our prompts are scattered f-strings buried in functions. We can’t version them, can’t reuse them easily, can’t A/B test them methodically.

I’m showing you BAML because it solves these problems elegantly. But the principles (type safety, separation of concerns, treating prompts as first-class citizens) matter more than any specific tool. If BAML disappeared tomorrow, you could apply these same principles with other tools.

What BAML Does

BAML comes from Boundary ML. It stands for Basically a Made-up Language.

Here’s how BAML works: you define your prompts and their expected output schemas in a separate .baml file. BAML generates type-safe Python (or TypeScript) code. Your IDE knows what types you’re working with. You get autocomplete. You get actual contracts that the LLM needs to fulfill.

The alternative? What we did in Part 1. Hope that your string manipulation produces valid prompts and that the LLM returns something usable.

When I first heard about BAML, I thought “do we really need a new language just for prompts?”

But it’s not about the language syntax. It’s about having a place where prompts live separately from your application code. When your prompts are scattered across ten different Python files as f-strings, nobody knows which version runs in production. BAML forces you to put them all in one place, give them types, and version them properly.

Could you do this without BAML? Sure. But you probably won’t because it’s annoying to set up from scratch.

Installing BAML

Assuming you’ve got Python 3.8 or higher, run pip install baml-py.

If you use VSCode, grab the BAML extension from the marketplace. The syntax highlighting alone is worth it, but you also get inline validation that catches errors before you run the code.

📝 Note: The VSCode extension makes a huge difference in your workflow. If you’re not a VSCode person, BAML will still work fine. You just won’t get those annoying (but important) red squiggly lines when you mess up the syntax.

Converting Our Messy Notebook to BAML

Remember the janky resume analyzer from Part 1?

The biggest issue: we’re treating the prompt like a throwaway string, and we don’t know what structure that JSON will have when it comes back.

Creating a BAML File

First, set up your project structure.

my-llm-project/
├── baml_src/
│   └── resume.baml
├── main.py
├── .env
└── requirements.txt

Inside baml_src/resume.baml, define your types and prompt function.

We defined a class ResumeAnalysis. This is our schema and our contract. When this function runs, it will return an object with these fields and these types. Not “hopefully returns” or “usually returns”. Will return.

Notice I added a clarification in the prompt: “if there’s a range, use the minimum.” Remember my crash? This prompt clarification prevents that crash. We’re explicit about what we expect.

The {{ ctx.output_format }} at the end is BAML magic. It automatically generates the JSON schema instructions for the LLM based on your class definition. You don’t have to manually write “return this as JSON with these exact fields.” BAML handles that for you.

⚠️ The prompt syntax uses {{ variable_name }} for variable interpolation, not Python’s f-string style.

Configuring Your LLM Client

Before our function works, we need to tell BAML how to call GPT-4. Create another file in baml_src/ called clients.baml.

We separated the configuration (which model, which API key, what temperature) from the function definition. This is separation of concerns in action. If you want to swap GPT-4 for GPT-5, or switch to a different provider entirely, you can change it in one place. Your function definitions don’t need to know or care about which model runs.

Generating the Python Code

Run the BAML compiler.

baml-cli generate

This creates a baml_client/ directory with generated Python code. Don’t edit these files manually (they’ll get overwritten next time you run the generator), but do check them into version control.

Using It in Your Python Code

Open up main.py.

Look at what we just did.

analysis.candidate_name - Your IDE knows this exists.

analysis.years_of_experience - Your IDE knows this is an int.

analysis.skills - Your IDE knows this is a list of strings.

If you try to access a field that doesn’t exist, your IDE yells at you before you run the code. If you try to do math on candidate_name, your type checker catches it.

This is type safety.

Handling Complex Nested Structures

The resume analysis returns flat data: a name, a number, a list of strings. Most real-world LLM tasks aren’t that simple.

Say you’re analyzing meeting transcripts. You need action items, but each action item has its own structure: a task, who it’s assigned to, a due date, a priority level. That’s nested data.

Here’s how you handle it in BAML. First, define the nested structure.

Each action item is its own object with typed fields. The Priority enum ensures the LLM can only return those four values, not random strings like “kinda important".

Now define what a full meeting analysis looks like.

The action_items field is an array of ActionItem objects. Your IDE understands this nesting.

Here’s the function 👇🏾

Using it in Python.

Your IDE knows item is an ActionItem. It knows item.priority is a Priority enum. You get autocomplete for nested fields.

Configuration Management Done Right

Remember that security nightmare of hardcoded API keys from Part 1? BAML handles this properly.

You can define multiple clients for different scenarios:

client<llm> GPT4 {
  provider openai
  options {
    model gpt-4
    api_key env.OPENAI_API_KEY
    temperature 0
  }
}

client<llm> GPT5 {
  provider openai
  options {
    model gpt-5-mini
    api_key env.OPENAI_API_KEY
    temperature 0
  }
}

client<llm> Claude {
  provider anthropic
  options {
    model claude-3-sonnet-20240229
    api_key env.ANTHROPIC_API_KEY
    temperature 0
  }
}

And in your function definition, you specify which client to use.

Want to test if Claude gives better results? Change one word. Want to swap out the model for a locally hosted one? Define a new client, point it at your local endpoint, change one word in your function. The rest of your Python code doesn’t change.

This is proper abstraction. You’ve separated what you want to do (analyze a resume) from how you’re doing it (which model, which provider).

I usually start with gpt-4.1-nano for development and testing because it’s cheaper. Once the prompts work well, I switch to gpt-4.1 for production. With BAML, that’s a one-word change in the client configuration.

The Principles Matter More than the Tool

BAML could disappear tomorrow and these lessons would still be valuable.

What matters:

• Type safety: Define schemas for your LLM outputs and enforce them.

• Separation of concerns: Keep prompts separate from application logic.

• Testability: Make it possible to test your prompts in isolation.

• Versioning: Treat prompts as code that can be versioned and reviewed.

• Configuration management: Separate what you’re doing from how you’re doing it.

You could implement these principles with Pydantic models for type safety, a custom prompt management system, a simple YAML file for prompts, or your own abstraction layer.

BAML makes it easier. But if you walk away from this thinking “I need to use BAML,” you’ve missed the point. Walk away thinking “I need type safety and structure for my LLM applications,” then choose whatever tool helps you get there.

Before BAML, I tried building my own prompt management system with Pydantic. It worked okay, but it was a lot of boilerplate code. BAML handles all that boilerplate for you, which is why I switched. But the underlying principles (schemas, separation of concerns, testability) were the same.

The VSCode Playground

One more thing about BAML: the VSCode extension gives you a playground where you can test your prompts without running your entire application.

You can click on a function in your .baml file, click “Test in Playground,” and try it out with different inputs. Instead of modifying your prompt, running your Python script, waiting for the API call, checking the output, repeat…you test right there in VSCode.

The first time I used this, I saved myself 30 minutes of iteration time. I was tweaking a prompt to get the output format right, and being able to test it directly in VSCode without context-switching to my terminal made a massive difference.

Exercise: Try Converting One Prompt

Pick one prompt from your codebase. Just one.

Install BAML (pip install baml-py), create a baml_src directory, define the output schema as a BAML class, convert the prompt to a BAML function, generate the Python code, and replace your old implementation with the new type-safe version.

When you define that output schema, you’ll probably realize your prompt was underspecified. You’ll find yourself adding clarifications like “if there’s a range, use the minimum” or “return dates in ISO format.” This is good. This is you thinking more carefully about what you actually want the LLM to do.

I’ve done this with probably a dozen prompts, and every time I realize my original prompt was vaguer than I thought. The act of defining a schema forces you to be explicit about your expectations.

⚠️ The biggest mistake I see: trying to define overly complex schemas on the first try. Start simple. Get one field working with type safety, then add more fields.

In the next post, we tackle testing. Now that we have type-safe, well-structured prompts, we can test them properly. We’ll cover mocking, test doubles, and how to build a comprehensive test suite without calling the OpenAI API a thousand times (and racking up a ridiculous bill).

Happy coding 👍🏾

I’ll never forget the first time I got an LLM to work locally. I was messing around with HuggingFace’s transformers library, running a sentiment analysis model on some text data. The model actually worked! It classified my test sentences correctly, the output looked reasonable, and I felt accomplished ☺️!

Not to mention the timing worked out well. My team had been talking about adding some LLM capability to one of our features for a few sprints. We kept deprioritizing it, moving it around in the backlog. Eventually, my tech lead asked if anyone wanted to spike it and see if it was feasible.

I volunteered. I mean, I’d just spent a weekend playing with transformers anyway, so why not?

I spent a few days tinkering in a Jupyter notebook. Got a basic prototype working, and showed it to people on my team during a sync. Everyone thought it was pretty cool. The PM got excited and my manager seemed interested. Apparently this prototype was what we needed.

Then sprint planning rolled around the next week.

“Okay, so the spike is done,” the PM said, pulling up the ticket. “How many story points would it take to actually build this properly?”

I stared at my screen. My brain went completely blank.

“Uh…more than 5?” I said. “Maybe 13? Honestly, I have no idea.”

And I really didn’t. My prototype was a 150-line Jupyter notebook with hardcoded API keys and string manipulation. How much work sits between “it works in my notebook” and “this is production-ready”?

Spoiler: a lot.

Why Notebooks Fail in Production

This isn’t a criticism of notebooks. I love using Jupyter notebooks or Google Colab! They’re perfect for experimentation, quick iterations, and showing your work. But there’s a massive gap between “it works in my notebook” and “this is ready to serve real traffic.” If you come from a data science or research background, you might not even know what that gap looks like yet.

This series bridges that gap. We’re going to take a messy notebook with LLM code and transform it into a production-ready system. No hand-waving, no shortcuts. We’ll cover type safety, testing, self-hosting, deployment, and everything else that separates a prototype from a product.

But first, we need to honestly assess the problem. Let me show you what makes notebook code so problematic for production systems.

The Mess We’re Starting With

Here’s what a typical “I just need to get this working” LLM notebook looks like.

At first glance, this seems fine. It’s about 30 lines and it works. You can run it, get results, and show your team. So what’s the problem?

I’ll show you.

The Type Safety Issue

Quick question: what does `analyze_resume()` return?

You might say “a dictionary.” But what keys are in that dictionary? What are their types? Is `years_of_experience` an int or a string? Is `skills` a list of strings or a comma-separated string?

You have to read the prompt, hope the LLM returns what you asked for, and pray it’s consistent. There’s no contract, no schema, no guarantee. Also, your IDE can’t help you with autocomplete because it has no idea what you’re working with.

I’ve run into this exact issue before. I had a function that was supposed to return a structured response from an LLM, and I just assumed `experience_years` would be an integer. Made sense, right? Turns out the LLM sometimes returned “5 years” as a string, sometimes “5” as a string, and occasionally (I kid you not) “five” spelled out. My downstream code that tried to do `if experience_years > 3` crashed in three different ways depending on which format showed up.

If I can’t tell what this function returns without running it, neither can anyone else on my team. When the LLM randomly decided to format the output differently, we’ll have runtime errors that should’ve been caught way earlier.

The Security Nightmare

See the API key on line 5? That’s a production incident waiting to happen.

Early in my career, I was reviewing a PR from someone on my team. The code looked fine, tests passed, everything seemed great. But I noticed a hardcoded HuggingFace API token.

“Did you push this to the repo?” I asked.

“Yeah, but I’ll change it right after!”

My stomach dropped. “How many commits ago?”

“Like, two commits back.”

Here’s the thing. Git is like an elephant. It NEVER forgets. Even if you change a secret in a later commit, it’s still sitting there in your repo’s history. Anyone who clones the repository can see it with a simple `git log` command. If your repo is public (or gets accidentally made public), that secret is now out in the wild.

GitHub has secret scanning, but it’s not instantaneous. By the time you get the notification email, bots have already scraped your token and added it to a database somewhere.

Right now, there’s probably a data scientist somewhere waking up to a $3,000 OpenAI bill because a bot found their exposed API key in a public repo and has been hammering GPT-4 endpoints for the past 12 hours. The hacker running that bot doesn’t care about your project or your career. They just want free compute.

What to Do If You’ve Leaked a Secret

If you realized you’ve committed a secret to Git:

1. Revoke the key immediately.

2. Report the incident.

3. Check your billing ASAP (OpenAI keys, AWS keys, anything that costs money).

4. Generate a new one.

5. Use `git filter-branch` or BFG Repo-Cleaner to remove it from history.

6. Force push (coordinate with your team first or get someone from security to help you).

The Right Way to Handle Secrets

⚠️ And add `.env` to your `.gitignore` file. Always. Every single time. No exceptions. I have this muscle memory now where if I create a new repo, `.gitignore` is literally the first file I create, forget the README.

The Testing Dilemma

How do you test `analyze_resume()`? Right now, you can’t. Well, you can, but you’d have to actually call the OpenAI API every single time you run your tests. This creates several problems.

• It costs money (every test run hits your API bill).

• It’s slow (GPT-4 responses can take several seconds each).

• It’s non-deterministic (same inputs potentially lead to different outputs).

• It’s fragile (tests fail when OpenAI has an outage).

This function is tightly coupled to the external API. There’s no way to test your logic without actually calling OpenAI. Your tests are expensive, slow, and flaky. Your CI/CD pipeline would take forever to run, cost a fortune, and randomly fail for reasons that have nothing to do with your code.

The String Manipulation Problem

Let’s take a look at that prompt construction again 👇🏾

This is fine for a quick prototype. But what happens when you need to reuse this prompt logic in three different places? What if you want to A/B test different prompt versions? What if you want to keep track of which prompt version produced which results?

Right now, your prompt is just a string buried in a function. You can’t version it, can’t test different variations systematically, can’t share it across files without copy-pasting.

I’ve been in codebases where we had seven different versions of basically the same prompt because everyone just copied and tweaked the string instead of having a centralized place for prompt templates. When we needed to change something fundamental, we had to hunt through the entire repo. Not fun.

The Error Handling Setback

What happens when OpenAI’s API is down? What if you hit rate limits? What if the response isn’t valid JSON? What if the model returns null for a required field?

Right now, the answer is: your code crashes.

There’s no retry logic, no fallback behavior, no graceful degradation. Just exceptions waiting to blow up your application.

The Observability Obstacle

When something goes wrong in production (and it will), how do you debug it?

The current notebook has a `print` statement. That’s it. You have no logs, no metrics, no tracing. You can’t answer basic questions like:

• How long do requests actually take?

• Which prompts are performing poorly?

• What’s our error rate?

• Which inputs consistently cause failures?

• How much are we spending on API calls?

You’re flying blind. I’ve been pulled into emergency debugging sessions where a service was failing intermittently. Having zero visibility into what’s happening is genuinely awful. You end up adding print statements, redeploying, waiting for it to fail again, checking logs, and repeating that cycle until you find the issue. It’s tedious and totally avoidable.

What Production-Ready Actually Means

I’ve thrown around the term “production-ready” a lot, but what does it actually mean?

Here’s my definition: production-ready code is code that you wouldn’t panic about if you got a Slack message saying it broke while you were in a meeting.

That means:

• Reliable (it handles errors gracefully and recovers from failures).

• Observable (you can see what it’s doing and diagnose problems quickly).

• Maintainable (other people can understand and modify it, including future you).

• Testable (you can verify it works without manual testing).

• Secure (it doesn’t leak secrets or expose vulnerabilities).

• Performant (it can handle your expected load, plus some buffer).

Notice I didn’t say perfect. Production-ready doesn’t mean bug-free. It means you’ve thought through the failure modes and built in safeguards.

Where We’re Going

Over the next four posts, we’re going to systematically fix every problem I just outlined. Here’s the roadmap 🧭.

Post 2: Treating Prompts Like Code with BAML

We’ll introduce BAML (a domain-specific language for LLM applications) and use it to add type safety and structure to our prompts. You’ll learn why treating prompts like untyped strings is asking for trouble.

Post 3: Testing Your LLM Applications (Without Going Broke)

We’ll build a testing strategy that doesn’t require calling OpenAI’s API for every single test. You’ll learn about mocking, test doubles, and how to test LLM applications in a way that’s actually maintainable.

Post 4: Self-Hosting LLMs with vLLM

We’ll set up vLLM (a high-performance inference engine) to self-host open-source models. You’ll learn when self-hosting makes sense, when it doesn’t, and how to actually do it with Docker and GPU support.

Post 5: The Complete Production Pipeline

We’ll bring everything together: Dockerize the application, set up CI/CD, add proper monitoring and logging, implement error handling and retries, and deploy it.

By the end of this series, you’ll have transformed a messy notebook into a production system that you’d actually want to maintain. More importantly, you’ll understand the principles behind each decision.

Homework - Audit Your Own Code

Before moving on to the next post, I want you to do an honest audit of your own LLM code (if you have any). If you don’t have LLM code yet, grab any data science notebook you’ve written recently.

Go through and honestly assess these problems.

1. Type safety: Can you tell what your functions return without running them?

2. Configuration: Are any secrets hardcoded? Did they ever make it into version control?

3. Testability: Can you test your logic without hitting external APIs?

4. String manipulation: Are your prompts scattered throughout your code as raw strings?

5. Error handling: What actually happens when things go wrong?

6. Observability: If this broke while you were away, could you figure out why within 15 minutes?

Be brutally honest with yourself. I’m not here to judge (I’ve written plenty of questionable notebook code over the years). But you can’t fix problems you don’t acknowledge.

Write down what you find. Take notes on which problems show up most often in your code. We’re going to address each one over this series.

In the next post, we’ll tackle type safety and string manipulation by introducing BAML and treating our prompts like the first-class citizens they deserve to be. We’ll add structure and sanity to our LLM applications. No more crossing your fingers and hoping the output is what you expect.

Happy coding 🚀!

When we finally figured it out, we discovered our test set had 50 variations for the same edge case and completely missed the actual failure modes. Random sampling failed us 😭.

I used to randomly sample everything. Test sets? Random sample. Few-shot examples? Random sample. Active learning? You guessed it, random sample. It seemed like the fair, unbiased approach. Then I stumbled onto research about maximally diverse subsets, and my entire approach to data selection changed.

Turns out, there’s a whole body of research on how to do this properly, and it’s been hiding in plain sight. Let me show you what I learned (and the mistakes I made along the way).

What Even Are Maximally Diverse Subsets?

The concept is simpler than the name suggests. Imagine you have 1,000 data points and need to pick 50 of them. Random sampling might cluster all 50 data points in one corner of your feature space. Diverse sampling spreads these data points out so they actually represent your full dataset.

What’s interesting is that there are different ways to measure diversity, and each measurement has different meanings.

Maximum pairwise distance picks points as far apart as possible from each other. If your data clusters naturally (like images of cats vs dogs vs birds), this approach grabs representatives from each cluster.

Coverage-based selection makes sure you’re hitting different regions of your feature space. Think of it like ensuring every neighborhood in a city has at least one representative on a committee.

Determinantal Point Processes (DPPs) use a probabilistic approach borrowed from quantum physics. Yes, quantum physics 🤯. DPPs naturally repel similar items, and the math behind them is honestly beautiful once you get past the intimidating name.

Why Should You Care?

I see four places where diversity-aware sampling makes a huge difference in your day-to-day work.

Building test sets: You want test cases covering different scenarios, not 100 variations of the same thing.

Few-shot prompting: Show your LLM different patterns. Three diverse examples teach more than five similar ones (I tested this and loved the results).

Hallucination detection: When an LLM hallucinates, it generates diverse outputs. Measure that diversity to catch hallucinations.

Data pruning with DPPs: Shrink your dataset by keeping the interesting stuff and dropping redundant examples. Your training time will decrease which means your AWS bill won’t make you nervous by the end of the month 💸.

Creating Better Test Sets

I was building a test set for a computer vision model a couple years ago. We had 10,000 labeled images but budget to manually review only 500 for our test set. I did what seemed reasonable: random sampling.

The test set looked fine at first glance. But when we deployed to production, the model failed on certain types of images that never appeared in our test set. Random sampling had oversampled common scenarios and completely missed rare but important edge cases.

When I dug into it, about 30% of my test images showed basically the same scene because the training data skewed heavily in that direction. Diverse sampling would have caught this immediately.

Two methods fix this problem.

K-means clustering groups similar images together and picks representatives from each cluster. It’s fast and works great for medium-sized datasets. I use this one most often now.

MaxMin Selection greedily picks points far from already-selected points. It’s slower on large datasets but guarantees good coverage across your entire feature space. (For large datasets, combine it with clustering first - see Pitfall #2 for the efficient approach.)

Here’s what I should have done.

MaxMin ensures your test set spreads throughout your feature space. No clumping allowed.

My Few-Shot Selection Disaster

I was building a sentiment classifier for movie reviews. I had 1,000 labeled examples and could fit 5 in my prompt. I did the “smart” thing and used k-nearest neighbors to pick the 5 most similar examples to my test query.

My results were disappointing. The model kept making the same mistakes over and over.

Then I realized that all 5 examples were teaching the LLM basically the same pattern. They were all positive reviews using similar language. The model learned this one way to express positivity and nothing else.

I switched to Maximum Marginal Relevance (MMR), which balances similarity with diversity. The first example matches your query closely. Each subsequent example adds new information instead of repeating what the model already saw.

The improvement was immediate and obvious.

Here’s the code I wish I’d written the first time 👇🏾

The paper, “Exploring the Role of Diversity in Example Selection for In-Context Learning”, confirmed what I saw in practice. MMR consistently beats pure similarity-based selection across different context sizes and similarity functions.

The Clever Hallucination Detection Trick

This one blew my mind when I first read about it. When an LLM hallucinates, it generates more diverse outputs across multiple samples. Why? Because it doesn’t actually know the answer, so it makes up different nonsense each time.

Think about it this way. Ask an LLM “What is the capital of France” five times. You’ll get five variations of “Paris” (maybe with different phrasing, but same core answer).

Now ask “What is the melting point of dragon scales?” five times. You’ll get five completely different made-up numbers because there’s no ground truth to anchor to. The diversity reveals the hallucination.

Several recent hallucination detection papers explore this approach. The intuition is solid. Confident and correct models give consistent answers. Hallucinating models generate different nonsense each time because there’s no ground truth to anchor to.

Data Pruning With Determinantal Point Processes

Imagine you have 100k training samples, but training is expensive and slow. Can you shrink your dataset to 10k samples while maintaining model performance? Random sampling gives you mediocre results. But what if you could select the 10k most diverse, informative examples?

That’s where DPPs shine.

DPPs model diversity through a kernel matrix (basically a similarity matrix). The probability of selecting a subset is proportional to the determinant of the kernel matrix for that subset. Larger determinants equal more diverse subsets.

The paper, “Post-training Large Language Models for Diverse High-Quality Responses”, used DPPs to train models that generate both high-quality AND diverse outputs. The results beat standard training approaches across multiple benchmarks.

Here’s a simplified implementation (this is the greedy approximation, exact sampling gets more complex).

The beauty of DPPs is that they give you a principled probabilistic framework for diversity. You’re not just greedily picking points. You’re sampling from a distribution that naturally prefers diverse subsets.

For data pruning, this means you can train on 10-20% of your data and match (or sometimes beat!) the performance of training on the full dataset. Less storage, faster training iterations, lower compute costs. Pretty cool, right?

You may be eager to start applying maximally diverse subsets in your work, but there are a couple pitfalls you need to watch out for.

⚠️ Pitfall #1: Diversity Without Quality

I made this mistake because I was focused on getting diverse test samples, but I ended up with really low-quality examples in my test set. Some samples were near-duplicates with slight corruption. Others had labeling errors. One even had a cat labeled as “car” (still not sure how that happened).

Were they diverse? Sure. Were they useful? Nope.

The fix is straightforward. Filter for quality first, THEN apply diversity selection.

Just like any analytics, data science, or machine learning problem, you need clean data. Garbage in, garbage out, am I right?

⚠️ Pitfall #2: The Computational Cost

Computing pairwise distances for large datasets is expensive.

My first implementation processed 10,000 data points to select 100 examples. It took about half an hour.

The problem? I was computing ~50 million pairwise comparisons. That’s O(n²) complexity. When you’re iterating and experimenting, that adds up fast.

The fix? Use clustering to reduce the problem size.

This is how the two-stage approach works.

Step 1: Cluster your data into more clusters than you need (maybe 2x your target). This step is O(n × k) where k is the number of clusters - much faster than O(n²).

Step 2: Pick one representative from each cluster - the point closest to each cluster center. Now you have maybe 200 representative points instead of 10,000.

Step 3: Run maxmin on the representatives to select your final diverse subset. Computing pairwise distances on 200 points is trivial compared to 10,000.

Why does this work for diversity? The clusters identify distinct regions of your feature space. One sample per cluster gives you coverage across different regions. Then maxmin on the representatives ensures you’re selecting the most spread-out samples from those regions.

This two-stage approach drops the runtime to under a few minutes for my use case. For 100k samples selecting 1k, this runs orders of magnitude faster than computing all pairwise distances.

Your teammates will appreciate you not hogging up all the resources for your experimental work 😉.

Note: Pick the Right Diversity Metric

Not all distance metrics are created equal, and choosing the wrong one can give you weird results.

Euclidean distance works for continuous embeddings where magnitude matters. I use this for image embeddings from ResNet or similar models.

Cosine distance works better for normalized embeddings where direction matters more than magnitude. I default to this for text embeddings from sentence transformers.

Semantic entropy captures meaning-level diversity. This requires clustering semantically equivalent outputs first. It’s more complex but sometimes worth it.

Your use case may result in using different distance metrics, and that’s fine as long as you’re aware that different metrics can render you different results.

The Numbers Don’t Lie

The research on diversity-aware sampling keeps getting better, and the results are honestly kind of exciting.

The February 2025 “Diversity as a Reward” paper showed a 27% improvement on math reasoning tasks. They used less than 10% of the original training data and got BETTER results. That’s awesome! Less data, less compute, better performance.

The few-shot learning improvements are equally impressive. The 2025 In-Context Learning paper showed consistent gains across different context sizes and similarity functions. No additional training required. Just change how you select examples.

The alignment work from “Efficient Alignment of Large Language Models via Data Sampling” demonstrated that you can match full-dataset performance using less than 10% of the data. Fewer labels to collect, less compute for training, faster iteration cycles. That’s a win every time.

These results come from different teams working on different models tackling different tasks. The pattern holds across domains. Diversity matters.

However, it’s not always the answer.

When Not To Use Diversity

Case 1: Debugging Specific Failures

Your model consistently chokes on questions about historical dates. You don’t want diverse examples. You want MORE historical date examples to understand the failure pattern.

Case 2: Narrow Specialization

You’re fine-tuning a model to generate SQL queries for your company’s specific database schema. You want homogeneous training data that teaches one thing really well. Diversity would dilute the signal.

Case 3: Consistency Checking

You’re testing whether slight input variations produce wildly different outputs. You need similar inputs on purpose.

Diversity is a tool for coverage and robustness. But sometimes you need focus instead.

Your Action Plan for Next Week

If you want to get started, start small by creating a function to help you select diverse test sets.

With this, you’ll build better test sets, select better few-shot examples, and catch failures earlier. Hopefully, you can spend less time debugging and more time creating.

Continued Reading

If this post clicked with you and you want to dive deeper, check out these papers. They’re all recent and fairly straightforward.

“Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Data” (2025)
https://arxiv.org/abs/2502.04380

“Post-training Large Language Models for Diverse High-Quality Responses” (2025)
https://arxiv.org/abs/2509.04784

“Exploring the Role of Diversity in Example Selection for In-Context Learning” (2025)
https://www.arxiv.org/abs/2505.01842

“Efficient Alignment of Large Language Models via Data Sampling” (2024)
https://arxiv.org/abs/2411.10545

Code Available Here 📎

All code snippets from this post are available in this GitHub repo.

Feel free to experiment with the MMR implementation and diversity parameter tuning!

Happy coding ✨