LLM Engineering in Production Crash Course

Goal: Connect transformer and tokenization internals to application behavior, then define the quality and operational contract for a real product.

Learning objectives:

• Explain the transformer components that influence generation behavior and inference cost
• Predict how tokenization affects multilingual inputs, context budgets, structured outputs, and billing
• Convert a product idea into explicit quality, latency, cost, and safety requirements

Syllabus:

• The production LLM lifecycle: data, tokenization, model, inference stack, application workflow, evaluation, and operations
• LLM architecture evolution
• Transformer fundamentals
• Attention mechanisms in transformers
• KV cache
• Tokenization as an engineering constraint: subword vocabularies, token budgets, multilingual variability, special tokens, and prompt-size estimation
• Byte Pair Encoding
• WordPiece
• SentencePiece
• Turn a feature idea into a production contract: user task, non-goals, error costs, data boundaries, quality threshold, latency SLO, cost budget, and fallback behavior
• AI solution compass: choosing rules, ML, and GenAI
• Key challenges for LLM deployment

Goal: Decide whether the application needs retrieval, fine-tuning, or both, then implement the smallest reliable architecture that supports the selected use case.

Learning objectives:

• Design a RAG architecture suited to the corpus, freshness requirements, and access-control model
• Choose an embedding model and retrieval configuration using task-specific evaluation evidence
• Decide when fine-tuning is worth its data, infrastructure, and maintenance cost
• Apply PEFT, LoRA, and quantization with clear quality and deployment trade-offs

Syllabus:

• RAG architecture patterns: ingestion, chunking, metadata, indexing, retrieval, reranking, context assembly, citations, and safe abstention
• Retrieval-Augmented Generation fundamentals
• Embedding models and retrieval quality: bi-encoders, similarity metrics, domain adaptation, vector databases, rerankers, and permission-aware metadata filters
• Text embeddings
• Embedding-layer architecture for LLMs
• Adaptation choices: prompting versus RAG versus supervised fine-tuning; data quality, train/validation splits, versioning, and rollback criteria
• Customizing LLMs with training, fine-tuning, and prompting
• Efficient adaptation with PEFT and LoRA: frozen base weights, adapter design, hyperparameters, evaluation, merging, and serving adapters safely
• Parameter-efficient fine-tuning
• LoRA for LLM fine-tuning
• Quantization trade-offs: weights, activations, post-training quantization, quantization-aware training, accuracy checks, and hardware compatibility
• Post-training quantization
• Quantization-aware training
• ML model quantization

Goal: Build an evaluation framework that turns model behavior into measurable evidence and makes hallucination reduction an end-to-end system property.

Learning objectives:

• Create representative offline test sets and define success criteria for each project risk
• Evaluate retrieval, generation, structured outputs, safety, latency, and cost separately and together
• Reduce unsupported claims through better context, constrained generation, verification, and safe abstention

Syllabus:

• Evaluation framework design: task taxonomy, golden datasets, dataset slices, versioning, deterministic checks, LLM-as-a-judge, human review, and regression gates
• How to measure LLM performance
• Testing machine learning code like a pro
• Evaluate RAG as a pipeline: retrieval recall, reranker quality, context relevance, citation correctness, grounded answer quality, and failure analysis
• Evaluating RAG systems
• How to debug a RAG workflow practically
• Hallucination reduction: task scoping, source selection, context quality, citation constraints, constrained decoding, verification, calibrated confidence, and abstention
• Hallucination as an LLM deployment challenge
• Model calibration
• Guardrails for LLMs
• Operational evaluation: quality and cost slices, canary releases, online feedback, incident labels, prompt and model comparison, and rollback thresholds
• MLflow
• OpenTelemetry for ML systems

Goal: Turn a reliable prototype into an observable service that meets latency, throughput, cost, and availability constraints.

Learning objectives:

• Diagnose where inference time and memory are spent during prefill and decoding
• Choose optimization levers that improve the service without silently degrading quality
• Design a GPU-serving architecture with batching, admission control, scaling, and observability

Syllabus:

• Inference performance fundamentals: prefill versus decoding, token throughput, latency percentiles, context length, concurrency, caching, routing, and model selection
• KV cache
• A practical guide to LLM cost reduction
• Inference-time scaling
• GPU serving architecture: model loading, memory planning, continuous batching, queueing, backpressure, autoscaling, multi-tenancy, streaming, health checks, and capacity tests
• Deploying efficient models: quantized artifacts, hardware-aware selection, ONNX deployment, edge versus cloud trade-offs, and safe fallback models
• Deploying ONNX models
• ONNX Runtime compaction for browser ML
• LLM deployment: cloud to edge
• Production operations: model, prompt, data, and index versioning; tracing; SLO dashboards; error budgets; incident response; rollback; and controlled updates
• MLOps
• Prompt development, externalization, and management