Free course · by Debugshala, Indore
LLMOps and MLOps Tutorial
21 free lessons, in the order we teach them in class: start at lesson 1 and work down. Every lesson has an analogy, a diagram, code you can run or copy, a quick quiz and an FAQ.
- 21Lessons
- 3Modules
- ~4 hReading time
- FreeAlways
01MLOps
- What is MLOps? A Beginner's Guide to Machine Learning OperationsMLOps explained for beginners: why models fail quietly after launch, what an MLOps system includes, MLOps vs DevOps, and three maturity levels, with Python.
- ML Lifecycle: The 8 Stages of a Machine Learning SystemThe machine learning lifecycle in 8 stages, from metric and baseline to monitoring and retraining, with runnable Python showing why time-based splits matter.
- ML Reproducibility and Versioning: Data, Code, Seeds, ModelsMake ML models reproducible: version code, data, config, environment and model files. Hash data in Python, fix seeds, and use DVC and the MLflow registry.
- Data Pipelines for ML: ETL, Validation and Train/Serve SkewBuild a data pipeline for machine learning: extract, validate, transform, load, idempotent backfills, Airflow scheduling, and a runnable train/serve skew demo.
- Feature Stores and Data Leakage: Point-in-Time Correct MLWhat a feature store is, how data leakage inflates scores, and how point-in-time joins with pandas merge_asof and Feast keep training and serving consistent.
- ML Model Deployment: Serve a Model as a REST API with FastAPIDeploy an ML model as an API: choose batch or online serving, serialise with joblib, validate input with FastAPI and Pydantic, and release by shadow or canary.
- Docker for Machine Learning: Package and Ship a Model APIDocker for machine learning: write a Dockerfile for a FastAPI model, use layer caching, keep images small and safe, tag by model version, and run Compose.
- Kubernetes for ML Engineers: Deploy and Scale a Model APIKubernetes for ML: Deployments, Services, readiness probes and HPA autoscaling for a model API, with YAML, kubectl commands and Python simulations you can run.
- ML Model Monitoring and Data Drift: PSI, KS Test and AlertsMonitor ML models in production: data drift vs concept drift, PSI and KS statistics computed by hand in Python, delayed labels, Evidently and Prometheus alerts.
- CI/CD for Machine Learning: Tests, Quality Gates and ReleasesCI/CD for machine learning: unit, data and behavioural tests, a quality gate against the champion model, a GitHub Actions workflow, canary release and CT.
02LLMOps
- What is LLMOps? Running LLM Apps in Production, ExplainedLLMOps explained: how it differs from MLOps, the lifecycle from prompt change to traced production, token-cost maths and a release gate, with runnable Python.
- LLM Evaluation Methods: Metrics, Golden Sets and a Python HarnessEvaluate LLM apps: exact match, token F1 and rubric scoring in Python, building a golden set, a bootstrap check for real gains, and a DeepEval G-Eval example.
- LLM-as-a-Judge: Rubrics, Biases and Cohen's Kappa CalibrationLLM-as-a-judge explained: single, pairwise and reference grading, JSON judge prompts, position bias, and calibration against humans with Cohen's kappa.
- Red Teaming LLM Apps: Attack Suites, Success Rates and FixesRed team an LLM chatbot: build a threat model, generate attacks from seeds and strategies, score attack success rate in Python, and automate with promptfoo.
- KV Cache Explained: How LLMs Reuse Attention and Use GPU MemoryWhat the KV cache is, why it makes LLM generation fast, prefill vs decode, a numpy speed-up demo, and the formula for KV cache memory with GQA and 8-bit caches.
- LLM Inference Optimization: Batching, Quantisation and SpeedLLM inference optimization: TTFT and TPOT, why decoding is memory-bound, continuous batching and speculative decoding simulated in Python, and a tuning order.
- vLLM Tutorial: Serve LLMs with PagedAttention and an OpenAI APIServe LLMs with vLLM: how PagedAttention works, vllm serve flags, OpenAI client calls, capacity planning maths, Docker, vllm bench serve and Prometheus metrics.
- LLM Observability and Tracing: Spans, Latency, Cost, FeedbackLLM observability explained: traces and spans, what to log, PII masking, p95 latency, cost by prompt version, and OpenTelemetry GenAI and Langfuse tracing code.
03Career Guide
- Generative AI Roadmap: Skills, Projects and Jobs for FreshersA Generative AI roadmap for freshers: the skills to learn, a six-stage path, three portfolio projects (RAG, MCP, agent) and how to study GenAI in Indore.
- Generative AI Projects: 10 Portfolio Ideas with ArchitectureTen Generative AI portfolio projects with architecture outlines, metrics and stretch goals, from a bilingual RAG bot to a vLLM load test and an eval suite.
- Generative AI Interview Questions: 30 Questions with Answers30 Generative AI interview questions with crisp answers on LLMs, prompting, RAG, fine-tuning, agents, evaluation and serving, plus a system design framework.