LLMOps / AI Ops | TechByteByByte

Begin with the problem

LLMOps connects prompts, models, datasets, evaluations, deployments, and production traces into one reproducible history. Without that connection, teams cannot explain why behavior changed.

experiment → version artifacts → evaluate → register → deploy → monitor → feed failures back

What you will learn

Distinguish LLMOps from model-training-focused MLOps.
Track prompts, datasets, model settings, retrieval configs, and results together.
Build reproducibility and governance across the complete lifecycle.

Current production grounding: OpenAI’s Evals documentation shows dataset- and grader-based evaluation for model applications.

These links document current public features and practices. They do not reveal a provider’s private implementation or guarantee that every product uses identical defaults.

1. The Engineering Problem

Modules 5, 18, 24-26 each built one piece — prompt versioning, dataset versioning, testing, deployment. This module closes Level 9 by naming and unifying the discipline that ties them together: LLMOps, the operational practice of managing an LLM-based system’s complete lifecycle, not just its individual artifacts in isolation.

2. LLMOps vs. MLOps — A Precise Distinction

MLOps: manages the lifecycle of a TRAINED model you own
      -- training pipelines, model versioning, retraining triggers.

LLMOps: manages the lifecycle of a SYSTEM built around a
       (often third-party) LLM you DON'T train -- prompt
       versioning, retrieval configuration, evaluation pipelines,
       and the model as ONE swappable component among several.

Module 1’s core framing: an AI Engineer typically treats the model as an external dependency. LLMOps is the operational discipline that follows from that framing — managing prompts, retrieval, and evaluation with the SAME rigor MLOps applies to model training, since these are now the components you actually control and iterate on.

3. The Artifacts LLMOps Manages

Artifact	Registry Discipline	Covered In
Prompts	Versioned, tested, rollback-able	Module 5
Models	Selected, routed, tracked per use case	Module 4
Datasets	Versioned, validated, governed	Module 18
Evaluation results	Tracked over time, per version	Module 10-11
Deployments	Staged, monitored, rollback-able	Module 25-26

4. Experiment Tracking — Connecting Changes to Outcomes

EXPERIMENT TRACKING records, for every meaningful change:

  - WHAT changed (which prompt version, model, retrieval config)
  - WHAT the evaluation scores were (Module 10-11)
  - WHAT the real production metrics were after deployment (Module
    12)

Without this teams can't answer "did switching from
model A to model B actually help?" with confidence -- only with
GUESSWORK.

5. Governance — Module 18, Extended System-Wide

Module 18's data governance, extended to EVERY artifact
this module covers: WHO can approve a prompt change reaching
production? WHO can approve a MODEL swap? Is there a audit
trail for every artifact that's EVER been in production?

6. A Real-World Analogy — The Hospital, Once More

Module 2, 10, and 20's doctor analogy: a hospital doesn't
just track WHICH doctor treated a patient -- it maintains a complete record: which PROTOCOL was followed, which
MEDICATIONS were prescribed, and the OUTCOME. If a treatment
protocol is later found to be worse than expected, the hospital can
trace back exactly which patients were affected and when
the protocol changed.

LLMOps provides EXACTLY this same traceability for an AI system's
prompts, models, and datasets.

7. The Complete AI Lifecycle

DEVELOP (write/test prompt, select model, curate dataset)
   |
   v
VALIDATE (Module 24's testing taxonomy)
   |
   v
EVALUATE (Module 10-11's evaluation gate)
   |
   v
DEPLOY (Module 25's canary progression)
   |
   v
MONITOR (Module 12's observability)
   |
   v
IMPROVE (Module 20's feedback-driven optimization hierarchy)
   |
   v
(loops back to DEVELOP)

8. A worked developer example

TechCorp’s unified registry, tracking a prompt’s complete lifecycle:

Version	Stage	Registered By
v1	Production	Alice
v2	Canary (currently being evaluated in production)	Bob

When a team member needs to know “what’s currently live,” the registry answers definitively — no ambiguity about which version is actually serving real traffic, and a complete, history of every version ever registered.

9. How Is This Used in the Industry?

🤖 How Is This Used in the Industry?

Mature AI engineering organizations implement a unified registry spanning prompts, models, and datasets — not three separate, disconnected tracking systems — because a real production incident often requires correlating changes across all three simultaneously (Module 18’s scenario: “was it the model, the prompt, or the dataset that changed?”).

10. Common Mistakes

Incorrect idea: Tracking model versions carefully while leaving prompts and datasets unversioned.

Why it is incorrect: As shown directly in Module 5 and 18’s scenarios, this leaves real gaps in traceability.

Incorrect idea: No experiment tracking connecting changes to actual outcomes.

Why it is incorrect: This shortcut removes a validation or control boundary, allowing an error to pass into later stages where it becomes harder and more expensive to detect. As shown directly in Section 4, this leaves teams guessing whether a change actually helped.

Incorrect idea: Treating LLMOps as identical to MLOps.

Why it is incorrect: As shown directly in Section 2, the focus is different — LLMOps manages the system around an often-external model, not a model you train yourself.

11. Code — A Unified AI Lifecycle Registry

What this shows: a working registry spanning prompts, models, and datasets in ONE unified system — directly implementing Section 3’s artifact list and Section 8’s worked developer example, supporting both “what’s currently in production” queries and a full audit trail.

from dataclasses import dataclass, field
from datetime import datetime
from enum import Enum

class ArtifactType(Enum):
    MODEL = "model"
    PROMPT = "prompt"
    DATASET = "dataset"

@dataclass
class RegistryEntry:
    artifact_type: ArtifactType
    name: str
    version: str
    metadata: dict
    registered_at: str

class AILifecycleRegistry:
    """A unified registry spanning models, prompts, AND
    datasets (Section 3) -- directly connecting Module 5's prompt
    versioning and Module 18's dataset versioning into ONE governed
    system, exactly what LLMOps means in practice."""

    def __init__(self):
        self.entries: list = []

    def register(self, artifact_type: ArtifactType, name: str, version: str, metadata: dict):
        entry = RegistryEntry(artifact_type, name, version, metadata, datetime.now().isoformat())
        self.entries.append(entry)
        return entry

    def get_production_artifact(self, artifact_type: ArtifactType, name: str) -> RegistryEntry:
        """Returns the CURRENT production version -- directly
        supporting governance and rollback (Section 5)."""
        candidates = [e for e in self.entries if e.artifact_type == artifact_type and e.name == name
                      and e.metadata.get("stage") == "production"]
        return candidates[-1] if candidates else None

    def audit_trail(self, artifact_type: ArtifactType, name: str) -> list:
        """Directly supports GOVERNANCE (Module 18) -- a full,
        history of every version ever registered for this
        artifact."""
        return [e for e in self.entries if e.artifact_type == artifact_type and e.name == name]

registry = AILifecycleRegistry()
# Exactly Section 8's worked developer example
registry.register(ArtifactType.PROMPT, "support_response", "v1", {"stage": "production", "author": "alice"})
registry.register(ArtifactType.PROMPT, "support_response", "v2", {"stage": "canary", "author": "bob"})
registry.register(ArtifactType.MODEL, "support_classifier", "gpt-mini-2026", {"stage": "production", "author": "carol"})

current_prod_prompt = registry.get_production_artifact(ArtifactType.PROMPT, "support_response")
print(f"Current production prompt version: {current_prod_prompt.version}")

audit = registry.audit_trail(ArtifactType.PROMPT, "support_response")
print(f"Full audit trail: {[(e.version, e.metadata['stage']) for e in audit]}")

Expected Output:

Current production prompt version: v1
Full audit trail: [('v1', 'production'), ('v2', 'canary')]

What this confirms: The registry identifies v1 as the current production version because v2 is still in canary. The audit trail preserves the complete history of both versions.

The system can therefore answer two separate questions: “What is live right now?” and “What is the full history?” This is the traceability described by Section 6’s hospital analogy.

12. Production Considerations

A real registry needs, durable storage (a database), not just in-memory objects — this data must survive across deployments and outlive any single process
Integrate the registry directly with Module 26’s CI/CD pipeline — every successful deployment should, automatically update the registry’s production-stage record

13. Trade-offs

A unified registry spanning models, prompts, AND datasets adds upfront implementation effort compared to three separate, simpler tracking systems — worthwhile for the cross-artifact correlation it enables during incident investigation
Comprehensive experiment tracking (Section 4) adds bookkeeping overhead to every change — worthwhile for the confidence it provides that changes are empirically validated

14. Chapter Summary

LLMOps is the operational discipline of managing an entire LLM-based system’s lifecycle — prompts, models, and datasets together, not the model alone — directly following from Module 1’s framing of the model as an external dependency you don’t train.

This differs from MLOps, which manages a trained model’s own lifecycle. A unified registry spanning all three artifact types, paired with experiment tracking connecting changes to real outcomes, gives teams the traceability needed to answer “what changed, and did it actually help” — closing this course’s Level 9 by tying together prompt versioning (Module 5), dataset versioning (Module 18), evaluation (Module 10-11), and deployment (Module 25-26) into one coherent operational practice.

15. Visual Cheat Sheet

DEVELOP -> VALIDATE -> EVALUATE -> DEPLOY -> MONITOR -> IMPROVE
   ^                                                        |
   +--------------------------------------------------------+

Unified registry: MODELS + PROMPTS + DATASETS, ONE system,
                  full audit trail, "what's live" answer

16. Top Takeaways

LLMOps manages a system built around an often-external LLM; MLOps manages a model you train yourself — different focuses.
A unified registry spanning prompts, models, AND datasets enables cross-artifact correlation during incident investigation.
Experiment tracking connects changes to real outcomes, replacing guesswork with empirical confidence.
Governance (who can approve what) should extend to every artifact type, not just models.
The complete AI lifecycle — develop, validate, evaluate, deploy, monitor, improve — loops back to develop, closing this course’s operational practices into one continuous cycle.

17. Interview Questions

Q: 1. Distinguish LLMOps from MLOps, and explain why this distinction matters for how a team organizes its operational practices.**

Ans: MLOps manages the lifecycle of a trained model you own — training pipelines, model versioning, retraining triggers. LLMOps manages the lifecycle of a system built around an LLM, often a third-party one you don’t train — prompt versioning, retrieval configuration, evaluation pipelines, with the model itself treated as one swappable, external component.

This matters because teams building on top of hosted models need LLMOps practices (prompt/dataset versioning, evaluation gates) even though they have no MLOps training pipeline at all.

Why it matters: Conflating these leads teams to under-invest in the relevant operational practices for their actual situation.
Real-world example: A team using a hosted API model has no training pipeline (no MLOps need there) but needs prompt versioning, evaluation gates, and deployment staging (LLMOps).
Common mistake: Assuming “we don’t train our own models” means no operational discipline is needed at all.
Interviewer is testing: Whether the candidate understands LLMOps as a distinct, necessary practice, not a subset of MLOps.
Likely follow-up: “What LLMOps practices would a team using only a hosted API model still need?” → Prompt versioning (Module 5), evaluation gates (Module 10-11), and deployment staging (Module 25) — all still necessary, MLOps or not.

Q: 2. Why should a registry track prompts and datasets with the same rigor traditionally applied only to model versions?**

Ans: A production regression can originate from a prompt change, a dataset change, or a model change — without versioning all three with equal rigor, diagnosing which one actually caused a specific issue becomes difficult or impossible, exactly Module 18’s scenario where an unversioned dataset change wasted significant investigation time.

Why it matters: Partial versioning creates blind spots precisely where root-cause investigation needs visibility most.
Real-world example: Section 11’s code — the unified registry lets a team query “what changed” across all three artifact types together, not just models.
Common mistake: Building rigorous model versioning while leaving prompts as inline strings and datasets as unversioned files.
Interviewer is testing: Whether the candidate applies consistent operational rigor across all the artifacts that affect system behavior, not just the most traditionally “ML” one.
Likely follow-up: “How would you retrofit this discipline onto an existing system that lacks it?” → Start by identifying every artifact type currently affecting production behavior (prompts, models, datasets), then incrementally introduce versioning and a unified registry (Section 11), prioritizing the artifact types most frequently changed or most implicated in past incidents.

18. Scenario-Based Question

Scenario: TechCorp experiences a production quality regression. The team has separate, disconnected systems for tracking model versions (a spreadsheet), prompt versions (git commit history, mixed in with application code), and dataset versions (no tracking at all).

Diagnosing the regression takes two full days because the team has to manually reconstruct a timeline of what changed when, across three disconnected sources.

Problem Analysis: Section 10’s common mistake — fragmented, inconsistent tracking across artifact types, exactly the gap Section 9’s unified registry is designed to close.
How to Think: The two-day diagnosis time isn’t a reflection of a hard technical problem — it’s a direct cost of missing, unified operational infrastructure.
Investigation: The team’s actual investigation process — manually cross-referencing a spreadsheet, git history, and no dataset record at all — directly illustrates why Section 9’s unified system matters.
Root Cause: No unified registry (Section 3, 11) — three disconnected, inconsistent tracking mechanisms for artifacts that all affect the same production system’s behavior.
Solution: Implement Section 11’s unified AILifecycleRegistry pattern — one system tracking models, prompts, and datasets together, with a queryable audit trail, directly integrated with Module 26’s CI/CD pipeline so every deployment automatically updates the registry.
Trade-offs: Migrating three disconnected systems into one unified registry requires, real migration effort — a worthwhile, one-time cost given the alternative is repeating this exact two-day diagnosis for every future regression.
Production Considerations: This scenario directly demonstrates Section 9’s real-world point — production incidents often require correlating changes across ALL artifact types simultaneously, and a fragmented tracking system makes this correlation far slower and more error-prone than it needs to be.

19. Next Step

Next: Module 28 — AI Architecture Patterns — Level 10 begins here: reusable, named architecture patterns (simple LLM app, RAG+reranking, agent+tools, enterprise knowledge assistant, and more) and exactly when each one applies.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026