TechByteByByte

Deployment

Local, Docker, cloud, serverless, and GPU deployment options; hosted vs. self-hosted serving; and the blue-green and canary deployment strategies that turn Module 11's evaluation-gated lifecycle into real, running infrastructure.

#AI Engineering#Deployment#Level 9

Begin with the problem

Deployment decides where components run and how a new version receives traffic safely. The central goal is not merely starting containers—it is observing, limiting, and reversing change.

artifact → environment/config → deploy → health checks → canary traffic → expand or roll back

What you will learn

  • Compare local, container, serverless, managed, and GPU deployment targets.
  • Separate configuration and secrets from application artifacts.
  • Use health checks, canaries, blue-green releases, and rollback plans.

Current production grounding: Kubernetes documents workload autoscaling and controlled Deployments for operating containerized services.

These links document current public features and practices. They do not reveal a provider’s private implementation or guarantee that every product uses identical defaults.

Topic-specific reference: Docker’s container overview explains portable application packaging, deployment, and scaling across development and production environments.

1. The Engineering Problem

Module 11 gave you a staged evaluation lifecycle — offline eval, shadow, canary, full rollout. This module covers the infrastructure that actually implements those stages: where an AI system’s components run, and how traffic actually moves between an old and new version safely, rather than a hard, all-at-once switch.

2. Deployment Targets — The Options

TargetWhat It’s For
LocalDevelopment and testing — never production
DockerContainerized, portable deployment — the standard unit for most modern AI services
Kubernetes (conceptual)Orchestrating many containers at scale — covered conceptually here, not hands-on
Cloud (managed)A cloud provider’s managed compute — less operational burden
Serverlesspay-per-execution, auto-scaling — well-suited to bursty, unpredictable traffic
GPU deploymentnecessary specifically for self-hosted model inference (Module 4)

3. Hosted vs. Self-Hosted Model Serving — Module 4,

Revisited

Module 4's hosted-vs-self-hosted DECISION, now covering
the DEPLOYMENT mechanics: a HOSTED model needs NO GPU deployment at
all -- you deploy the SURROUNDING application, and the provider
handles model serving. A SELF-HOSTED model requires you to deploy and manage inference infrastructure (GPU
instances, model serving software) yourself.

4. Blue-Green Deployment

BLUE (current, LIVE version) and GREEN (NEW version) run
SIMULTANEOUSLY, on separate infrastructure.

Traffic switches from BLUE to GREEN in ONE deliberate cutover, once
GREEN is verified healthy.

If a problem is discovered, traffic switches BACK to BLUE
immediately -- a fast, clean rollback since BLUE never stopped
running.

5. Canary Deployment — Module 11, Made Into Real

Infrastructure

Module 11's canary STAGE: a SMALL percentage of traffic
routes to the new version FIRST (e.g., 5%), with the
REMAINDER staying on the current version.

If metrics stay healthy, the percentage GRADUALLY
increases (5% -> 25% -> 100%) -- exactly the staged, graduated
exposure Module 11 introduced, now as REAL, running traffic-routing
infrastructure.

6. Rollback — A Non-Negotiable Requirement

ANY deployment strategy MUST support a FAST rollback --
directly Module 14's reliability principle, applied to DEPLOYMENTS
specifically: a deployment that CANNOT be reverted quickly is
a production risk, regardless of how well-tested the
change was beforehand.

7. Configuration and Secrets Management

configuration (model names, API endpoints, feature flags)
should be EXTERNALIZED from code -- directly Module 17's
stateless-service principle, extended to CONFIGURATION.

secrets (API keys, database credentials) should NEVER be
hardcoded or committed to version control -- managed through a
secrets manager, with access scoped per
environment.

8. A Real-World Analogy — The Power Grid, Once More

Module 14's power-grid analogy: a canary deployment is
like testing a new power source on a SMALL, isolated
section of the grid before connecting it to the ENTIRE network --
if something's wrong, only that small section is
affected, and it can be DISCONNECTED immediately without a
widespread outage.

9. A worked developer example

TechCorp’s canary rollout for a new prompt version, tracing exactly Module 11’s staged lifecycle through real deployment infrastructure:

StepActionTraffic Split
1Start canary95% current version, 5% new version
2Metrics healthy after 1 hourPromote to 25% new version
3Metrics still healthy after 2 more hoursPromote to 100% new version
(Alternative)Error rate spikes at any stageImmediate rollback to 100% current version

10. How Is This Used in the Industry?

🤖 How Is This Used in the Industry?

Production AI teams automate this exact canary progression as part of their CI/CD pipeline (Module 26) — traffic percentages increase automatically as monitored metrics (Module 12) stay within healthy thresholds, and automatically roll back the moment they don’t, removing manual judgment calls from the riskiest moments of a deployment.

11. Common Mistakes

Incorrect idea: Deploying directly to 100% traffic with no canary stage.

Why it is incorrect: As shown directly in Module 11, this skips the risk-graduated exposure that catches issues a golden dataset alone couldn’t anticipate.

Incorrect idea: Hardcoding secrets or configuration in application code.

Why it is incorrect: As shown directly in Section 7, this is a security and operational risk.

Incorrect idea: No fast rollback path.

Why it is incorrect: As shown directly in Section 6, this turns a bad deployment into a prolonged incident.

12. Code — A Canary Deployment Orchestrator

What this shows: a working orchestrator implementing Section 5’s canary progression and Section 9’s real developer example — directly turning Module 11’s staged evaluation lifecycle into actual, running traffic-management logic.

from dataclasses import dataclass
from enum import Enum

class DeploymentStrategy(Enum):
    CANARY = "canary"

@dataclass
class DeploymentPlan:
    strategy: DeploymentStrategy
    traffic_split: dict  # version -> percentage
    rollback_trigger: str

class DeploymentOrchestrator:
    """A deployment orchestrator implementing canary
    deployment (Section 5) for an AI system -- directly connecting
    Module 11's evaluation-gated lifecycle to actual, working
    deployment state."""

    def __init__(self):
        self.current_plan = None
        self.deployment_log = []

    def start_canary(self, new_version: str, initial_percentage: int = 5):
        plan = DeploymentPlan(
            strategy=DeploymentStrategy.CANARY,
            traffic_split={"current": 100 - initial_percentage, new_version: initial_percentage},
            rollback_trigger="error_rate_exceeds_baseline_by_50_percent",
        )
        self.current_plan = plan
        self.deployment_log.append(f"Started canary: {new_version} at {initial_percentage}%")
        return plan

    def promote_canary(self, new_version: str, new_percentage: int):
        if self.current_plan is None or self.current_plan.strategy!= DeploymentStrategy.CANARY:
            return {"success": False, "reason": "No active canary deployment"}
        self.current_plan.traffic_split = {"current": 100 - new_percentage, new_version: new_percentage}
        self.deployment_log.append(f"Promoted canary: {new_version} to {new_percentage}%")
        return {"success": True, "traffic_split": self.current_plan.traffic_split}

    def rollback(self, reason: str):
        """Directly implements Section 6's non-negotiable fast-
        rollback requirement."""
        self.current_plan = None
        self.deployment_log.append(f"ROLLED BACK: {reason}")
        return {"success": True, "action": "rolled_back", "reason": reason}

orchestrator = DeploymentOrchestrator()
# Exactly Section 9's worked developer example, steps 1-3
orchestrator.start_canary("v2.3.0", initial_percentage=5)
orchestrator.promote_canary("v2.3.0", new_percentage=25)
orchestrator.promote_canary("v2.3.0", new_percentage=100)

print("Deployment log:")
for entry in orchestrator.deployment_log:
    print(f"  {entry}")

Expected Output:

Deployment log:
  Started canary: v2.3.0 at 5%
  Promoted canary: v2.3.0 to 25%
  Promoted canary: v2.3.0 to 100%

What this confirms: the orchestrator correctly progresses through the exact staged percentages from Section 9’s real developer example, with a full, auditable log of each promotion — directly implementing Module 11’s staged lifecycle as real, working deployment infrastructure rather than a purely conceptual process.

13. Production Considerations

  • Automate canary promotion/rollback decisions based on real, monitored metrics (Module 12) rather than manual judgment calls — humans are slower and less consistent under deployment pressure
  • Secrets management (Section 7) should support per- environment scoping — a development environment’s credentials should never grant production access

14. Trade-offs

  • Blue-green deployment requires running two full environments simultaneously during the cutover window — a real, temporary infrastructure cost in exchange for a very fast rollback
  • Canary deployment’s gradual promotion takes longer to reach full rollout than a direct deployment — a real, worthwhile trade-off for the risk reduction it provides

15. Chapter Summary

Deployment infrastructure is what turns Module 11’s staged evaluation lifecycle into real, running traffic management — canary deployments gradually shift real traffic to a new version while monitoring health, blue-green deployments enable near-instant rollback via parallel environments, and configuration/secrets management keeps environment- specific and sensitive values out of application code.

A fast, reliable rollback path is a non-negotiable requirement for any deployment strategy, directly extending Module 14’s reliability principles to the deployment process itself.

16. Visual Cheat Sheet

Canary:      5% -> 25% -> 100% (gradual, metric-gated)
Blue-Green:  parallel environments, ONE cutover, instant rollback

Config/secrets: EXTERNALIZED, never hardcoded, scoped per environment

17. Top Takeaways

  1. Deployment infrastructure implements Module 11’s staged evaluation lifecycle as real, running traffic management.
  2. Canary deployment gradually shifts real traffic while monitoring health, promoting or rolling back based on real metrics.
  3. Blue-green deployment enables near-instant rollback via parallel, already-running environments.
  4. Configuration and secrets must be externalized from code, never hardcoded.
  5. A fast, reliable rollback path is non-negotiable for any real deployment strategy.

18. Interview Questions

Q: 1. Explain how canary deployment directly implements the staged evaluation lifecycle covered earlier in this course.**

Ans: Canary deployment routes a small percentage of real traffic to a new version first, monitors health metrics, and gradually increases that percentage only if metrics stay healthy — exactly the graduated risk exposure the staged evaluation lifecycle (offline eval, shadow, canary, full rollout) describes.

Canary deployment is the concrete infrastructure that makes the “canary” stage of that lifecycle actually happen with real traffic.

  • Why it matters: Understanding this connection shows the candidate can translate an evaluation concept into deployable infrastructure.
  • Real-world example: Section 9’s TechCorp rollout — 5% → 25% → 100%.
  • Common mistake: Treating the evaluation lifecycle and deployment infrastructure as unrelated concerns.
  • Interviewer is testing: Whether the candidate connects evaluation theory to deployment practice.
  • Likely follow-up: “What would trigger an automatic rollback during a canary deployment?” → A monitored metric (error rate, latency, or evaluation score) exceeding a defined threshold relative to the current baseline version.

Q: 2. Why is a fast, reliable rollback path considered non-negotiable for any deployment strategy, especially for AI systems?**

Ans: AI systems can fail in ways that are harder to predict in advance than traditional software (Module 2’s non-determinism) — a change that passed offline evaluation can still reveal a problem only visible under real production traffic.

Without a fast rollback path, a bad deployment becomes a prolonged incident rather than a quickly-contained one, directly extending Module 14’s reliability principles to the deployment process itself.

  • Why it matters: This is one of the highest-leverage investments for limiting the blast radius of an inevitable bad deployment.
  • Real-world example: Section 4’s blue-green pattern — the previous version never stops running, so rollback is just a traffic switch, not a redeploy.
  • Common mistake: Optimizing deployment speed and automation while treating rollback as an afterthought.
  • Interviewer is testing: Whether the candidate treats rollback as a first-class deployment requirement, not a nice-to-have.
  • Likely follow-up: “How would you decide between blue-green and canary for a given system?” → Canary offers more granular, gradual risk exposure; blue-green offers a faster, cleaner full cutover and rollback — the right choice depends on the project’s risk tolerance and traffic patterns (Module 23’s decision framework).

19. Scenario-Based Question

Scenario: TechCorp deploys a new model version directly to 100% of traffic (no canary stage) because the team was confident after strong offline evaluation results. Within 20 minutes, error rates spike significantly due to an issue offline evaluation didn’t catch — a edge case triggered only by a specific, real production query pattern.

The team scrambles to identify how to revert, since there’s no defined rollback procedure.

  • Problem Analysis: Section 11’s two common mistakes, compounded — no canary stage AND no fast rollback path.
  • How to Think: Two separate, avoidable gaps combined to turn a contained, small-blast-radius issue into a full, significant incident.
  • Investigation: Confirm the deployment process skipped canary staging entirely, and that no automated or well-documented rollback procedure existed.
  • Root Cause: Missing canary deployment infrastructure (Section 5) meant the entire user base was exposed simultaneously; missing rollback infrastructure (Section 6) meant reverting took longer than necessary once the problem was identified.
  • Solution: Implement Section 12’s canary orchestrator for all future deployments; establish a tested, fast rollback procedure (ideally automated, triggered by monitored metric thresholds) before any future direct-to-production deployment.
  • Trade-offs: Building proper canary and rollback infrastructure takes, upfront engineering investment — a real, worthwhile cost given this incident’s actual impact and duration.
  • Production Considerations: This scenario directly demonstrates why Section 6 treats rollback as non-negotiable — even with a canary stage, a slow rollback path prolongs any incident that does occur; both gaps compound each other’s real impact.

20. Next Step

Next: Module 26 — AI CI/CD — how AI changes the traditional CI/CD pipeline: prompt tests, evaluation tests, dataset validation, and deployment gates, assembled into one complete pipeline.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed