How AI Applications Differ from Traditional Software

Begin with the problem

Traditional code follows rules written by developers. A model generates probable outputs, so correctness is often a score or range rather than one exact sentence. That changes how we test and operate the system.

fixed code path → exact test | probabilistic output → criteria + repeated evaluation

What you will learn

• Compare deterministic software with probabilistic model behavior.
• Choose exact assertions for code and graded evaluation for generated content.
• Design for variation without treating every different answer as a failure.

Current production grounding: OpenAI’s Evals documentation shows dataset- and grader-based evaluation for model applications.

Current production grounding: Google’s Gemini tools documentation distinguishes managed built-in tools from custom functions executed by the application.

These links document current public features and practices. They do not reveal a provider’s private implementation or guarantee that every product uses identical defaults.

1. The Engineering Problem

You know how to test traditional software: given input X, assert output equals Y. This assumption — that correctness is a fixed, checkable target — quietly breaks the moment an LLM enters the system. This module makes that break explicit, because every later module in this course (evaluation, testing, reliability) exists specifically to compensate for it.

---

2. Traditional Software vs. AI Software

TRADITIONAL SOFTWARE:

  Input --> Deterministic Logic --> Output
  (same input ALWAYS produces the same output)

AI SOFTWARE:

  Input --> Probabilistic Model --> Output
  (same input CAN produce a DIFFERENT output)

This single structural difference is the root cause of nearly everything this course covers. Retries behave differently. Testing behaves differently. Evaluation has to exist as a separate discipline. None of this is optional complexity — it’s a direct, unavoidable consequence of the input-to-output relationship no longer being fixed.

---

3. Five Sources of Difference

Source	What It Means
Non-determinism	The same prompt can produce different outputs across calls, due to sampling (temperature, top-p)
Model dependency	System behavior depends on a component (the model) you don’t fully control and can’t inspect internally
Context/data dependency	Output quality depends on what was retrieved or included in context — a data problem, not just a code problem
Prompt dependency	Small prompt wording changes can shift behavior in ways no traditional code review would catch
Evaluation difficulty	“Correct” is often a matter of degree (partially helpful, mostly grounded) rather than a binary pass/fail

---

4. Why Traditional Unit Testing Falls Short

# A traditional unit test -- fine for deterministic code
def test_addition():
    assert add(2, 2) == 4

# The SAME pattern, applied naively to an LLM call, is broken
def test_llm_response():
    response = call_llm("Summarize this document.")
    assert response == "This document is about X."  # will FAIL most runs,
                                                       # even when the summary
                                                       # is GOOD

Important clarification: A traditional assert output == expected doesn’t work for most LLM outputs — there are MANY valid phrasings of a good summary. This isn’t a testing failure; it’s a mismatch between the testing PARADIGM and the nature of the system under test. Module 24 covers exactly what CAN and CANNOT be made deterministic in AI testing.

Why it matters: This shortcut removes a validation or control boundary, allowing an error to pass into later stages where it becomes harder and more expensive to detect.

---

5. A Real-World Analogy — The Hospital

A traditional software function is like a VENDING MACHINE: insert
the right coins, the SAME snack comes out, every single time.

An LLM call is more like consulting a DOCTOR: given the SAME
symptoms, two competent doctors might suggest slightly
different, both reasonably valid, treatment plans. You don't test
a doctor's judgment with `assert diagnosis == "flu"` -- you evaluate
whether the diagnosis was REASONABLE, GROUNDED in the symptoms, and
SAFE.

This is why Module 10 (Evaluation) exists as such a large, dedicated part of this course — it’s the AI-system equivalent of peer review, not a simple pass/fail test.

---

6. What Stays the Same

It’s worth being precise about what does NOT change, so you don’t over-correct:

STILL fully testable and deterministic:

  - Input validation logic
  - Retrieval FILTERING logic (e.g., access control checks)
  - Structured-output PARSING and schema validation
  - Retry/timeout/fallback CONTROL FLOW
  - Cost and token-counting logic

The important skill this course builds: knowing WHICH parts of your system are deterministic (test them traditionally) and WHICH parts are probabilistic (evaluate them, don’t just test them). Most real AI systems are a mix of both — Module 24 revisits this split precisely.

---

7. A worked developer example

TechCorp’s support assistant, showing where determinism ends and probabilistic behavior begins:

Component	Deterministic?	How It’s Verified
Parsing the user’s request into a structured ticket	✅ Yes	Traditional unit tests
Retrieving relevant policy documents	Mostly — retrieval logic is deterministic, but relevance quality varies	Retrieval evaluation (Module 10)
Generating the actual response text	❌ No	Evaluation (groundedness, helpfulness)
Validating the response is well-formed JSON before sending to the UI	✅ Yes	Traditional unit tests + schema validation
Deciding whether to escalate to a human	Depends on implementation — a rule-based check is deterministic; an LLM judgment call is not	Mixed — test the rule, evaluate the judgment

---

8. How Is This Used in the Industry?

🤖 How Is This Used in the Industry?

Mature AI engineering teams explicitly draw this boundary in their architecture and their test suites — deterministic components get traditional CI tests with hard assertions; probabilistic components get evaluation pipelines with graded scores and thresholds, run separately and treated as a different kind of quality gate.

---

9. Common Mistakes

Incorrect idea: Writing assert response == expected_text for LLM-generated content.

Why it is incorrect: As shown directly in Section 4, this fails most runs regardless of actual quality.

Incorrect idea: Assuming non-determinism means “untestable.”

Why it is incorrect: As shown directly in Section 6, large parts of a real AI system remain deterministic and should be tested normally.

Incorrect idea: Treating a single successful test run as proof of reliability.

Why it is incorrect: This shortcut removes a validation or control boundary, allowing an error to pass into later stages where it becomes harder and more expensive to detect. Because behavior can vary across runs, a single pass proves less than it would for deterministic code — Module 11 covers regression testing for probabilistic systems.

---

10. Code — Demonstrating Determinism vs. Non-Determinism

What this shows: a direct, side-by-side comparison of a deterministic function against a simulated LLM call with sampling — making Section 2’s core distinction observable rather than just asserted.

import random

def deterministic_add(a: int, b: int) -> int:
    """Traditional software: SAME input always produces the SAME
    output., provably deterministic."""
    return a + b

def probabilistic_llm_call(prompt: str, temperature: float = 0.7) -> str:
    """Simulates an LLM call -- even with the IDENTICAL prompt, the
    output can vary because sampling introduces randomness
    (this mirrors real temperature/top-p sampling, simplified)."""
    responses = ["The answer is 42.", "The result is forty-two.", "It's 42."]
    return random.choice(responses)

# Deterministic: run 3 times, ALWAYS the same result
det_results = [deterministic_add(20, 22) for _ in range(3)]
print(f"Deterministic function, 3 runs: {det_results}")

# Probabilistic: run 3 times with the SAME prompt -- may differ
random.seed(1)  # fixed only so THIS example is reproducible for the reader
prob_results = [probabilistic_llm_call("What is 20+22?", temperature=0.7) for _ in range(3)]
print(f"Probabilistic call (temp=0.7), 3 runs: {prob_results}")

Expected Output:

Deterministic function, 3 runs: [42, 42, 42]
Probabilistic call (temp=0.7), 3 runs: ['The answer is 42.', "It's
42.", 'The answer is 42.']

What this confirms: the deterministic function produces the identical result on every run, while the simulated LLM call produces different phrasings across runs despite an identical prompt — exactly Section 2’s structural distinction, made directly observable.

Note both answers are arguably “correct,” which is precisely why assert output == expected (Section 4) is the wrong testing tool here.

---

11. Production Considerations

• Non-determinism complicates debugging: “it worked when I tried it” is weaker evidence for AI systems than for deterministic code, since the same input might not reproduce the same output
• Setting temperature=0 reduces (but does not eliminate) variability for some models — useful for testing, but not a complete substitute for evaluation (Module 10)

---

12. Trade-offs

• Lower temperature increases reproducibility but can reduce output quality or creativity for tasks that benefit from variation (e.g., brainstorming) — the right setting depends on the task
• Treating everything as “needs evaluation, not testing” is wasteful — Section 6’s deterministic components should stay in fast, traditional test suites

---

13. Chapter Summary

AI applications differ from traditional software in one structural way that cascades into everything else: the input-to-output relationship is probabilistic, not fixed. This makes traditional assert output == expected testing insufficient for model-generated content, while leaving large parts of a real system — validation, parsing, control flow — just as deterministic and testable as ever.

The core AI Engineering skill this module introduces: correctly identifying which parts of your system are which, and applying the right quality-assurance approach to each.

---

14. Visual Cheat Sheet

DETERMINISTIC parts  -->  traditional unit tests, hard assertions
PROBABILISTIC parts  -->  evaluation pipelines, graded scores,
                          thresholds (Module 10)

Same input, same output   = deterministic  = TEST it
Same input, varying output = probabilistic = EVALUATE it

---

15. Top Takeaways

1. Traditional software is deterministic; AI software is probabilistic — this is the root cause of most AI-specific engineering practices.
2. assert output == expected fails for most LLM-generated content, even when the content is good.
3. Large parts of a real AI system remain fully deterministic and should be tested traditionally.
4. Evaluation (Module 10) exists specifically to fill the gap traditional testing leaves for probabilistic components.
5. temperature=0 reduces but doesn’t eliminate variability — it’s a testing aid, not a substitute for evaluation.

---

16. Interview Questions

Q: 1. Why does assert response == expected_text fail as a testing strategy for LLM-generated content, even when the model is performing well?**

Ans: Because the same prompt can produce multiple valid, differently-worded outputs due to sampling — a good response might use different phrasing than the exact expected string, causing the assertion to fail despite acceptable quality.

• Why it matters: Teams that don’t understand this waste effort chasing “flaky tests” that are actually working as designed, or worse, hard-code overly rigid expectations that produce false negatives.
• Real-world example: A team writes assert summary == "This document discusses X." and the test fails when the model correctly produces “This document is about X.” — a equivalent, valid summary.
• Common mistake: Trying to fix this by making prompts more rigid rather than switching to evaluation.
• Interviewer is testing: Whether the candidate understands the structural reason traditional testing patterns don’t transfer directly to LLM outputs.
• Likely follow-up: “What would you use instead?” → LLM-as-judge or rubric-based evaluation (Module 10), checking properties like groundedness and relevance rather than exact string match.

Q: 2. Give an example of a component in an AI system that should still be tested with traditional, deterministic unit tests.**

Ans: Input validation, structured-output schema validation, retry/timeout control flow, cost/token-counting logic, and any rule-based routing logic (e.g., “if user is unauthenticated, reject the request”) — all fully deterministic and testable exactly like any traditional software.

• Why it matters: Recognizing this prevents the opposite mistake — assuming an entire AI system is “untestable” and skipping quality assurance on the deterministic parts.
• Real-world example: A JSON schema validator checking an LLM’s structured output is fully deterministic — the SAME malformed JSON should ALWAYS fail validation, and this is worth a standard unit test.
• Common mistake: Treating “this system uses AI” as license to skip normal software testing discipline entirely.
• Interviewer is testing: Whether the candidate can correctly draw the boundary between deterministic and probabilistic components in a real system.
• Likely follow-up: “How would you structure your test suite to reflect this split?” → Separate CI stages: fast, traditional unit tests for deterministic components; a slower, separately-gated evaluation suite for probabilistic ones.

---

17. Scenario-Based Question

Scenario: A new engineer joins TechCorp and writes a CI test that asserts the AI assistant’s exact response text for 20 sample questions. The test suite is now flaky — failing roughly 30% of the time even though the responses look correct when reviewed manually.

• Problem Analysis: The engineer applied Section 4’s traditional testing pattern to a probabilistic component.
• How to Think: Flakiness here isn’t a bug in the system — it’s the test itself using the wrong paradigm for this kind of output.
• Investigation: Manually review the “failed” cases — are the responses actually wrong, or just differently worded but equally valid?
• Root Cause: Exact-string assertions applied to non-deterministic, model-generated text.
• Solution: Replace exact-match assertions with evaluation criteria — does the response contain the required facts, is it grounded in the retrieved context, does it pass an LLM-as-judge rubric check (Module 10) — with a pass/fail threshold rather than exact equality.
• Trade-offs: Evaluation-based checks are slower and more complex to set up than a string comparison, but reflect real quality rather than producing false failures.
• Production Considerations: This exact mistake is common enough that it’s worth explicitly training new team members on the deterministic/probabilistic split from Section 6 before they write their first AI-system test.

---

18. Next Step

Next: Module 3 — AI Application Architecture — the complete layered system design (client through observability) that every subsequent module in this course will reference by name.