TechByteByByte

Architecture Decision Making

Closing Level 8: the complete, senior-engineer decision framework spanning scale, latency, cost, security, reliability, data sensitivity, and team expertise — with decision matrices for real trade-offs.

#AI Engineering#Architecture Decisions#Level 8

Begin with the problem

Architecture is not selected by fashion. It is selected by making priorities visible, comparing options against them, and documenting why one trade-off fits this project.

requirements → weighted priorities → candidate architectures → evidence → decision record

What you will learn

  • Turn quality, cost, latency, security, scale, and team constraints into explicit priorities.
  • Use decision matrices without hiding judgment behind numbers.
  • Record assumptions, rejected alternatives, and conditions for revisiting a decision.

Current production grounding: OpenAI’s Evals documentation shows dataset- and grader-based evaluation for model applications.

Current production grounding: Google’s Gemini tools documentation distinguishes managed built-in tools from custom functions executed by the application.

These links document current public features and practices. They do not reveal a provider’s private implementation or guarantee that every product uses identical defaults.

1. The Engineering Problem

Every module so far taught you individual decisions — model selection (Module 4), RAG vs. fine-tuning (Module 21), workflow vs. agent (Module 22). This module closes Level 8 with the meta-skill: how a senior engineer weighs ALL of these dimensions together for a specific, real project — because the “right” architecture isn’t universal, it depends on what a specific project actually, prioritizes.

2. Why There’s No Universal “Best” Architecture

A STARTUP shipping fast with NO sensitive data yet prioritizes speed-to-market and team expertise over security
hardening.

A REGULATED BANK prioritizes security and data sensitivity
above almost everything else, even at real cost to speed or team
familiarity.

The SAME architectural options (hosted API vs. self-hosted, Module
4) produce OPPOSITE correct answers for these two
projects.

This is the core insight this module builds toward: a “best practice” recommendation without knowing a project’s ACTUAL priorities is incomplete advice — the right architecture decision requires knowing what THIS project weighs most.

3. The Decision Factors

FactorWhat It Captures
ScaleCurrent and expected future request volume (Module 17)
LatencyHow time-sensitive the user experience is (Module 16)
CostHow cost-sensitive the project is at its expected volume (Module 15)
SecurityHow much risk a security incident would represent (Module 13)
ReliabilityHow much impact downtime or errors would have (Module 14)
Data sensitivityWhether the data involved is regulated, private, or sensitive (Module 4, 13)
Operational complexityHow much ongoing maintenance burden the team can absorb
Team expertiseWhat the team already knows how to operate well
Build vs. buyWhether building custom infrastructure is worth it vs. using an existing service

4. Building a Decision Matrix

1. Score EACH candidate architecture on EACH factor (Section 3),, on a consistent scale (e.g., 1-5)

2. WEIGHT each factor by how much THIS SPECIFIC project cares about it -- weights should sum to 1.0

3. Compute a WEIGHTED score per candidate -- the highest
   score is the recommended choice, FOR THIS PROJECT'S priorities

The critical step is 2 — the weights are NOT universal. A senior engineer’s real skill here is correctly eliciting a project’s ACTUAL priorities (often from stakeholders who haven’t explicitly articulated them) before ever scoring the options.

5. A Real-World Analogy — The Bank, Once More

Module 13's bank analogy: a bank weighs security
and compliance FAR more heavily than a startup prototyping a new
feature -- not because the bank's engineers are more cautious
PEOPLE, but because a security incident's cost (regulatory
penalties, customer trust) is categorically higher for
a bank than for an early prototype with no real users yet.

The SAME weighted-decision discipline a bank applies to its
architecture choices is EXACTLY what Section 4's matrix formalizes.

6. A worked developer example

TechCorp’s decision matrix, comparing hosted API vs. self-hosted models (Module 4) for two different, real projects:

FactorStartup’s WeightBank’s Weight
Team expertise0.50 (dominant priority)0.05
Security0.050.35 (dominant priority)
Data sensitivity0.050.30

With these different weights applied to the SAME candidate architectures, the startup’s weighted score favors the hosted API, while the bank’s weighted score favors self-hosting — exactly Section 2’s point, made concrete and computed rather than asserted.

7. How Is This Used in the Industry?

🤖 How Is This Used in the Industry?

Senior AI engineers and architects conduct this weighting exercise explicitly — often in a design document — before committing to a major architectural decision, precisely because it makes the trade-offs and priorities visible and debatable, rather than leaving them as an unstated, individual engineer’s intuition.

8. Common Mistakes

Incorrect idea: Applying a “best practice” architecture without checking whether it matches THIS project’s actual priorities.

Why it is incorrect: As shown directly in Section 2, the same architecture can be right for one project and wrong for another.

Incorrect idea: Leaving priority weights implicit and unstated.

Why it is incorrect: As shown directly in Section 4, this makes the decision harder to debate, review, or revisit later.

Incorrect idea: Scoring candidates without honestly weighing all relevant factors

Why it is incorrect: — e.g., ignoring team expertise entirely because it feels like a “soft” factor. As shown directly in Section 6, this is a real decision input.

9. Code — A Weighted Architecture Decision Matrix

What this shows: implementing Section 4’s weighted decision process directly — working code that produces different recommendations for different project priorities, exactly Section 6’s worked developer example made concrete and computed.

from dataclasses import dataclass
from enum import Enum

class DecisionFactor(Enum):
    SCALE = "scale"
    LATENCY = "latency"
    COST = "cost"
    SECURITY = "security"
    DATA_SENSITIVITY = "data_sensitivity"
    TEAM_EXPERTISE = "team_expertise"

@dataclass
class ArchitectureOption:
    name: str
    scores: dict  # factor -> 1-5, higher is better on that factor

def weighted_decision(options: list, weights: dict) -> dict:
    """A weighted decision matrix (Section 4) -- different
    real projects weight factors differently, and the
    right architecture choice depends on those weights,
    not a universal ranking."""
    results = {}
    for option in options:
        score = sum(option.scores.get(factor, 0) * weight for factor, weight in weights.items())
        results[option.name] = round(score, 2)
    return results

hosted_api = ArchitectureOption("Hosted API Model", {
    DecisionFactor.SCALE: 4, DecisionFactor.LATENCY: 4, DecisionFactor.COST: 3,
    DecisionFactor.SECURITY: 3, DecisionFactor.DATA_SENSITIVITY: 2, DecisionFactor.TEAM_EXPERTISE: 5,
})
self_hosted = ArchitectureOption("Self-Hosted Model", {
    DecisionFactor.SCALE: 5, DecisionFactor.LATENCY: 3, DecisionFactor.COST: 4,
    DecisionFactor.SECURITY: 5, DecisionFactor.DATA_SENSITIVITY: 5, DecisionFactor.TEAM_EXPERTISE: 2,
})

# Exactly Section 6's two real projects -- different weights
startup_weights = {
    DecisionFactor.SCALE: 0.1, DecisionFactor.LATENCY: 0.15, DecisionFactor.COST: 0.15,
    DecisionFactor.SECURITY: 0.05, DecisionFactor.DATA_SENSITIVITY: 0.05, DecisionFactor.TEAM_EXPERTISE: 0.5,
}
bank_weights = {
    DecisionFactor.SCALE: 0.1, DecisionFactor.LATENCY: 0.1, DecisionFactor.COST: 0.1,
    DecisionFactor.SECURITY: 0.35, DecisionFactor.DATA_SENSITIVITY: 0.3, DecisionFactor.TEAM_EXPERTISE: 0.05,
}

startup_result = weighted_decision([hosted_api, self_hosted], startup_weights)
bank_result = weighted_decision([hosted_api, self_hosted], bank_weights)

print(f"Startup's priorities -> {startup_result}")
print(f"Bank's priorities -> {bank_result}")

Expected Output:

Startup's priorities -> {'Hosted API Model': 4.2,
'Self-Hosted Model': 3.05}
Bank's priorities -> {'Hosted API Model': 3.0, 'Self-Hosted
Model': 4.55}

What this confirms: The same two candidates produce opposite recommendations when the project priorities change. The hosted API wins for the startup because team expertise receives more weight.

Self-hosting wins for the bank because security and data sensitivity receive more weight. The calculation makes Section 2’s central idea concrete instead of leaving it as an abstract claim.

10. Production Considerations

  • Document the weights and scores used for a real architectural decision — this helps future engineers understand WHY a choice was made, and whether changed priorities (Module 20’s feedback loop) warrant revisiting it
  • Revisit weights periodically — a startup’s priorities (Section 6) shift as it scales and accumulates real sensitive data

11. Trade-offs

  • A formal weighted matrix takes, real time to construct properly — worthwhile for significant, hard-to-reverse architectural decisions, less necessary for small, easily-reversible ones
  • Scores are subjective estimates, not precise measurements — the matrix’s value is in making trade-offs EXPLICIT and DEBATABLE, not in producing a mathematically perfect answer

12. Chapter Summary

There is no universal “best” AI system architecture — the right choice depends on a specific project’s actual priorities across scale, latency, cost, security, reliability, data sensitivity, operational complexity, and team expertise. A weighted decision matrix — scoring candidates on each factor, then weighting by what THIS project cares about — makes these trade-offs explicit and debatable, rather than leaving them as an individual engineer’s unstated intuition.

This is the meta-skill senior AI engineers apply, tying together every module-specific decision covered throughout this course.

13. Visual Cheat Sheet

Score each candidate on: scale, latency, cost, security,
reliability, data sensitivity, team expertise, operational complexity

Weight each factor by THIS PROJECT'S priorities
(weights sum to 1.0)

Highest weighted score = recommended choice, FOR THIS PROJECT
(not universally)

14. Top Takeaways

  1. There is no universal “best” AI system architecture — the right choice depends on a specific project’s real priorities.
  2. A weighted decision matrix makes trade-offs explicit and debatable, rather than leaving them as unstated intuition.
  3. The same two candidate architectures can produce opposite correct recommendations for different projects.
  4. Team expertise is a real decision factor — not a “soft” consideration to dismiss.
  5. Document weights and scores for significant architectural decisions — this supports future review as priorities shift.

15. Interview Questions

Q: 1. Why might the “best practice” architecture recommendation be wrong for a specific project?**

Ans: Best-practice recommendations are generic — they don’t account for a specific project’s actual priorities.

A security-hardened, self-hosted architecture might be “best practice” in general, but wrong for an early-stage startup with no sensitive data yet and limited infrastructure expertise, where a simpler hosted API is the better fit given that project’s real constraints and priorities.

  • Why it matters: Blindly applying generic best practices without weighing them against a project’s actual, priorities produces real, unnecessary cost or risk in either direction.
  • Real-world example: Section 6’s startup-vs-bank comparison.
  • Common mistake: Treating “best practice” as universally correct rather than context-dependent.
  • Interviewer is testing: Whether the candidate can reason about architecture decisions contextually, not by rote.
  • Likely follow-up: “How would you elicit a project’s priorities before making a recommendation?” → Directly asking stakeholders about their real tolerance for risk, cost, and operational complexity — often surfacing priorities they hadn’t explicitly articulated.

Q: 2. Design a weighted decision matrix approach for choosing between two candidate architectures on a real project. What steps would you take?**

Ans: I’d first identify the relevant decision factors (scale, latency, cost, security, reliability, data sensitivity, team expertise, operational complexity) for this specific decision. I’d score each candidate architecture on each factor, on a consistent scale. Then, critically, I’d work with stakeholders to determine weights reflecting this specific project’s actual priorities, summing to 1.0.

Finally, I’d compute the weighted score per candidate and use the highest-scoring option as the recommendation — while documenting the weights and scores for future review.

  • Why it matters: This process makes an otherwise implicit, individual judgment call into an explicit, reviewable, and revisitable decision.
  • Real-world example: Section 9’s code, applied as a real process rather than just a demonstration.
  • Common mistake: Skipping the explicit weighting step and jumping straight to a recommendation based on general intuition.
  • Interviewer is testing: Whether the candidate can structure a rigorous decision process, not just state an opinion.
  • Likely follow-up: “How would you handle disagreement among stakeholders about the weights?” → surface the disagreement explicitly — differing weights often reveal an underlying disagreement about risk tolerance or priorities worth resolving directly, rather than papering over with an arbitrary compromise.

16. Scenario-Based Question

Scenario: TechCorp’s startup-phase support assistant was architected around a hosted API model, reasonably weighted heavily toward team expertise and speed-to-market at the time. Two years later, TechCorp has scaled significantly and now handles sensitive customer financial data as part of a new product line. A new architecture review is proposed.

  • Problem Analysis: Section 11’s point — the project’s priorities have shifted substantially (data sensitivity is now a real, significant factor it wasn’t before), warranting a re-weighting of the original decision.
  • How to Think: This isn’t evidence the original decision was wrong — it was correct given the priorities and constraints at the time. It’s evidence that priorities have changed enough to warrant revisiting.
  • Investigation: Re-run Section 9’s weighted decision process with updated weights reflecting TechCorp’s current, priorities — data sensitivity and security now weighted far more heavily than at launch.
  • Root Cause: Not a mistake — a natural evolution in project priorities that the original architecture wasn’t re-evaluated against.
  • Solution: Conduct the re-weighted decision matrix (Section 9); if self-hosting or additional security controls now score higher under the updated weights, plan a deliberate migration rather than treating this as an emergency fix.
  • Trade-offs: Migrating architecture at this scale involves real, significant engineering effort — but the alternative is operating with a security/data-sensitivity posture that no longer matches the project’s actual current risk profile.
  • Production Considerations: This scenario directly demonstrates Section 10’s point — documenting the ORIGINAL weights and scores makes this exact re-evaluation conversation easier, since the team can see precisely what changed (the weights) rather than re-deriving the entire decision from scratch.

17. Next Step

Next: Module 24 — AI Testing — Level 9 begins here: unit, integration, contract, prompt, model, evaluation, RAG, agent, and chaos testing — and what can and cannot be made deterministic.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed