Interview Questions

Before you continue: three tools for this module

• Token: a piece of text processed by the model.
• Parameter: a learned number controlling the model’s transformations.
• Inference: using the trained model without updating its parameters.

You do not need to memorize these yet. Use this map when the terms reappear.

Begin with the central question

What hidden problem does Interview Questions solve inside a real language-model system?

Keep that central question about Interview Questions in mind. The definitions, numbers, diagrams, and code examples below answer it one piece at a time.

question → definition → mechanism → example → limitation → production connection

1. What This Module Does

Learning outcomes

• Answer LLM questions using a clear definition, intuition, mechanism, and example.
• Explain important distinctions without relying on memorized slogans.
• Discuss limitations, failure modes, and production tradeoffs.
• Structure scenario answers so another engineer can follow the reasoning.

In one sentence

💡 Big picture

This module helps you practise explaining LLM ideas clearly, using mechanisms, examples, limitations, and production tradeoffs instead of memorized definitions.

---

2. Why This Module Exists

The problem this module solves

• Interviews test how you think and communicate, not only what terms you remember.
• A repeatable answer structure helps you stay clear even when several topics appear in one unfamiliar question.

---

Beginner-Level Questions

Q: What exactly happens when you send a prompt to an LLM?

Ans: The prompt is tokenized into sub-word units (Module 2), converted to token IDs, looked up in an embedding table and combined with positional information (Module 4), processed through stacked Transformer blocks using causal self-attention (Module 10-11), and the final hidden state is projected through the LM head into logits (Module 5) — a raw score per vocabulary token.

Softmax converts these into a probability distribution, a token is selected (Module 15), appended, and the entire process repeats (Module 7) until a stopping condition is met.

Q: Why do LLMs use tokens instead of words?

Ans: Sub-word tokenization (Module 2, building on the NLP course) keeps vocabulary size manageable while gracefully handling words never seen during training by decomposing them into smaller, familiar pieces — whole-word tokenization would require an impractically large vocabulary and fail entirely on new words.

Q: What is the difference between pretraining and fine-tuning?

Ans: Pretraining (Module 8) starts from random weight initialization and trains on a massive, general corpus using next-token prediction, to build broad language capability.

Fine-tuning (Module 16) starts from those pretrained weights and continues training on a much smaller, task/domain-specific dataset, adapting rather than building capability from scratch.

---

Intermediate-Level Questions

Q: Why can an LLM generate coherent text even though it’s trained only for next-token prediction?

Ans: The chain rule of probability (Module 6) means the probability of an entire sequence can be decomposed into a product of one-token-at-a-time conditional probabilities.

Training the model to predict each next token well, across an enormous amount of text, teaches it the statistical patterns of coherent language — generating a full response is simply repeating this well-trained prediction step (Module 7), nothing more.

Q: Why does temperature affect randomness in generation?

Ans: Temperature divides logits by a value before softmax (Module 15) — values below 1 sharpen the resulting probability distribution (amplifying relative differences before exponentiation), while values above 1 flatten it.

Verified directly: lower temperature produced noticeably lower entropy (a sharper, more confident distribution) than higher temperature on the identical underlying logits.

Q: What is the difference between RAG and fine-tuning?

Ans: Fine-tuning (Module 16) bakes patterns into the model’s weights through additional training, well-suited for consistent style/format but poorly suited for frequently-changing information.

RAG (Module 20) retrieves relevant, current context from an external, easily-updatable knowledge source at query time, without any training — better suited for information that changes regularly.

---

Advanced-Level Questions

Q: Why do decoder-only models use causal attention?

Ans: The model’s task — predicting the next token given everything before it — requires that no position ever “see” tokens that come after it, since during real generation, those future tokens genuinely don’t exist yet (Module 11).

Causal masking enforces this during training (where the full sequence is technically present but must be masked) and is automatically satisfied during inference (Module 7) by the sequential nature of generation itself. Verified directly: a causal mask produces a strictly triangular attention pattern, where each position can only attend to itself and earlier positions.

Q: What is KV cache and why does it improve inference?

Ans: The KV cache (Module 14) stores computed Key and Value vectors for already-processed tokens, avoiding redundant recomputation at every decode step — since causal masking (Module 11) guarantees these vectors never depend on future tokens, they never need to change once computed.

Verified directly: caching reduced total decode FLOPs by over 127x for a realistic 50-token generation scenario compared to naively recomputing the entire sequence’s attention at every single step.

Q: Why does an LLM hallucinate?

Ans: Next-token prediction (Module 5) always produces a full probability distribution and selects a token using the exact same mechanism, regardless of whether the model’s training data contained strong, reliable signal on the specific topic.

There’s no separate “I genuinely don’t know this” pathway distinct from confident, correct prediction — verified directly, a query about a likely fabricated fact still produced a substantially confident (70.81%) top answer through the identical mechanism used for well-established facts.

---

Scenario-Based Questions

Q: A team’s chatbot seems to “forget” details mentioned early in a long conversation. Diagnose this.

Thought process: This connects directly to context window management (Module 3) rather than any reasoning failure.

Investigation: As conversations grow, total token usage can approach or exceed the context window limit — verified directly in Module 3, a growing conversation eventually required truncating the oldest turns to fit a new message within budget.

Correct answer: This is very likely a context management issue — either naive oldest-first truncation dropping genuinely relevant early details, or the context window being exceeded entirely.

The fix is a more deliberate context management strategy: summarization of older turns, or retrieval (RAG, Module 20) of specifically relevant earlier context when needed, rather than relying purely on recency-based truncation.

Production consideration: Accurate, model-specific token counting (Module 2) is essential for reliably managing this budget.

---

Q: A production RAG system occasionally produces hallucinated claims despite retrieving relevant documents. What would you investigate?

Thought process: RAG substantially reduces but doesn’t structurally eliminate hallucination risk (Module 21).

Investigation: Check whether the specific hallucinated claim can be traced to (or is contradicted by) the actually retrieved documents — this distinguishes a retrieval failure (Module 20’s retrieval-failure case) from a generation-grounding failure (the model misrepresenting or extrapolating beyond what was actually retrieved).

Correct answer: If retrieval found relevant documents but the model still fabricated an unsupported claim, this points to a generation- grounding issue — potentially addressed with prompting that more strongly anchors the model to only use provided context. If retrieval itself failed to find relevant documents, this points to a retrieval quality issue instead.

Production consideration: RAG evaluation (Module 23) should assess retrieval and generation quality separately for exactly this kind of diagnosis.

---

LLMs — What You Should Know Now

The complete mental model, in one page:

An LLM is a decoder-only Transformer (the exact architecture from the Transformers course), trained at large scale across data, parameters, and compute. It processes text by tokenizing it into sub-word units, converting these to embeddings, and passing them through stacked Transformer blocks using causal self-attention — each position only ever attending to itself and earlier positions.

The final layer’s output is projected through an LM head into logits — a raw score per vocabulary token — which softmax converts into a genuine probability distribution.

A token is selected from this distribution (greedy, or via temperature/top-k/top-p sampling), appended to the sequence, and the entire process repeats for each subsequent token — generation is fundamentally iterative, one full forward pass per token.

The model’s weights are learned through pretraining — the same forward-pass/loss/backpropagation/gradient-descent loop from your Neural Networks and Optimization courses, applied to next-token prediction as a self-supervised objective across massive amounts of raw text, requiring no manual labeling.

This produces a base model capable of plausible text continuation, but not reliably helpful, instruction- following behavior — instruction tuning (supervised fine-tuning on curated instruction-response pairs) closes this gap. RLHF and DPO further align the model using human preference data, targeting nuanced quality dimensions that demonstration alone doesn’t capture.

At inference time, real serving systems distinguish prefill (processing the full prompt once, quadratic cost) from decode (generating tokens one at a time, made practical by the KV cache, which avoids redundant recomputation). Production systems apply further optimization — quantization, continuous batching, speculative decoding, distillation — to manage cost and latency at scale.

LLMs have genuine, structural limitations: hallucination (no built-in truth-detector), context limits (quadratic attention cost), knowledge cutoff (fixed training data), reasoning limitations (sequential generation, no explicit planning step), and prompt injection risk (uniform processing of all input text).

None of these are bugs to be patched away entirely — they’re consequences of the underlying mechanism, requiring deliberate mitigation (RAG, verification, careful system design) rather than assumption that “a better model” alone solves them.

Choosing between prompting, fine-tuning, and RAG — and combining them deliberately — is a first-order practical decision for any real system, as is choosing between open-weight self-hosted and closed/API-based deployment.

A modern GenAI system assembles these pieces — prompt, optional RAG, the LLM, optional tool calling — into a working whole; an agent is this same architecture run iteratively, the natural next step in this learning path.

---

LLM Cheat Sheet

Term	One-line definition
Token	The actual unit of LLM processing — often a sub-word piece
Context window	Maximum tokens (input + output) a model can process per request
Embedding	A learned vector representation of a token
Logits	Raw, unbounded scores over the vocabulary, before softmax
Softmax	Converts logits into a genuine probability distribution
Autoregressive	Generating one token at a time, each conditioned on all previous tokens
Causal attention	Attention masked so no position sees future positions
KV cache	Stored Key/Value vectors reused across decode steps
Prefill	Processing the full prompt in one initial forward pass
Decode	Generating new tokens one at a time
Perplexity	exp(average cross-entropy loss) — lower is better
Temperature	Rescales logits before softmax — controls randomness
Top-k / Top-p	Restrict token selection to a subset of the distribution
Pretraining	Initial, massive-scale, self-supervised training
Fine-tuning	Further training on a smaller, task-specific dataset
Instruction tuning	Fine-tuning on (instruction, response) pairs
RLHF	Reward model + reinforcement learning, aligning to human preference
DPO	Direct preference optimization — simpler alternative to RLHF
Hallucination	Confidently generated, factually incorrect content
Quantization	Reducing weight precision to save memory/improve speed
RAG	Retrieval-augmented generation — grounding via retrieved context

---

LLM Interview Questions — Final Bank

Q: What makes an LLM “large”?

Ans: The combination of large training data, large parameter count, and large training compute together (Module 1, 12-13) — not any single dimension alone.

Q: Why is “the model looked up the answer” an inaccurate description of how an LLM produces a factual response? A: Verified directly (Module 1, 5): the model always produces a full probability distribution over its entire vocabulary via the same mechanism, regardless of whether the answer is well-known — there’s no separate lookup or retrieval step in the base generation mechanism.

Q: Why does scaling show diminishing returns?

Ans: Verified directly (Module 13): each additional order-of-magnitude increase in compute produces a progressively smaller loss improvement than the previous one, following an approximately power-law relationship.

Q: Why is DPO considered simpler to implement than RLHF?

Ans: DPO (Module 19) optimizes directly on preference pairs using a single, supervised-style loss function and ordinary gradient descent — no separate reward model training stage, no full reinforcement learning loop, unlike RLHF’s (Module 18) two-stage pipeline.

Q: Why can’t perplexity alone determine whether a model is suitable for a production assistant application? A: Perplexity (Module 6, 23) measures statistical prediction quality — not helpfulness, safety, or instruction-following behavior, dimensions it simply isn’t designed to capture.

Q: What’s the practical difference between prefill and decode costs?

Ans: Prefill scales quadratically with prompt length (verified directly, Module 14: doubling length quadrupled cost); decode, with KV caching, scales much more favorably per new token (verified directly: 127x reduction from caching in a realistic scenario).

Q: Why is prompt injection a structural risk rather than a rare edge case? A: The Transformer architecture (Module 10-11, 22) processes all input text uniformly — there’s no built-in mechanism distinguishing trusted instructions from untrusted content, making this a genuine, systematic concern for any system processing external content through an LLM.

Q: How would you decide between self-hosting and using an API-based model for a new application? A: Evaluate data privacy requirements (a potential hard constraint), expected usage volume (favoring self-hosting at high, sustained scale), customization depth needed, and deployment complexity tolerance (Module 25) — a genuine, multi-factor decision, not a default choice.

---

You have now completed the Large Language Models course — from tokenization through embeddings, the Transformer architecture, pretraining, next-token prediction, inference, fine-tuning, alignment, and production deployment — with every mechanism traced, and most verified with real, executed code.

You’re prepared for the upcoming Agentic AI, LangChain, and LangGraph material, which builds directly on the complete GenAI system architecture established in Module 27.