Generative AI Interview Questions: 30 Questions with Answers

A GenAI interview is a bit like a viva at the end of a college lab course. The examiner skips textbook definitions and asks what you did, why, and what you would change.

Learn each short answer below, then attach a real example from your own work.

How Generative AI interviews are structured

A Generative AI interview checks whether you understand how large language model applications work, can build and evaluate them, and can reason about quality, cost and safety trade-offs.

Typical loops include a Python or SQL coding round, a concepts round like the one below, a project deep-dive and, for experienced roles, system design. Strong answers follow a pattern: one-line definition, mechanism, trade-off, example.

LLM fundamentals

1. What is a large language model?

A neural network, usually a decoder-only transformer, trained on very large text corpora to predict the next token. See what an LLM is.

2. What is a token, and why does it matter?

A token is a chunk of text, often a word piece, that the model reads and writes. Context limits, latency and pricing are all counted in tokens; Hindi often needs more tokens per word than English.

3. What does self-attention do?

Each token builds a query and compares it with every earlier token's key; the scores weight those tokens' values.

4. What is the difference between temperature and top-p?

Temperature reshapes the whole probability distribution: low is focused, high is varied. Top-p keeps only the smallest set of tokens whose probabilities sum to p, then samples among them. Usually tune one, not both.

5. Why do LLMs hallucinate, and how do you reduce it?

The model generates plausible text, not verified facts, so it fills gaps confidently. Reduce it with retrieval grounding, citations, permission to say "I don't know" and faithfulness checks.

Prompting and context

6. What is the difference between zero-shot, few-shot and chain-of-thought prompting?

Zero-shot gives only instructions. Few-shot adds worked examples to show format and style. Chain-of-thought asks the model to reason step by step before answering, trading tokens for accuracy.

7. How do you get reliable JSON output from an LLM?

Use the provider's structured-output or JSON mode with a schema, validate with a library such as Pydantic, and retry with the validation error on failure.

8. What is prompt injection?

Text that tries to override the app's instructions, from a user or a retrieved document. Defend with least-privilege tools, permission checks in code, separating untrusted content and filtering outputs, as in prompt injection defence.

Embeddings and RAG

9. What are embeddings, and how is similarity measured?

Vectors that place texts with similar meaning close together, usually compared by cosine similarity: the dot product divided by the product of the lengths.

import numpy as np

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

refund = np.array([0.9, 0.1, 0.3])
money_back = np.array([0.8, 0.2, 0.35])
weather = np.array([0.1, 0.9, 0.0])
print(f"refund vs money back: {cosine(refund, money_back):.2f}")
print(f"refund vs weather   : {cosine(refund, weather):.2f}")

# a favourite whiteboard question: KV cache size for one 4,000-token chat
layers, kv_heads, head_dim, bytes_per = 32, 8, 128, 2
print(f"KV cache: {2 * layers * kv_heads * head_dim * bytes_per * 4000 / 2**20:.0f} MiB")

10. What is RAG, and why use it?

Retrieval-augmented generation fetches relevant documents at question time and puts them in the prompt. It adds fresh, private knowledge with citations and no retraining; see what RAG is.

11. How do you choose a chunk size?

Small chunks retrieve precisely but lose context; large ones waste tokens. Start at a few hundred tokens with overlap, split on headings, then tune by retrieval recall on a test set.

12. What is hybrid search, and why add a re-ranker?

It combines BM25 keyword scoring with vector search, catching exact terms like order IDs that embeddings miss. A cross-encoder re-ranker then rescores the top candidates.

13. How do you evaluate a RAG system?

Separately. Score retrieval with recall@k or hit rate on questions with known source chunks, and generation with faithfulness and answer correctness, often using a calibrated judge. Fix retrieval first.

Fine-tuning, agents and MCP

14. When should you fine-tune instead of using RAG or prompting?

Fine-tune to change behaviour, style or format, or to specialise a small model. Use RAG for changing knowledge. Try prompting first; it is cheapest.

15. What is LoRA?

Low-rank adaptation freezes the base model and trains small low-rank matrices added to some weight layers. Training a tiny fraction of parameters needs far less memory, and adapters swap per task.

16. What is the difference between SFT and preference tuning?

Supervised fine-tuning trains on example prompt–response pairs. Preference methods such as RLHF or DPO learn from which of two responses is preferred, shaping helpfulness and safety.

17. What is an AI agent?

An LLM in a loop that decides which tool to call, observes the result and repeats until the task is done. It needs tools, memory, stopping rules and guardrails. See AI agents explained.

18. What is function calling?

The model returns a structured request naming a tool and its arguments instead of plain text. Your code validates and executes the call, then sends the result back to the model.

19. What is MCP?

The Model Context Protocol is an open standard for connecting AI applications to tools and data through MCP servers. One server works with many AI clients.

LLMOps, evaluation and serving

20. How do you evaluate an LLM application?

Build a golden set from real queries, tagged by category. Score short answers with exact match or F1, policies with code rubrics, open-ended quality with a calibrated judge, and compare versions with a confidence interval. Details in LLM evaluation methods.

21. What are the risks of LLM-as-a-judge?

Position, verbosity and self-preference biases, plus leniency. Mitigate with small anchored scales, swapping answer order, JSON verdicts and measuring agreement with human labels using Cohen's kappa.

22. What is the KV cache?

Stored keys and values for previous tokens, so each decode step computes only the new token. It speeds generation but costs GPU memory per token. Covered in the KV cache lesson.

23. What are TTFT and TPOT?

Time to first token covers queueing plus prefill of the prompt. Time per output token is the gap between streamed tokens during decode.

24. How would you reduce LLM latency and cost?

Shorten prompts and context, cap output length, use prefix caching and continuous batching, quantise if evaluation allows, and route easy queries to a smaller model.

25. What is continuous batching, and why is vLLM fast?

Continuous batching admits new requests whenever any running request finishes, keeping the GPU busy. vLLM adds PagedAttention, which stores the KV cache in small blocks so far more requests fit in memory.

26. What should you monitor in a production LLM app?

Traces with a span per step, p95 latency, errors, cost per request, guardrail blocks and feedback by prompt version, with personal data masked.

27. What is red teaming?

Attacking your own app with jailbreaks, injections and data-leak attempts, measuring attack success rate per category, fixing causes in code and keeping successful attacks as regression tests.

System design and behavioural questions

28. Design a support chatbot for 10,000 chats a day.

Clarify requirements first: languages, latency target, what the bot may do, escalation. Then walk through the framework below, naming a trade-off at each step.

Seven steps keep a system design answer organised. Spend the most time where the interviewer pushes.

29. How would you choose between a hosted model API and self-hosting?

Hosted APIs start fastest and need no GPUs. Self-hosting gives data control and cheaper high volume, but needs serving and on-call skills. Compare cost per 1,000 requests, latency and compliance.

30. Tell me about a project that did not work as planned.

Pick a real failure, such as a judge model that disagreed with your labels. Explain how you measured it, what you changed and the result in numbers.

Common interview mistakes

• Reciting definitions with no example. Always connect to something you built.
• No numbers. Mention test-set size, metric and latency.
• Bluffing. "I haven't used that, but here is my approach" earns respect.

Key takeaways

• GenAI interviews test LLM basics, prompting, RAG, fine-tuning and agents, and LLMOps.
• Answer with a definition, the mechanism, a trade-off and an example from your work.
• Use a fixed framework for system design questions.