CS50x: Introduction to Computer Science Artificial Intelligence Syllabus

Define reinforcement learning (RL).

A learning paradigm in which an agent learns from experience by interacting with an environment: it takes actions, receives rewards for good outcomes and penalties for bad ones, and gradually learns a strategy (policy) that maximizes cumulative reward.

Name the five core components of a reinforcement learning problem.

Agent (the learner/decision-maker), environment (the world it acts in), state (the current situation), action (a choice the agent makes), and reward (numeric feedback signaling how good the action's outcome was).

Describe the interaction loop in reinforcement learning.

The agent observes the current state, chooses an action, the environment transitions to a new state, and the agent receives a reward (or punishment). This state $\to$ action $\to$ reward cycle repeats, and the agent updates its behavior to earn more reward over time.

What is the exploration vs. exploitation tradeoff in reinforcement learning?

Exploitation means choosing actions already known to yield high rewards; exploration means trying new actions that might lead to even better rewards. An agent must balance both: too much exploitation misses better strategies, too much exploration wastes time on poor actions.

How does reinforcement learning differ from supervised learning?

Supervised learning requires labeled examples of correct outputs for each input. Reinforcement learning has no labeled answers; the agent only receives reward signals — often delayed — from its own trial-and-error interaction with the environment.

Give two examples of problems well suited to reinforcement learning.

Training game-playing agents (e.g., learning Nim or Go, as in AlphaGo, by playing many games and being rewarded for wins) and robotics tasks like teaching a robot to walk, where success is rewarded and falling is penalized.

What biological system inspired artificial neural networks, and what is the analogy?

The brain's network of neurons. Artificial units are analogous to neurons, and the weighted connections between units are analogous to synapses: each unit receives signals, and if the combined input is strong enough it "activates" and passes a signal onward.

What are the three types of layers in a typical neural network?

The input layer (receives the feature values), one or more hidden layers (intermediate units that learn internal representations), and the output layer (produces the final prediction).

Write the formula for the output of a single artificial neuron with inputs $x_1,\dots,x_n$.

$$h(x) = g\left(\sum_{i=1}^{n} w_i x_i + b\right)$$ where $w_i$ are the weights, $b$ is the bias, and $g$ is the activation function applied to the weighted sum.

What is an activation function? Give two common examples with their formulas.

A function applied to a unit's weighted sum that determines its output and introduces non-linearity. Examples: the sigmoid $\sigma(x)=\frac{1}{1+e^{-x}}$, which squashes values into $(0,1)$, and ReLU, $g(x)=\max(0,x)$, which passes positive values and zeroes out negatives.

How is a neural network trained, at a high level?

By gradient descent: the network makes predictions on training data, a loss function measures the error, and the weights are repeatedly adjusted in the direction that decreases the loss, i.e., $w \leftarrow w - \alpha \cdot \frac{\partial L}{\partial w}$, where $\alpha$ is the learning rate.

What is backpropagation?

The algorithm for training multilayer neural networks: it propagates the output error backward through the network, layer by layer, using the chain rule to compute the gradient of the loss with respect to every weight, so hidden-layer weights can be updated by gradient descent.

What is a deep neural network?

A neural network with multiple hidden layers between input and output. Each successive layer can learn increasingly abstract features, which is what "deep learning" refers to.

Why are hidden layers and non-linear activations necessary in neural networks?

A single unit with a linear function can only learn linearly separable decision boundaries. Hidden layers combined with non-linear activation functions allow the network to model complex, non-linear relationships between inputs and outputs.

At its core, what task does a large language model (LLM) perform?

Next-token prediction: given a sequence of text (the context), it outputs a probability distribution over the vocabulary for the most likely next token, then repeats this process token by token to generate text.

What is a token in the context of LLMs?

The basic unit of text an LLM processes — often a word, part of a word, or punctuation mark. Text is split into tokens before being fed to the model, and models have a maximum context window measured in tokens.

What neural network architecture underlies modern LLMs, and what is its key mechanism?

The transformer architecture, whose key mechanism is attention: it lets the model weigh how relevant every other token in the context is when interpreting or generating each token, capturing long-range relationships between words.

What are word embeddings?

Representations of tokens as high-dimensional numeric vectors that capture meaning: words with similar meanings map to vectors that are close together, allowing the model to compute with semantic relationships mathematically.

How is an LLM trained, at a high level?

In pretraining, the model is shown massive text corpora and its billions of weights (parameters) are tuned by gradient descent to predict the next token accurately. It is then typically fine-tuned — e.g., with reinforcement learning from human feedback (RLHF) — to be more helpful and follow instructions.

Why does the attention mechanism matter for understanding a sentence like "The dog didn't cross the street because it was too tired"?

Attention lets the model determine which earlier word "it" refers to (the dog, not the street) by assigning high relevance weights between "it" and "dog," resolving ambiguity using the whole context rather than only nearby words.

What is the temperature parameter in LLM text generation?

A setting that controls randomness when sampling the next token from the probability distribution: low temperature makes the model pick the most likely tokens (more deterministic, focused output), while high temperature flattens the distribution (more varied, creative output).

What is prompt engineering?

The practice of deliberately crafting the input text (prompt) given to an LLM — its wording, structure, context, and examples — to steer the model toward more accurate, relevant, and useful outputs.

What is the difference between a system prompt and a user prompt?

A system prompt is an instruction set by the developer that defines the model's role, rules, and behavior for the whole conversation (e.g., "You are a helpful tutor; never give away answers"). A user prompt is the message the end user actually types for the model to respond to.

Compare zero-shot and few-shot prompting.

Zero-shot prompting asks the model to perform a task with instructions only and no examples. Few-shot prompting includes a handful of worked input–output examples in the prompt, which usually improves accuracy because the model can imitate the demonstrated pattern.

What is chain-of-thought prompting and why does it help?

Prompting the model to reason step by step (e.g., "think through this step by step") before giving its final answer. Generating intermediate reasoning tokens improves performance on multi-step problems like math and logic, since each step conditions the next.

Why does adding context and constraints to a prompt improve LLM output?

Because the model generates text conditioned on everything in its context window: relevant background, explicit requirements, desired format, and examples narrow the space of plausible continuations, making the statistically likely output also the correct and useful one.

What is a hallucination in the context of LLMs?

When a model confidently generates text that is false or fabricated — invented facts, citations, dates, or events — that sounds plausible and fluent but does not correspond to reality.

Why do LLMs hallucinate?

Because they are trained only to produce statistically plausible next tokens, not to verify truth: they have no built-in fact database or fact-checking step, so when the training data is sparse or the question is ambiguous, a fluent but false continuation can be the most probable output.

Name three key limitations of LLMs besides hallucination.

(1) Knowledge cutoff: they know nothing about events after their training data ends. (2) Bias: they can reproduce biases present in their training text. (3) No true understanding or guaranteed reasoning: they can fail at arithmetic, logic, and consistency, and they cannot verify their own claims.

What strategies help mitigate hallucinations when using an LLM?

Verify important claims against authoritative sources; ask the model for citations and check them; ground the model with retrieved documents (retrieval-augmented generation, RAG) or provide the relevant facts in the prompt; and avoid relying on unverified output for high-stakes decisions.