NPTEL · Generative AI for Computer Vision · 2026 run · Assignments 1 to 12

Gen AI CV Assignment Drill

All 120 questions from this year's 12 assignments, with the options jumbled the way NPTEL does it. Pick an answer to check it, then open the short explanation and the trap to avoid. Progress is saved in this browser. Reset clears everything and reshuffles.

Correct 0 / 0 attempted
All 120 questions from the 2026 assignments with NPTEL's accepted answers. Weeks 4 and 7 were not submitted, so they are marked and show up under "Only questions I got wrong". Options are shuffled each time; the solution gives the answer's position in NPTEL's original order.
NPTEL · Generative AI for Computer Vision · 2026

Week Handbook

One page per assignment week: the core ideas from the lectures, the formulas, exactly what the 2026 assignment asked (answer in short), and the traps. Week 5's assignment was only about the course itself, so that page carries LSTM, GRU, attention and the vision-tasks lecture that no assignment tested.

Week 1

GenAI intro, probability, MLE and Bayes

Lecture 1: Intro, generative modelling, generative vision models · Lecture 2: Linear algebra, Probability I and II

Core ideas

  • GenAI learns patterns in data and makes new content. Core tech: GANs, VAEs, LLMs, diffusion. Generative modelling is (mostly) unsupervised.
  • Discriminative learns \(P(Y\mid X)\) (decision boundary). Generative learns \(P(X,Y)\) or \(P(X)\): how the data is made.
  • Generator: noise \(z \sim p(z)\) → \(G(z)\); loss = distance between the real and generated distributions.
  • 5 families: VAE (encoder → distribution), GAN (generator vs discriminator), autoregressive (PixelRNN/CNN, one pixel at a time), diffusion (add noise, learn to remove it), normalizing flows (invertible, exact likelihood).
  • Applications: synthesis, restoration (inpainting, super-resolution), augmentation, image-to-image, video, 3D.
  • Synthetic data: cheap, perfect labels, rare events, privacy. Sim2Real gap → domain randomization, fine-tune on real.
  • Joint = variables together; marginal = sum the others out; covariance = do they move together; correlation = strength + direction in [−1, 1].
  • MLE: parameters that make the observed data most likely. MAP adds a prior. ML = MAP when priors are equal.
  • Bayesian learning = prior + data → posterior, updated as new evidence arrives. Bayes optimal classifier weights all hypotheses by posterior.

Formulas

  • Bayes: \(P(A\mid B) = \dfrac{P(B\mid A)P(A)}{P(B)}\) · posterior = likelihood × prior / evidence
  • \(\rho = \dfrac{\text{Cov}(X,Y)}{\sigma_X\sigma_Y}\) · \(\text{Cov} = \frac{1}{n-1}\sum (x_i-\bar x)(y_i-\bar y)\)
  • MLE: \(\theta^* = \arg\max_\theta \sum_i \log P(x_i\mid\theta)\) · MAP: \(\arg\max_h P(D\mid h)P(h)\)
  • \(P(A\cup B) = P(A)+P(B)-P(A\cap B)\) · independent: \(P(A\cap B)=P(A)P(B)\)
  • Eigen: \(\det(A-\lambda I)=0\) · 2×2 inverse \(\frac{1}{ad-bc}\begin{bmatrix}d&-b\\-c&a\end{bmatrix}\)

Asked in the 2026 assignment

  • Generative model → GAN (not ResNet / U-Net / AlexNet).
  • Purpose of a generative vision model → generate new images.
  • Applications (synthesis, restoration, augmentation) → All of the above.
  • Joint distribution → two or more variables occurring together.
  • Correlation → strength and direction. Covariance → whether they change together.
  • MLE → parameters that make observed data most likely.
  • Bayesian learning → prior + observed data; updates with evidence (True).
  • GenAI creates new images resembling training data (True).

Traps

  • Correlation has strength and direction; covariance is only "do they move together".
  • U-Net appears in diffusion models, but it is not the generative model itself.
Week 2

Neurons, perceptron, MLP, backprop, overfitting

Lecture 3: Perceptron, multi-layer perceptron, backpropagation, overfitting and regularization

Core ideas

  • Neuron (basic unit): multiply by weights → sum + bias → activation.
  • Single-layer perceptron: step activation, one hyperplane → only linearly separable problems. Fails on XOR.
  • MLP adds hidden layers: layer 1 = half-spaces, layer 2 = convex regions, output = any region → solves XOR.
  • Activation = non-linearity. Sigmoid (0..1, vanishes), tanh (−1..1, zero-centred), ReLU (fast, dying ReLU), Leaky ReLU, softmax (multi-class output, sums to 1).
  • Forward pass: input → output, compute loss. Backprop: chain rule gradients from output → input, then gradient descent.
  • Checkpointing: store all activations (O(N) memory) vs recompute (O(1) memory, O(N²) time).
  • Overfitting: great on train, poor on unseen data (high variance). Underfitting = high bias.
  • Fixes: regularization (L1 Lasso → zeros / feature selection, L2 Ridge → weight decay), early stopping, k-fold CV (k = 5 or 10).
  • λ too big → underfit; too small → overfit.

Formulas

  • \(y = f(\sum_i w_i x_i + b)\) · \(\sigma(z) = \frac{1}{1+e^{-z}}\), \(\sigma' = \sigma(1-\sigma) \le 0.25\)
  • Softmax \(\frac{e^{z_i}}{\sum_j e^{z_j}}\) · ReLU \(\max(0,z)\)
  • Update: \(w \leftarrow w - \eta\,\partial E/\partial w\) · \(E = \tfrac12\sum (y-d)^2\)
  • Total error = Bias² + Variance + irreducible · #weights \(=\sum n_{l-1} n_l\)
  • L1: \(J + \lambda\sum|w|\) · L2: \(J + \lambda\sum w^2\)

Asked in the 2026 assignment

  • Basic unit → Neuron.
  • NN applications (image, speech, NLP) → All of the above.
  • SLP solves → linearly separable problems. Limitation → can't do XOR.
  • Activation function → introduce non-linearity.
  • Softmax → multi-class classification.
  • Forward pass → input layer toward output layer.
  • Backprop → compute gradients to update weights.
  • Overfitting → good on train, poor on unseen. Fixes → All of the above.

Traps

  • "Cannot solve linearly separable problems" is the reversed (wrong) SLP limitation.
  • Kernel is a CNN word, not the basic ANN unit.
Week 3

CNNs and classic architectures

Lecture 4: CNN basics, CNN architectures (LeNet, AlexNet, VGG, GoogLeNet), CNN optimization

Core ideas

  • CNN = Convolutional Neural Network. Two ideas: local receptive field (each neuron sees a small patch) + weight sharing.
  • Conv layer = feature extraction (edges → textures → parts → objects). Pooling = shrink spatial size (max or average).
  • Pipeline: conv → activation → normalization → pooling → FC → softmax.
  • LeNet-5 (1998): 32×32 grayscale digits, tanh, avg pool.
  • AlexNet (2012): 5 conv + 3 FC, ReLU, dropout, augmentation, 2 GPUs, ~60M params. CONV1 = 96 × 11×11, stride 4.
  • VGG (2014): only 3×3 convs; three 3×3 = one 7×7 receptive field with fewer params.
  • GoogLeNet (2014): Inception module (parallel 1×1, 3×3, 5×5, pool, concatenated), 22 layers, ~5M params, no FC, 1×1 bottlenecks.
  • Very deep nets → vanishing gradient (sigmoid′ ≤ 0.25 multiplied many times) and degradation.
  • Fixes: ReLU, residual connections, batch norm, good init (Xavier/He). Deeper is NOT always better.

Formulas

  • Output size \(= \dfrac{N - F + 2P}{S} + 1\) · 32, 5×5, S1 → 28
  • Conv params \(= F\cdot F\cdot C_{in}\cdot C_{out}\) (+ biases)
  • VGG: three 3×3 = 27 vs one 7×7 = 49 (per channel pair)
  • Inception naive 854M ops → with 1×1 bottleneck 358M

Asked in the 2026 assignment

  • CNN → Convolutional Neural Network.
  • Conv neuron connects to → a small local patch.
  • Conv layer purpose → extract features. Pooling → reduce spatial size.
  • Famous CNN → AlexNet.
  • Inception module → GoogLeNet; GoogLeNet known for Inception.
  • Deep-net problem → vanishing gradient. Fix → ReLU.
  • "More layers always better" → False.

Traps

  • Pooling reduces size; conv extracts features. Don't swap them.
  • VGG = 3×3 only; Inception = GoogLeNet.
Week 4

ResNet, DenseNet, MobileNet, EfficientNet, RNN intro

Lecture 5: ResNet; DenseNet, MobileNet, EfficientNet · Lecture 6 Part 1: Sequence modelling, RNN types

Core ideas

  • ResNet: skip (residual) connections that jump over one or more layers; learns \(F(x)=H(x)-x\). Fixes vanishing gradient, trains 50 to 152+ layers, adds no params.
  • ResNet-18/34 = basic block (two 3×3); 50/101/152 = bottleneck (1×1 → 3×3 → 1×1).
  • DenseNet: each layer connects to every previous layer (concatenation). Growth rate k, transition layers.
  • MobileNet: for handheld / embedded devices. Depthwise separable conv = depthwise (per channel) + pointwise 1×1.
  • EfficientNet: compound scaling of depth, width and resolution together. B0 from NAS, MBConv, SE, Swish.
  • Sequence modelling: ordered data (text, speech, video, time series, DNA).
  • RNN = Recurrent Neural Network: hidden state carries temporal context, same weights every step.
  • Types: one-to-many (captioning), many-to-one (sentiment), many-to-many same length (frame labels), different length (translation).

Formulas

  • ResNet: \(a^{[l+2]} = g(z^{[l+2]} + a^{[l]})\) · \(\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y}\left(\frac{\partial F}{\partial x}+1\right)\)
  • DenseNet connections \(= \frac{L(L+1)}{2}\) (6 layers → 21)
  • EfficientNet: \(d=\alpha^\phi,\ w=\beta^\phi,\ r=\gamma^\phi,\ \alpha\beta^2\gamma^2 \approx 2\)
  • RNN: \(h_t = \tanh(W_h h_{t-1} + W_x x_t)\), \(y_t = W_o h_t\)

Asked in the 2026 assignment

  • ResNet idea → residual (skip) connections.
  • Skip connection → skip one or more layers.
  • Advantage → reduces vanishing gradient.
  • DenseNet vs ResNet → every layer connected to all previous layers.
  • Handheld / embedded → MobileNet.
  • EfficientNet → scale depth, width, resolution together.
  • Sequence modelling → sequential data. RNN = Recurrent. Application → translation. Feature → temporal context.

Traps

  • RNN is Recurrent, not Residual (that's ResNet).
  • ResNet adds (+), DenseNet concatenates.
Week 5

BPTT, LSTM, attention, GRU + GenAI vision tasks

Lecture 6: BPTT, LSTM, seq2seq with attention, GRU · Lecture 7: GenAI for vision tasks. (Assignment 5 only asked about the course itself, so this page covers what no assignment tested.)

Core ideas

  • BPTT: unroll the RNN over time; gradients summed across steps (shared weights). Long products of tanh′ → vanishing. Truncated BPTT cuts it to k steps.
  • LSTM (Hochreiter 1997): forget, input, output gates + cell state. Additive cell update (Constant Error Carousel) → long memory.
  • GRU (Cho 2014): update + reset gates, no separate cell, faster.
  • Seq2seq = encoder + decoder; one fixed context vector is the bottleneck. Attention builds a new context at each decoder step.
  • Bidirectional RNN uses past and future; deep RNN stacks layers.
  • CV challenges: viewpoint, scale, deformation, occlusion, illumination, clutter, intra-class variation.
  • GenAI for vision: classification (augment rare classes), detection (YOLO, Faster R-CNN), segmentation (semantic vs instance, SAM), super-resolution, inpainting, image-to-image, medical synthesis, anomaly detection (low likelihood = anomaly), style transfer, video (temporal coherence).
  • Assignment 5 itself: the course covers maths, CV features, CNN/RNN/LSTM, GAN/VAE/diffusion, VLMs and multimodal, agentic AI, all in one course.

Formulas

  • LSTM: \(C_t = f_t\odot C_{t-1} + i_t\odot \tilde C_t\), \(h_t = o_t\odot\tanh(C_t)\)
  • GRU: \(h_t = z_t\odot h_{t-1} + (1-z_t)\odot\tilde h_t\)
  • Attention: \(\alpha_{t,i} = \text{softmax}(e_{t,i})\), \(c_t = \sum_i \alpha_{t,i} h_i\)
  • Vanishing: \(0.9^{50}\approx 0.005\)

Asked in the 2026 assignment

  • A5: every answer was the one sensible AI option (first option in all 10).
  • Why the course matters → modern AI for images and videos.
  • Covers → maths foundations, feature extraction, CNN/RNN/LSTM, GAN/VAE/diffusion, VLMs, agentic AI.
  • Unique → combines CV, GenAI, multimodal and agentic AI in one course.

Traps

  • LSTM has 3 gates, GRU has 2.
  • Semantic segmentation doesn't separate two cars; instance segmentation does.
Week 6

Generative models, MLE, autoregressive, captioning

Lecture 8 Parts 1 to 2: Generative model taxonomy, MLE, autoregressive models (FVBN, PixelRNN, PixelCNN), Show-Attend-Tell

Core ideas

  • Generative model goal: learn \(p(x)\), then make new data similar to the training data.
  • Density estimation = estimating the probability distribution of data.
  • Taxonomy: explicit density (tractable: FVBN, PixelRNN/CNN, flows; approximate: VAE) vs implicit (GAN).
  • Autoregressive: predict each value from the previous ones (chain rule). Slow sequential sampling, exact likelihood.
  • FVBN (Fully Visible Belief Network) = explicit-density generative model using the chain rule over pixels.
  • PixelRNN (RNN = Recurrent): pixel by pixel from the top-left, Row LSTM / Diagonal BiLSTM. Slow.
  • PixelCNN: masked convolutions, generates images pixel by pixel, trains in parallel.
  • Spatial context = relationships between nearby / different regions of an image.
  • Image captioning = textual description of an image. Attention lets each word focus on the relevant region (Show, Attend and Tell).

Formulas

  • MLE: \(\theta^* = \arg\max \sum_i \log p_\theta(x_i)\) = minimize NLL
  • Autoregressive: \(p(x) = \prod_i p(x_i \mid x_1,\dots,x_{i-1})\)
  • Colour: \(p(R)\,p(G\mid R)\,p(B\mid R,G)\)
  • Captioning attention: \(c_t = \sum \alpha_{t,ij}\, z_{ij}\)

Asked in the 2026 assignment

  • Goal of generative model → new data like training data.
  • MLE → parameters making observed data most likely.
  • Density estimation → estimate probability distribution.
  • Autoregressive → predict current value from previous values.
  • PixelCNN → generate images pixel by pixel. PixelRNN "RNN" → Recurrent.
  • Spatial context → relationships between regions.
  • Captioning → text description; attention → focus on relevant parts.
  • FVBN → a generative model.

Traps

  • PixelCNN is still sequential at generation time; only training is parallel.
  • FVBN is explicit + tractable; VAE is explicit + approximate; GAN is implicit.
Week 7

Attention and Transformers

Lecture 8 Parts 3 to 6: Attention, self-attention, masked and multi-head attention, Transformer encoder/decoder, ViT intro

Core ideas

  • Attention: score every input, softmax the scores, take a weighted sum → focus on the most relevant parts.
  • Q, K, V from linear layers; scale by \(\sqrt{d_k}\) so softmax doesn't saturate.
  • Self-attention: Q, K, V from the same sequence → relates different parts of the same input.
  • Masked self-attention: future positions set to −∞ → can't attend to the future (decoder / GPT).
  • Multi-head: several heads capture different relationships, then concatenate (e.g. 512 → 4 × 128).
  • Transformer = attention-based. Encoder → contextual representations; decoder adds masked + cross-attention.
  • Input embedding: token → vector (512-d). Positional encoding (sin/cos or learned) adds order.
  • Encoder block: MHA → Add & Norm → FFN → Add & Norm. BERT = encoder-only, GPT = decoder-only.
  • Transformer vs CNN: CNN is local and builds up; attention relates all elements globally from layer 1.

Formulas

  • \(\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V\)
  • Masked: add \(M\) with \(-\infty\) on future positions
  • Add & Norm: \(\text{LayerNorm}(x + \text{Sublayer}(x))\) · FFN: \(W_2\,\sigma(W_1x+b_1)+b_2\)
  • PE: \(\sin(\omega_k t),\ \cos(\omega_k t)\)

Asked in the 2026 assignment

  • Attention → focus on most relevant parts.
  • Self-attention → parts of the same sequence.
  • Masked → block attending to future positions.
  • Multi-head → several heads, different relationships.
  • Transformer based on → attention.
  • ViT → image split into patches as tokens.
  • Transformer vs CNN → attention models relations between elements.
  • Encoder → contextual representations. Embedding → tokens to vectors. Positional encoding → order.

Traps

  • Self-attention = same sequence; cross-attention = two different ones.
  • Attention has no sense of order by itself; that's why positional encoding exists.
Week 8

ViT, DETR, Swin, autoencoders, VAE

Lecture 8 Part 7: ViT, DETR, Swin · Lecture 9: Autoencoder, Variational Autoencoder

Core ideas

  • ViT (Dosovitskiy 2021): 16×16 patches → flatten (768) → linear embed + CLS token + learned position → encoder-only.
  • DETR = DEtection TRansformer: encoder-decoder, object queries, bipartite matching, no NMS.
  • Swin: attention in local windows + shifted windows to link them, patch merging → hierarchy.
  • Autoencoder = encoder + decoder; learns a compact (bottleneck) representation by reconstructing. Uses: compression, denoising, anomaly detection.
  • Plain AE is a poor generator: random latent points decode to junk.
  • VAE: encoder outputs a distribution (μ, σ), not a point → you can sample and generate.
  • Order: Encode → Sample → Decode. Latent z = a sampled representation; sampling gives different outputs.
  • Problem formulation: model data via latent z, \(p(x)=\int p(x\mid z)p(z)dz\) (intractable) → train with the ELBO.
  • KL term keeps the latent space smooth and close to N(0, I) so sampling works.

Formulas

  • ELBO \(= \mathbb E_q[\log p(x\mid z)] - D_{KL}(q(z\mid x)\,\|\,p(z))\le \log p(x)\)
  • Reparameterization: \(z = \mu + \sigma\odot\epsilon,\ \epsilon\sim\mathcal N(0,I)\)
  • ViT patches \(N = HW/P^2\) · patch dim \(16\cdot16\cdot3 = 768\)
  • \(\log p(x) = \text{ELBO} + D_{KL}(q\,\|\,p(z\mid x))\)

Asked in the 2026 assignment

  • ViT → patches as tokens. DETR → Detection Transformer. Swin → shifted windows.
  • Autoencoder → compact representation; parts → encoder + decoder.
  • VAE vs AE → VAE learns a probability distribution for the latent.
  • VAE sequence → Encoding → Sampling → Decoding.
  • Latent variable → sampled representation of the input.
  • Why sample → generate different possible outputs.
  • Problem formulation → define how data and its distribution are modelled.

Traps

  • VAE does have both an encoder and a decoder and DOES generate.
  • Swin is still pure attention, just windowed.
Week 9

GANs and diffusion

Lecture 11: GAN 1 and 2 · Lecture 10: Diffusion 1 and 2 (DDPM, guidance, Stable Diffusion)

Core ideas

  • GAN (Goodfellow 2014): Generator makes fakes from noise; Discriminator tells real (1) from fake (0). Minimax game; equilibrium D = 0.5.
  • Problems: mode collapse (little variety), instability, non-convergence, D too strong. Minibatch discrimination helps diversity.
  • cGAN: condition on extra info (class label, text) for both G and D.
  • WGAN: Wasserstein distance + 1-Lipschitz critic → stable training, less mode collapse.
  • Pix2Pix: paired image-to-image (cGAN + λL1). CycleGAN: unpaired (horse ↔ zebra) with cycle-consistency loss.
  • Diffusion forward: gradually add Gaussian noise until pure noise. Reverse: network removes noise step by step → new image.
  • DDPM = Denoising Diffusion Probabilistic Model (Ho 2020). Network predicts the added noise ε.
  • ELBO = the training objective for VAEs and diffusion (lower bound on log-likelihood).
  • Guidance: classifier guidance (separate classifier) vs classifier-free (scale s). Stable Diffusion = latent diffusion: VAE latent + text cross-attention U-Net.

Formulas

  • \(\min_G\max_D\ \mathbb E[\log D(x)] + \mathbb E[\log(1-D(G(z)))]\)
  • Forward: \(x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon\), \(\bar\alpha_t = \prod(1-\beta_i)\)
  • Loss: \(\|\epsilon - \epsilon_\theta(x_t,t)\|^2\) · CFG: \(\epsilon_\varnothing + s(\epsilon_c - \epsilon_\varnothing)\)
  • Cycle: \(\|F(G(x))-x\|_1 + \|G(F(y))-y\|_1\)

Asked in the 2026 assignment

  • GAN purpose → new data like training data. Two networks → Generator + Discriminator.
  • Generator role → make fake samples.
  • Common GAN problem → mode collapse.
  • cGAN → generate using extra info like class labels.
  • WGAN → stable GAN training.
  • Forward diffusion → gradually add noise. Reverse → gradually remove noise.
  • DDPM → Denoising Diffusion Probabilistic Model.
  • ELBO → objective for learning the generative model.

Traps

  • "Distinguish real and fake" is the Discriminator, not the Generator.
  • Forward = noising, reverse = generating. Don't flip them.
Week 10

LLMs and Vision-Language Models

Lecture 12: LLM 1 and 2, Vision-Language Models

Core ideas

  • LLM: Transformer trained with next-token prediction to understand and generate human-like text.
  • Architectures: encoder-only (BERT), encoder-decoder (T5, Whisper), decoder-only (GPT, Llama, Claude).
  • Tokenization: text → sub-word units (BPE: merge the most frequent pair, e.g. e + s → es). 1 token ≈ 4 chars ≈ 0.75 words.
  • Pre-training: learn general patterns from huge text (Common Crawl, C4 ~156B tokens). GPT-3: 175B params.
  • Fine-tuning: further train for a task/domain: full, instruction, PEFT (LoRA, adapters), RLHF, DPO.
  • Decoding: greedy, top-k, top-p (nucleus), temperature. Perplexity = how "surprised" (lower is better).
  • VLM: understands images + language. Vision encoder (ViT/CNN) extracts image features → projection → LLM.
  • Alignment: image and text in one common space. Cross-modal attention: one modality attends to the other.
  • Contrastive learning: matching image-text pairs get similar representations.

Formulas

  • \(P(X) = \prod_t P(x_t\mid x_{\lt t})\) · \(L = -\sum_t \log P(x_t\mid x_{\lt t})\)
  • Temperature: \(P_i = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}}\) (T < 1 sharper)
  • Perplexity \(= \prod p(x_i\mid x_{\lt i})^{-1/L}\)
  • Contrastive: \(L_{CL} = \tfrac12(L_{I\to T} + L_{T\to I})\), cosine with temperature τ

Asked in the 2026 assignment

  • LLM purpose → understand and generate human-like text. Architecture → Transformer.
  • Tokenization → text into smaller units.
  • Pre-training → general patterns from lots of text. Fine-tuning → further training for a task/domain.
  • VLM → images + language. Vision encoder → extract image features.
  • Alignment → common representation space.
  • Cross-modal attention → one modality interacts with another.
  • Contrastive learning → similar representations for matching pairs.

Traps

  • Pre-training = general; fine-tuning = specific. Don't swap them.
  • BERT is encoder-only, GPT is decoder-only.
Week 11

CLIP and multimodal generative models

Lecture 13: CLIP, Multimodal Generative Models 1 and 2

Core ideas

  • CLIP (Radford 2021): learns the relationship between images and text in one shared embedding space.
  • Two encoders: image encoder (ResNet/ViT) + text encoder (Transformer). Trained on web image-text pairs.
  • Contrastive learning: matched pairs (diagonal) close, mismatches far. Cosine similarity of normalized embeddings.
  • Zero-shot classification: prompts "a photo of a {class}" → pick the class with the highest cosine. No retraining, open vocabulary.
  • Prompt engineering: design good text prompts; prompt ensembling averages several templates.
  • Multimodal fusion: combine modalities. Early (input), middle (cross-modal interaction), late (decisions). Bilinear / gated variants.
  • MLLM: LLM + modality encoders + projector → handles text + images.
  • Text-to-image: text description → image (Reed 2016 GAN, diffusion). Captioning is the reverse.
  • Training stages: pretraining → alignment → instruction tuning → preference optimization. Challenge: aligning modalities, hallucination.

Formulas

  • \(s_{ij} = \frac{v_i^\top t_j}{\|v_i\|\|t_j\|}\) · zero-shot \(\hat c = \arg\max_c s(v, t_c)\)
  • Symmetric InfoNCE: \(\tfrac12(L_{I\to T}+L_{T\to I})\) with temperature τ
  • Prompt ensemble \(t_c = \frac1K\sum_k f_T(T_{c,k})\)
  • MLLM: \(P(y\mid I,q) = \prod_t P(y_t\mid I,q,y_{\lt t})\)

Asked in the 2026 assignment

  • CLIP → relationship between images and text.
  • Two encoders → image encoder + text encoder.
  • Learning type → contrastive.
  • Key capability → zero-shot classification.
  • Prompt engineering → design suitable text prompts.
  • Training data → image-text pairs.
  • Multimodal fusion → combine modalities. MLLM → text + images.
  • Text-to-image → image from a text description. Challenge → aligning modalities.

Traps

  • CLIP's two parts are image + text encoders, not encoder + decoder.
  • Zero-shot means no training on the new classes.
Week 12

Agentic AI and research trends

Lecture 14: Agentic AI 1 and 2, research trends and conclusion

Core ideas

  • Agentic AI: systems that autonomously pursue goals by planning and taking actions. GenAI makes content; agentic AI acts.
  • The agent interacts with the environment (tools, APIs, files, web).
  • Loop: Goal → Perception → Reasoning → Planning → Action → Observation → Evaluation → Updated state.
  • Planning: decide the steps to reach the goal; re-plan after observations. Task decomposition: complex task → subtasks (DAG).
  • LLM = brain: reasons and decides; the framework executes. Tool use: external tools via function calls tool(name, args).
  • Memory: working (current context) + long-term (vector stores, knowledge graphs) to reuse past info.
  • RAG: retrieve relevant info → generate a grounded answer. Agentic RAG = iterative retrieval in the loop.
  • Patterns: ReAct (reason → act → observe), reflection (critique → revise), plan-and-execute, human-in-the-loop. Multi-agent: supervisor, pipeline, debate, blackboard.
  • Challenges: safety and reliability (prompt injection, data leakage, excess permissions), cost, latency. Trends: DiT, flow matching, text-to-video, world models.

Formulas

  • Policy: \(a_t \sim \pi(a_t \mid s_t)\)
  • Tool call: tool(name, arguments)

Asked in the 2026 assignment

  • Agentic AI → autonomously perform tasks and take actions.
  • Agent interacts with → the environment.
  • Planning → decide steps to reach a goal. Task decomposition → break into subtasks.
  • LLM role → reasoning / decision-making brain.
  • Tool use → external tools for specific actions. Memory → store and reuse past info.
  • RAG → retrieve relevant info to help answer.
  • Multi-agent → multiple agents working together.
  • Major challenge → safety and reliability of autonomous actions.

Traps

  • The LLM decides; it doesn't replace tools.
  • Multi-agent means multiple agents, not multiple GPUs.
Built from all 43 lecture decks and the 12 assignments of the 2026 run. "Asked" lines follow NPTEL's accepted answers.
NPTEL · Generative AI for Computer Vision · formulas

Formula Book

96 formulas in 15 topics, in course order. Each one has what it means in plain words, and 52 have a small worked example. The 2026 assignments were all theory, so treat these as backup for any numerical that shows up in the exam: CNN shapes, Bayes, softmax, attention shapes, ViT patches and diffusion steps are the likeliest.

Probability basics

Must knowW1

Counting chances. Union adds and removes the overlap, conditional zooms into one event, Bayes flips the condition around.

Union
\[P(A\cup B)=P(A)+P(B)-P(A\cap B)\]
Add both, subtract the overlap you counted twice.
P(A)=0.5, P(B)=0.4, P(A∩B)=0.2 → 0.5+0.4−0.2 = 0.7
Conditional
\[P(B\mid A)=\frac{P(A\cap B)}{P(A)}\]
Chance of B once you know A happened.
P(A∩B)=0.2, P(A)=0.5 → 0.4
Independence
\[P(A\cap B)=P(A)\,P(B)\]
Knowing one tells you nothing about the other.
Total probability
\[P(A)=\sum_i P(B_i)\,P(A\mid B_i)\]
Split A over all the ways it can happen.
Bayes rule
\[P(B\mid A)=\frac{P(A\mid B)\,P(B)}{P(A)}\]
posterior = likelihood × prior / evidence.
Allergy slide: P(+|allergic)=0.98, P(allergic)=0.008, P(+|not)=0.03. P(+) = 0.98·0.008 + 0.03·0.992 = 0.0376 → P(allergic|+) = 0.00784/0.0376 ≈ 0.21. Rare condition, so even a good test gives a low posterior.
Uniform PDF
\[f(x)=\frac{1}{b-a},\quad a\le x\le b\]
Flat density; area must be 1.
a=2, b=6 → f = 0.25, mean = (a+b)/2 = 4
Gaussian
\[f(x)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}\]
Bell curve, centre μ, spread σ.
Expectation and variance
\[E[X]=\sum x_i p_i,\quad \mathrm{Var}(X)=E[(X-\mu)^2]\]
Weighted average, and average squared distance from it.
Fair die: E = 3.5, Var = 35/12 ≈ 2.92

Joint, covariance, correlation, MLE, MAP

Must knowW1

How two variables move together, and how to pick parameters or hypotheses from data.

Joint and chain rule
\[P(A,B)=P(A)\,P(B\mid A),\quad P(X_1..X_n)=\prod_i P(X_i\mid X_{\lt i})\]
Joint = both together. The chain rule is the base of autoregressive models.
Marginal
\[P(x)=\sum_y P(x,y)\]
Sum the other variable out.
Covariance
\[\mathrm{Cov}(X,Y)=\frac{1}{n-1}\sum_i (x_i-\bar x)(y_i-\bar y)\]
Positive: move together. Negative: opposite.
Correlation
\[\rho=\frac{\mathrm{Cov}(X,Y)}{\sigma_X\sigma_Y}\in[-1,1]\]
Covariance scaled to strength and direction.
Cov = 6, σx = 2, σy = 4 → 6/8 = 0.75 (strong positive)
Likelihood / MLE
\[L(\theta)=\prod_i P(x_i\mid\theta),\quad \theta^*=\arg\max_\theta \sum_i \log P(x_i\mid\theta)\]
Parameters that make the observed data most likely. Log turns products into sums.
Coin: 7 heads in 10 → MLE p = 0.7
MAP vs ML
\[h_{MAP}=\arg\max_h P(D\mid h)P(h),\quad h_{ML}=\arg\max_h P(D\mid h)\]
MAP adds the prior. Equal priors → MAP = ML.
Bayes optimal classifier
\[v^*=\arg\max_v \sum_h P(v\mid h)\,P(h\mid D)\]
Let every hypothesis vote, weighted by its posterior.
h1 (0.4) says +, h2 and h3 (0.3 each) say −. MAP picks h1 → +. Bayes optimal: + = 0.4, − = 0.6 → −

Linear algebra

Good to knowW1

Shapes, determinants, inverses and eigenvalues. Mostly mechanical once you know the recipe.

Dot product
\[u\cdot v=\sum_i u_iv_i=\|u\|\|v\|\cos\theta\]
Large when vectors point the same way. Used in attention and CLIP.
(1,2)·(3,4) = 3+8 = 11
Matrix product shape
\[[m\times n]\,[n\times p]=[m\times p]\]
Inner sizes must match; outer sizes survive.
[3×4][4×5] → [3×5]
2×2 inverse
\[A^{-1}=\frac{1}{ad-bc}\begin{bmatrix}d&-b\\-c&a\end{bmatrix}\]
Swap the diagonal, negate the off-diagonal, divide by det.
A = [[4,7],[2,6]] → det = 24−14 = 10 → A⁻¹ = (1/10)[[6,−7],[−2,4]]
Eigenvalues
\[\det(A-\lambda I)=0\]
Directions the matrix only stretches.
[[2,1],[1,2]] → (2−λ)² − 1 = 0 → λ = 3, 1. Diagonal [[3,0],[0,2]] → 3, 2
Singular matrix
\[\det(A)=0\]
No inverse; collapses a dimension. Fix with pseudo-inverse or regularization.

Neurons, activations, backprop

Must knowW2

A neuron is a weighted sum plus an activation. Backprop is the chain rule run backwards.

Neuron
\[y=f\Big(\sum_i w_ix_i+b\Big)\]
Multiply, sum, activate.
w = (0.5, −1), x = (2, 1), b = 0.5 → z = 1 − 1 + 0.5 = 0.5 → ReLU = 0.5
Sigmoid and its slope
\[\sigma(z)=\frac{1}{1+e^{-z}},\quad \sigma^{\prime}(z)=\sigma(1-\sigma)\le 0.25\]
Squashes to (0, 1). The max slope 0.25 is why deep sigmoid nets vanish.
σ(0) = 0.5 → σ′ = 0.25. Ten layers: 0.25¹⁰ ≈ 10⁻⁶
Softmax
\[\mathrm{softmax}(z)_i=\frac{e^{z_i}}{\sum_j e^{z_j}}\]
Turns scores into probabilities that add to 1.
z = (2, 1, 0) → e = 7.39, 2.72, 1 → sum 11.11 → (0.67, 0.24, 0.09)
ReLU / Leaky ReLU
\[\max(0,z),\quad \max(\alpha z,z)\]
Gradient 1 for positive inputs. Leaky keeps a small slope for negatives (no dying ReLU).
Squared error
\[E=\tfrac12\sum (y-d)^2\]
Half the squared gap; the ½ cancels the 2 on differentiation.
Gradient descent update
\[w\leftarrow w-\eta\,\frac{\partial E}{\partial w}\]
Step against the gradient.
w = 0.8, η = 0.1, ∂E/∂w = 0.5 → 0.75
MLP weight count
\[W=\sum_l n_{l-1}\,n_l\ (+\ \text{biases}\ \sum_l n_l)\]
Every neuron connects to every neuron in the next layer.
3-4-2 net: 3·4 + 4·2 = 20 weights, + 6 biases = 26

Overfitting and regularization

Must knowW2

Penalize big weights, or stop early, so the model generalizes.

Error decomposition
\[\text{Error}=\text{Bias}^2+\text{Variance}+\text{Irreducible}\]
High bias = underfit, high variance = overfit.
k-fold CV score
\[\text{CV}=\frac1k\sum_{i=1}^k S_i\]
Train on k−1 folds, test on 1, average. k = 5 or 10.
Fold accuracies 80, 82, 78, 84, 81 → 81
L1 (Lasso)
\[J+\lambda\sum_i|w_i|\]
Pushes weights to exactly 0 → feature selection.
w = (3, −4), λ = 0.1 → penalty = 0.1·7 = 0.7
L2 (Ridge / weight decay)
\[J+\lambda\sum_i w_i^2\]
Shrinks all weights, never to 0. Handles correlated features.
w = (3, −4), λ = 0.1 → 0.1·25 = 2.5

CNN shapes and parameter counts

Must knowW3

Most CNN numericals are just these three: output size, parameter count, and stacking channels.

Output size
\[O=\frac{N-F+2P}{S}+1\]
N input, F filter, P padding, S stride.
32, F5, P0, S1 → 28. AlexNet CONV1: 227, F11, S4 → (227−11)/4 + 1 = 55
Output depth
\[\text{depth}=\#\text{filters}\]
Each filter gives one feature map.
Six 5×5×3 filters on 32×32×3 → 28×28×6
Conv parameters
\[F\cdot F\cdot C_{in}\cdot C_{out}+C_{out}\]
Weights per filter times number of filters, plus one bias each.
6 filters of 5×5×3 → 5·5·3·6 + 6 = 456. VGG conv1: 3·3·3·64 = 1,728
FC parameters
\[n_{in}\times n_{out}\]
Flatten first.
32×32×3 = 3072 inputs → 10 classes: 3072 × 10 weights
3×3 stack vs big filter
\[3\times(3\cdot3)=27\ \text{vs}\ 7\cdot7=49\]
Three 3×3 convs see 7×7 with fewer params and more non-linearity (VGG).
Inception concat depth
\[C_{out}=\sum \text{branch depths}\]
Branches are stacked along depth.
Naive module on 28×28×256: 128 + 192 + 96 + 256 (pool) = 672
Conv ops (multiplies)
\[H_{out}\,W_{out}\,C_{out}\cdot F\,F\,C_{in}\]
Every output value costs one filter-sized dot product.
Inception: naive ≈ 854M ops, with 1×1 bottlenecks ≈ 358M

ResNet, DenseNet, MobileNet, EfficientNet

Must knowW4

Ways to go deeper or cheaper: skip connections, dense links, split convolutions, balanced scaling.

Residual block
\[y=F(x)+x,\quad a^{[l+2]}=g(z^{[l+2]}+a^{[l]})\]
Learn only the change. If F = 0 the block is the identity.
Gradient through skip
\[\frac{\partial L}{\partial x}=\frac{\partial L}{\partial y}\Big(\frac{\partial F}{\partial x}+1\Big)\]
The +1 keeps gradient alive even when ∂F/∂x is tiny.
∂F/∂x = 0.01 → factor 1.01 instead of 0.01
DenseNet connections
\[\frac{L(L+1)}{2}\]
Every layer links to every earlier layer.
L = 5 → 15. L = 6 → 21
DenseNet input channels
\[k_0+k\,(l-1)\]
Each layer adds k (growth rate) maps.
k₀ = 16, k = 4, layer 5 → 16 + 16 = 32
Depthwise separable cost
\[\frac{D_K^2 M D_F^2+M N D_F^2}{D_K^2 M N D_F^2}=\frac1N+\frac1{D_K^2}\]
Depthwise (per channel) + pointwise 1×1 instead of one big conv.
3×3, M = 16, N = 32, 32×32 map: standard 4.72M vs separable 0.67M → ratio 0.14 (1/32 + 1/9)
EfficientNet compound scaling
\[d=\alpha^\phi,\ w=\beta^\phi,\ r=\gamma^\phi,\ \alpha\beta^2\gamma^2\approx2\]
Scale depth, width and resolution together; each φ step ≈ doubles FLOPs.
α = 1.2, β = 1.1, γ = 1.15 → 1.2 · 1.21 · 1.32 ≈ 1.92 ≈ 2

RNN, BPTT, LSTM, GRU

Must knowW4 to W5

A loop that carries a hidden state. Gates decide what to keep so gradients survive.

RNN step
\[h_t=\tanh(W_hh_{t-1}+W_xx_t),\quad y_t=W_oh_t\]
Same weights at every time step.
hidden 4, input 3, output 2: 16 + 12 + 8 = 36 weights (+ biases)
BPTT
\[\frac{\partial L}{\partial W}=\sum_t\sum_{i\le t}\frac{\partial L_t}{\partial h_t}\frac{\partial h_t}{\partial h_i}\frac{\partial h_i}{\partial W}\]
Sum gradients over time because weights are shared.
Why it vanishes
\[\frac{\partial h_t}{\partial h_i}=\prod_{m=i+1}^{t}W_h^\top\mathrm{diag}(\tanh^{\prime}(a_m))\]
A long product of numbers below 1.
0.9⁵⁰ ≈ 0.005
LSTM cell
\[C_t=f_t\odot C_{t-1}+i_t\odot\tilde C_t,\quad h_t=o_t\odot\tanh(C_t)\]
Forget old, add new, expose some. The + is the "constant error carousel".
LSTM gates
\[f,i,o=\sigma(W x_t+U h_{t-1}+b),\quad \tilde C=\tanh(\cdot)\]
Three sigmoid gates plus a tanh candidate.
Params ≈ 4·(h(h+d) + h). h = 4, d = 3 → 128
GRU
\[h_t=z_t\odot h_{t-1}+(1-z_t)\odot\tilde h_t\]
Update gate z blends old and new; reset gate r controls the candidate. 2 gates, no cell.
Params ≈ 3·(h(h+d) + h) → h = 4, d = 3 → 96
Attention context (seq2seq)
\[\alpha_{t,i}=\mathrm{softmax}_i(e_{t,i}),\quad c_t=\sum_i\alpha_{t,i}h_i\]
A fresh weighted mix of encoder states for every decoder step.
α = (0.7, 0.3), h₁ = (1, 0), h₂ = (0, 1) → c = (0.7, 0.3)

Generative modelling and autoregressive models

Good to knowW6

Learn p(x). Autoregressive models factor it pixel by pixel with the chain rule.

MLE for generative models
\[\theta^*=\arg\max_\theta\sum_i\log p_\theta(x_i)=\arg\min_\theta -\frac1N\sum_i\log p_\theta(x_i)\]
Maximize likelihood = minimize negative log-likelihood.
Autoregressive factorization
\[p(x)=\prod_{i=1}^{n^2}p(x_i\mid x_1,\dots,x_{i-1})\]
One pixel at a time, from the top-left, given all earlier pixels.
Colour channels
\[p(x_i)=p(R)\,p(G\mid R)\,p(B\mid R,G)\]
Even inside one pixel, R then G then B.
Captioning attention
\[e_{t,ij}=f_{att}(h_{t-1},z_{ij}),\ a_t=\mathrm{softmax}(e_t),\ c_t=\sum a_{t,ij}z_{ij}\]
Each word gets its own weighted look over the H×W feature grid.

Attention, Transformers, ViT

Must knowW7 to W8

Scores = query · key, softmax, mix the values. Everything else is bookkeeping on shapes.

Scaled dot-product attention
\[\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\Big(\frac{QK^\top}{\sqrt{d_k}}\Big)V\]
Scale by √d_k so softmax doesn't saturate.
d_k = 64 → divide scores by 8
Attention shapes
\[X\in\mathbb R^{T\times d}\Rightarrow QK^\top\in\mathbb R^{T\times T}\]
One score for every pair of tokens.
6 tokens × 512 → score matrix 6×6, output 6×512
Multi-head split
\[d_{head}=\frac{d_{model}}{h}\]
Split the width across heads, then concatenate.
512 with 4 heads → 128 each (8 heads → 64)
Masking
\[\mathrm{softmax}\Big(\frac{QK^\top}{\sqrt{d_k}}+M\Big),\ M_{ij}=-\infty\ (j>i)\]
Future tokens get weight 0.
Add & Norm, FFN
\[\mathrm{LayerNorm}(x+\mathrm{Sublayer}(x)),\quad \mathrm{FFN}=W_2\sigma(W_1x+b_1)+b_2\]
Residual + normalization around every sub-layer.
Positional encoding
\[PE_{t,2k}=\sin(\omega_kt),\ PE_{t,2k+1}=\cos(\omega_kt)\]
Adds order. Fixed sinusoids or a learned T×d table.
ViT patches
\[N=\frac{HW}{P^2},\quad \text{patch dim}=P^2C\]
Cut into P×P patches and flatten each.
224×224×3, P = 16 → 14×14 = 196 patches (+1 CLS = 197 tokens), each 16·16·3 = 768

Autoencoders and VAE

Must knowW8

A VAE encodes to a distribution, samples, decodes. The ELBO is what it maximizes.

Latent model
\[p_\theta(x)=\int p_\theta(x\mid z)\,p(z)\,dz\]
Intractable integral, so we can't maximize it directly.
Encoder output
\[q_\phi(z\mid x)=\mathcal N(\mu(x),\sigma^2(x))\]
A distribution, not a point. Prior p(z) = N(0, I).
ELBO
\[\log p(x)\ge\mathbb E_{q}[\log p(x\mid z)]-D_{KL}(q(z\mid x)\,\|\,p(z))\]
Reconstruction minus KL. From Jensen's inequality.
Gap to the true likelihood
\[\log p(x)=\mathrm{ELBO}+D_{KL}(q\,\|\,p(z\mid x))\]
KL ≥ 0, so ELBO ≤ log p(x).
VAE loss
\[L=-\mathbb E_q[\log p(x\mid z)]+D_{KL}(q\,\|\,p(z))\]
Minimize = maximize ELBO.
Reparameterization
\[z=\mu+\sigma\odot\epsilon,\ \epsilon\sim\mathcal N(0,I)\]
Move the randomness into ε so gradients reach μ and σ.
μ = 2, σ = 0.5, ε = −1 → z = 1.5

Diffusion (DDPM, guidance)

Must knowW9

Add noise in a fixed way, learn to predict that noise, then remove it step by step.

Forward step
\[q(x_t\mid x_{t-1})=\mathcal N(\sqrt{1-\beta_t}\,x_{t-1},\ \beta_tI)\]
A little Gaussian noise each step; β grows over time.
Alphas
\[\alpha_t=1-\beta_t,\quad \bar\alpha_t=\prod_{i\le t}\alpha_i\]
ᾱ shrinks toward 0, so x_T ≈ pure noise.
β₁ = 0.1, β₂ = 0.2 → ᾱ₂ = 0.9 · 0.8 = 0.72
Jump to any step
\[x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon\]
No need to loop: one formula for any t.
ᾱ = 0.64, x₀ = 1, ε = 0.5 → 0.8·1 + 0.6·0.5 = 1.1
Training loss
\[L_{simple}=\mathbb E\,\|\epsilon-\epsilon_\theta(x_t,t)\|^2\]
Predict the noise that was added.
Reverse mean
\[\mu_\theta=\frac{1}{\sqrt{\alpha_t}}\Big(x_t-\frac{1-\alpha_t}{\sqrt{1-\bar\alpha_t}}\epsilon_\theta\Big),\ x_{t-1}=\mu_\theta+\sigma_tz\]
Subtract predicted noise, rescale, add a bit of fresh noise.
Classifier guidance
\[\nabla\log p(x_t\mid c)=\nabla\log p(x_t)+\nabla\log p(c\mid x_t)\]
A separate classifier's gradient steers toward class c.
Classifier-free guidance
\[\hat\epsilon=\epsilon_\theta(\varnothing)+s\,[\epsilon_\theta(c)-\epsilon_\theta(\varnothing)]\]
One model, with and without the condition. Bigger s = more faithful, less diverse.
ε(∅) = 0.2, ε(c) = 0.5, s = 3 → 0.2 + 3·0.3 = 1.1

GANs

Must knowW9

A two-player game: D maximizes, G minimizes the same value.

Minimax objective
\[\min_G\max_D\ \mathbb E_x[\log D(x)]+\mathbb E_z[\log(1-D(G(z)))]\]
D wants D(x) → 1, D(G(z)) → 0. G wants D(G(z)) → 1.
Optimal discriminator
\[D^*(x)=\frac{p_{data}(x)}{p_{data}(x)+p_g(x)}\]
When p_g = p_data, D* = 0.5 everywhere.
p_data = 0.3, p_g = 0.1 → D* = 0.75
BCE view
\[L=-[y\log\hat y+(1-y)\log(1-\hat y)]\]
Real: y = 1. Fake: y = 0.
WGAN
\[W=\sup_{\|f\|_L\le1}\mathbb E[f(x)]-\mathbb E[f(G(z))]\]
Critic outputs a score, not a probability. Smoother gradients.
WGAN losses
\[L_C=-\mathbb E[C(x)]+\mathbb E[C(G(z))],\quad L_G=-\mathbb E[C(G(z))]\]
Pix2Pix
\[G^*=\arg\min_G\max_D L_{cGAN}+\lambda\,\mathbb E\|y-G(x)\|_1\]
Paired translation: fool D and stay close to the target.
CycleGAN cycle loss
\[L_{cyc}=\|F(G(x))-x\|_1+\|G(F(y))-y\|_1\]
Unpaired: going there and back should give the input.

LLMs: loss, sampling, perplexity

Must knowW10

Predict the next token. Sampling knobs reshape that distribution; perplexity measures surprise.

Next-token factorization
\[P(X)=\prod_tP(x_t\mid x_{\lt t}),\quad L=-\sum_t\log P(x_t\mid x_{\lt t})\]
Cross-entropy on the next token.
Temperature
\[P_i=\frac{e^{z_i/T}}{\sum_je^{z_j/T}}\]
T < 1 sharper, T > 1 flatter.
z = (2, 1): T = 1 → (0.73, 0.27). T = 0.5 → (0.88, 0.12). T = 2 → (0.62, 0.38)
Top-k
\[\text{keep the }k\text{ largest, renormalize}\]
(0.5, 0.3, 0.1, 0.1), k = 2 → (0.625, 0.375)
Top-p (nucleus)
\[\text{smallest set with }\sum p\ge p\]
Same probs, p = 0.9 → keep the first three (0.5 + 0.3 + 0.1 = 0.9)
Perplexity
\[PPL=\prod_iP(x_i\mid x_{\lt i})^{-1/L}\]
Like choosing uniformly among PPL tokens. Lower is better.
Each token p = 1/4 → PPL = 4. p = 0.5 then 0.25 → (0.125)^(−1/2) ≈ 2.83
Token rule of thumb
\[1\ \text{token}\approx4\ \text{chars}\approx0.75\ \text{words}\]
1,000 words ≈ 1,333 tokens

VLMs and CLIP

Must knowW10 to W11

Put images and text in one space; matching pairs get high cosine similarity.

VLM pipeline
\[V=f_v(I),\ Z_v=P(V),\ Z=[Z_v;Z_t]\]
Vision encoder → projector → concatenate with text tokens → LLM.
Cross-attention
\[Q=Z_tW_Q,\ K=Z_vW_K,\ V=Z_vW_V\]
Text asks (queries), image answers (keys and values).
Cosine similarity
\[s(v,t)=\frac{v^\top t}{\|v\|\|t\|}\]
Angle only, not length.
v = (1, 0), t = (0.6, 0.8) → 0.6
Contrastive (InfoNCE)
\[L_{I\to T}=-\frac1N\sum_i\log\frac{e^{s(v_i,t_i)/\tau}}{\sum_je^{s(v_i,t_j)/\tau}}\]
Make the matching pair win a softmax over the batch.
Symmetric CLIP loss
\[L_{CL}=\tfrac12(L_{I\to T}+L_{T\to I})\]
Both directions, averaged.
Zero-shot
\[\hat c=\arg\max_c\ s(v,\ f_T(\text{"a photo of a }c\text{"}))\]
Score the image against a prompt for each class.
Cosines (cat 0.31, dog 0.27, car 0.12) → cat
Prompt ensembling
\[t_c=\frac1K\sum_kf_T(T_{c,k})\]
Average text embeddings over K templates.
MLLM / captioning
\[P(y\mid I,q)=\prod_tP(y_t\mid I,q,y_{\lt t})\]
Same next-token loss, conditioned on the image.
Formulas from the lecture slides. Examples are worked by hand with round numbers.
NPTEL · Generative AI for Computer Vision · lecture recall bank

Lecture Recall

255 questions across all 12 lecture weeks, straight from the slides. Green rows are one-line recall answers; amber "Open" rows are short "why" answers. Cover the right column and test yourself.

W1W2W3W4W5W6W7W8W9W10W11W12

WEEK 1Intro, generative modelling, math prelims

1Generative AI, in one line?AI that creates new content (images, text, audio, video) by learning patterns from existing data.
2Four core GenAI technologies named on the intro slides?GANs, VAEs, LLMs, diffusion models.
3Is generative modelling supervised or unsupervised?Generally unsupervised: it learns the data distribution without labels.
4Discriminative model learns what?P(Y|X), i.e. the decision boundary (e.g. logistic regression β0 + βᵀx = 0).
5Generative model learns what?P(X, Y) or P(X): how the data itself is generated.
6Input to a generator G?Random noise z sampled from a simple distribution p(z).
7Generator loss in the basic setup is?A statistical distance between the real data D and generated data D′ (slide: L1/L2 norm).
8The 5 generative vision model families?VAE, GAN, autoregressive, diffusion, normalizing flows.
9A VAE encoder maps an input to?A distribution (Gaussian with μ, σ), not a single point.
10Why are GAN images sharper than MSE-trained ones?MSE averages possible outputs, so it blurs; the adversarial loss rewards realism and high-frequency detail.
11PixelRNN vs PixelCNN speed?PixelRNN is slow (sequential RNN); PixelCNN uses masked convolutions and trains much faster.
12Diffusion backward process samples using?Annealed Langevin dynamics (as stated on the slide), denoising step by step.
13Key property of normalizing flows?Invertible, differentiable transforms, so exact likelihood via change of variables; train by minimizing NLL.
14Name 4 reasons to use synthetic data.Lower collection/annotation cost, perfect ground truth, rare events, diversity, safe training, domain adaptation, privacy.
15Sim2Real gap fixes?Domain randomization, photorealistic rendering, GenAI augmentation, fine-tuning on real data.
16Main challenges of GenAI for vision?Fidelity/artefacts, diversity (mode collapse), scene consistency, bias, compute cost, deepfakes.
17A singular matrix means?det = 0, no inverse, it collapses a dimension (multicollinearity); use pseudo-inverse or regularization.
18Eigenvalues of [[3,0],[0,2]]?3 and 2 (diagonal entries; from det(A − λI) = 0).
19In Bayes P(A|B) = P(B|A)P(A)/P(B), name each term.P(A|B) posterior, P(B|A) likelihood, P(A) prior, P(B) evidence.
20MAP vs ML hypothesis?MAP = argmax P(D|h)P(h). ML = argmax P(D|h); same as MAP when all priors are equal.
21Bayes optimal classifier vs MAP?MAP picks the single most probable hypothesis; Bayes optimal gives the most probable classification by weighting every hypothesis by its posterior.
O1Open: P(h1|D)=0.4 says +, h2 and h3 (0.3 each) say −. What do MAP and Bayes optimal predict?MAP takes h1, so +. Bayes optimal sums: + gets 0.4, − gets 0.6, so −. They can disagree.
O2Open: Why does a generator need a distance between distributions rather than per-sample labels?There is no "correct" output for a given noise vector; only the overall generated distribution should match the real one.

WEEK 2Perceptron, MLP, backprop, overfitting

1Three steps of an artificial neuron?Multiply inputs by weights, sum (plus bias), apply activation.
2Biological neuron parts and roles?Dendrites take input, soma processes, axon sends output, synapse connects to the next neuron.
3Single-layer perceptron activation?Step function, binary output.
4Biggest SLP limitation?Only solves linearly separable problems; cannot do XOR (no hidden layer).
5XOR as a Boolean expression?x′y + xy′.
6How does an MLP solve XOR (slide weights)?h1 = σ(−x1 + x2 + 1), h2 = σ(x1 − x2 + 1), y = σ(h1 + h2).
7What does each MLP layer carve out geometrically?Layer 1: half-spaces (hyperplanes). Layer 2: convex regions. Output: unions, so any (non-convex) region.
8Sigmoid range and main problem?(0, 1); saturates, so vanishing gradient.
9tanh vs sigmoid?tanh is a shifted sigmoid, range (−1, 1), zero-centred; still vanishes.
10Dying ReLU, and the fix?Negative inputs give 0 output and 0 gradient, so the neuron stops learning; Leaky ReLU keeps a small negative slope.
11Where is softmax used?Output layer for multi-class classification; outputs sum to 1.
12Sigmoid derivative and its max?σ(1 − σ), max 0.25 at z = 0.
13Backprop weight update rule?w(t+1) = w(t) − η · δ · I_prev (learning rate × local error × input).
14Total number of weights in an MLP?W = Σ n_(l−1) · n_l over consecutive layers.
15Gradient checkpointing trade-off?Store all activations: O(N) memory. Recompute all: O(1) memory, O(N²) time. Checkpointing sits between.
16Formal definition of overfitting?h overfits if some h′ has higher training error but lower test error than h.
17Total error decomposition?Bias² + Variance + irreducible error.
18High bias means? High variance means?High bias = underfit (high train and test error). High variance = overfit (low train, high test).
19k-fold cross-validation: typical k and score?k = 5 or 10; train on k−1 folds, test on 1, score = average of the k scores.
20L1 vs L2 regularization?L1 (Lasso, λΣ|w|): exact zeros, feature selection. L2 (Ridge / weight decay, λΣw²): shrinks, never zero, handles multicollinearity.
21λ too large vs too small?Too large (e.g. 1000): flat line, underfit. Too small (e.g. 0.0001): overfit.
O1Open: Why is L1 a poor choice with correlated features?It arbitrarily keeps one of the correlated features and zeroes the rest; L2 spreads weight across them.
O2Open: Why is training error a bad predictor of test error?A complex model can memorize the training set (near-zero train error) and still generalize badly.

WEEK 3CNNs and classic architectures

1Two key ideas that make CNNs efficient?Local receptive fields and weight sharing.
2Typical CNN pipeline?Conv → activation → normalization → pooling → FC → softmax.
3Conv output size formula?(N − F + 2P)/S + 1.
432×32×3 image, six 5×5×3 filters, stride 1, no pad?28×28×6.
5FC weights from a 32×32×3 image to 10 classes?3072 × 10.
6Max vs average pooling?Max keeps the strongest activation; average smooths. Both shrink size and add robustness.
7Local contrast normalization?Subtract the local mean, divide by local std: I′ = (I − m)/σ; robust to illumination.
8Why padding?Keeps spatial size and preserves border information.
9LeNet-5 key facts?LeCun 1998, 32×32 grayscale, conv/subsample layers C1 S2 C3 S4 C5 F6, tanh/sigmoid, avg pooling, 10 outputs.
10AlexNet key facts?2012, 227×227×3, 5 conv + 3 FC, ReLU, overlapping max pool, LRN, dropout, augmentation, ~60M params, 2 GPUs.
11AlexNet CONV1?96 filters, 11×11, stride 4 (227 → 55, so 55×55×96).
12VGG design rule?Only 3×3 conv (stride 1, pad 1) and 2×2 max pool; VGG-16/19, 224×224 input.
13Why three 3×3 convs instead of one 7×7?Same 7×7 receptive field, fewer params (27 vs 49 per channel pair), more non-linearity.
14VGG first conv params?3·3·3·64 = 1,728 (second conv 3·3·64·64 = 36,864).
15GoogLeNet key facts?ILSVRC 2014 winner, 22 layers, 6.7% top-5, ~5M params (12× fewer than AlexNet), no FC, global avg pool, auxiliary classifiers.
16What is an Inception module?Parallel 1×1, 3×3, 5×5 convs and 3×3 max pool, concatenated along depth.
17Role of 1×1 convs in Inception?Bottleneck: cut channels while keeping H×W; naive 854M ops drops to 358M.
18Degradation problem?Deeper plain networks get higher training error, so it is an optimization problem, not overfitting.
19Max derivative of sigmoid and tanh?Sigmoid 0.25, tanh 1. Multiplying many of them shrinks gradients.
20Fixes for vanishing gradient?ReLU, residual connections, batch norm, Xavier/He init, adaptive optimizers.
O1Open: Why isn't deeper always better?Vanishing gradients, degradation (harder optimization), overfitting and the credit assignment problem.
O2Open: Why does a 1×1 conv save so much compute?It shrinks channel depth before expensive 3×3/5×5 convs, and cost scales with input channels.

WEEK 4ResNet, DenseNet, MobileNet, EfficientNet, RNN intro

1What does a residual block learn?The residual F(x) = H(x) − x; output is F(x) + x.
2Why is identity easy for ResNet?If W, b go to 0, a[l+2] = g(a[l]), so the block just passes input through.
3Gradient through a skip connection?∂L/∂x = ∂L/∂y · (∂F/∂x + 1); the +1 keeps gradient flowing.
4Extra parameters added by skip connections?None.
5ResNet stem and head?Stem: 7×7 conv 64 stride 2, BN, ReLU, 3×3 max pool. Head: global avg pool + one FC (1000) + softmax.
6Basic vs bottleneck block?ResNet-18/34: two 3×3 convs. ResNet-50/101/152: 1×1 → 3×3 → 1×1 (256 → 64 → 256).
7DenseNet connection count for L layers?L(L+1)/2; 6 layers give 21.
8DenseNet vs ResNet merge?DenseNet concatenates all previous feature maps; ResNet adds.
9DenseNet growth rate k?Number of new feature maps each layer adds.
10DenseNet transition layer?BN + 1×1 conv + average pool (between dense blocks).
11Depthwise separable convolution?Depthwise (one filter per input channel) + pointwise 1×1 (mixes channels).
12MobileNet V2 and V3 additions?V2: inverted residuals, linear bottlenecks. V3: squeeze-excitation and NAS.
13EfficientNet compound scaling?Depth α^φ, width β^φ, resolution γ^φ with α·β²·γ² ≈ 2.
14Where does EfficientNet-B0 come from?Neural architecture search; MBConv blocks, squeeze-excitation, Swish. B1 to B7 are scaled up.
15RNN hidden-state update?h_t = tanh(W_h h_(t−1) + W_x x_t), y_t = W_o h_t.
16What is shared across RNN time steps?The same weights at every step.
17RNN type for image captioning? sentiment?Captioning: one-to-many. Sentiment: many-to-one.
18RNN type for MT? for video frame labelling?MT: many-to-many with different lengths (seq2seq). Frame labelling: many-to-many, same length.
O1Open: Why does depthwise separable conv cost so much less?It splits spatial filtering and channel mixing, so you never do a full K·K·C_in filter for every output channel. Ratio ≈ 1/C_out + 1/K².
O2Open: Why scale depth, width and resolution together?They are coupled: higher resolution needs more depth (receptive field) and width (fine patterns); scaling one alone saturates fast.

WEEK 5BPTT, LSTM, attention, GRU

1BPTT in one line?Unroll the RNN over time and backprop; gradients are summed over steps because weights are shared.
2Total RNN loss?L = Σ_t L_t.
3Why do RNN gradients vanish?∂h_t/∂h_i is a product of W_hᵀ · diag(tanh′) terms, and tanh′ ≤ 1.
40.9^50 ≈ ?≈ 0.005 (shows how fast products shrink).
5Truncated BPTT?Backprop only a fixed number of steps; keep carrying the hidden state forward.
6Char-level RNN example on slides?Vocab {e, a, g, r} as one-hot vectors, training word "eager".
7How does a char-RNN generate at test time?Sample from the softmax and feed the sampled char back as the next input.
8Bidirectional RNN uses?Both past and future context (forward + backward pass).
9LSTM proposed by?Hochreiter and Schmidhuber, 1997.
10LSTM gates?Forget f, input i, output o (plus candidate cell C̃).
11LSTM cell update?C_t = f_t ⊙ C_(t−1) + i_t ⊙ C̃_t.
12LSTM hidden state?h_t = o_t ⊙ tanh(C_t).
13Why LSTM resists vanishing gradients?Additive cell update (Constant Error Carousel) lets gradient flow nearly unchanged.
14Seq2seq structure?Encoder (many-to-one) + decoder (one-to-many).
15Seq2seq bottleneck?Whole input squeezed into one fixed-length context vector; fails on long inputs.
16Attention context vector?c_t = Σ_i α_(t,i) h_i, with α = softmax(e), e_(t,i) = f_att(s_(t−1), h_i).
17GRU gates and author?Update z and reset r; no separate cell. Cho et al., 2014.
18GRU hidden update?h_t = z_t ⊙ h_(t−1) + (1 − z_t) ⊙ h̃_t.
19GRU vs LSTM?GRU: 2 gates, fewer params, faster; LSTM: 3 gates + cell state.
O1Open: How does attention fix the seq2seq bottleneck?The decoder looks at all encoder states at every step and builds a fresh context vector, so nothing has to fit in one vector.
O2Open: "The kids ... are" vs "The kid ... is": what does this example show?Long-term dependency: the verb depends on a word far back, which a plain RNN forgets because of vanishing gradients.

WEEK 6GenAI for vision tasks (theory)

1The 7 CV challenges?Viewpoint, scale, deformation, occlusion, illumination, background clutter, intra-class variation.
2How does GenAI help classification?Synthetic augmentation, more data for rare classes, semi/unsupervised feature learning.
3Conventional object detection pipeline?Hand-crafted features (HOG, SIFT) + sliding window or selective search + classifier.
4Deep detectors named on slides?YOLO and Faster R-CNN.
5How does GenAI help detection?Generates diverse object variants and rare-class samples for training.
6Semantic vs instance segmentation?Semantic labels every pixel by class; instance also separates each object of the same class.
7SAM stands for?Segment Anything Model.
8Super-resolution with GenAI vs classic?Generates realistic textures and detail instead of blurry upsampling.
9Image inpainting?Filling missing or masked regions plausibly (e.g. Stable Diffusion inpainting).
10Image-to-image translation?Change the domain, keep the structure (sketch → photo, day → night).
11Why synthesize medical images?Limited data, class imbalance, patient privacy.
12Anomaly detection with generative models?Learn the normal-data distribution; flag low likelihood / high reconstruction error. VAE, GAN, diffusion, flows.
13Style transfer?Content of one image + style of another.
14Main challenge in video generation?Spatial quality plus temporal coherence across frames.
15Name some CV application areas.Face recognition, medical imaging, autonomous vehicles, surveillance, agriculture, manufacturing, OCR, AR, robotics.
O1Open: Why is anomaly detection a natural fit for generative models?Anomalies are rare and unlabelled; a model of "normal" data flags anything it cannot explain well.
O2Open: Why is video harder than image generation?Every frame must look real and also stay consistent with neighbours in motion, identity and lighting.

WEEK 7Generative models, autoregressive, Transformers, ViT

1Explicit vs implicit density examples?Explicit: autoregressive, flows, VAE. Implicit: GAN.
2Tractable vs approximate explicit density?Tractable: FVBN (PixelRNN/CNN, NADE, MADE), flows. Approximate: VAE (variational), Boltzmann machine (Markov chain).
3MLE objective?θ* = argmax Σ log p_θ(x_i), i.e. minimize negative log-likelihood.
4Autoregressive factorization of an image?p(x) = Π p(x_i | x_<i) over pixels.
5Per-pixel colour factorization?p(R) · p(G|R) · p(B|R, G).
6PixelRNN generation order?From the top-left pixel, one pixel at a time (raster order).
7Row LSTM vs Diagonal BiLSTM?Row LSTM: row by row, triangular context, misses some pixels. Diagonal BiLSTM: skews the map, full context.
8PixelCNN training vs generation?Masked convs let training run in parallel; generation is still sequential.
9Show, Attend and Tell (Xu 2015)?Captioning with attention over H×W×D CNN features; a new context vector for each word.
10Why divide attention scores by √D?Large dot products saturate softmax; scaling keeps gradients healthy.
11Scaled dot-product attention?softmax(QKᵀ/√d_k) V.
12Self-attention?Q, K, V all come from the same sequence.
13Masked self-attention?Future positions get −∞ before softmax, so they get zero weight.
14Multi-head example on slides?512-d input split into 4 heads of 128, run in parallel, concatenated.
15Positional encoding options?Fixed sinusoids (sin/cos) or a learned lookup table (T × d params).
16Add & Norm?LayerNorm(x + Sublayer(x)).
17BERT and GPT architectures?BERT: encoder-only. GPT: decoder-only. Original Transformer: encoder-decoder.
18ViT pipeline?16×16 patches, flatten (16·16·3 = 768), linear embed, add CLS token + learnable position embeddings, encoder-only. Dosovitskiy ICLR 2021.
19DETR?Encoder-decoder with learned object queries, set loss with bipartite matching, no NMS. Carion 2020.
20Swin Transformer?Attention in local windows (linear complexity), shifted windows for cross-window links, patch merging for hierarchy.
O1Open: CNN vs Transformer on images?CNNs build global context slowly through stacked local filters; self-attention relates every patch to every other from the first layer.
O2Open: Why does DETR not need NMS?Bipartite matching assigns exactly one prediction per object during training, so duplicates are penalized.

WEEK 8VAE and diffusion

1Autoencoder uses?Dimensionality reduction, feature learning, denoising, reconstruction, anomaly detection.
2Why is a plain autoencoder a poor generator?Its latent space is not regularized, so random latent points decode to garbage.
3VAE encoder output?μ and σ of q_φ(z|x) = N(μ, σ²).
4Why is p(x) intractable in a VAE?p(x) = ∫ p(x|z)p(z) dz integrates over all z.
5ELBO?E_q[log p(x|z)] − KL(q(z|x) || p(z)): reconstruction minus KL.
6Which inequality gives the ELBO?Jensen: log E[Y] ≥ E[log Y].
7Relation of ELBO and log p(x)?log p(x) = ELBO + KL(q || p(z|x)), so ELBO ≤ log p(x).
8VAE prior?Standard normal N(0, I).
9Reparameterization trick?z = μ + σ ⊙ ε, ε ~ N(0, I), so gradients flow through sampling.
10Why the KL term?Regularizes the latent space: smooth, continuous, sampleable.
11Diffusion forward step?q(x_t | x_(t−1)) = N(√(1−β_t) x_(t−1), β_t I).
12Closed-form noising?x_t = √ᾱ_t x_0 + √(1−ᾱ_t) ε, with α_t = 1−β_t, ᾱ_t = Π α.
13β schedule direction?β increases over time (α decreases), so x_T ≈ N(0, I).
14What does the DDPM network predict?The added noise ε_θ(x_t, t); loss L_simple = E‖ε − ε_θ(x_t, t)‖².
15DDPM paper?Ho, Jain, Abbeel, NeurIPS 2020.
16Reverse sampling step?x_(t−1) = μ_θ(x_t, t) + σ_t z.
17Classifier guidance?Separate classifier; add ∇ log p(c | x_t) to steer sampling toward class c.
18Classifier-free guidance?One model trained with and without condition: ε̂ = ε(∅) + s(ε(c) − ε(∅)). Scale s trades diversity for fidelity.
19Why latent diffusion (LDM)?Diffusion runs in a compressed VAE latent space, much cheaper.
20Stable Diffusion pipeline?Text encoder → U-Net denoising in latent with cross-attention → VAE decoder. Rombach CVPR 2022.
O1Open: Why predict noise instead of the clean image?Noise is a simple, well-scaled target at every timestep, so training is easier and more stable.
O2Open: VAE vs diffusion quality?VAEs are fast but blurry (Gaussian decoder, KL pressure); diffusion is slower but much sharper via many small denoising steps.

WEEK 9GANs

1GAN paper?Goodfellow et al., 2014.
2G and D roles?G makes fakes from noise; D outputs the probability that an input is real.
3GAN objective?min_G max_D E[log D(x)] + E[log(1 − D(G(z)))].
4How are D and G updated?Alternately: D by gradient ascent (maximize V), G by descent (minimize V).
5Equilibrium?p_g = p_data and D outputs 0.5 everywhere.
6Mode collapse?G produces only a few kinds of outputs; little variety.
7Other GAN training problems?Instability, non-convergence (oscillation), D too strong (vanishing G gradients), hyperparameter sensitivity, hard evaluation.
8Minibatch discrimination?D sees statistics across the batch; similar samples look fake, forcing diversity.
9WGAN distance?Wasserstein (Earth Mover) with a 1-Lipschitz critic that outputs a score, not a probability.
10WGAN losses?Critic: −E[C(x)] + E[C(G(z))]. Generator: −E[C(G(z))].
11cGAN?Condition y is fed to both G and D.
12Pix2Pix?Paired image-to-image translation; loss = cGAN + λ L1. Isola 2017.
13Why L1 in Pix2Pix?Keeps output close to the ground-truth pair; L1 blurs less than L2.
14CycleGAN?Unpaired translation with G: X→Y, F: Y→X and two discriminators. Zhu ICCV 2017.
15Cycle-consistency loss?‖F(G(x)) − x‖₁ + ‖G(F(y)) − y‖₁.
16CycleGAN examples?Horse ↔ zebra, summer ↔ winter.
O1Open: Why does CycleGAN need cycle consistency?Without pairs, G could map every horse to any zebra; requiring F(G(x)) ≈ x forces it to keep the input's content.
O2Open: Why does WGAN help training?Wasserstein distance gives smooth, useful gradients even when real and fake distributions do not overlap, reducing instability and mode collapse.

WEEK 10LLMs and VLMs

1LLM training objective?Next-token prediction: P(X) = Π P(x_t | x_<t).
2LLM loss?L = −Σ log P(x_t | x_<t) (cross-entropy).
3Three LLM architectures with examples?Encoder-only (BERT, HuBERT); encoder-decoder (T5, Flan-T5, Whisper); decoder-only (GPT, Claude, Llama, Mixtral).
4Encoder-only pretraining?Masked language modelling, bidirectional.
5LLM output layer?Hidden state (size d) → linear to |V| → softmax.
6Tokenization levels?Character, word, sub-word.
7BPE algorithm?Start from characters, repeatedly merge the most frequent pair. Slide corpus low/lower/newest/widest: first merge e + s = es.
8Why BPE?Handles unknown words, small vocab, reuses common sub-words.
9Teacher forcing?During training the true previous token is fed in, not the model's own prediction.
10Inference loop?Predict a token, append it, repeat until [EoS].
11Greedy decoding downside?Deterministic and repetitive.
12Top-k vs top-p?Top-k: keep the k most probable tokens. Top-p (nucleus): smallest set with cumulative prob ≥ p (e.g. 0.9).
13Temperature effect?P_i = e^(z_i/T)/Σ e^(z_j/T). T < 1 sharper, T > 1 flatter / more random.
14Token rule of thumb?1 token ≈ 4 characters ≈ 0.75 words.
15Pretraining data sources?Common Crawl, C4 (~156B tokens), The Pile; filtered and deduplicated.
16GPT-3 scale on slides?175B parameters, ~300B training tokens.
17Perplexity of 10 means?The model is as unsure as picking uniformly among 10 tokens.
18Fine-tuning types?Full, instruction, domain, PEFT (LoRA, adapters), SFT, preference (RLHF, DPO).
19VLM components?Vision encoder (CNN/ViT) + projection layer + language model.
20VLM training stages?Vision-language pretraining → alignment (train projector) → multimodal instruction tuning.
21Known VLM weaknesses?Hallucination, counting, spatial reasoning, OCR, fine detail, video/temporal reasoning.
O1Open: Why compare perplexity only on the same dataset and tokenizer?Perplexity is per token; a different tokenizer changes what a token is, so the numbers are not comparable.
O2Open: In a VLM, why do text queries attend to image keys/values?Cross-attention lets each text token pull in the visual details it needs from the image features.

WEEK 11CLIP and multimodal generative models

1CLIP paper?Radford et al., ICML 2021 (OpenAI).
2CLIP encoders?Image: ResNet or ViT. Text: Transformer.
3CLIP training objective?Contrastive: matching image-text pairs close, all other pairs in the batch far apart.
4CLIP similarity?Dot product of L2-normalized embeddings, i.e. cosine similarity.
5CLIP loss?Symmetric: ½(L_I→T + L_T→I), InfoNCE with temperature τ.
6Zero-shot classification steps?Turn class names into prompts, encode them, compare cosine with the image embedding, take argmax.
7Standard prompt template?"a photo of a {class}".
8Prompt ensembling?Average text embeddings over K templates: t_c = (1/K) Σ f_T(T_c,k).
9Why CLIP needs no classification head?Classes are just text, so the label set is open-vocabulary.
10Sparse vs dense image features?Sparse: region features from detectors (R-CNN, Faster R-CNN, YOLO, Mask R-CNN). Dense: CNN feature maps or ViT tokens.
11Text feature shape?Batch × sequence length × hidden size.
12Early vs middle vs late fusion?Early: combine at input/feature level. Middle: separate encoders then cross-modal interaction. Late: combine decisions.
13Fusion operators named?Max, multiplicative / gated tanh(Wu) ⊙ Vv, weighted concat, bilinear uᵀWv, compact bilinear (FFT).
14Multimodal LLM?Pretrained LLM as the reasoning core + modality encoders + projection.
15Visual-semantic embedding?Joint image-text space for retrieval in both directions.
16Reed et al. 2016?GAN text-to-image: text embedding conditions both G and D; birds and flowers datasets.
17Captioning model on slides?Show, Attend and Tell: P(y|I) = Π P(y_t | I, y_<t).
18Multimodal training stages?Unimodal pretraining → cross-modal alignment → multimodal instruction tuning → preference optimization.
19Multimodal challenges?Hallucination, cross-modal misalignment, fine-grained understanding.
O1Open: Why does CLIP generalize zero-shot?Trained on huge, varied image-text pairs, it learns concepts tied to language, so any new class name can be matched.
O2Open: Early vs late fusion trade-off?Early fusion captures fine interactions but is heavy and rigid; late fusion is simple and modular but misses cross-modal detail.

WEEK 12Agentic AI and research trends

1Agentic AI definition?AI that pursues goals by making decisions and taking actions: interpret, decompose, pick tools, execute, inspect, revise.
2Generative vs agentic AI?Generative makes content in response to input; agentic runs an action loop toward a goal.
3Core agent properties?Goal-directedness, perception/context, reasoning, planning, tool use, memory, feedback/adaptation.
4The agentic loop?Goal → Perception → Reasoning → Planning → Action → Observation → Evaluation → Updated state.
5Orchestrator role?Coordinates the reasoning model, tools, memory and state through the loop.
6Policy / governance layer?Permissions, budgets, safety rules, human approval across the whole system.
7"LLM as the brain" means?The LLM decides the action; the framework / tool layer actually executes it.
8Re-planning?Revising the plan after new observations or failures.
9Task decomposition structure?A DAG of subtasks; mind the granularity (too fine adds overhead).
10Function calling?Structured tool(name, arguments) actions with argument validation.
11Working vs long-term memory?Working: current goal, plan, recent observations. Long-term: databases, vector stores, knowledge graphs.
12Agentic RAG vs plain RAG?Retrieval happens iteratively inside the loop, deciding what to fetch next.
13Reflection?Critique → revise → verify own output.
14ReAct?Interleave Reason → Act → Observe.
15Other planning strategies?Plan-and-execute, tree/graph search, hierarchical planning.
16When is human-in-the-loop needed?High-impact or irreversible actions; use approval gates.
17Multi-agent patterns?Supervisor, peer-to-peer, pipeline, debate/review, blackboard.
18Workflow types?Deterministic, agentic, hybrid.
19Agent safety risks?Prompt injection, data leakage, excessive permissions, unauthorized tool use.
20Agent evaluation dimensions?Task completion, reasoning quality, tool correctness, robustness, efficiency (cost, latency).
21Research trends on the conclusion slides?Diffusion Transformers (DiT), flow matching / rectified flow, VLMs, text-to-video, 3D/4D and world models, synthetic data.
O1Open: Why separate deciding (LLM) from executing (framework)?The execution layer can enforce permissions, validate arguments and log actions, which is safer and easier to debug.
O2Open: When is a deterministic workflow better than an agent?When steps are fixed and known: cheaper, faster, predictable. Use an agent only where the path depends on what it observes.