Deep Learning Interview Questions for 2 Years Experience

Backpropagation computes gradients of the loss with respect to all parameters using the chain rule. Forward pass: compute predictions and loss. Backward pass: starting from loss, compute ∂L/∂output, propagate backwards through each layer using chain rule: ∂L/∂W = ∂L/∂output × ∂output/∂W. Gradients accumulate in .grad attributes. Then optimizer updates: W = W - lr × ∂L/∂W. Key insight: gradients flow backward through the same computational graph used in forward pass. PyTorch autograd builds this graph dynamically — every tensor operation records itself for backward computation.

CNN: applies learned filters over local receptive fields, translation-invariant, excels at spatial data (images). O(n) operations, fixed context size. RNN/LSTM: processes sequences step-by-step, maintains hidden state, good for sequential data but slow (sequential dependency) and struggles with long-range. O(n) sequential operations. Transformer: self-attention connects all positions directly, O(n²) memory but O(1) sequential depth, excellent long-range dependencies, fully parallelizable. Trend: transformers replaced RNNs for NLP entirely; Vision Transformers (ViT) are replacing CNNs for many vision tasks at scale.

Batch normalization normalizes layer activations to zero mean, unit variance across the batch, then scales/shifts with learned parameters γ and β. Benefits: (1) Reduces internal covariate shift — stable input distributions to each layer, (2) Allows higher learning rates — gradients don't explode, (3) Acts as mild regularizer — adds noise via batch statistics, (4) Reduces sensitivity to weight initialization. Placement: before or after activation (before is more common). Variants: LayerNorm (normalizes across features, not batch — used in transformers), GroupNorm (for small batches), InstanceNorm (style transfer).

Dropout randomly zeros out neuron activations with probability p during training, forcing the network to learn redundant representations. During inference, no dropout — activations scaled by (1-p) to match expected values. Why it works: prevents co-adaptation (neurons relying on specific other neurons), approximates ensemble of 2^n different networks. Use dropout: (1) After dense/linear layers (p=0.1–0.5), (2) Not after batch norm (conflicting noise), (3) Less effective in CNNs (use instead: DropBlock), (4) Transformers use attention dropout + residual dropout (p=0.1). Too high dropout (>0.5) causes underfitting — tune on validation set.

Classification: CrossEntropyLoss (multi-class), BCEWithLogitsLoss (binary/multi-label — numerically stable). Regression: MSELoss (when outliers are few), L1Loss/MAELoss (robust to outliers), HuberLoss (best of both). Object detection: Focal Loss for class imbalance, IoU loss for bounding boxes. Generative: adversarial loss (GANs), ELBO (VAEs), reconstruction + KL divergence. LLMs: CrossEntropy on next-token prediction. Tips: always use logit-space losses (CrossEntropyLoss not NLLLoss+softmax) for numerical stability; class weights for imbalanced data; label smoothing (0.1) to prevent overconfident predictions.

Transfer learning reuses a pretrained model's weights as initialization for a new task. Strategies by how different the target domain is: (1) Feature extraction — freeze all layers, only train new head (target data very small or similar domain), (2) Fine-tuning last few layers — unfreeze top layers, small LR, (3) Full fine-tuning — unfreeze all, very small LR (2e-5), (4) Progressive unfreezing — start by training only head, gradually unfreeze layers from top to bottom (ULMFiT approach). Key: use discriminative learning rates — lower LR for early layers (generic features), higher for later layers (task-specific features).

SGD with momentum: accumulates velocity in gradient direction, dampens oscillations, good for convex problems. Adam (Adaptive Moment Estimation): maintains per-parameter adaptive learning rates using first moment (mean) and second moment (variance) of gradients; default for deep learning. AdamW: Adam + decoupled weight decay (L2 regularization applied to weights, not adaptive gradients) — better generalization, preferred for transformers. Lion: sign-based optimizer, memory-efficient, recently popular for LLM pretraining. Learning rate schedule: warmup (0 → peak over first 10% steps) + cosine decay or linear decay.

Underfitting (high bias): model too simple, high training loss, high validation loss — both similar. Fix: increase model capacity, train longer, reduce regularization, better features. Overfitting (high variance): model memorized training data, low training loss, high validation loss — large gap. Fix: more data, dropout, L2 regularization, data augmentation, early stopping, reduce model size. Diagnosis: plot training vs validation loss curves over epochs. Ideal: both decrease, then validation plateaus. Use validation set (not test set!) for all tuning decisions — the test set is touched exactly once for final evaluation.

CNN processes images through: (1) Convolutional layers — learnable filters slide over input, detect local patterns (edges → textures → shapes → objects), (2) ReLU activation — non-linearity after each conv, (3) Pooling — MaxPool/AvgPool reduces spatial dimensions, builds spatial invariance, (4) Global Average Pooling — collapses spatial dimensions before dense layers, (5) Dense layers + softmax — final classification. Key properties: parameter sharing (same filter across all positions), local connectivity, translation equivariance. Modern CNNs: ResNet (skip connections for very deep networks), EfficientNet (compound scaling of depth/width/resolution).

A residual connection adds the input of a block directly to its output: output = F(x) + x, where F(x) is the learned transformation. Why critical: (1) Solves vanishing gradients in very deep networks — gradients can flow directly through skip connections without passing through many layers, (2) Makes optimization easier — the network only needs to learn the residual (difference) rather than a full transformation from scratch, (3) Allows training of 100+ layer networks (ResNet-152). Without residual connections, deep networks performed worse than shallow ones due to optimization difficulty — not overfitting.

Early stopping monitors a validation metric and stops training when it stops improving, saving the best checkpoint. Implementation: track best_val_loss, counter for patience. Each epoch: if val_loss < best_val_loss: save model, reset counter; else: counter++; if counter >= patience: break. Use patience=5–10 epochs. Save with torch.save(model.state_dict(), "best_model.pt") at each improvement. Load best model after training ends. Important: checkpoint the model state, not just the metric — you'll restore to the best point, not the last point. Use with ReduceLROnPlateau for automatic LR reduction before stopping.

Image augmentation: random horizontal/vertical flip, rotation (±15°), crop + resize, color jitter (brightness/contrast/saturation), cutout/cutmix/mixup, perspective transforms, elastic deformations. For medical imaging: conservative augmentation (no unnatural colors). Text augmentation: synonym replacement, random insertion/deletion, back-translation, EDA. Audio: time stretching, pitch shifting, noise addition, SpecAugment (mask frequency bands). Key principle: augmentations must preserve the label — flipping "6" to "9" in digit recognition changes the label! Albumentations library for image, torchvision.transforms for standard pipelines.