Deep Learning Interview Questions for Senior (7+ Years)

Transformer components: (1) Input embeddings + positional encodings (sine/cosine or learned), (2) Multi-head self-attention: Q=XW_Q, K=XW_K, V=XW_V; attention = softmax(QK^T/√d_k)V; multiple heads capture different relationships, (3) Add & Norm: residual connection + LayerNorm after each sublayer — critical for training stability, (4) Feed-forward network: two linear layers with GELU activation — processes each position independently, (5) Encoder stacks these; decoder adds cross-attention to encoder output. Key design choices: residual connections prevent vanishing gradients; LayerNorm stabilizes training; positional encoding injects sequence order.

Knowledge distillation trains a small "student" model to mimic a large "teacher" model's output probabilities (soft labels), not just the hard labels. Soft labels encode richer information — e.g., 0.85 dog, 0.10 wolf, 0.05 cat tells the student about feature relationships. Loss = α × CrossEntropy(student_hard, labels) + (1-α) × KLDiv(student_soft, teacher_soft) with temperature T to soften distributions. Use cases: (1) Deploy smaller model on mobile/edge, (2) Compress BERT → DistilBERT (40% smaller, 97% performance), (3) Ensemble distillation. For LLMs: distill GPT-4 outputs into a smaller model for a specific task.

Bottleneck identification: torch.profiler shows CPU/GPU time per operation. Common bottlenecks: (1) Data loading — use DataLoader(num_workers=4, pin_memory=True, prefetch_factor=2), (2) GPU underutilization — increase batch size until GPU memory is ~80% full, (3) Float32 → use Automatic Mixed Precision (torch.cuda.amp.autocast()) for 2–3x speedup on modern GPUs, (4) Gradient accumulation — simulate larger batches without OOM, (5) torch.compile() (PyTorch 2.0) fuses operations for ~20% speedup, (6) Gradient checkpointing — trades compute for memory in very deep models. Always profile before optimizing.