NLP Interview Questions for Freshers

Tokenization splits text into tokens (words, subwords, or characters). Approaches: (1) Word tokenization — split on spaces/punctuation, simple but can't handle unknown words, (2) Subword tokenization — BPE (GPT), WordPiece (BERT), SentencePiece (T5) — breaks rare words into known subword units, handles any input, (3) Character tokenization — each character is a token, no OOV but very long sequences. Modern LLMs use subword tokenization as the best trade-off between vocabulary size and sequence length.

TF-IDF (Term Frequency-Inverse Document Frequency): TF = how often a word appears in a document; IDF = log(N/df) penalizes common words. TF-IDF produces sparse, interpretable vectors. Use TF-IDF when: (1) You need keyword-based exact matching, (2) Your corpus is small and domain-specific (legal, medical jargon), (3) Inference speed is critical (milliseconds vs. 50ms for neural), (4) Interpretability matters (explain which words caused a classification). Use neural embeddings when: semantic similarity matters ("car" ≈ "automobile"), you have multilingual text, or handling paraphrases.

Transfer learning pretrains a model on massive text corpora (internet-scale) to learn general language understanding, then fine-tunes on small task-specific datasets. Why effective: (1) Language has universal structure — grammar, semantics, world knowledge — transferable across tasks, (2) LLM pretraining costs millions of dollars; fine-tuning costs hundreds, (3) Fine-tuning needs 1000s of examples vs. millions for training from scratch. The paradigm shift: GPT-3 showed that sufficiently large pretrained models can do new tasks with just a few examples (few-shot), eliminating fine-tuning entirely for many tasks.