How would you design a scalable RAG system that handles 10,000 queries per minute?
Architecture: (1) API Layer — FastAPI behind ALB, auto-scale ECS tasks, (2) Query Processing — async request handling, query expansion with LLM, (3) Retrieval — Pinecone with multiple index replicas, connection pool, (4) Cache Layer — Redis for semantic caching (embed query, find nearest cached query within cosine sim 0.95), (5) LLM Layer — multiple OpenAI API keys with round-robin + fallback to Azure OpenAI, (6) Monitoring — request tracing, latency p99, cost per query. At 10k QPM, caching becomes critical — aim for 60%+ cache hit rate.
Design a real-time ML feature pipeline for an AI recommendation system.
Components: (1) Data Ingestion — Kafka streams for user events, (2) Stream Processing — Flink/Spark Streaming for real-time feature computation, (3) Feature Store — Feast with online store (Redis for <5ms reads) + offline store (S3/BigQuery for training), (4) Model Serving — TorchServe/Triton with A/B testing support, (5) Monitoring — Evidently for data drift, Prometheus for latency. Key challenge: keeping online and offline features consistent. Use same feature computation logic for both.
Unlock 3,000+ Interview Questions
Get full access to all interview questions with detailed answers, explanations, and real company context
Unlock Full Access →