System Design Reliability, Security, and Operations Flashcards

24 sample cards from the Reliability, Security, and Operations deck

Sampled from the end of the deck, so these are different cards from the ones shown on the syllabus page.

1. What is a Kubernetes Deployment vs a ReplicaSet vs a Service?

   A ReplicaSet ensures a specified number of identical Pod replicas are running. A Deployment manages ReplicaSets to provide declarative updates and rolling upgrades/rollbacks. A Service provides a stable virtual IP/DNS name and load-balances traffic across the matching Pods, decoupling clients from ephemeral Pod IPs.

2. What does the Horizontal Pod Autoscaler (HPA) do?

   The HPA automatically scales the number of Pod replicas up or down based on observed metrics (e.g., CPU utilization or custom metrics) against a target, so capacity tracks load. It changes replica count, unlike the Vertical Pod Autoscaler which changes per-Pod resource requests/limits.

3. Compare rolling, blue-green, and canary deployment strategies.

   Rolling: gradually replace old instances with new ones in batches — no extra full environment, but old and new run together briefly. Blue-green: run two full environments (blue=current, green=new); switch all traffic at once and roll back by switching back — fast rollback, double resources. Canary: release the new version to a small % of traffic first, monitor, then gradually ramp — limits blast radius of a bad release.

4. What is a feature flag and how does it decouple deployment from release?

   A feature flag is a runtime toggle that turns a code path on/off without redeploying. It lets teams deploy code to production dark (disabled) and later release (enable) it — possibly to a subset of users — enabling canary releases, A/B tests, and instant kill-switch rollback independent of the deploy pipeline.

5. What is Infrastructure as Code (IaC) and its key benefits?

   IaC is managing and provisioning infrastructure through machine-readable definition files rather than manual configuration. Benefits: version control, repeatability/consistency across environments, automated and auditable changes, easy disaster recovery/rebuild, and code review of infrastructure changes.

6. What is the difference between declarative and imperative IaC, and between provisioning and configuration-management tools?

   Declarative IaC specifies the desired end state and the tool computes how to reach it (e.g., Terraform); imperative specifies the exact steps/commands to execute. Provisioning tools (e.g., Terraform, CloudFormation) create infrastructure resources; configuration-management tools (e.g., Ansible, Chef, Puppet) configure software on existing machines. They are complementary.

7. What is the difference between immutable and mutable infrastructure?

   Mutable infrastructure is updated in place (patch/config existing servers), risking configuration drift and snowflake servers. Immutable infrastructure never modifies running servers — to change anything you build a new image/instance and replace the old one, guaranteeing consistency and easy rollback by redeploying a prior image.

8. What is the difference between Continuous Integration, Continuous Delivery, and Continuous Deployment?

   Continuous Integration (CI): developers merge to a shared branch frequently, each change auto-built and tested. Continuous Delivery: every passing build is automatically prepared and can be released to production with a manual approval click. Continuous Deployment: every passing build is automatically released to production with no manual gate.

9. What are typical stages of a CI/CD pipeline?

   Source/trigger (commit or PR) → Build (compile, package artifacts) → Test (unit, integration, security/lint scans) → Artifact/publish (store image) → Deploy to staging → Automated acceptance/smoke tests → Deploy to production (with a strategy like canary), often followed by monitoring/rollback. Each stage gates the next.

10. Why is clock synchronization hard in distributed systems, and what does NTP provide?

    Each machine has its own physical clock that drifts, and message delays are variable, so no two clocks agree exactly. NTP (Network Time Protocol) synchronizes clocks against reference time servers, estimating and correcting offset and round-trip delay, typically achieving millisecond-level accuracy — but it cannot guarantee perfectly synchronized clocks, so ordering by wall-clock timestamps is unreliable.

11. What problem do Lamport logical clocks solve, and what is their key rule?

    Lamport clocks provide a consistent logical ordering of events without synchronized physical clocks, capturing the 'happens-before' relation. Rule: each process increments its counter on every event; when sending a message it attaches its counter; on receipt the receiver sets its counter to $\max(\text{local}, \text{received}) + 1$. If event $a$ happens-before $b$, then $C(a) < C(b)$.

12. What is the limitation of Lamport clocks that vector clocks overcome?

    With Lamport clocks, $a \to b$ implies $C(a) < C(b)$, but the converse fails: $C(a) < C(b)$ does not tell you whether $a$ happened-before $b$ or they are concurrent. Vector clocks (a per-process counter vector) capture causality exactly, letting you detect concurrency: two events are concurrent iff neither vector dominates the other.

13. Why not use a single centralized auto-increment counter for unique ID generation in a distributed system?

    A single central counter is a single point of failure and a scalability/throughput bottleneck: every ID request must round-trip to it, and it can't keep up at scale or across regions. Distributed systems instead use schemes (UUIDs, Snowflake, ranges/tickets) that generate IDs locally without global coordination.

14. Describe how a Twitter Snowflake ID is structured.

    A Snowflake ID is a 64-bit integer, typically: 1 unused sign bit, a 41-bit timestamp (milliseconds since a custom epoch), a ~10-bit machine/worker ID, and a ~12-bit per-millisecond sequence number. This yields globally unique, roughly time-sortable (k-sorted) IDs generated locally without central coordination, with up to $2^{12}=4096$ IDs per machine per millisecond.

15. What are the trade-offs of UUIDs vs Snowflake-style IDs?

    UUIDs (128-bit) are generated with no coordination and are collision-safe, but are large and random (v4) — poor for database index locality and not time-ordered. Snowflake IDs are 64-bit, compact, and time-sortable (good index locality), but require coordinated worker-ID assignment and depend on clock behavior (clock skew/rollback can cause issues).

16. State the CAP theorem.

    In a distributed data store, during a network Partition you cannot have both Consistency (every read sees the latest write) and Availability (every request gets a non-error response). You must choose: a CP system rejects/blocks requests to stay consistent, or an AP system stays available but may return stale data. Without a partition, you can have both C and A.

17. What is a network partition (split-brain) and why is it dangerous?

    A network partition is when a failure splits nodes into groups that cannot communicate. Split-brain occurs when both sides independently believe they are the primary and accept writes, leading to divergent, conflicting state and data corruption. It is prevented with quorum/consensus so only one side (the majority) can act.

18. How does quorum prevent split-brain, and what is the majority quorum condition?

    By requiring an action (e.g., electing a leader or committing a write) to be approved by a majority of nodes, only the partition containing more than half the nodes can proceed; the minority side cannot form a quorum and stops accepting writes. For a cluster of $N$ nodes the majority quorum is $\lfloor N/2 \rfloor + 1$, which is why clusters use an odd node count.

19. What is the quorum condition for consistency using read/write quorums (W + R > N)?

    In a replicated store with $N$ replicas, if every write must be acknowledged by $W$ replicas and every read must query $R$ replicas, then setting $W + R > N$ guarantees the read set overlaps the latest write set, so a read sees the most recent write. Tuning $W$ and $R$ trades off read vs write latency and availability.

20. What is the difference between failover and failback?

    Failover is switching operation from a failed primary to a standby/secondary to keep service running. Failback is the subsequent, controlled switch back to the original (now-recovered) primary once it is healthy, restoring the normal configuration — often done during a low-traffic window to avoid another disruption.

21. What is graceful degradation and how does it improve reliability?

    Graceful degradation is designing a system to keep providing reduced but useful functionality when some components fail or are overloaded, instead of failing completely — e.g., serving cached/stale data, disabling non-essential features, or returning default responses. It preserves core user value and contains the impact of partial failures.

22. What is a circuit breaker pattern and its three states?

    A circuit breaker wraps calls to a failing dependency to prevent cascading failures. States: Closed (calls pass through; failures counted), Open (failure threshold exceeded → calls fail fast without contacting the dependency, giving it time to recover), and Half-Open (after a timeout, a few trial calls test recovery; success closes the breaker, failure re-opens it).

23. What is the difference between a leader-based (single-leader) and leaderless replication approach for handling writes?

    Single-leader: all writes go through one leader that orders them and replicates to followers — simple consistency and ordering, but the leader is a bottleneck and a failover point. Leaderless (e.g., Dynamo-style): any replica accepts writes and the system uses quorums and conflict resolution (versioning/read-repair) — higher availability during failures but requires handling conflicts and eventual consistency.

24. Why is exponential backoff with jitter used when retrying failed requests?

    Exponential backoff increases the wait between retries multiplicatively (e.g., $base \cdot 2^{n}$) to avoid hammering a struggling service. Jitter adds randomness to the delay so that many clients don't retry in synchronized waves (the 'thundering herd'), which would otherwise re-overload the recovering service. Together they spread retries out and improve recovery.

What this deck covers

Reliability, Security, and Operations flashcards FAQ