System Design Reliability, Security, and Operations Syllabus

What is 'alert fatigue' and how is it mitigated?

Alert fatigue is desensitization from too many alerts (especially false positives), causing responders to ignore or miss real ones. Mitigations: alert only on actionable, user-impacting symptoms; tune thresholds; deduplicate/group alerts; use severity tiers; and route non-urgent issues to tickets instead of pages.

What are the roles of primary and secondary in an on-call rotation, and what is an escalation policy?

The primary on-call is first to receive and respond to pages; the secondary is a backup who is paged if the primary doesn't acknowledge within a set time. An escalation policy defines the ordered chain (primary → secondary → manager) and timeouts for escalating unacknowledged alerts.

What is a runbook (playbook) in operations?

A runbook is a documented, step-by-step procedure for diagnosing and resolving a specific alert or incident. It lets on-call engineers respond consistently and quickly without re-deriving the solution, reducing MTTR.

Define MTTR, MTBF, and MTTF.

MTTR (Mean Time To Recovery/Repair): average time to restore service after a failure. MTBF (Mean Time Between Failures): average operating time between failures for a repairable system. MTTF (Mean Time To Failure): average lifetime of a non-repairable component. Availability $\approx \frac{\text{MTBF}}{\text{MTBF}+\text{MTTR}}$.

What is the difference between a liveness check and a readiness check?

A liveness check tests whether a process is alive/healthy — if it fails, the orchestrator restarts the instance. A readiness check tests whether the instance is ready to receive traffic — if it fails, the instance is removed from the load balancer but not restarted (e.g., still warming up or a dependency is down).

What is a heartbeat in distributed systems?

A heartbeat is a periodic 'I'm alive' signal a node sends (or a monitor polls) at a fixed interval. Missing a configured number of consecutive heartbeats marks the node as failed/dead, triggering failover or removal. It is the basis of failure detection.

What is the trade-off in setting the heartbeat timeout/interval?

A short timeout detects failures fast but risks false positives (marking a slow/temporarily-partitioned node as dead), causing unnecessary failovers. A long timeout avoids false positives but delays detection, increasing downtime. The value balances detection speed against network jitter and load variability.

What is the difference between authentication and authorization?

Authentication (AuthN) verifies who you are (identity — e.g., password, token, certificate). Authorization (AuthZ) determines what you are allowed to do (permissions/access to resources). Authentication always precedes authorization.

Compare RBAC and ABAC access control models.

RBAC (Role-Based Access Control) grants permissions to roles, and users get permissions by being assigned roles — simple and coarse-grained. ABAC (Attribute-Based Access Control) makes decisions from attributes of the user, resource, action, and environment via policies — fine-grained and context-aware but more complex to manage.

What is the difference between OAuth 2.0 and OpenID Connect (OIDC)?

OAuth 2.0 is an authorization framework: it issues access tokens so a client can access resources on a user's behalf (delegated access). OIDC is an authentication layer built on top of OAuth 2.0 that adds an ID token (a JWT) proving the user's identity. OAuth = authorization; OIDC = authentication.

What are the three parts of a JWT and what is each for?

A JWT has three Base64URL-encoded parts separated by dots: Header (algorithm and token type), Payload (claims such as subject, expiry, roles), and Signature (a MAC/signature over header+payload used to verify integrity and authenticity). Only the signature — not the payload — is protected from tampering; the payload is not encrypted by default.

Distinguish encryption in transit from encryption at rest.

Encryption in transit protects data moving over a network (e.g., TLS/HTTPS) from eavesdropping and tampering on the wire. Encryption at rest protects stored data (disks, databases, backups) so it is unreadable if the storage medium is stolen or accessed directly. Comprehensive security needs both.

What is the difference between symmetric and asymmetric encryption, and how does TLS use both?

Symmetric encryption uses one shared key for encrypt and decrypt (fast, e.g., AES). Asymmetric uses a public/private key pair (slower, e.g., RSA/ECC). TLS uses asymmetric crypto during the handshake to authenticate and securely exchange/derive a symmetric session key, then uses the fast symmetric key for the bulk data — combining the strengths of both.

What does a TLS handshake accomplish?

It authenticates the server (and optionally the client) via certificates, negotiates the protocol version and cipher suite, and establishes a shared symmetric session key (e.g., via ephemeral Diffie-Hellman) — providing confidentiality, integrity, and authenticity for the subsequent session, plus forward secrecy when ephemeral keys are used.

What is a DDoS attack and what are the broad categories?

A Distributed Denial-of-Service attack overwhelms a target with traffic from many sources to exhaust resources and deny legitimate users. Categories: volumetric (saturate bandwidth, e.g., UDP floods, amplification), protocol/state-exhaustion (e.g., SYN floods exhausting connection tables), and application-layer (Layer 7, e.g., HTTP floods mimicking real requests).

What is a WAF and what layer/threats does it address?

A Web Application Firewall operates at Layer 7 (HTTP/HTTPS), inspecting request content to block application-layer attacks like SQL injection, XSS, and malicious payloads using rule sets. Unlike a network firewall (L3/L4), it understands web semantics; it complements but does not replace network-level DDoS protection.

How does rate limiting help mitigate DDoS and abuse, and name one algorithm.

Rate limiting caps the number of requests a client (IP/user/key) can make per time window, throttling floods and abusive callers while allowing normal use. Common algorithms: token bucket, leaky bucket, fixed window, and sliding window log/counter. Token bucket allows bursts up to bucket size while enforcing an average rate.

State the Principle of Least Privilege.

Every user, process, or component should be granted only the minimum permissions necessary to perform its function, and nothing more. This limits the blast radius of compromised or misused credentials by reducing what an attacker can reach.

What is defense in depth?

A security strategy of layering multiple independent controls (network firewalls, WAF, authn/authz, encryption, monitoring, least privilege) so that if one layer is breached, others still protect the system. No single control is a single point of failure for security.

What do 'fail secure / fail closed' and 'secure by default' mean as secure design principles?

Fail secure (fail closed) means that when a component fails or is uncertain, it denies access rather than granting it (default deny). Secure by default means the out-of-the-box configuration is the safe one (e.g., encryption on, ports closed), so security doesn't depend on the user hardening it.

In Kubernetes, what is a Pod and how does it relate to containers?

A Pod is the smallest deployable unit in Kubernetes. It wraps one or more tightly coupled containers that share the same network namespace (IP/port space) and can share storage volumes. Containers in a Pod are always co-scheduled on the same node and scaled together.

What are the roles of the kube-scheduler, kubelet, and etcd in Kubernetes?

kube-scheduler assigns unscheduled Pods to suitable nodes based on resources/constraints. kubelet is the per-node agent that starts/monitors the containers for Pods assigned to its node. etcd is the distributed key-value store holding all cluster state (the single source of truth) for the control plane.

What is a Kubernetes Deployment vs a ReplicaSet vs a Service?

A ReplicaSet ensures a specified number of identical Pod replicas are running. A Deployment manages ReplicaSets to provide declarative updates and rolling upgrades/rollbacks. A Service provides a stable virtual IP/DNS name and load-balances traffic across the matching Pods, decoupling clients from ephemeral Pod IPs.

What does the Horizontal Pod Autoscaler (HPA) do?

The HPA automatically scales the number of Pod replicas up or down based on observed metrics (e.g., CPU utilization or custom metrics) against a target, so capacity tracks load. It changes replica count, unlike the Vertical Pod Autoscaler which changes per-Pod resource requests/limits.

Compare rolling, blue-green, and canary deployment strategies.

Rolling: gradually replace old instances with new ones in batches — no extra full environment, but old and new run together briefly. Blue-green: run two full environments (blue=current, green=new); switch all traffic at once and roll back by switching back — fast rollback, double resources. Canary: release the new version to a small % of traffic first, monitor, then gradually ramp — limits blast radius of a bad release.

What is a feature flag and how does it decouple deployment from release?

A feature flag is a runtime toggle that turns a code path on/off without redeploying. It lets teams deploy code to production dark (disabled) and later release (enable) it — possibly to a subset of users — enabling canary releases, A/B tests, and instant kill-switch rollback independent of the deploy pipeline.

What is Infrastructure as Code (IaC) and its key benefits?

IaC is managing and provisioning infrastructure through machine-readable definition files rather than manual configuration. Benefits: version control, repeatability/consistency across environments, automated and auditable changes, easy disaster recovery/rebuild, and code review of infrastructure changes.

What is the difference between declarative and imperative IaC, and between provisioning and configuration-management tools?

Declarative IaC specifies the desired end state and the tool computes how to reach it (e.g., Terraform); imperative specifies the exact steps/commands to execute. Provisioning tools (e.g., Terraform, CloudFormation) create infrastructure resources; configuration-management tools (e.g., Ansible, Chef, Puppet) configure software on existing machines. They are complementary.

What is the difference between immutable and mutable infrastructure?

Mutable infrastructure is updated in place (patch/config existing servers), risking configuration drift and snowflake servers. Immutable infrastructure never modifies running servers — to change anything you build a new image/instance and replace the old one, guaranteeing consistency and easy rollback by redeploying a prior image.

What is the difference between Continuous Integration, Continuous Delivery, and Continuous Deployment?

Continuous Integration (CI): developers merge to a shared branch frequently, each change auto-built and tested. Continuous Delivery: every passing build is automatically prepared and can be released to production with a manual approval click. Continuous Deployment: every passing build is automatically released to production with no manual gate.

What are typical stages of a CI/CD pipeline?

Source/trigger (commit or PR) → Build (compile, package artifacts) → Test (unit, integration, security/lint scans) → Artifact/publish (store image) → Deploy to staging → Automated acceptance/smoke tests → Deploy to production (with a strategy like canary), often followed by monitoring/rollback. Each stage gates the next.

Why is clock synchronization hard in distributed systems, and what does NTP provide?

Each machine has its own physical clock that drifts, and message delays are variable, so no two clocks agree exactly. NTP (Network Time Protocol) synchronizes clocks against reference time servers, estimating and correcting offset and round-trip delay, typically achieving millisecond-level accuracy — but it cannot guarantee perfectly synchronized clocks, so ordering by wall-clock timestamps is unreliable.

What problem do Lamport logical clocks solve, and what is their key rule?

Lamport clocks provide a consistent logical ordering of events without synchronized physical clocks, capturing the 'happens-before' relation. Rule: each process increments its counter on every event; when sending a message it attaches its counter; on receipt the receiver sets its counter to $\max(\text{local}, \text{received}) + 1$. If event $a$ happens-before $b$, then $C(a) < C(b)$.

What is the limitation of Lamport clocks that vector clocks overcome?

With Lamport clocks, $a \to b$ implies $C(a) < C(b)$, but the converse fails: $C(a) < C(b)$ does not tell you whether $a$ happened-before $b$ or they are concurrent. Vector clocks (a per-process counter vector) capture causality exactly, letting you detect concurrency: two events are concurrent iff neither vector dominates the other.

Why not use a single centralized auto-increment counter for unique ID generation in a distributed system?

A single central counter is a single point of failure and a scalability/throughput bottleneck: every ID request must round-trip to it, and it can't keep up at scale or across regions. Distributed systems instead use schemes (UUIDs, Snowflake, ranges/tickets) that generate IDs locally without global coordination.

Describe how a Twitter Snowflake ID is structured.

A Snowflake ID is a 64-bit integer, typically: 1 unused sign bit, a 41-bit timestamp (milliseconds since a custom epoch), a ~10-bit machine/worker ID, and a ~12-bit per-millisecond sequence number. This yields globally unique, roughly time-sortable (k-sorted) IDs generated locally without central coordination, with up to $2^{12}=4096$ IDs per machine per millisecond.

What are the trade-offs of UUIDs vs Snowflake-style IDs?

UUIDs (128-bit) are generated with no coordination and are collision-safe, but are large and random (v4) — poor for database index locality and not time-ordered. Snowflake IDs are 64-bit, compact, and time-sortable (good index locality), but require coordinated worker-ID assignment and depend on clock behavior (clock skew/rollback can cause issues).

State the CAP theorem.

In a distributed data store, during a network Partition you cannot have both Consistency (every read sees the latest write) and Availability (every request gets a non-error response). You must choose: a CP system rejects/blocks requests to stay consistent, or an AP system stays available but may return stale data. Without a partition, you can have both C and A.

What is a network partition (split-brain) and why is it dangerous?

A network partition is when a failure splits nodes into groups that cannot communicate. Split-brain occurs when both sides independently believe they are the primary and accept writes, leading to divergent, conflicting state and data corruption. It is prevented with quorum/consensus so only one side (the majority) can act.

How does quorum prevent split-brain, and what is the majority quorum condition?

By requiring an action (e.g., electing a leader or committing a write) to be approved by a majority of nodes, only the partition containing more than half the nodes can proceed; the minority side cannot form a quorum and stops accepting writes. For a cluster of $N$ nodes the majority quorum is $\lfloor N/2 \rfloor + 1$, which is why clusters use an odd node count.

What is the quorum condition for consistency using read/write quorums (W + R > N)?

In a replicated store with $N$ replicas, if every write must be acknowledged by $W$ replicas and every read must query $R$ replicas, then setting $W + R > N$ guarantees the read set overlaps the latest write set, so a read sees the most recent write. Tuning $W$ and $R$ trades off read vs write latency and availability.

What is the difference between failover and failback?

Failover is switching operation from a failed primary to a standby/secondary to keep service running. Failback is the subsequent, controlled switch back to the original (now-recovered) primary once it is healthy, restoring the normal configuration — often done during a low-traffic window to avoid another disruption.

What is graceful degradation and how does it improve reliability?

Graceful degradation is designing a system to keep providing reduced but useful functionality when some components fail or are overloaded, instead of failing completely — e.g., serving cached/stale data, disabling non-essential features, or returning default responses. It preserves core user value and contains the impact of partial failures.

What is a circuit breaker pattern and its three states?

A circuit breaker wraps calls to a failing dependency to prevent cascading failures. States: Closed (calls pass through; failures counted), Open (failure threshold exceeded → calls fail fast without contacting the dependency, giving it time to recover), and Half-Open (after a timeout, a few trial calls test recovery; success closes the breaker, failure re-opens it).

What is the difference between a leader-based (single-leader) and leaderless replication approach for handling writes?

Single-leader: all writes go through one leader that orders them and replicates to followers — simple consistency and ordering, but the leader is a bottleneck and a failover point. Leaderless (e.g., Dynamo-style): any replica accepts writes and the system uses quorums and conflict resolution (versioning/read-repair) — higher availability during failures but requires handling conflicts and eventual consistency.

Why is exponential backoff with jitter used when retrying failed requests?

Exponential backoff increases the wait between retries multiplicatively (e.g., $base \cdot 2^{n}$) to avoid hammering a struggling service. Jitter adds randomness to the delay so that many clients don't retry in synchronized waves (the 'thundering herd'), which would otherwise re-overload the recovering service. Together they spread retries out and improve recovery.