System Design Reliability, Security, and Operations Syllabus

Reliability, Security, and Operations syllabus — full chapter and topic list

Expand any chapter to see its topics and sub-topics. This is the whole examinable outline for Reliability, Security, and Operations in System Design, not a summary of it.

1. 1

   High Availability and Disaster Recovery

   4 topics

   • Redundancy and Failover Strategies
   • Active-Active vs Active-Passive
   • Multi-Region and Multi-AZ Deployments
   • Backups, RPO, and RTO
2. 2

   Observability and Monitoring

   3 topics

   • Metrics, Logs, and Traces
   • Alerting and On-Call
   • Health Checks and Heartbeats
3. 3

   Security Fundamentals

   4 topics

   • Authentication and Authorization
   • Encryption in Transit and at Rest
   • DDoS Protection and WAF
   • Secure Design Principles and Least Privilege
4. 4

   Deployment and Infrastructure

   4 topics

   • Containers and Orchestration with Kubernetes
   • Deployment Strategies
   • Infrastructure as Code
   • CI/CD Pipelines
5. 5

   Distributed System Challenges

   4 topics

   • Clock Synchronization and Logical Clocks
   • Unique ID Generation
   • Handling Failures and Network Partitions
   • Chaos Engineering

Reliability, Security, and Operations flashcards for System Design

19 of 66 cards from the Reliability, Security, and Operations deck — real questions with worked answers.

1. What is the difference between redundancy and failover?

   Redundancy is having duplicate components (extra capacity/replicas) so no single component is a single point of failure. Failover is the process/mechanism that detects a component failure and switches traffic to a redundant standby. Redundancy is the state; failover is the action that exploits it.

2. Define N+1, N+2, and 2N redundancy.

   N is the capacity needed to run the workload. N+1 provides one extra unit beyond need (survives 1 failure). N+2 provides two extra units (survives 2 concurrent failures). 2N is full duplication — a complete second set equal to N (survives loss of an entire set).

3. Compare Active-Active and Active-Passive failover configurations.

   Active-Active: all nodes serve traffic simultaneously, load is balanced across them, giving better resource utilization and instant failover, but requires state synchronization/conflict handling. Active-Passive: only the primary serves traffic while the standby waits idle; on failure traffic fails over to the passive node. Simpler, but wastes standby capacity and has failover delay.

4. What is a 'hot' vs 'warm' vs 'cold' standby?

   Hot standby: fully running and continuously synced, takes over almost instantly (lowest RTO). Warm standby: running but partially synced/scaled down, needs some time to take over. Cold standby: powered off or unprovisioned, must be started and loaded before use (longest RTO, lowest cost).

5. What is a Multi-AZ deployment and what failure does it protect against?

   A Multi-AZ (Availability Zone) deployment spreads resources across multiple isolated data centers within the same region, each with independent power, cooling, and networking. It protects against the failure of a single data center/AZ (e.g., power loss, fire) while keeping low inter-AZ latency.

6. What is a Multi-Region deployment and what does it protect against that Multi-AZ does not?

   Multi-Region spreads resources across geographically distant regions. Beyond AZ-level failures, it protects against an entire region outage and reduces latency for globally distributed users. It also aids data-residency compliance, at the cost of higher cross-region latency and replication complexity.

7. Why is cross-AZ communication generally preferred over cross-region for synchronous replication?

   AZs within a region are connected by low-latency, high-bandwidth links (typically single-digit milliseconds), so synchronous replication adds little latency. Cross-region links have much higher latency (tens to hundreds of ms), making synchronous replication expensive and slow, so regions usually use asynchronous replication.

8. Define RPO (Recovery Point Objective).

   RPO is the maximum acceptable amount of data loss measured in time — how far back in time your recovery point can be. It answers 'how much data can we afford to lose?' An RPO of 1 hour means backups/replication must be at most 1 hour behind.

9. Define RTO (Recovery Time Objective).

   RTO is the maximum acceptable duration to restore service after a failure — how long the system can be down. It answers 'how quickly must we recover?' An RTO of 15 minutes means service must be back within 15 minutes of an outage.

10. How do RPO and RTO relate to backup frequency and recovery mechanism?

    RPO is driven by how often you back up/replicate data (more frequent = smaller RPO, less data loss). RTO is driven by how fast you can restore/failover (hotter standby, faster restore = smaller RTO). Lower RPO and RTO both cost more.

11. Contrast full, incremental, and differential backups.

    Full: copies all data every time — largest, slowest, but self-contained restore. Incremental: copies only data changed since the last backup (full or incremental) — smallest/fastest, but restore requires the full plus every increment in order. Differential: copies all data changed since the last full backup — larger than incremental but restore needs only the full plus one differential.

12. What is the 3-2-1 backup rule?

    Keep at least 3 copies of your data, on 2 different types of storage media, with 1 copy stored off-site (e.g., a different region or cloud). This protects against media failure, site disasters, and single-location loss.

13. Name the three pillars of observability.

    Metrics, logs, and traces. Metrics are aggregated numeric measurements over time; logs are discrete timestamped event records; traces follow a single request across services showing spans and timing.

14. What distinguishes a metric from a log from a trace?

    A metric is a numeric value sampled/aggregated over time (e.g., request rate, CPU%), cheap to store. A log is a discrete, often textual record of a specific event with context. A trace records the end-to-end path of one request through a distributed system as a tree of spans, showing where latency is spent.

15. In distributed tracing, what are a trace, a span, and a trace/context propagation?

    A trace is the whole journey of one request. A span is a single named, timed operation within that trace (e.g., a DB call), with parent-child relationships. Context propagation passes the trace ID and span ID across service boundaries (usually via headers) so spans can be stitched into one trace.

16. What are the four 'golden signals' of monitoring (per Google SRE)?

    Latency (time to serve a request), Traffic (demand on the system, e.g., requests/sec), Errors (rate of failed requests), and Saturation (how full the system is, e.g., resource utilization). Monitoring these four gives strong coverage of service health.

17. What is the difference between an SLI, an SLO, and an SLA?

    SLI (Service Level Indicator): a measured metric of service quality (e.g., % of successful requests). SLO (Objective): the internal target for an SLI (e.g., 99.9% success). SLA (Agreement): a contractual promise to customers with consequences (penalties) if the SLO-like target is missed. SLA is usually looser than the internal SLO.

18. What is an error budget and how is it calculated from an SLO?

    An error budget is the allowed amount of unreliability = $1 - \text{SLO}$. For a 99.9% availability SLO, the error budget is $0.1\%$ of requests/time. If monthly (~30 days), that is about $43.2$ minutes of allowed downtime. When the budget is exhausted, teams freeze risky releases.

19. Roughly how much downtime per year corresponds to 'three nines', 'four nines', and 'five nines' availability?

    99.9% (three nines) ≈ 8.76 hours/year. 99.99% (four nines) ≈ 52.6 minutes/year. 99.999% (five nines) ≈ 5.26 minutes/year. General formula: $\text{downtime} = (1 - A)\times 525600$ minutes/year.

See more Reliability, Security, and Operations flashcards →

Planning Reliability, Security, and Operations for System Design

Reliability, Security, and Operations (System Design) FAQ