System Design Foundations

Introduction

The Skill That Separates Seniors from Everyone Else

There is a clear and uncomfortable gap in most engineering teams. Ask a developer to add a feature to an existing system and they will likely do it well. Ask that same developer to design a system from scratch — with vague requirements and a blank whiteboard — and most will freeze. Not because they lack intelligence, but because designing systems from the ground up is a fundamentally different skill from writing code within an established architecture.

Senior engineers earn their title not just by writing better code, but by making architectural decisions with incomplete information. They choose which database fits the access patterns. They decide how the system should scale before it needs to. They identify failure modes before they become incidents. They turn ambiguous product requirements into concrete technical blueprints.

Companies are not paying senior-level salaries for people who can implement tickets. They are paying for judgment — architectural decisions that affect performance, reliability, and scalability at the system level.

This post walks through the foundational layer of that skill: how requests flow through a basic system, how to choose and structure data storage, how to scale when a single server isn't enough, and how to design systems that survive failures gracefully.

Foundations

Starting with a Single Server

The most powerful thing you can do when approaching system design is to resist the urge to jump straight to microservices, distributed caches, and Kubernetes clusters. Every complex system starts simple. Understanding a single-server setup thoroughly gives you the mental model you need to evolve it intelligently.

In a single-server setup, everything lives on one machine: the web application, the database, the cache, and every other component. This is not a toy — it is the correct starting point for most early-stage products. The diagram below shows how it handles a user request end to end.

Single server setup showing DNS resolution, IP address lookup, and HTTP request/response flow between web browser, mobile app, and a single server — A single-server system: the DNS resolves the domain to a server IP, the client sends an HTTP request, and the server returns either an HTML page or a JSON response depending on the client type

When a user types app.demo.com into their browser, the browser does not know the server's location directly — it only has a human-readable name. It first contacts a DNS (Domain Name System) provider, which acts as a directory service mapping domain names to IP addresses. The DNS returns the server's IP, and the client then sends an HTTP request directly to that address. The server processes the request and returns the appropriate response: an HTML page for a browser or a JSON payload for a mobile app.

This flow — DNS lookup → IP resolution → HTTP request → server response — is the fundamental cycle that underpins everything in web infrastructure. Every layer of complexity you add later sits on top of this basic exchange.

Foundations

How Web and Mobile Clients Differ

The single server handles two meaningfully different kinds of clients, and understanding the distinction matters for how you design your APIs and data layer.

Comparison showing web browser receiving HTML/CSS/JavaScript from server versus mobile app receiving JSON via API calls — Web browsers expect the server to assemble a complete renderable response (HTML/CSS/JS), while mobile apps drive their own UI and only need data, delivered as JSON through API calls

Web browsers traditionally offload most presentation work to the server. The server handles business logic, data retrieval, and assembles a complete HTML/CSS/JavaScript response that the browser renders. The browser is a dumb display layer in this model.

Mobile applications work differently. They contain their own UI logic and rendering capabilities — they just need data. Communication happens through API calls over HTTP, and the server responds with JSON: a lightweight, structured format that is easy to parse on any device. A typical API response to a GET /products/456 request might include the product ID, name, description, price, stock count, category tags, and seller information — everything the app needs to render the product page, without the server prescribing how it looks.

Modern applications increasingly use this API-first pattern even for web clients, which is why REST, GraphQL, and gRPC have become so central to backend design.

Data Layer

Separating the Data Tier

As user traffic grows, the single server hits a wall. The first meaningful architectural step is separating the web tier — which runs application logic and serves traffic — from the data tier — which manages persistent storage. Running them on the same machine means a spike in database queries degrades the API's responsiveness, and vice versa. Separating them lets each layer scale independently.

Architecture diagram showing web tier and data tier separated, with web browser and mobile app on the left connecting through HTTP to the web server, which connects to a separate database on the right — Separating the web tier from the data tier is the first architectural step toward scalability — each layer can now be scaled, optimized, and replaced independently

Once you have a dedicated data tier, you face the most consequential early design decision: which database? There are two major families, and choosing the wrong one creates expensive technical debt.

Two main database categories showing Relational (PostgreSQL, MySQL, SQLite, Oracle) on the left and Non-Relational (Cassandra, Redis, MongoDB, Neo4j) on the right — The two fundamental database families — relational databases enforce structure and relationships; non-relational databases prioritize flexibility, speed, and scale

Databases

Relational Databases

Relational databases organize information into tables — a structure analogous to a spreadsheet. Each table has defined columns (the fields or attributes, like id, name, email) and rows (individual records). You interact with this data using SQL — Structured Query Language — which gives you a powerful, declarative way to query, filter, join, and aggregate data across multiple tables.

SQL Customers table showing a CREATE TABLE statement defining id, name, age, email columns alongside an INSERT statement and the resulting table rows — A SQL table defines a strict schema upfront. Every row must conform to that schema, which is what enables reliable joins and transactional integrity across multiple tables

The defining capability of relational databases is the join operation. When you have a Customers table and a Products table, you can create an Orders table that links them through foreign keys — recording which customer ordered which product without duplicating data in either original table. This normalization keeps your data consistent and queryable from any angle, and it scales surprisingly well when combined with proper indexing.

Well-known relational databases include PostgreSQL (the go-to for most production systems), MySQL, SQLite (embedded, great for development), and Oracle (enterprise). PostgreSQL in particular is worth learning deeply — it handles JSON, full-text search, time-series data, and geospatial queries in addition to standard relational workloads.

Databases

ACID Transactions

The other defining strength of relational databases is transactional integrity. A transaction is a group of one or more SQL operations executed as a single indivisible unit. Every transaction in a relational database adheres to the ACID model.

Consider a bank transfer as the canonical example: moving $500 from Account A to Account B requires two separate operations — a deduction and a credit. If the system crashes between them, the money cannot simply disappear. ACID properties ensure this can never happen.

ACID transaction model diagram showing five numbered steps in a loop with the four ACID properties listed: Atomicity (completely succeeds or fails), Consistency (valid state to valid state), Isolation (isolated from other transactions), Durability (data remains in case of system failure) — The ACID model is what makes SQL databases the correct choice for financial systems, healthcare records, and any domain where data correctness is non-negotiable

Property	What it guarantees
Atomicity	The entire transaction either completely succeeds or completely rolls back. There is no such thing as a half-applied transaction. The bank transfer either completes fully or neither side changes.
Consistency	Every transaction moves the database from one valid state to another valid state, respecting all constraints, foreign keys, and business rules defined in the schema.
Isolation	Transactions running concurrently are invisible to each other until they commit. A transaction reading a row while another is updating it sees a consistent snapshot, not an in-progress partial state.
Durability	Once a transaction commits, it stays committed. Even if the server crashes the moment after the commit, the data survives — it has been written to persistent storage.

Databases

Non-Relational (NoSQL) Databases

NoSQL databases trade SQL's rigid structure for flexibility, horizontal scalability, and lower latency. They are not a single technology — the term covers four distinct database models, each designed for a different data shape and access pattern.

The four NoSQL families each solve a different problem — choosing between them depends entirely on how your application reads and writes data

Document Stores

MongoDB

Records are stored as JSON-like documents, allowing deeply nested structures within a single record. A customer document can embed their entire order history inline — no joins required. Ideal for content systems, user profiles, and any schema that evolves frequently.

Wide-Column Stores

Cassandra / Cosmos DB

Data is organized in tables with dynamic column families. Architected specifically for massive write throughput across distributed nodes — the standard choice for time-series data, event logs, analytics pipelines, and IoT sensor streams where writes vastly outnumber reads.

Key-Value Stores

Redis / Memcached

The simplest model: every value is retrieved by a unique key. Because the data lives in RAM, reads and writes are sub-millisecond — orders of magnitude faster than disk-backed databases. The default choice for caching, session storage, real-time leaderboards, and rate limiting.

Graph Databases

Neo4j / AWS Neptune

Models data as nodes (entities) and edges (relationships). Traversing connections — "find all friends of friends who bought this product" — is where graph databases dramatically outperform SQL joins. Amazon uses Neptune to power its recommendation engine.

Wide-Column Store architecture diagram showing column families, row keys, super columns, and the WRITES label indicating high write throughput capability — Wide-column stores like Cassandra organize data into flexible column families rather than rigid rows — this structure enables exceptional write performance at distributed scale

Databases

SQL vs NoSQL: Choosing the Right Tool

The most common mistake engineers make is treating this as a religious debate. SQL is not better than NoSQL, nor the reverse — they solve different problems. The question is always: what are the access patterns, consistency requirements, and scale characteristics of this specific workload?

Use Case	Best Fit	Why
E-commerce (orders, payments)	SQL	Well-structured relational data; ACID transactions are essential for payment integrity
Banking & financial systems	SQL	Strong consistency and transactional guarantees are non-negotiable
Sub-millisecond latency	NoSQL	Key-value stores (Redis) in RAM deliver <1ms reads impossible with disk-backed SQL
Dynamic / evolving schemas	NoSQL	Document stores handle schema changes without painful migrations
Massive write throughput	NoSQL	Wide-column stores (Cassandra) are purpose-built for distributed high-volume writes
Social graphs & recommendations	NoSQL	Graph databases traverse relationship networks far more efficiently than SQL joins

Most mature production systems use both — a PostgreSQL database for core transactional records, Redis for caching and sessions, and sometimes a document store for content. The goal is not to pick one; it is to match each data problem to the right tool.

Scaling

Vertical vs Horizontal Scaling

A single server has a finite capacity. As user traffic grows, requests queue up, response times climb, and eventually the system degrades. At this point you have two fundamentally different approaches to adding capacity.

Vertical Scaling — Scale Up

Add more resources to the existing server: more CPU cores, more RAM, faster NVMe storage. The appeal is simplicity — no code changes, no new architecture, no redeployment complexity. For moderate traffic growth, this is often the right first move.

The problem is that vertical scaling has two hard limits. There is a physical ceiling — you can only buy so much RAM or so many CPU cores for a single machine before you hit the top of what hardware vendors offer, at an exponentially increasing cost. And there is a resilience ceiling — no matter how large you make one machine, a single hardware failure, OS crash, or software bug brings the entire application offline. One server means one point of total failure.

Horizontal Scaling — Scale Out

Instead of making one server larger, you replicate it — three identical servers where there was one, each handling a share of the traffic. This delivers two properties that vertical scaling fundamentally cannot provide:

Fault tolerance: If one server fails, the other two absorb its traffic and continue serving users while the failed instance recovers. The system degrades gracefully rather than going fully offline.

Elastic scalability: You can add servers as traffic increases and remove them as it falls — without touching the application code. This is the architecture that powers every high-traffic web platform.

Load Balancing

Load Balancers

Horizontal scaling immediately raises a question: with three servers behind your application, how does an incoming request know which one to go to? You need a component that sits between clients and your server pool, receives every request, and makes an intelligent routing decision. That component is a load balancer.

Load balancer diagram showing client devices on the left sending traffic to a central load balancer, which distributes requests to three server groups on the right — The load balancer is the traffic director for a horizontally scaled system — it receives every incoming request and routes it to the appropriate server based on the chosen algorithm

A load balancer is simultaneously doing three things. It distributes load so no single server is overwhelmed while others are idle. It monitors server health and stops routing to any server that becomes unavailable. And it enables elasticity — new servers can be added to or removed from the pool at any time, and the load balancer automatically incorporates them.

Load balancer fault tolerance diagram showing server three marked with a red X (offline), with traffic being rerouted from that server to servers one, two, and a newly added server four — When a server fails, the load balancer detects the failure through health checks and immediately reroutes traffic to the remaining healthy servers — with no manual intervention required

Load Balancing

Load Balancing Algorithms

The load balancer doesn't pick servers randomly — it uses an algorithm. The right choice depends on your server specifications, session requirements, and traffic patterns. There are seven commonly used strategies.

Algorithm 1

Round Robin

Requests cycle through servers in fixed rotating order — server 1, then 2, then 3, then back to 1. Simple and effective when all servers have identical specs and workloads are uniform.

Algorithm 2

Least Connections

Routes each new request to the server with the fewest active connections. Smarter than Round Robin when session lengths vary dramatically — a server processing many short requests can accept more while another handles a slow upload.

Algorithm 3

Least Response Time

Balances two signals: fastest average response time and fewest active connections. The best fit when your fleet has mixed hardware capabilities and minimizing perceived latency is the top priority.

Algorithm 4

IP Hash

Hashes the client's IP address to consistently map them to the same server across requests. Essential when servers store session state locally — routing a returning client to a different server would lose their session context.

Least Connections algorithm diagram showing three servers with active connection counts of 10, 9+1 (new request being directed here), and 30, with the load balancer routing the new request to server 2 with the fewest connections — Least Connections routes traffic to whichever server has the smallest active connection count at that moment — a more intelligent distribution for workloads with variable session lengths

Algorithm 5

Weighted Algorithms

A modifier applied to Round Robin or Least Connections: servers are assigned weights based on their capacity (RAM, CPU). A 64GB server naturally receives proportionally more traffic than a 16GB one.

Algorithm 6

Geographical Routing

Routes users to the server geographically nearest to them — US East traffic to a US East server, European traffic to a European server. Critical for global applications where network latency directly affects user experience.

Geographical Algorithms diagram showing a load balancer distributing traffic to servers labeled East (E), West (W), and Europe (Eur), with US and European users being routed to their nearest regional server — Geographical routing minimizes round-trip time by directing users to the nearest server — users in Europe should never be hitting a server in US East if an alternative exists

The seventh strategy, Consistent Hashing, maps both servers and client IPs onto a circular hash ring. Each request hashes to a point on the ring, and traffic is routed to the nearest server clockwise from that point. This guarantees the same session consistency as IP Hash while handling server additions and removals more gracefully — only the keys nearest a changed node need remapping, rather than rehashing the entire pool.

Load Balancing

Health Checks

A load balancer's fault tolerance only works if it actually knows which servers are alive. This is solved through health checks — the load balancer continuously sends lightweight probe requests to every server in its pool on a fixed interval. Any server that responds successfully stays in rotation. Any server that fails to respond is immediately removed from the pool.

When the failed server recovers and health checks begin succeeding again, the load balancer automatically adds it back to the active pool — no manual intervention, no deployment required.

Load balancers are available in three forms. Software load balancers like Nginx and HAProxy are installed on commodity servers and configured through text files — flexible, open-source, and widely deployed. Hardware load balancers like the F5 BIG-IP are dedicated physical appliances built for extreme throughput, common in large enterprise environments. Cloud-managed solutions — AWS Elastic Load Balancing, Azure Load Balancer, Google Cloud Load Balancing — offer the easiest operational path, integrating natively with cloud infrastructure, scaling automatically, and including health monitoring without any configuration overhead.

Reliability

Single Points of Failure

A Single Point of Failure (SPOF) is any component in your system whose failure brings the entire system down. In a typical architecture where multiple API servers sit behind a load balancer and all connect to a single database, that database is a SPOF. The load balancer itself is a SPOF. A single authentication service, a single message queue, a single DNS provider — any component with no redundancy is a hidden risk.

Single Point of Failure diagram showing clients connecting to a load balancer routing to API servers, all sharing one database — the database is highlighted as the SPOF with arrows indicating its failure would cascade to all API servers and all clients — A single database serving the entire API tier is the classic SPOF. Its failure cascades instantly — every API server loses its data layer, every client request returns an error, and the entire application goes offline

SPOFs create three distinct categories of risk. The reliability risk is obvious: one failure takes everything offline. The scalability risk is subtler: adding more servers or services amplifies the blast radius of the SPOF, because more components now depend on that single vulnerable point. The security risk is equally important: a known SPOF is a high-value attack target — overwhelming a single load balancer with traffic (a DDoS attack) brings down the entire system behind it.

Eliminating the Load Balancer SPOF

Redundancy solution diagram showing two load balancers in parallel, both routing traffic to the same pool of API servers, eliminating the single load balancer as a SPOF — Running two load balancers in parallel immediately eliminates the LB as a SPOF. Traffic can be split 50/50 under normal conditions, with one absorbing all traffic if the other fails

Three strategies address the load balancer SPOF specifically. Redundancy is the most direct: run two load balancers simultaneously, with traffic split between them. If one fails, all traffic reroutes through the other automatically.

Self-healing system diagram showing a failed load balancer being automatically replaced by a new load balancer instance, with clients transparently redirected to the replacement — Self-healing infrastructure automatically replaces a failed load balancer with a fresh instance — clients experience no interruption because the replacement happens before the failure is visible to end users

Health checks on the load balancers themselves apply the same monitoring logic one tier up — continuously polling the LBs so failures are detected and traffic rerouted before users notice. Self-healing systems go one step further: they automatically spin up a replacement instance when a failure is detected, bringing the system back to full capacity without any manual intervention. This is the pattern used by modern cloud infrastructure at scale — Kubernetes, AWS Auto Scaling, and similar platforms implement this loop automatically.

The discipline of eliminating SPOFs is not just operational — it is a design practice. Every time you draw an architecture diagram, ask: which single component, if it failed right now, would take down everything downstream? Then add redundancy before you ship.

Conclusion

Building the Foundation First

System design mastery is not about memorizing architectures. It is about developing a reasoning process: understanding why each component exists, what tradeoffs it introduces, and when a simpler solution is better than a sophisticated one. The engineers who design reliable systems are the ones who resisted premature complexity early — who started with a single server, proved it needed to scale, then scaled deliberately.

Principle

Start Simple

A single server handling everything is the right architecture for an early product. Add tiers only when you have evidence that a simpler setup is no longer sufficient.

Principle

Match Database to Workload

SQL for structured, transactional data. NoSQL for flexibility, scale, or speed. Most mature systems use both — the goal is the right tool for each data problem.

Principle

Scale Out, Not Just Up

Vertical scaling has a ceiling and creates fragility. Horizontal scaling provides fault tolerance and elastic capacity — design for it from the start even if you don't need it yet.

Principle

Design Out SPOFs Early

Identify every single point of failure on the whiteboard — not during an incident at 2am. Redundancy is cheapest when it is designed in, not retrofitted after an outage.