Site Topics Fundamentals 3 in Practice: Lessons From Real Deployments

Cost Controls: A design that cannot be rolled back is a design that cannot be changed safely. Cost Controls: Latency budgets are easier to defend when every hop has a stated ceiling. Cost Controls: Caching helps only until the invalidation rules become the bottleneck.

A queue smooths spikes but also hides how far behind you are. This is most visible in api design. Consider api design specifically. Retries without jitter turn a small outage into a large one. API Design: Separating the reads from the writes buys room to change either side.

You can often replace a coordination problem with an idempotency key. The same reasoning holds for api design. For api design, the constraint matters more than the feature list. Anything that grows without a bound will eventually hit one. Teams working on api design usually discover this the hard way. Documentation that is not tested tends to describe the previous version.

Edge Caching: The interesting number is not the average, it is the 99th percentile. Edge Caching: Adding a cache in front of a slow query is a fix; fixing the query is a cure. Edge Caching: Every abstraction you add is a place where behaviour can differ from intent.

Content Delivery: If the rollback plan needs a meeting, it is not a rollback plan. Content Delivery: Small pages that stay small are easier to keep fast than large ones made fast. Content Delivery: Write the invariant down; otherwise it lives only in someone's memory.

Search Indexing: Serving static bytes is the cheapest thing you can do at the edge. Search Indexing: A schema is an interface; changing it is a migration, not an edit. Search Indexing: Track the denominator as carefully as the numerator.

A design that cannot be rolled back is a design that cannot be changed safely. The same reasoning holds for cloud infrastructure. For cloud infrastructure, the constraint matters more than the feature list. Latency budgets are easier to defend when every hop has a stated ceiling. Teams working on cloud infrastructure usually discover this the hard way. Caching helps only until the invalidation rules become the bottleneck.

Load Balancing: Periodic jobs should be safe to run twice, because they will be. Load Balancing: You rarely need a new component to fix a boundary problem. Load Balancing: The signal you want is often already logged, just not aggregated.

In practice, data pipelines behaves differently: A queue smooths spikes but also hides how far behind you are. Retries without jitter turn a small outage into a large one. The same reasoning holds for data pipelines. For data pipelines, the constraint matters more than the feature list. Separating the reads from the writes buys room to change either side.

If a metric has no owner, it will drift until it causes an incident. This is most visible in cloud infrastructure. Consider cloud infrastructure specifically. The cheapest optimisation is usually removing work nobody asked for. Cloud Infrastructure: Aggregating at write time trades flexibility for predictable read cost.

Data Pipelines: Configurations should be reviewable in a diff, not only in a console. Data Pipelines: The best time to add an index is before the table gets large. Data Pipelines: Failures are usually correlated, so plan for the shared dependency.

Backup Strategy: Serving static bytes is the cheapest thing you can do at the edge. Backup Strategy: A schema is an interface; changing it is a migration, not an edit. Backup Strategy: Track the denominator as carefully as the numerator.

In practice, search indexing behaves differently: Periodic jobs should be safe to run twice, because they will be. You rarely need a new component to fix a boundary problem. The same reasoning holds for search indexing. For search indexing, the constraint matters more than the feature list. The signal you want is often already logged, just not aggregated.

Schema Markup: Periodic jobs should be safe to run twice, because they will be. Schema Markup: You rarely need a new component to fix a boundary problem. Schema Markup: The signal you want is often already logged, just not aggregated.

Crawl Budget: A queue smooths spikes but also hides how far behind you are. Crawl Budget: Retries without jitter turn a small outage into a large one. Crawl Budget: Separating the reads from the writes buys room to change either side.

If the rollback plan needs a meeting, it is not a rollback plan. The same reasoning holds for data pipelines. For data pipelines, the constraint matters more than the feature list. Small pages that stay small are easier to keep fast than large ones made fast. Teams working on data pipelines usually discover this the hard way. Write the invariant down; otherwise it lives only in someone's memory.

Observability: The interesting number is not the average, it is the 99th percentile. Observability: Adding a cache in front of a slow query is a fix; fixing the query is a cure. Observability: Every abstraction you add is a place where behaviour can differ from intent.

Data Pipelines: You can often replace a coordination problem with an idempotency key. Data Pipelines: Anything that grows without a bound will eventually hit one. Data Pipelines: Documentation that is not tested tends to describe the previous version.

Access Control: A queue smooths spikes but also hides how far behind you are. Access Control: Retries without jitter turn a small outage into a large one. Access Control: Separating the reads from the writes buys room to change either side.

Cost Controls: You can often replace a coordination problem with an idempotency key. Cost Controls: Anything that grows without a bound will eventually hit one. Cost Controls: Documentation that is not tested tends to describe the previous version.

Log Analysis: A design that cannot be rolled back is a design that cannot be changed safely. Log Analysis: Latency budgets are easier to defend when every hop has a stated ceiling. Log Analysis: Caching helps only until the invalidation rules become the bottleneck.

The first thing to settle is the failure mode, not the happy path. This is most visible in backup strategy. Consider backup strategy specifically. Measurements taken once are anecdotes; you need a baseline that repeats. Backup Strategy: Costs usually concentrate in a small number of operations, so find those first.

The interesting number is not the average, it is the 99th percentile. That applies to release process as well. In practice, release process behaves differently: Adding a cache in front of a slow query is a fix; fixing the query is a cure. Every abstraction you add is a place where behaviour can differ from intent. The same reasoning holds for release process.

Monitoring Alerts: The first thing to settle is the failure mode, not the happy path. Measurements taken once are anecdotes; you need a baseline that repeats. That applies to monitoring alerts as well. In practice, monitoring alerts behaves differently: Costs usually concentrate in a small number of operations, so find those first.