The big idea
Airbnb built a database traffic capture and replay system around a proxy that already sits between applications and databases. It uses real queries, rather than relying solely on synthetic workloads, to test capacity, compare database behavior and investigate performance problems offline.
Why it matters
Synthetic benchmarks do not represent the full range of production queries, and abbreviated diagnostic records may not contain enough detail to reproduce a problem. A replay system can make testing more representative, but it also has to preserve transaction context and protect the production data that queries may contain.
Key insights
- 01
A shared capture point replaced fragmented, language-specific logging.
The previous approach depended on separate logging frameworks for each language and custom replay tools for different uses. It was difficult to maintain and did not provide a complete enough view of transactions. Airbnb moved capture to ProxySQL, a database proxy already in the traffic path, without requiring application changes. A companion process moves the resulting logs from local disk to object storage.
- 02
Replay quality depends on reconstructing transaction order.
Proxy logs contain queries from multiple database clusters and interleave statements from concurrent connections. An offline processor separates the records by cluster and restores the original order of statements within each transaction. It also groups queries into five-minute windows so the replayer can control the pace and load of a test.
- 03
Captured workloads require production-level protection.
Queries can contain personal data, so Airbnb treats the logs as production data: it encrypts them in transit and at rest, limits access and enables capture only for the required test window. Replays run in production-equivalent environments under the same standards, not in lower environments or a separate development tier.
Practical takeaways
- 01
Use captured workloads to test database capacity at current and forecast traffic levels.
Replaying production traffic at its original rate or higher multiples helps evaluate headroom. The results can inform whether a cluster needs more capacity or is over-provisioned.
- 02
Compare the same captured queries across database targets before an upgrade or migration.
Result comparisons can expose behavioral incompatibilities that might otherwise break applications after a move to a different version or compatible engine.
- 03
Replay complete statements with their transaction context when investigating performance problems.
Abbreviated query diagnostics may not reproduce an issue. A fuller workload can support offline investigation and help validate a proposed fix before it ships.