The big idea
Netflix’s TimeSeries platform separates frequently accessed recent events from older data, moving the latter to object storage while keeping a consistent query interface. Its first approach streams events into a cold tier and reorganizes them for efficient reads. The source also introduces a Cassandra-native approach that uses backups already in object storage.
Why it matters
As datasets grow and users ask to retain events for longer, keeping everything on fast storage becomes expensive. Historical data is queried less often, but it still needs to be available for audits, investigations, and internal analysis—sometimes with sub-second latency. Netflix’s design shows how a storage tier can trade some operational complexity for lower costs without asking users to learn a separate query interface.
Key insights
- 01
Time-based slices make older data eligible for different storage treatment.
TimeSeries divides data into tables that each cover a time window. Once a window passes the write boundary, its table can become immutable, meaning it no longer changes. A control plane—the part of the system responsible for managing configuration—maintains slice metadata, while the data plane uses that metadata to route reads and writes.
- 02
Files optimized for streaming writes need additional work for fast lookups.
The first design writes events to Cassandra and streams them through Kafka and Flink into S3. Flink batches events into Parquet files, a format suited to high-throughput writes but not initially to low-latency reads. A daily compaction job merges and reorganizes those files, using small row groups and time ordering to reduce the data needed for a lookup. Cached filters and file metadata help the query planner avoid unnecessary S3 reads.
- 03
A shared query interface can span tiers with different performance characteristics.
The data plane sends queries to both hot and cold storage and combines the results. Netflix reports that its cold-tier planning limits a full-day retrieval to at most three sequential S3 reads. Reported per-read latency was about 150 milliseconds at the median and 500 milliseconds at the 99th percentile. The streaming tier reduced both the Cassandra storage footprint and its backup footprint.
Practical takeaways
- 01
Separate recent and older data when their access patterns differ.
Netflix keeps recent events on fast storage and moves older, less frequently accessed events to cheaper storage, while preserving an interface that can query both.
- 02
Use a write boundary to identify windows that can be reorganized safely.
In TimeSeries, a table can become immutable after crossing that boundary. That property enables further storage optimizations for the completed window.
- 03
Choose compaction speed in light of compute cost.
Netflix’s daily compaction can take around 12 hours on some of its largest datasets because the job favors lower compute cost over finishing sooner.