The tick store is bounded
Table of Contents
1. Summary
The tick store holds a bounded window, so its size follows from the tick rate and the window, not from how long the system has run. Chunks are narrow enough that the index of the chunk being written fits in memory: at 29,000 rows a second, one hour is about 104 million rows. Closed chunks are compressed, segmented by stream and ordered by time. Retention drops chunks past the window, and bars that summarise each period are kept longer than the raw ticks.
2. Detail
2.1. The parts and their windows
| Part | Window |
|---|---|
| Raw ticks | a short window, such as a week, set by the class policy |
| Bars of one second and one minute | longer, set by the class policy |
The policy table of The tick ingest pipeline: freshness first states the window of each class.
2.2. Compression
A closed chunk is compressed. The points hold no tenant and carry no row-level security (A source tick is stored once and entitled when it is read), which compression requires. The instance runs TimescaleDB under the Timescale licence, the edition that compresses chunks and runs the compression and retention jobs.
2.3. Retention and compression as jobs
A retention job drops chunks past the window, and a compression job compresses chunks as they close. Both are TimescaleDB policies on the table, so dropping old data costs a chunk drop, never a delete of rows.
2.4. Size
The store holds the window's rows, compressed. Its size is the tick rate times the window, divided by the compression ratio. A week of raw ticks at 29,000 rows a second is about 17.5 billion rows, or about 2.8 TB at 162 bytes a row before compression. The target is that week in under 300 GB on disk, which sets the compression the store reaches: about ten to one.
3. See also
- The data path — the structure note that orders this cluster.
- Ticks reach the store through sharded binary COPY in bounded batches — how the rows arrive.