Export to
Thursday, May 18, 2017 at 1:08pm.
Papers We Love @ PDX - Gorilla: A Fast, Scalable, In-Memory Time Series Database
Website
Description
This paper will be presented by Johannes Hoff.
This paper describes how Facebook stores 700 million data points (timestamp and value) per minute and keeps 26 hours of it available in memory for fast access. We will mainly look at the compression algorithms making this possible.
Here is the abstract:
"""
Large-scale internet services aim to remain highly available and responsive in the presence of unexpected failures. Pro-viding this service often requires monitoring and analyzing tens of millions of measurements per second across a large number of systems, and one particularly effective solution is to store and query such measurements in a time series database (TSDB).
A key challenge in the design of TSDBs is how to strike the right balance between efficiency, scalability, and relia-bility. In this paper we introduce Gorilla, Facebook’s in-memory TSDB. Our insight is that users of monitoring sys-tems do not place much emphasis on individual data points but rather on aggregate analysis, and recent data points are of much higher value than older points to quickly detect and diagnose the root cause of an ongoing problem. Gorilla op-timizes for remaining highly available for writes and reads, even in the face of failures, at the expense of possibly drop-ping small amounts of data on the write path. To improve query efficiency, we aggressively leverage compression tech-niques such as delta-of-delta timestamps and XOR’d floating point values to reduce Gorilla’s storage footprint by 10x. This allows us to store Gorilla’s data in memory, reduc-ing query latency by 73x and improving query throughput by 14x when compared to a traditional database (HBase)-backed time series data. This performance improvement has unlocked new monitoring and debugging tools, such as time series correlation search and more dense visualization tools. Gorilla also gracefully handles failures from a single-node to entire regions with little to no operational overhead.
"""