Log-Gap Charts for Time Series

Sometimes you want to visualize your time-series data in a chart. Traditionally, this is done by slicing your time-scale into equal-sized bins and counting the events that fall into each one. The big players—Grafana, Datadog, CloudWatch—all do this, handing you a nice, neat histogram of events over time.

But binning the data destroys resolution.

Choose too large a bin, and you lose time resolution. You completely miss high-density burst events and long idle gaps; it all just averages out into a meaningless block. Choose too small a bin, and you lose aggregate resolution. You end up with hundreds of atomized bins containing zero, one, or two events each, flattening your chart into a needle-bed of noise. If you're lucky, you might find an intermediate bin size that still shows you something marginally interesting on both the count and time axes. But more often than not, there is no reasonable sweet spot. If you're particularly unlucky, a bin boundary will arbitrarily divide a burst right down the middle, and you'll miss the anomaly altogether.

Furthermore, binned data is highly sensitive to scale. If you zoom in on a bin, you don't get a closer look at the behavior; you just get a histogram with a single lonely bar in it. If you zoom out, you don't get a higher-elevation view of systemic trends; you simply add more bins.

Let's look at a live example. Here is a chart of the past 48 hours of HTTP requests hitting my web site, sliced into 3-hour bins:

We can see that some 3-hour blocks obviously have more requests than others, but the resolution is crude. We only have about fifteen or so blocks of data. It tells us almost nothing about how the traffic arrived. Was it steady across the three hour interval or did it come in bursts?

Here is the exact same 48 hours of live data, but rendered with 15-minute bins:

Now we can more clearly see the bursts, but for the steady state the y-axis is practically useless. Most of the bins have zero or one element in them. We get absolutely no feel for the macro, hourly rate.

Because we take the logarithm of the gap, the data become insensitive to scale. We can plot small intervals of a single second right next to large intervals of hours on the same axis without losing the shape of either. The steady state appears as a cloud of points high up in the chart, while burst traffic shows as vertical lines.

To read this kind of chart, you don't look at the individual points, but at the overall shape of the point cloud, the envelope, and the streaks. The highest points in the chart are the longest intervals between events, the lowest are the shortest. The middle of the point cloud indicates the median time between events. I have been using charts like this to plot time series events and they give a good feel of how events arrive.

Note: if you are getting broken links, that is probably the rate limiting on my site. Give it some time and reload later.