For some reason I decided that RRD and mrtg just weren’t complicated enough tools to monitor my weight. And the temperature of my wine racks. Thus, I went down the rabbit hole that is OpenTSDB, after quickly discarding the idea to roll my own.
Now, there’s a pretty big technology stack behind OpenTSDB. This makes it a little silly on a single box in my office. Getting the dumb thing up and working was not trivial. Table creation was running on 30 minutes, with 8 hours of data (keep in mind we’re just talking the temperature of a wine rack) consuming gigabytes of disk space. Absurdly this seemed to be caused by a faulty IPV6 configuration on my box.
With that beyond me, I thought I was in the clear, until my HBase crashed after being up for just 5 hours. I couldn’t get the thing to re-start, either, and the log messages were useless. I kept getting inexplicable socket failures in zk. I sifted through megabytes upon megabytes of logs, and just could not figure out what was going on.
Turns out, I was experiencing an SSD failure due to a bug relating to interactions between SSDs and EXT4 in the Linux kernel. It had caused the filesystem to alternate between pretending it was read-only and just discarding I/O operations. Relatively hardware combined with a relatively dated distribution was not a good mix!
I eventually sorted it out. So now I can do stupid stuff like this:
I like the idea of this. The stock OpenTSDB front-end is powerful, but not particularly elegant, and is definitely lacking in some regards. The back-end is pretty awesome and appears well thought-out. Apparently there aren’t any good front-ends for it that are open source, though I guess you can build your own queries or your own entire front-end.
I’m still not delighted by the height of the technology stack for a small project (go figure), but it’s sort of neat what can be done. Especially if you already have a giant HBase/ZK thing going, it’s probably more than compelling.