This post is part of our Non Sequitur Fridays series, which will feature a different Wistia team member's take on a non-Wistia-related topic each week. It's like our "employee of the month" but less "of the month"-y. Jason Lawrence is an engineer at Wistia.
Generating information is easy... collecting, storing, and visualizing it is not.
Most people encounter interesting sources of information on a fairly regular basis. Sometimes this information can be readily processed without manipulation (i.e., it's human readable). In many cases, however, our puny human brains can't cope with the complexity of the raw data and we resort to readily available tools such as spreadsheets, graphing utilities, parsers, etc. to help us grok otherwise inscrutable pieces of information.
Okay, fine. Spreadsheets are easy. Problem solved. But what if getting at the data is the hard part? This, too, might seem to be a triviality. If we only need to acquire the information once, a little extra effort might be worth the reward. But what if we aren't just interested in a snapshot?
Things really start to get hairy when we want our data fast, presentable, and/or compiled from from a number of separate sources.
As a computer scientist, I often encounter situations in which data is readily available but inconvenient to access. This sort of thing crops up a lot when I want to know things that: A.) must be sourced from multiple places (for example, a dynamic collection of network hosts); B.) require a lot of processing before a human should be expected to make heads-or-tails of them; C.) are best viewed in real-time (or close to it).
This entry will detail a simple real-world example of automated data acquisition, storage, and visualization. Basically, I'm going to run through a recipe for building a browser-based, near real-time visualization tool for atmospheric data harvested from an Arduino UNO.
I'm gonna try to keep this post digestible, so you won't find a full build log below. What you will find, however, is a reasonable outline of what I used and how it all fits together.
- Arduino UNO + Ethernet Shield
- BMP085 atmospheric pressure and temperature sensor
- HIH-4030 relative humidity sensor
- Redis, for data storage.
- Sinatra, a lightweight Ruby web framework.
- EventMachine, a Ruby event processing framework.
The design is fairly straightforward. Data is harvested from the sensors hooked up to the Arduino UNO periodically and sent over the network to the data collection server via a UDP packet. Each of these packets contains a timestamp and samples from each of the onboard sensors, encoded as a JSON object. I used a 250ms interval for sampling and transmitting data (faster than needed, really). The sensor also makes use of NTP to provide accurate timestamps.
These encoded samples are received by a simple EventMachine based UDP server. All it really does is decode and insert them in a time-ordered Redis data structure.
Serving it up hot!
The video above is a screencast from Firefox. The average latency from sensor to browser is generally well under a second. The purple curve is atmospheric pressure, yellow is ambient temperature, and blue is relative humidity (the spikes are caused by breathing on the sensor).
Hopefully you've enjoyed my overkill approach to what is essentially a basic DIY weather sensor. Sometimes want trumps need ;)