0
0
Non Sequitur Fridays

Data Acquisition

This post is part of our Non Sequitur Fridays series, which will feature a different Wistia team member's take on a non-Wistia-related topic each week. It's like our "employee of the month" but less "of the month"-y. Jason Lawrence is an engineer at Wistia.

Generating information is easy... collecting, storing, and visualizing it is not.

Most people encounter interesting sources of information on a fairly regular basis. Sometimes this information can be readily processed without manipulation (i.e., it's human readable). In many cases, however, our puny human brains can't cope with the complexity of the raw data and we resort to readily available tools such as spreadsheets, graphing utilities, parsers, etc. to help us grok otherwise inscrutable pieces of information.

Okay, fine. Spreadsheets are easy. Problem solved. But what if getting at the data is the hard part? This, too, might seem to be a triviality. If we only need to acquire the information once, a little extra effort might be worth the reward. But what if we aren't just interested in a snapshot?

Things really start to get hairy when we want our data fast, presentable, and/or compiled from from a number of separate sources.

Things really start to get hairy when we want our data fast, presentable, and/or compiled from from a number of separate sources.

As a computer scientist, I often encounter situations in which data is readily available but inconvenient to access. This sort of thing crops up a lot when I want to know things that: A.) must be sourced from multiple places (for example, a dynamic collection of network hosts); B.) require a lot of processing before a human should be expected to make heads-or-tails of them; C.) are best viewed in real-time (or close to it).

This entry will detail a simple real-world example of automated data acquisition, storage, and visualization. Basically, I'm going to run through a recipe for building a browser-based, near real-time visualization tool for atmospheric data harvested from an Arduino UNO.

I'm gonna try to keep this post digestible, so you won't find a full build log below. What you will find, however, is a reasonable outline of what I used and how it all fits together.

Hardware

Software

  • Redis, for data storage.
  • Sinatra, a lightweight Ruby web framework.
  • EventMachine, a Ruby event processing framework.
  • Code: C/C++ for the UNO, Ruby for the web app, and JavaScript for drawing pretty moving pictures.

Data Collection

The design is fairly straightforward. Data is harvested from the sensors hooked up to the Arduino UNO periodically and sent over the network to the data collection server via a UDP packet. Each of these packets contains a timestamp and samples from each of the onboard sensors, encoded as a JSON object. I used a 250ms interval for sampling and transmitting data (faster than needed, really). The sensor also makes use of NTP to provide accurate timestamps.

These encoded samples are received by a simple EventMachine based UDP server. All it really does is decode and insert them in a time-ordered Redis data structure.

Serving it up hot!

Getting the accumulating data into a browser is accomplished using Sinatra, a lightweight Ruby DSL for building web apps. All our app really needs to do is serve our sensor data and some fancy JavaScript for drawing graphs.

Whenever our graph view is loaded, client-side JavaScript kicks into action and makes periodic requests to the web server for the latest atmospheric sensor readings. This data is then used to create an interactive near real-time graph (thanks, <canvas>!).

The video above is a screencast from Firefox. The average latency from sensor to browser is generally well under a second. The purple curve is atmospheric pressure, yellow is ambient temperature, and blue is relative humidity (the spikes are caused by breathing on the sensor).

Hopefully you've enjoyed my overkill approach to what is essentially a basic DIY weather sensor. Sometimes want trumps need ;)

Follow Wistia on Twitter

We are constantly summarizing our Sequitur and Non-Sequitur antics in 140 characters or less.

Keep Learning
Here are some related guides and posts that you might enjoy next.