Tweet-A-Watt: Beyond the Twitter

As I’ve written before, I really like the idea behind Adafruit’s Tweet-A-Watt project, which uses a modified Kill-A-Watt box and an Xbee transmitter to create an inexpensive home energy monitor. I’ve been running one 24/7 for over a year, and it has proven to be remarkably stable.

However, while the eponymous Twitter feature is (was?) novel and catchy, it’s not the most convenient way to track your energy usage. The Tweet-A-Watt project page includes a well written tutorial for using Google App Engine to graph your energy usage. It looks like a great weekend project, especially if you want an excuse to learn Python or the Google App Engine API, but I never got around to freeing up a weekend for it.

As an alternative, you can hook up your Tweet-A-Watt to an existing data tracking service with just a few lines of code. I’ve used a couple of “cloud-based” services for this, Pachube and Nimbits. They each have some features and advantages worth considering. I’ll cover Pachube in this article, and Nimbits in the next one.

Pachube

Pachube is a good choice for those who aren’t DYI software enthusiasts. It’s designed to get you up-and-running quickly, with as little software configuration as possible. Their site is easy to use, well documented, and filled with tutorials and examples. It has been around for a few years and is now widely used, so you can Google “Pachube” to find a lot of videos and articles showing how to use the platform.

The downside to using Pachube is that your data is stored for only a month, unless you sign up for one of their (fairly expensive) paid plans. There are also limits on how much data you frequently you can log or retrieve data, though those limits won’t affect Tweet-A-Watt usage. Still, there is nothing to stop you from using Pachube for day-to-day monitoring, and other software (such as Nimbits) for historical archiving and other more advanced usage.

Getting Connected

Here’s a walkthrough of how to get your Tweet-A-Watt connected to Pachube:

1. Register at Pachube.com.

2. You’ll need an API key, which you’ll paste into the Tweet-A-Watt wattcher.py script. While you can use your Master API Key, it is much safer to generate a separate API key for use by Tweet-A-Watt. (Wouldn’t want evil environmentally-conscious hackers posting their Tweet-A-Watt data to your feed, would you?)

Go to “My Settings” page in Pachube. Click on “New API Key”, then select the “all” checkbox as shown in the screenshot below. (There are other options that restrict how the API key is used, such as setting the source domain to your ISP, but when I played with those options in February the only one that worked for me was plain old “All”).

Pachube API Keys window

3. Register a feed. There is a link for this near the top right of most Pachube pages when you are logged in. For a Tweet-A-Watt feed, you should set the Feed Type to “Manual”. Most of the other fields are optional and up to you: I filled in a title, picked my location using the map, and left the other fields alone. There is an “Add a datastream” button near the bottom of the page. You can use it to manually create a datastream for each of your Tweet-A-Watts. However, since new streams from the wattcher.py script are automatically added here, I think it’s easiest to add your tags later, after the data shows up.

When you’re finished, Pachube will display a page listing various URLs for your feed. For the Tweet-A-Watt Python script you won’t actually use any of these – all you need to know is the unique numeric ID of your feed. For example, if your feed URL is http://api.pachube.com/v2/feeds/12345.xml, then your numeric ID is 12345.

Pachube Feed ID. That's my blur - go get your own.

4. That’s it for the Pachube end. Now for the Tweet-A-Watt end.

The least intuitive part of the process is getting the data you send into the format that Pachube prefers, called EEML. To do this, you’ll need to install a small piece of software, python-eeml on the computer running the Python wattcher.py script.

Go to the python-eeml page and click the Downloads button. Unzip (or “tar –xzvf”) the download file. Alternatively, if you have the git client installed (which you can do on Ubuntu by installing the git-core package), you can download the python-eeml by running the command “git clone http://github.com/petervizi/python-eeml.git”. Either way, you then open a command prompt, change to the “python-eeml” directory and run “python setup.py install”.

5. As a test, save the following Python code into a file (e.g. eemltest.py), fill in your API Key and Feed ID where indicated, and try running it (e.g. python eemltest.py).

import eeml

# parameters
# the key is a string, so include the quotes
API_KEY = '[the API key from step 2 above]'
# this isn’t a real URL, just a number, so don’t put quotes around it
API_URL = [the numeric ID of your feed from step 3 above]

pac = eeml.Pachube(API_URL, API_KEY)
# 99 is an ID for your senso
# 123.45 is the value in watts
# tags can be anything you like such as a text description of your sensor location
pac.update([eeml.Data(99, 123.45, tags=['Test1', ‘Tweet-A-Watt’], unit=eeml.Watt())])
pac.put()

Run the script. If you get an error, make sure you’ve got the data types right: API_KEY Is a string, API_URL is a 5-digit number, and the sensor ID (99) and watt value (123.45) are also just numbers without quotes. The “eeml.Watt” thing isn’t freeform text – it needs to be entered exactly as above, with no quotes.

If you experimented with API Key settings in step #2, an error might indicate that the settings you tried aren’t working – generate an API Key with the “All” setting to see if that’s the problem.

If the script runs without errors, then switch back to the Pachube web page, go to the “My Feeds” page, and click on the “Edit Feed” link. You should see a “datastream” 99 at the bottom of the page. Note that the tags you specified in the python code are displayed here – this is a good way to give a text description for your sensor ID.

6. Modify the Tweet-A-Watt wattcher.py script:

Add the import eeml statement to the top of the file:

import serial, time, datetime, sys
from xbee import xbee
import eeml  # add this!!

Add constants with your API key and URL number anywhere in the script – I added mine after the NUMWATTDATASAMPLE line:

CURRENTNORM = 15.5  # conversion to amperes from ADC
NUMWATTDATASAMPLES = 1800 # how many samples to watch in the plot window, 1 hr @ 2s samples

# Add this!!  Pachube settings:
API_KEY = '[the API key from step 2 above]'
API_URL =  [the numeric ID of your feed from step 3 above]

Then add the code to pass your Tweet-A-Watt data to Pachube. A good place to put this code is in the area where it is currently writing to the log file, which by default happens every 5 minutes.

Search for the following code in wattcher.py, which is in the update_graph method:

# Lets log it! Seek to the end of our log file
if logfile:
    logfile.seek(0, 2) # 2 == SEEK_END. ie, go to the end of the file
    logfile.write(time.strftime("%Y %m %d, %H:%M")+", "+
                  str(sensorhistory.sensornum)+", "+
                  str(sensorhistory.avgwattover5min())+"n")
    logfile.flush()

# Or, send it to the app engine
if not LOGFILENAME:
    appengineauth.sendreport(xb.address_16, avgwattsused)

The variables sensorhistory.sensornum and avgwattsused are what we want to send to Pachube. Add the following new lines underneath. (If you’ve never written Python code, the whitespace at the start of the line has to be exactly right – the “try:” line should start at the same column as the “if not LOGFILENAME” line above. The alignment of the other lines, relative to it, should be exactly as shown.

try:
    pac.update([eeml.Data(int(PACHUBE_ID), avgwattsused, unit=eeml.Watt())])
    pac.put()
except:
    print 'Error sending to Pachube'

The try/pass is included in case something temporarily goes wrong (like your Internet connection drops) – the try/pass ensures that wattcher.py will keep truckin’ rather than displaying an Exception message and terminating.

Run the wattcher.py script, give it 5 minutes to do its thing, then, go back to your feed page on Pachube. If everything is working, your data should show up as datastreams. You can fill in the descriptive fields for your datastreams as you like: whatever you type in won’t be overwritten by the incoming data.

Pachube Datastreams list

Once things are up and running, you can consider logging data to Pachube more frequently (the limit is 10 updates per minute). To do this, locate the code in the main loop (where it prints the data to the screen every time it gets data from an Xbee). However, you’ll have to use a counter or timer to ensure you don’t write to Pachube more than 10 times per minute. If you put the code there, you also have the option of logging other data to Pachube, like voltage or amperage or the RSSI signal strength of the XBee.

Output Feeds

By default, the feed’s page shows the most recent value and a tiny graph of the last 24 hours’ readings for each datastream. (Actually, they won’t resemble graphs until you’ve logged about half an hour of data). These graphs are quite useful for spotting any unexpected energy usage, but you probably don’t want to leave them tucked away in your feed page on Pachube.

Pachube Google Gadget. The dotted line indicates downtime (at my end in this case).

One quick and easy way to make these graphs more accessible is as a Google Gadget, which you can get from the Gadget gallery, or create and customize your own version using this page on Pachube. I have the gadget embedded in my iGoogle home page, where the data can be conveniently monitored.

There are variety of other graphs and notifications that can be configured for your data. You can read about them on Pachube’s Use A Feed page. Some of them can be used just by filling in a form, such as the SMS notifications, but most of them require some “assembly”.

As a test, or a 1-time view, you can manually enter a URL in a browser like the examples below. However, these URLs are intended to be embedded in a web page on your own site or blog, or used by software to retrieve data. For programmers there are a lot of possibilities available through Pachube’s extensive API.

Most of the request formats include 2 parameters: interval (how far apart the readings are) and duration (how many readings you want to display). If you are posting Tweet-A-Watt data every 5 minutes, like my sample code, then the shortest interval is 5 minutes. With this interval, the longest duration you can request is 5 days. Pachube restricts the amount of data you can request at once, so going beyond this will result in a somewhat confusing “Forbidden” web page.

For example, to see the raw data with timestamps over the last 24 hours for datastream 1 (replace “12345″ with your 5 digit feed ID):

This will return data in JSON format, suitable for processing by other software. Note that the times are all UTC (aka Greenwich Mean Time) – that’s the default format, which you can change on your Pachube settings page:


{"at":"2011-02-09T23:10:14.035059Z","tags":["DuinoFarm","Tweet-A-Watt"],"current_value":"72.2982427511","max_value":"164.251237339","unit":{"type":"derivedSI","label":"Watt","symbol":"W"},"min_value":"12.2650462797","id":"1","datapoints":[{"value":"32.5504473426","at":"2011-02-09T22:15:13.503066Z"}
…


To return data for all datastreams for the last hour in CSV format

which returns (datastream #, timestamp, value):


0,2011-02-09T22:25:03.179157Z,58.6875577135
1,2011-02-09T22:25:13.480201Z,77.7378909685
0,2011-02-09T22:30:05.235568Z,55.9543778625
1,2011-02-09T22:30:12.307615Z,75.1620522158
0,2011-02-09T22:35:02.070760Z,58.0097389617
1,2011-02-09T22:35:13.445684Z,76.802283106
…


As mentioned in the introduction, an alternative to Pachube is Nimbits. (The name is apparently unrelated to the Canadian delicacy Timbits, which, contrary to what my brother claimed when we are kids, are apparently not manufactured from bits of the desiccated corpse of restauranteur/hockey player Tim Horton. But I digress.)

Nimbits uses a similar interface for posting data, and has similar functionality for accessing your data, but with some extra prepackaged options, like sending updates through Google Chat or Twitter. (Hey, posting your Tweet-A-Watt to Twitter — there’s an idea!). I’ll cover Nimbits in my next blog post.