Netduino: Watching the Watcher

Note: This article is an installment in a series about building a plant light controller (and beyond) on the Netduino microcontroller platform.  For previous articles in the series, click here.

 It’s been a few months now since the Netduino in this project, with the help of its Arduino cousin, learned to measure the temperature and humidity in the room.  It’s faithfully reporting this data on its LCD… I think.  Who really knows what it’s doing when I’m not around?  It might be sleeping, or watching Fox News, or doing God-knows-what with its “cousin”.

 Clearly, this situation demands a nanny cam.  But my eyes ache at the thought of reviewing hours of footage of an LCD displaying temperature and humidity data (… I think).

Fortunately, now that the Netduino is connected to the Internet, we have a better option: online data monitoring.

 The Asocial Network: Rise of the Machines

 You’ve probably read recently of a rising trend, The Internet of Things.  It is becoming increasingly easy to hook devices and sensors up to the Internet, where they can report their status, receive instructions, play Farmville, and murder us in our sleep.

 The backbone of this system is the online data repository, which records the data and allows you to view the results..  I’ve written before of two of the early entrants in the field, Pachube and Nimbits. 

In this article, I’ll be connecting the Netduino to Nimbits via a Python middleman.  As explained in my last article, I prefer this approach to a common alternative, connecting the Netduino directly to an Ethernet connection.  If you’re interested in trying that approach instead, there’s a whole book that shows you how.

 Both Pachube and Nimbits use a REST API to receive data from sensors.  In past articles, I showed the Python code I use to send Tweet-A-Watt data to Pachube and to Nimbits.  I’m using the same API here.

 My article on using Tweet-A-Watt with Nimbits also covers the process of creating an account on the Nimbits server, getting an API key, and creating Data Points.  You’ll need to do all these things in order to receive data from the Netduino.  Your Nimbits settings are saved in the Python configuration file, explained below.

Netduino Configuration

The Netduino hardware and software is unchanged from the previous stage of the project, Netduino Meets World.  You’ll need to have XBees configured at both the Netduino and Python ends of the configuration – details are in Netduino Meets World.

Well, actually, you don’t need XBees if the Netduino is located quite close to the PC that’s running Python.  This was true for the Netduino Meets World article as well, I just forgot to mention in there.  Since we’re using the XBees as a generic serial device, a USB connection will also work with no software changes needed.  I have one of my Netduino’s connected by USB to a Beagleboard XM running Python. 

You can’t use the Netduino’s USB port for this, but you can connect the pins to TTL-to-USB adapter.  One option is the 3.3V FTDI cable that I described way back in the Netduino: Time and Weather article.  A slightly cheaper option is an FTDI breakout board – I’m using this one from Sparkfun and can confirm that it works fine with the code accompanying this article.

Configuring the Python Interface

The Python code, TimeRelay.py, builds upon the foundation of the Python program used last time, TimeServer.py.  TimeServer was configured using command line switches – since there are a lot more settings now, I’ve switched to using a configuration file.  You can download the Python code and sample configuration file from my Google Code page, here.

The configuration file contains the following settings.   The ones that you have to fill in are marked in italics.

# Configuration file for TimeRelay.py
# SERIALPORT = \.COMxx <-- Windows COM port must be specified in this format
SERIALPORT = /dev/ttyUSB0
LOGFILENAME = TimeRelay.log
DEBUG = True
NIMBITS_SERVER = http://app.nimbits.com
NIMBITS_USERID = 
NIMBITS_API_KEY = 
NIMBITS_UPDATE_INTERVAL = 300
# Sensor IDs - format is <Netduino Sensor ID> = <Nimbits Data Point ID>[,Nimbits Update Interval in seconds]
H1 = PlantRoomHumidity,1800
T1 = PlantRoomTemperature

The first 3 settings correspond to the command line parameters in the program’s predecessor, as listed in the last article:

• SERIALPORT – Specifies the serial port that the XBee is connected to.  Note the unusual format of the serial port name for Windows PCs — this is required by the PySerial library.
• LOGFILENAME – The logfile has an expanded role – in addition to any error messages, it also records the data readings received from the Netduino in a comma-delimited format.  You can disable the local logging feature by leaving this setting blank (i.e. deleting “TimeRelay.log” from that line).
• DEBUG – Turns on debug messages written to the console.  This is on by default, since it is quite useful for troubleshooting.

The Nimbits settings are taken from your Nimbits account:

• NIMBITS_SERVER -  The default setting, app.nimbits.com, is the public Nimbits server.  If you don’t want to rub shoulders with the unwashed masses, you can setup your own private Nimbits server on Google App Engine – you would then enter your GAE server name here (e.g. whatever.appspot.com).  (Both Nimbits and GAE are free when used for this purpose.  I briefly described the process of setting up a Nimbits server in the “Rolling Your Own” section of my Nimbits article)
• NIMBITS_USERID – This is the Gmail address you used when creating your Nimbits account.
• NIMBITS_API_KEY – This key is e-mailed to you when you click the “Secret Key” toolbar button in your Nimbits server console. 
• NIMBITS_UPDATE_INTERVAL – This specifies how often Nimbits will be updated with the latest sensor reading.  The default is 300 seconds (i.e. 5 minutes), but you can override this for individual sensors on the sensor lines at the bottom of hte configuration file.  Note that the Netduino will send a new measurement every 60 seconds, regardless of what you configure here – the Python app will  send Nimbits an average of all the readings since the last update.

The sensor settings are used to link the sensor IDs hard-coded in the Netduino application with the Data Point Names that you created in Nimbits.  The default settings assume Data Point names of PlantRoomTemperature and PlantRoomHumidity – if you use other names (as you might if you don’t actually have a Plant Room), then fill them in on these lines of the configuration file.

Note that the settings shown above will send a new humidity reading to Nimbits every 30 minutes, while the temperature will be sent using the default interval of 5 minutes.  Feel free to change this depending on how meteorologically active your home is.

Activate Sensors

After you’ve finished updating TimeRelay.cfg, run TimeRelay.Py and watch the console.

The program will start by listing the settings it found in the configuration file, then will open the serial port and wait for something to happen.

If a Netduino running the code from the last article is powered on,  and you left the “Debug” setting turned on in the TimeRelay.cfg file, you should see a temperature and humidity reading displayed within a minute.  A message showing the REST API call that sends the data to Nimbits should be displayed in 5 minutes (or whatever interval you used in the TimeRelay.cfg file). 

Switch to your Nimbits web page, double-click on the Data Point in the left hand pane, and the Data Channels pane should update with the newly received value.  

There are a bunch of useful settings that you can experiment with in the Data Points dialog.  I use the Idle Alarm setting to tell me when something has gone wrong with Netduino/Python configuration, and the High and Low Value alert to tell me when something has gone terribly wrong with my home.

The Nimbits web page also has some nifty built-in graphing capabilities.  These have recently changed with the release of Nimbits 3.2, and will continue to evolve, so see the Nimbits web page and Nimbits blog for the latest.

Since I have a fairly large number of data points that I like to glance at from time to time, I wrote a Google Gadget to display line charts in the iGoogle page.  I also wrote a blog post describing its development.  Check it out and let me know of any new features you’d like to see. (Or feel free to take the code and add the features yourself).

Programmer’s Show and Tell

Although the Netduino code isn’t new, I haven’t previously explained in the sensor data handling.

The upload process is run on yet another timer.  This is hardcoded to occur every 60 seconds — nothing magical about that interval, and it could certainly be changed as needed.

Since we’re using the XBee as a generic serial interface, all the Netduino needs to do is write the data to the serial port.  The sensor ID is hard-coded, and each reading is ended with a CR/LF – a “crude but effective” message format. I haven’t noticed any problems with data being garbled or truncated:

// upload sensor readings once a minute
timerUploadSensors = new Timer(new TimerCallback(uploadSensorData),
 null, 60000, 60000);

. . .
private static void uploadSensorData(object data)
{
 string strReading = "";
 if (blnHumidityUpdated)
 {
  blnHumidityUpdated = false;
  // make copy of humidity reading, since it
  //    could be overwritten at any time
  strReading += "H1:" + strHumidity + "rn";
 }
 if (blnTempUpdated)
 {
  blnTempUpdated = false;
  // make copy of temperature reading, since it
  //    could be overwritten at any time
  strReading += "T1:" + strTempDHT11 + "rn";
 }
 uart.WriteToUART(strReading);
}

The Python code is a little more interesting.

First, I’ve added a workaround for a known problem with the PySerial library when running on Windows.  When PySerial tries to open the COM port it thinks that its already open – the workaround is to always close it before opening it.  Trippy but effective, and the workaround doesn’t cause problems on other OSes:

ser = serial.Serial(SERIALPORT, BAUDRATE, timeout=TIMEOUT)
ser.close() # workaround for known problem when running on Windows
ser.open()

The code which reads TimeRelay.cfg demonstrates some of Python’s file- and string-handling power.  Five lines of code is all that is required to open a file, read its contents, parse the lines, and store them in a dictionary. 

def readConfigFile():

    #global config

    for line in open(CONFIGFILE):

        if line.strip()[0] != "#": # skip comment lines

            parts = line.split("=", 1)

            if len(parts) == 2:

                config[parts[0].strip()] = parts[1].strip()

The sensor readings received from the Netduino are stored in a dictionary.  The key is the sensor ID (e.g. “T1”) and the value is a List with format:

<# of readings since last upload>,
<total of readings since last update>,
<date/time of last upload> 

Python dictionaries are cool because they can store any object as their value, and the object type can be different for different entries in the same dictionary.  Less cool is the fact that Python throws an exception if you try to read a dictionary key that doesn’t exist yet, so be sure to check to see if an earlier reading is already stored there.

 
# add this reading to the sensor's list
if sensor in sensorValues:
    sensorValues[sensor][0] = sensorValues[sensor][0] + 1
    sensorValues[sensor][1] = sensorValues[sensor][1] + float_value
    if DEBUG:
        print "found sensorValue", sensor, sensorValues[sensor]           
else:           
    sensorValues[sensor] = [1, float_value, datetime.datetime.now()]
    if DEBUG:
        print "adding new sensorValue", sensor, sensorValues[sensor] 

Note that Python isn’t too picky about variable types except when it is.  If “sensorValues[sensor][1] ”or “float_value” weren’t both floating point values, an Exception would be thrown when they are added together.

The code that sends the data to Nimbits uses the CurrentValue REST API.  This API has a dual role of writing new values and reading the most recent value, but we’re only using it to write data here.

The code starts out with a hack, to round the value to 3 decimal places before sending it. This is done for purely asthetic reasons, so that the readings displayed in the Nimbits web page look nice and tidy.  I call it a hack because rounding a number shouldn’t require this degree of string manipulation.  (Note to Pythonistas: I’m criticizing my code, not your language. Peace out.)

 
# HACK round to 3 decimal places and drop trailing zeros
#  - technically not needed, but makes Nimbits output nicer looking
stringValue = ("%.3f" % value).rstrip('0').rstrip('.') 

try:
    LogMsg(dataPoint + "," + stringValue)
    url = (NIMBITS_SERVER + "/service/currentvalue?value=" +
        stringValue + "&point=" + dataPoint.replace(" ", "+") +
   "&email=" + NIMBITS_USERID +
        "&secret=" + NIMBITS_API_KEY)

    urllib.urlopen(url)
    return True
except IOError:
    print 'Error sending to nimbits'
    print sys.exc_info()[0]

The REST API call is a 2-liner, another example of Python’s power and simplicity compared to .Net.  (Note to Microsoft Fanboys: I’m on your side. Peace out.)  The only trick is to replace spaces in the Data Point name with plus signs, this being a URL and all.

Note that it isn’t necessary to send a timestamp with the data – the CurrentValue API assumes the value has a current timestamp unless otherwise specified.  Good thing, that, since working with time zones in Python can be a nightmare.  (Note to Pythonistas: yeah, now I’m criticizing your language).

Unfortunately the Nimbits CurrentValue API doesn’t return a value to indicate whether the API call was successful, so the Python code won’t write an error message to the console or log file if something goes wrong.  My first indication that something is wrong is generally when Nimbits’ “Idle Alarm” sends me an e-mail.

 Wrapping Up

The Netduino project has admittedly become quite unwieldy, with each stage heaping on more hardware and/or software to the previous stage.    It’s unlikely that anyone but me has all of the Netduino hardware add-ons needed to make use of all the software’s features.

In the next article in the series, I’m going to switch to a slimmed-down Netduino software image for a sensor node, that consists of only a Netduino, an XBee, and a few new analog sensors.