Beagleboard XM / Angstrom Next Steps: NTP Time Daemon and Cron

At about 8 am on January 1st, I was awakened by the beep-beep-beep of an unhappy UPS system.  The power had gone off, and it stayed off for about half an hour. Mercifully (this being 8 am on January 1), my UPS bleated for only about 15 minutes before going to sleep, as did I soon afterwards.

At about 8:30, my Beagleboard was rebooted and promptly resumed its role as Tweet-A-Watt server.  You’ll have to take my word for that, since it abandoned its role as Tweet-A-Watt Twitterer.  I didn’t notice this latter fact until earlier today.  So, if you read my Beagleboard XM And Angstrom – Getting the Big Dog to Run At Full Speed
post sometime between Jan 1 and Jan 3, you might have wondered if the Big Dog had, indeed, begun to fight with Angstrom.  But, rest assured, the failure was mine, not the Beagleboard’s or Angstrom’s.

Hello, Dali
Hello, Dali
What I had failed to do was configure an NTP daemon, so that it would automatically get the correct date and time in just such an event.  Oddly, it ended up being about 6 days off – when I checked it earlier today the system date was December 28th.  This led to a “date out of range” error from the Twitter API when it attempted to tweet.

There is a post on the Mechomaniac blog which explains the process of configuring your timezone and installing the NTP package.  After following those instructions, you can enter ntpdate pool.ntp.org, and your system date and time should be set correctly.  (If the time is off, you probably didn’t set the timezone correctly – as stated in that blog entry, the /etc/timezone file used by some Linux distros seems to be ignored by Angstrom.)

Although I didn’t include the NTP configuration in my initial article about setting up Angstrom the Beagleboard XM, I actually had done it.  I also took the next logical step, which is to configure ntpdate to be automatically run periodically, so that your system date is updated after, say, a power outage.

Or, so I thought.

What I did was

opkg install cron

[Update Aug 13: Oh, hai! Dan from the Future, here. Sorry to interrupt your reading, but I just installed a new Angstrom image generated with Narcissus, and I found that the ntpdate entry was placed in the correct crontabs file after installing cron: /var/cron/tabs/root. So, if you're configuring NTP using an up-to-date cron package, then you're probably done! However, I'm sure you'll want to continuing reading, to see what madcap adventures Dan from the Past got himself into this time...]

Period.  The next step should have been to configure cron to run ntpdate.  However, when I checked to see how cron was configured, I noticed that a folder named /etc/cron/crontabs existed, and in there was a file named root that contained:

30 * * * *    /usr/bin/ntpdate -s -u pool.ntp.org

And, furthermore, the cron package automatically added a file to etc/init.d so that cron was automatically started when the system booted.

Hey yeah! That was easy, especially by Angstrom standards.  Surprisingly easy.  Suspiciously easy.  Hmmm.

On January 1 I belatedly learned that it isn’t quite that easy.  The problem is that /etc/cron/crontabs isn’t where the configuration tables for cron are stored, at least not in the world of Angstrom.  They are in /var/cron/tabs.  One way to configure cron to run ntpdate is to copy the file from /etc/cron/crontabs to here. Then, restart the cron daemon

cp /etc/cron/crontabs/root /var/cron/tabs/
/etc/init.d/cron restart

In the interests of Linux orthodoxy, I should mention that this isn’t the “proper” way.  The proper way is to make use of two crontab commands.

To see what is currently configured (for root – for another user’s crontab, replace “root” in the following commands with the user-ID):

crontab -u root -l

If you are already root (which you should be for the purpose of configuring ntpdate), you can leave out the user name:

crontab -l

To change what is configured

crontab -e

This last one will launch vi so that you can edit the file.

Oops, maybe I should have mentioned that earlier. If you are now staring dumbfounded at the world’s most user-hostile editor, press the Esc key, enter :quit! and take a deep breath. Then, copy the root file from the crontabs folder, as specified earlier. Should you need to change the contents of this file, you can just edit the “root” file using your preferred editor, then copy the resulting file into /var/cron/tabs.  If you go this route, you’ll need to manually restart cron afterwards, whereas changes made with crontab -e are automatically detected by cron.

Actually, its possible to configure crontab to use another editor — Google it if you’re interested.

The above crontab line will run ntpdate at the 30th minute of every hour (i.e. 12:30, 1:30).  That’s probably fine, but if you want to stand out from the crowd you can change it to something less common like 13 * * etc, or change ntp.pool.org to your favorite neighbourhood NTP server.

Not working? Not sure that it’s working, and anal retentive? By default cron will log its results to /etc/cron/log, in a format like this

root (01/03-17:03:00-1237) CMD (-s -u pool.ntp.org )
root (01/03-17:03:00-1236) MAIL (mailed 27 bytes of output but got status 0x0001)

root (01/03-17:08:00-1259) CMD (/usr/bin/ntpdate -s -u pool.ntp.org )

The “MAIL” entry is how it handles errors – presumably you have to configure mail on your Linux box in order for that error message to actually go somewhere.  It wouldn’t matter much in the case of this ntpdate command, since the “-s” tells it to log its output to syslog.  And syslog is another thing that works differently in Angstrom (and doesn’t work at all by default). Sigh. We won’t go there.

If you want to know what went wrong when ntpdate goes wrong, you’ll have to change the crontab entry to something like the following:

13 * * * *  /usr/bin/ntpdate -u pool.ntp.org ›› /home/root/ntpdatelog.txt

This will append the results of the ntpdate command to a text file.

Oh, and a belated Happy New Year from my Beagleboard!

Posted in Gadgets | Tagged , | Leave a comment

Netduino and Analog Input: Look Ma, No Breadboard!

If you carefully compare the photos in my first and second posts about the Netduino, you may have noticed a couple of things:

Before
Before
-
After
After

- There is no longer a mass of wires on the breadboard connecting the potentiometer to the Netduino.  In fact, the potentiometer is connected directly to the Netduino – no breadboard required.

- There is a small gray knob embedded in the potentiometer’s cable.

So, what’s up with that?

Well, only the greatest invention that I’ve ever made this week! (But it’s only Wednesday).

Ladies and Gentlemen, I give you the incredible, inedible (I assume) Voltage Dividing Cable!!

Voltage Divider Cable (TM).  Yeah, ground is red and signal is black.  Go figure!
Voltage Divider Cable (TM). Yeah, ground is red and signal is black. Go figure!

The idea was to replace the complicated-but-accurate Sparkfun Logic Level Converter with a simple-but-less-accurate voltage divider circuit that could be embedded in the Grove connector cable. (The innovative use of black for signal and red for ground wasn’t my idea – I give full credit to Seeed Studio for that one.)

The use of a voltage divider to connect 5V sensors with a 3.3V microcontroller is, of course, not an original one. For example, here’s an article that describes this approach in as simple terms as possible.

The selection of 2 resistor values likewise not difficult. There are plenty of voltage divider calculators on the Internet, such as this.

You have to be careful, though, to take into account the variation between the resistor’s marked level and its actual level. Most resistors have a 5% variation (as marked with a gold band). For example, common resistor values of 330R and 620R should, in theory, convert a 5V signal to a nice safe 3.26V. However, if both resistors have variations that break the wrong way, you could end up with a 3.5V signal, which may not be healthy for a 3.3V microcontroller.

One approach to dealing with this is to play it safe and go with a circuit that will bring you well short of 3.3V. Your code can then upscale the analog reading as necessary. For example, the 10K and 15K resistor values mentioned in the Making Things article will result in a 5V signal being converted to 3V. Your code can then increase the analog reading by 10% to compensate.

Another approach – the one I used – was to replace one of the resistors with a small potentiometer. Using a multimeter, you can then adjust that potentiometer’s setting so that you come as close to 3.3V as you like. This approach could also be used if you happen to be working with a microcontroller whose ADC’s tolerance is even lower than 3.3V.

Posted in Electronics | Tagged , | Leave a comment

Netduino: Time and Weather

Note: The source code for the project described in this post can be downloaded here.

This post is a continuation of my “Netduino Enters the Grove” article.   Building on the project in that post, I’m going to make the Netduino a more productive member of society by having it display the time, in addition to the temperature.

I’m also going to add to the mix a couple more “twigs” from the Seeed Grove Starter Kit: the 8 x 2 LCD, and the piezo buzzer.

Lastly, I’m going to make use of the Netduino’s serial port to show how to make it communicate with a PC.

Adding the Grove Parallel LCD

The Seeed Grove LCD - all the news that 8 x 2 can fit.   Click to enlarge.
The Seeed Grove LCD - all the news that 8 x 2 can fit. Click to enlarge.

The Grove Starter Kit comes with a rather odd LCD: a cramped 8 x 2 board bristling with 24 pins, 6 of them for voltage and 6 for ground.

Since documentation isn’t Seeed’s strong suit, this has led to some head-scratching about the best way to connect the LCD to an Arduino. I wasn’t too optimistic about getting it to work with a Netduino, which is one of the reasons I went with a serial LCD in my earlier post.

However, it turned out to be relatively straightforward to connect, thanks to Szymon Kobalczyk’s excellent MicroLiquidCrystal library.

The LCD connectors that you need to use are the 2 on the right and the 2 on the bottom. The pins are labelled on the back of the LCD’s  ”stem” board – clockwise from the upper right  connector (and ignoring the voltage and ground pins) they are RW, NC (unused), RS, EN, D4, D5, D6 and D7. These are the pins used in almost any LCD library based on the HD44780 chipset, including one of the constructors of the MicroLiquidCrystal library. (MicroLiquidCrystal supports other types of connections that use fewer digital ports, but for the purpose of testing the Grove LCD’s default configuration I’ll stick to the more traditional GPIO connection for this project).

The 2 key lines of code are:

GpioLcdTransferProvider lcdProvider =
     new GpioLcdTransferProvider(rs, rw, enable, d4, d5, d6, d7);
Lcd lcd = new Lcd(lcdProvider);

In my project, I’ve added a wrapper class to MicroLiquidCrystal in order to make to compatible with the 2 other devices I use when testing: a Matrix Orbital serial LCD and a Sparkfun Serial LCD Backpack (as described in my earlier article). So, the code which calls the above methods is in the clsLCD_MLC constructor in the attached source code:

public clsLCD_MLC(Cpu.Pin rs, Cpu.Pin rw, Cpu.Pin enable, Cpu.Pin d4,
    Cpu.Pin d5, Cpu.Pin d6, Cpu.Pin d7, byte rows, byte cols)
{
// NOTE - RW pin is needed for Seeed's Grove LCD
lcdProvider = new GpioLcdTransferProvider(rs, rw, enable, d4, d5, d6, d7);
//lcdProvider = new GpioLcdTransferProvider(rs, enable, d4, d5, d6, d7);
lcd = new Lcd(lcdProvider);
SetScreenSize(rows, cols);
lcd.Write("hello, world!");
Thread.Sleep(500); // greet the world and give the LCD time to get its act in order

The call to this code specifies what pins are being used. The configuration shown in the photos uses the following:

lcd = new clsLCD_MLC(Pins.GPIO_PIN_D4, Pins.GPIO_PIN_D13, Pins.GPIO_PIN_D5,
Pins.GPIO_PIN_D9, Pins.GPIO_PIN_D10, Pins.GPIO_PIN_D11, Pins.GPIO_PIN_D12, 2, 8);

So, the 4 Grove cables are connected to the Grove shield using (clockwise from the upper right): D13/NC, D4/D5, D9/D10, DD11/D12. Actually, I’m pretty sure that any 4 connectors (or 7 pins) would work just as well. I went with these in order to save the serial port pins and one of the PWM pins  for other uses.

The LCD tutorial in the Grove documentation suggests that a 6 pin connection is also possible (minus the RW pin), but I couldn’t get that to work with either the Arduino or Netduino.

By the way, if you are trying your Grove LCD for the first time and nothing appears on the screen, the first thing to check is the contrast control. This is a little metallic potentiometer on the LCD “stem” board – it looks like a brass screw-head. If you don’t see a line of block characters when you first power-up the LCD, then turn the potentiometer counter-clockwise, then try again.

There are a few drawbacks to the Grove LCD compared to other parallel LCDs. The first and most obvious is the size – 8 x 2 characters is OK for the time and temperature, but won’t be enough for future projects. The backlight is “always on”, with no easy way to control that. Presumably, it can be controlled by connecting one (hopefully not all 4!) of the connectors’ voltage pins to a digital PWM pin. However, I haven’t tried this myself yet). The 2 connectors on the bottom, combined with the stiff connector cables, makes mounting the LCD somewhat of a mechanical challenge, as you can see in the photo.

And yet it works. I tested the Netduino Time/Temperature gadget for a couple of days, and the LCD didn’t display any garbled characters or have any other problems.

Adding the Grove Piezo Buzzer

The buzzer connection is quite straightforward. You can connect it to any digital pin if you just want it to buzz, or to a PWM pin if you want to give it a greater musical range. The Netduino has just four PWM pins, 5, 6, 9 and 10. For the purpose of this project, I’m using pin 6.

Since PWM is not supported by the underlying Micro Net Framework, its implementation is left up to the hardware maker. As with Seeed Studios, documentation doesn’t seem to be Secret Labs’ strong suit – they definitely know how to keep a secret. This makes the process of controlling the buzzer a little tricky to figure out. I found the best source to be the full-featured and well-documented code in this post by Ray Mckaig on the Netduino forums.

This is a low-end buzzer, so its musical range runs the gamut from annoying to tortuous. The following code makes 2 types of sounds: an annoying ding-a-ling somewhat like a cordless phone (when the parm is true), or a tortuous beeeeeep that my cat really, really dislikes.

public static void Beep(bool blnUp)
{

blnBeeping = true;
float fltFreq = 10;
int intDuration = 1000;
if (!blnUp)
{
fltFreq = 30;
intDuration = 1000;
}

uint period = (uint)(1000000 / fltFreq);
buzzer.SetPulse(period, period / 2);
Thread.Sleep(intDuration);
buzzer.SetDutyCycle(0);

}

Setting the time through a serial port

Netduino's serial connection - click to enlarge
Netduino's serial connection - click to enlarge
One of the most noticeable differences between the .Net Micro Framework and the Arduino is that it is considerably more difficult to have a PC send commands to a NMF device. The Arduino uses the USB connection for this, but the NMF reserves the USB connection for use by the debuggers.

(Secret Labs has recently added a firmware feature that allows your program to use the USB port as a serial device, and use one of the Netduino’s serial ports for deploying code and debugging, but I haven’t tried that out yet.)

When using a Netduino serial port for communication with a PC, you’ve got 2 problems: the voltage level sent and received on the Netduino’s pins (3.3V) is much lower than what the PC sends and receives, and the communication protocol used by the Netduino (TTL) is not the same as what the PC uses (RS-232).

Fortunately, there is a relatively inexpensive device that solves both problems: a 3.3V FTDI cable. This is widely available at hobby electronics retailers such as Adafruit and Sparkfun. If you use an Adafruit Boarduino, Solarbotics Ardweeny, Secret Labs’ own Netduino Mini, or some other microcontroller-based device that doesn’t include a USB port, then you probably already have this cable.

With the cable, connection of the Netduino to a PC pretty simple. Only 3 of the cable’s coloured connectors are used: the black cable (ground) should be connected to ground, the orange cable (TX) should be connected to the RX pin of one of the Netduino’s 2 COM RX pins (pin 0 or 2) and the yellow cable (RX) to one of the Netduino’s TX pins (1 or 3).

In the photo of my setup, I’ve inserted a 5 pin male header like this into the breadboard.   You’ll need to adjust the pins in the header slightly so that they stick out about the same amount on both sides of the header.  Since I’m using COM2 on the Netduino,  the white jumper cable goes from the breadboard to pin 2 and the yellow jumper to pin 3.

The code to open the COM port is in the constructor of my “clsUART” class:

public clsUART(string strPort, Int16 intBaud, int intReadBufferLen, bool blnSendResponse)
{

SerialPort UART = new SerialPort(strPort, intBaud, Parity.None, 8, StopBits.One);
UART.Open();
UART.DataReceived += new SerialDataReceivedEventHandler(UART_DataReceived);
Debug.Print(strPort + " Opened at baud " + intBaud);
WriteToUART("Hello World");

And the call to this constructor is:

uart = new clsUART(SerialPorts.COM2, 9600, 50, true);
uart.Command += new CommandEventHandler(uart_Command);

A couple of things to note about using the MNF’s SerialPort:
- In order to use the Net MF’s SerialPort class, the Project References must include a reference to Microsoft.SPOT.Hardware.Serial. This isn’t included by Visual Studio in a new Netduino project by default – it has to be added manually

PuTTY settings, with some (partially overwritten) terminal commands in the background.  Click to enlarge
PuTTY settings, with some (partially overwritten) terminal commands in the background. Click to enlarge
- The SerialPort constructor allows you to specify not only the COM port, but also the baud rate and the other, more esoteric serial connection settings. Be sure that the serial terminal program on your PC it using the same settings. In my case, I’m using PuTTY, and the serial settings are as shown to the right. The only settings that I changed from the default were Flow Control (set it off) and Local Echo (force it on, in order  to see what you are typing).

One of the nice things about the FTDI cable is that the chip which handles the RS-232 communications with the PC is embedded in the cable, and therefore it is ready to go as soon as you plug the cable into the PC. You can then use PuTTY (or your serial terminal of choice) to connect to whatever COM port the PC assigns to the cable. If there is nothing at the other end of the cable, other terminal screen will remain empty, but you won’t get any errors either.

More PuTTY settings - note the Local Echo option.  Click to enlarge
More PuTTY settings - note the Local Echo option. Click to enlarge
The Netduino is not too picky about its own COM port – you can power it on with nothing connected to the COM port pins and it will happily open that COM port without complaint.

However, if you “hot plug” the FTDI cable into your Netduino when it is already up and running, you’ll likely find that the connection doesn’t automagically start working – you have to reset the Netduino’s COM port. I added some code to handle this by pressing and holding the red button. This toggles the COM state on or off, sends a message  to the LCD, Visual Studio’s Debug window, and the PC serial’s terminal, and makes a buzzer noise (the annoying one when the port goes down, the tortuous one when the port comes back up).

If your PC’s serial terminal displays “Hello world” when the Netduino boots up, or “Hello again world” when you toggle the port open with the red button, then the Netduino is yours to command. It knows the following commands:

Set time:
T:hh:mm[:ss]
e.g. T:14:35

Set date:
D:yyyy-mm-dd
e.g. D:2010-12-29

Set temperature format (Celcius or Fahrenheit):
F:F|C
e.g. F:C

Set LCD backlight:
L:ON|OFF
e.g. L:OFF

Set LCD brightness:
B:nnn where nnn is 0-255
e.g. B:128

The Netduino will respond to a valid command by sending “OK” to the terminal.  Generally the LCD will immediately be updated with whatever setting you changed. However, in the case of the 8 x 2 Grove LCD setting the date and changing the backlight or brightness has no effect.

Should you enter a command that the Netduino doesn’t recognize, it will send a “?” to the terminal and making the annoying sound with the buzzer, so type carefully!

All things considered, this is probably the simplest way to set the time on the Netduino, but also the least convenient.  (A clock that has to be set using a serial port – Grandma’s worst nightmare!).

In future posts, I plan to demonstrate a few other methods: an on-screen menu (more convenient, but  lots of coding), an Ethernet shield (very convenient for the user – it sets itself – but not so convenient if you don’t happen to have an Ethernet port nearby) and a wireless XBee connection (very very convenient, très cool, and my platform of choice for a lot of projects).

Note: The source code for the project described in this post can be downloaded here.  The .zip file contains 2 folders: one (ClockAndTemperature) with my source and one (MicroLiquidCrystal) with the MicroLiquidCrystal library.  The latter is a separate project that is loaded by the ClockAndTemperature solution file.  The 2 folders should therefore be stored in the same parent folder.

Posted in .Net, Electronics, Programming | Tagged , , | 1 Comment

Beagleboard XM / Angstrom Next Steps: Assigning a Static IP

A few weeks ago I wrote about configuring a Beagleboard XM to run Angstrom Linux.

At the time I was somewhat dubious about whether Angstrom would prove to be a stable platform. So far, so good. My Beagleboard XM has been faithfully working 24/7, receiving Tweet-A-Watt data via XBee and posting power usage updates to Twitter.

Over that time, I’ve made a few tweaks to the initial configuration and have begun to give the new dog some new responsibilities.

The first of the tweaks is to have it use a static IP address rather than using one assigned by the router’s DHCP service.

This tweak is necessary because the XM’s Ethernet port is not a dedicated piece of hardware with a “burnt-in” Ethernet MAC address. Instead, it makes use of the same USB hub chipset’s ability to handle Ethernet communications. (The chipset is an SMSC LAN9514). Saving and restoring the MAC address is the responsibility of the software, and the Linux driver used for this chipset doesn’t support this feature currently.

The end result is that the DHCP server thinks the Beagleboard XM is a new PC every time it is rebooted, and assigns it a new IP address. This is a major nuisance if you are using the XM as a headless server. After each reboot you have to hunt down the XM over the network by checking the DHCP logs, or by physically connecting to its serial port.

One solution to the problem is to manually assign a MAC address by changing the XM’s boot parameters, as described in Max Galemin’s blog post here.

Another solution is to manually assign the IP address, instead. This is somewhat easier to configure, and it allows you to choose an easy-to-remember address. One downside to this approach is that Wake-On-Lan support won’t work, since you need to configure a MAC address for the “magic packet”. I went with a static IP, but if you can’t let sleeping dogs lie (nyuck nyuck nyuck) then go with the static MAC instead.

The process of configuring a static IP address is the same in Angstrom as in most Linux distros. The only catch is that the device name is /dev/usb0, rather than the /dev/eth0 used on most PCs.

1. Switch to root, backup /etc/network/interfaces, then edit it:

user@beagleboard ~  su -
Password:
root@beagleboard:~# cd /etc/network
root@beagleboard:/etc/network# mkdir back122610
root@beagleboard:/etc/network# cp interfaces back122610/

2. You’ll need to know the IP address of your gateway and your DNS server. If you aren’t sure, you can see what the XM is currently using (as assigned by the DHCP server) by running a couple of commands:

root@beagleboard:/etc/network# netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 usb0
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 usb0

root@beagleboard:/etc/network# cat /etc/resolv.conf
search lan
nameserver 111.222.333.444

Your gateway address is displayeded in the 2nd row of output from netstat -nr (192.168.1.1 in the above example). Your DNS is displayed in the 2nd line of /etc/resolv.conf (111.222.333.444).

3. Edit the interfaces file (e.g. vi interfaces). Comment out the line that says “iface usb0 inet dhcp” and replace it with the following lines, specifying your own IP addresses. In the following example, I’m assigning the XM address 192.168.1.2. The gateway (my router) is address 192.168.1.1. Since my router is configured as a DNS server, my DNS server address is also 192.168.1.1.

# Ethernet/RNDIS gadget (g_ether) or LAN9514
auto usb0
# iface usb0 inet dhcp
iface usb0 inet static
      address 192.168.1.2
      netmask 255.255.255.0
      network 192.168.1.0
      broadcast 192.168.1.255
      gateway 192.168.1.1
      dns-nameservers 192.168.1.1

4. If you are connected from a remote PC, reboot. Otherwise, if you are connected to the Beagleboard’s serial console or with a keyboard and LCD, you can just start and stop the network interface:

root@beagleboard:/etc/network# shutdown -r now

…or…

root@beagleboard:/etc/network# ifdown usb0
root@beagleboard:/etc/network# ifup usb0
Posted in Gadgets | Tagged , | 4 Comments

Netduino Enters the Grove

Note: The source code for the temperature LCD program described in this article can be downloaded here.

I recently received one of Seeed Studio’s new Grove Starter Bundles. Since many others will put GROVE through its paces using an Arduino — and Seeed recently posted a series of projects that do just that — I thought it would be interesting to try it out with a Netduino instead.

For those unfamiliar with these two products, Grove is a set of plug-and-use electronics components, like sensors and controls, intended to be used with an Arduino-compatible shield. The Netduino is a .Net Micro Framework-based system that uses the same form factor and pinout as the Arduino.

Why Grove?

Netduino meets Seeed Grove
Netduino meets Seeed Grove
The Grove system (and Seeed’s previous generation Electronic Bricks) are great for prototyping. Wiring components to a breadboard-based Arduino system can be rather frustrating, since you have to be sure that all the wires and components stay firmly plugged in. Buttons and potentiometers are particularly tricky. With the Grove system, wiring a component to an Arduino is a snap, literally. Each component connects using a latched cable, ensuring that things stay put.

The closest competitor to Grove that I know of is the Phidgets systems. Seeed’s product is considerably less expensive and, in my experience, the quality is just as good. The Phidgets system, on the other hand, has top quality documentation, and can be directly interfaced with PCs without using an *duino go-between.

The advantages of GROVE over Seeed’s Electronic Bricks are, frankly, yet to be revealed. The GROVE components can use 2 pins, not just 1. In some cases this allows one component (they’re called twigs, actually) to include 2 separate items, such as 2 pushbuttons or 2 LEDs.

However, I think the real reason for the double-pin support is that the “twigs” contain edge connectors which would allow them to be directly connected to one another. You can see what I mean by looking at the photos of the various components on this page. Those markings on the edges of each twig’s circuit board (VCC, GND, D1, D2 etc.) are pads that align with those on other twig’s boards. I’m honestly not sure where that is leading, since the Grove was just released and the documentation is a work in progress, but I assume that the goal isn’t just to reduce cable usage. Stay tuned.

Why Netduino?

If you are a programmer, or learning to become one, then the advantages of Secret Lab’s Netduino over the Arduino are clear.

The Netduino’s IDE is Microsoft’s venerable Visual Studio 2010, which absolutely bristles with features when compared to the Arduino’s IDE. The free Visual Studio C# 2010 Express is well suited to Netduino development, since the extra features found in the other Visual Studio versions generally don’t apply to microcontroller programming.

And, even more importantly, the Netduino supports In Circuit Debugging, using nothing more than a USB cable. No more trial-and-error debugging and testing! As Joe Biden would say, that is a Big F’ing Deal!

If you aren’t a Windows user,the the disadvantages of the Netduino over the Arduino are also clear. ‘Nuff said. The Arduino also has a larger and more established hobbyist community, and is probably a better choice for somebody new to coding.

The Netduino has a close competitor, the FEZ Panda from GHI. GHI has actually been in the .Net Micro Framework game a lot longer than Secret Labs, and their firmware and documentation are therefore much, much better. When Netduino first came along this summer, it had a clear price advantage over GHI’s first Arduino-compatible board, the Domino. GHI quickly responded with a budget board called the Panda, Secret Labs responded with a feature-rich board, the Netduino Plus, and the fight is on!

But, back to the point of this article: Netduino’s adventures in the Grove.

Digital Components

The first things I tried were some basic digital components, and they worked pretty much seamlessly.

Grove shield and twigs
Grove shield and twigs
As shown in the photo, the basic GROVE shield fits on top of the Netduino. In keeping with the botanological theme, the shield is called a “stem”, actually, but since I can’t quite wrap my mind around a stem with twigs sticking out of it, I’ll call it a shield. (I’ll use the term twig for the component boards, though, since I don’t know what else to call them).

I tried connecting a button twig and LED twig (2 per twig in each case), and programmed them so that the buttons triggered the lights to glow and dim, using PWM. Initially, the button event (yes, the .Net Micro Framework fully supports raised events, just like Bill Gates invented :-) ) didn’t fire. A bit of trial and error revealed that the pulldown resistor had to be disabled, which is easily done in the source code.

The Netduino has relatively limited PWM support when compared to the Arduino (you can set the period, but frequency is fixed at 10Khz), but it worked fine with the LEDs. GHI’s firmware allows the frequency to be set, which might come in handy with the Grove’s piezo buzzer.

I’m using a serial LCD connected to digital pin 3 and the 5V and Ground pins. That setup works as expected. The Grove Starter Bundle includes a parallel LCD,  However, since the Netduino has 2 serial ports, neither of which are used to program or debug the board, and its support for parallel LCDs is less well established than the Arduino’s a Serial LCD might be the best way to go with the Netduino any way.

I’m using an elderly Matrix Orbital serial LCD, but the Sparkfun Serial LCD Backpack is a relatively inexpensive way to convert a parallel LCD to serial, including 4×20 LCDs. I’ve included support for it in the attached source code — just comment out the corresponding #define at the top of Program.cs. I took advantage of the .Net Compact Framework’s support for Interfaces to create compatible classes for the 2 serial LCD APIs.

Incidentally, both the analog and digital pins of the Netduino supply 5V, even though the processor uses 3.3V, which improves compatibility with Arduino shields. However, while the digital pins can accept 5V input, the analog pins (or, more accurately, the analog-to-digital converter) cannot go beyond 3.3V. That presents a problem when using 5V sensors and controls like the Grove’s.

Analog Components

For example, the potentiometer twig would, when turned all the way up, flood the ADC and cause all of the analog pins to read weird values. The proper solution for this is to connect the pot (and any other analog twig you use) through a circuit that converts the 5V signal to 3.3V or lower.

A voltage divider works well for something like a pot, since accuracy isn’t all that important. However, I tried a more accurate (though more cumbersome) approach using Sparkfun’s Logic Level Converter.

Sparkfun Logic Level Converter - Simplicity is for wimps!
Sparkfun Logic Level Converter - Simplicity is for wimps!
In theory, this converter takes 5V signals and steps them down to a 3.3V signal. In practice, things are a little more complicated. You need to connect the converter to the 5V line, and to the 3.3V line, and to ground (actually to ground twice, once on the 5V side and once on the 3.3V side). And, of course, you need to connect the signal from the twig to the converter, and the output from the converter to the analog in pin on the shield. All of which makes for a very messy setup, as you can see from the photo.

Lastly, to make things interesting, Sparkfun decided to step the signal down to 2.5V, not 3.3V as the product description suggests. The step up, on the other hand, always goes to 5V. Presumably, this decision was made to allow compatibility with microprocessors that use other voltage levels, but it means an extra step (pun intended) when what you want is a 3.3V signal.

Generally, you would correct the reading in code (by multiplying it by 3.3/2.5). Another option is to use the Netduino’s AREF connector to feed in the actual analog voltage you want to use. So, I connected it to AREF to a 2.5V signal from the Sparkfun converter, requiring yet another pair of wires – a connection to the 5V bus on the input side, and a connection from the output side to AREF.

Messy, but effective. The pot twig’s signal now moves smoothly from 0 to 1023 while staying well within the Netduino’s tolerance.

Next up, the temperature sensor twig. I tried the same connection approach as with the pot. Actually, since the Sparkfun Logic Level Converter only supports 2 input signals, I setup a second converter on the breadboard for this. While the converter correctly stepped the analog voltage down, the resulting temperature calculation was 34C (93F). This overshot the correct reading by about 12C.

I’m not 100% sure what the cause is, but I think the problem is that the temperature twig’s sensor uses a logarithmic scale, and simple fractional conversions don’t work in that world. (If anyone knows differently, please post a comment with an explanation).

A good solution, which has been described in the Netduino forums, is to use a temperature sensor that connects via I2C rather than an analog pin. However, I went with a hack. The analog signal returned by the Grove system’s temperature sensor (assuming you are using it for room temperature, and aren’t in Hell) is safely below the 3.3V threshold supported by the Netduino’s ADC. So, you can get away with connecting it directly to the shield, and the same temperature calculation formula used with the Arduino then works correctly with the Netduino.

Well, it almost works. The formula in the Grove system’s tutorial makes use of the Arduino’s (technically, AVR Libc‘s) log() function, which calculates a natural logarithm. The .Net Micro Framework doesn’t have a log() function. .Net’s Math function was pared down for size, and log() didn’t make the cut.

I first tried an alternative formula, taken from a forum posting for the older Electronic Brick temperature sensor. This came close, but was about 0.5C too high.

So, I next tried incorporating Elze Kool’s exMath library, which includes a log() function. After waiting, and waiting, for the logarithm calculation to complete, I broke into the debugger (again, in-circuit debugging is a Big Freaking Deal!!) and found it caught in an infinite loop.

while (partial >= double.Epsilon)
{
   if (x >= newBase)
   {
      fractional += partial;
      x = x / newBase;
   }
   partial *= 0.5F;
   x *= x;
}

The comparison of partial to epsilon kept returning true, even though the debugger showed the value of partial to be 0. In fact, entering partial > epsilon in Visual Studio’s Immediate window correctly returned false - only the compiled code was getting it wrong.

I can only assume that this is a bug in the compiler — epsilon is, after all, the mother of all edge cases. Adding (partial == 0) to the while loop’s condition fixed the problem. The temperature calculated by this formula is almost dead-on.

This leaves us with one last problem: the hack of using the “raw” signal from the 5V temperature sensor isn’t compatible with the earlier hack of feeding a 2.5 voltage into the AREF pin. Since the AREF problem is easily fixed in software, I changed the AREF signal to 5V, and doubled the reading from the potentiometer twig.

Watch it Spidey, it's Dr Octopus!
Watch it Spidey, it's Dr Octopus!
I really like Seeed Studio’s products, and the Grove looks like a solid, low-cost starter bundle for *duino experimentation and prototyping. I’m not entirely convinced of the advantages of the Grove over the older Electronic Brick system. The new connector cables are quite stiff, and since the twigs don’t have mounting holes they tend to stick up in the air like, um, branches of a tree. However, the twigs’ edge connectors are innovative, and it will be interesting to see what uses they are put to.

Source Code Notes

A few notes on the hardware setup used by my code:

  1. The button twig is connected to D7/D8, and the LED twig to D9/D10.
  2. The temperature sensor twig is connected to A2/A3,
  3. The potentiometer twig is connected to A0. As explained this connection is via the Sparkfun Logic Level Converter. The “D1″ (why D?) pin of the pot is actually connected by a jumper cable to the RI pin of the Converter, and the corresponding RO pin of the Converter to the A0 pin of the shield. If you want to skip the hassle of the analog voltage conversion, you can omit the potentiometer. It is used only to set the backlight level of the LCD. Just comment out the call to setLCDBrightness in Program.cs.
  4. The Serial LCD is connected to D3. You can run the code without a serial LCD by commenting out both #define statements at the top of Program.cs. You can see the temperature value in the Visual Studio Debug output window, and the buttons and LEDs will still work as normal.

The MathEx library is included in its entire, original form, except for the one change I made as explained above.  The change is commented with date 12/12/10.

And one final note: if you are thinking that a temperature display is a horrible waste of the Netduino’s capabilities, I agree entirely. This is just a starting point for a series of refinements that I’ll blog about in the future. I currently have various Arduinos scattered about my house, controlling plant lights, displaying Twitter feeds and weather forecasts, and logging temperature data. The code required a lot of trial-and-error debugging, and relies on the use of a Beagleboard as a go-between to the Internet. I’m looking forward to seeing how much more (or less?) the Netduino can achieve.

The source code for the temperature LCD program described in this article can be downloaded here.

Posted in .Net, Electronics, Programming | Tagged , , | Leave a comment

Beagleboard XM and Angstrom – Getting the Big Dog to Run At Full Speed

When I received a Beagleboard XM low-power computer a few weeks ago, I promptly installed Ubuntu Maverick 10.10 on it. I’ve found Ubuntu’s build for Beagleboard‘s ARM-based processors to be remarkably stable on my previous generation Beagleboard C4, so this seemed to be a safe bet. On the XM, Ubuntu seems remarkably solid, but remarkably slow.

I noticed some early signs of trouble on the initial boot: an ominous “”Kernel is not ready for 1Ghz limiting to 800Mhz” message, and a Windows-like leisurely boot time. My first impressions were confirmed by running the HardInfo benchmarks. Every one of them except BogoMIPs was lower on the XM than the C4 by 10-20%. BogoMIPs was a disappointment too, at just over 600. On the Angstrom demo build that is bundled with the XM, the BogoMIPs benchmark was close to 1000.

I tried installing Ubuntu 10.04 on the XM, but the HardInfo benchmarks were just as slow. A Google search confirms that others have run into the same problem with the XM, and as of now (early December 2010) there doesn’t seem to be a workaround. It would appear that Ubuntu isn’t yet optimized for the XM’s faster DM3730 processor.

Performance aside, Ubuntu 10.10 ran well on the XM. But it broke my heart to see the old C4 running rings around the new pup. So goodbye Ubuntu, hello Ångström, at least for now.

Angstrom (sorry, I’m not going to type in those umlauts again) is a Linux distribution that seems to be designed for speed and not comfort. As the name suggests, its roots are as a relatively tiny OS that can run on portable devices that are slower and more memory constrained than the XM, such as Pandora handheld gaming console and the Sharp Zaurus. The XM lets Angstrom stretch its legs, and it responds with blazing speed!

Unfortunately, it’s also much trickier than Ubuntu to configure. Angstrom’s default installation doesn’t include a lot of the packages that are included in every Ubuntu install (and most other distros too). Many of the packages available for Angstrom have generic or misleading names, so it can be hard to track down the one you want. There isn’t a lot of end user documentation for Angstrom, and the user base is much smaller than Ubuntu’s, so Googling won’t always find you the answer either.

Here’s a step-by-step description of how I got Angstrom up and running. To take optimal advantage of Beagleboard’s ultra low power design, my goal was a “headless” application server (i.e. no monitor and therefore no GUI).

1. Angstrom’s has an online image builder named Narcissus. While its pretty easy to generate an image, it isn’t obvious how to boot the image on your Beagleboard. The best instructions that I found were in this article on Trey Weaver’s blog. The only thing that I did differently than what it describes is the command to extract files from the Angstrom image file. Since Narcissus is (currently) using a default image format of gzip (i.e. the downloaded image file ends with .gz), the command to extract the boot files has slightly different parms:

tar --wildcards -xzvf [your-download-file] ./boot/*

Similarly, the command to extract the root filesystem is:

sudo tar -xvz -C /media/Angstrom -f [your-download-file]

2. The Beagleboard XM has a built-in serial port, and the default Angstrom build supports it, so you can initially login by connecting a PC to the serial port (a USB-to-serial adapter works fine) and running a terminal program. (Settings are 115200 bps, N-8-1 no flow control.) On Windows, WinPutty works well – on a Linux PC, minicom is, I think, the best bet.

However, you may run into an irritating problem with the display and keyboard input being garbled. I’ve had that problem when I’ve installed Angstrom on both my Beagleboards, and I’m not aware of any workaround.

Fortunately, the garbled terminal only occurs when connecting through the Beagleboard’s serial port, so you’ll want to switch to some other type of connection as soon as possible.

[Update: It turns out that the garbling occurred only when using a PCI Serial Port card.  I switched to a USB-Serial converter (which uses the common Prolific chipset), and WinPutty works flawlessly.]

Since I wasn’t interested in hooking the Beagleboard up to a monitor and keyboard, I switched to using SSH to bring up a command line connection from another PC. Here is another area where Angstrom is easier to use than Ubuntu – Angstrom activates the Ethernet port by default, and includes an SSH server named Dropbear by default, so you may be able to connect by SSH on your very first boot, bypassing the serial port entirely.

In any case, the default user-ID is root, and it has no password.

3. To set a password, use the passwd command. I’d suggest you do so even if you aren’t at all concerned about security, since some utilities will expect a password.

passwd

Angstrom makes some suggestions about the length and types of characters in your password, but it doesn’t enforce them.

4. You might want to add a regular user account too. The command for that is useradd

useradd youruserid

You need to assign a password to this user-ID before it can be used to login:

passwd youruserid

5. If the only Linux distro you’ve used before is Ubuntu, you might be puzzled about the command used to run root commands when logged in as another user. “sudo” isn’t installed in Angstrom by default. Configuring sudo in Angstrom is surprisingly difficult, so you might want to stick to running as root, or get used to switching from a user account to root by entering:

su -

Enter the root password when prompted, not the user password à la Ubuntu.

6. The command to install packages in opkg (which, like pretty much every other command I list, must be run as root). Entering “opkg –help” will tell you all you need to know, but here are some commonly used commands:
To bring your installed packages up-to-date

opkg update
opkg upgrade

To install a package:

opkg install packagename

To remove a package:

opkg remove packagename

If the removal of a package fails due to a long list of dependent packages (and you are sure you don’t want to keep any of them!):

opkg remove packagename --force-removal-of-dependent-packages

Unlike Ubuntu’s aptitude, opkg doesn’t automagically remove conflicting packages for you. So, when an install fails due to conflicts, you need to either remove them manually, or (as a last resort) grit your teeth, cross your fingers, and enter:

opkg install packagename --force-overwrite

To list the files in an installed package:

opkg files packagename

If you want to install something and are trying to figure out its packagename, you can get a list of packages whose name includes a search term:

opkg list | grep "whatever"

To get a description of a package:

opkg info packagename

A better way of finding a package is to use the package browser on Angstrom’s web site. Better, but frankly not great, since Angstrom tends to use different package names from other distros, and the package descriptions are usually vague.

To get a list of packages already installed:

opkg list_installed

7. As with most Linux distros, the default text editor is “vi”. Lots of alternatives are available, but the one I use is emacs:

opkg install emacs

8. You’ll need an editor to change the hostname to something other than the default of “beagleboard”. The 2 files you’ll need to edit are /etc/hostname and /etc/hosts. For example, to change your hostname to bigdog, the contents should be:

youruserid@bigdog:/etc$ cat hostname
bigdog
youruserid@bigdog:/etc$ cat hosts
127.0.0.1       localhost.localdomain           localhost
127.0.0.1       bigdog

9. [Update: This step may no longer be necessary - I noticed that the Angstrom 11.03 Narcissus image uses bash as the default user shell] By now, you’ve probably noticed that the shell is missing some features that you’re used to, like color coding. Strangely, one of the things that Angstrom doesn’t install by default is the nearly-universal bash shell. To install it:

opkg install bash

After installing it, you need to make it the default for your user-ID (though you can run it temporarily by just entering “bash”). The command for doing this is

chsh /bin/bash

Since you have to be root to do this, if you want to change a non-root user to the bash shell, enter

chsh /bin/bash youruserid

10. You’ll probably want to make some folders on the Beagleboard accessible over the network. In my case, I primarily use Windows PCs, so I needed to install Samba. Since configuring samba can be tedious using its command line tools, I also installed the browser-based samba configuration tool, SWAT:

opkg install xinetd
opkg install samba swat

xinetd is the standard way of invoking SWAT (and other port-based utilities). You’ll probably have to manually edit the SWAT settings in the xinetd configuration file — the following settings will allow access from any PC:

The updated /etc/xinetd.conf file should look like this:

# Simple configuration file for xinetd
#
# Some defaults, and include /etc/xinetd.d/

defaults
{

}

service swat
{
   disable = no
   port    = 901
   socket_type = stream
   wait    = no
#       only_from = localhost
   user = root
   server = /usr/sbin/swat
   log_on_failure += USERID
}
includedir /etc/xinetd.d

After changing the xinetd.conf file, you need to restart xinetd:

/etc/init.d/xinetd restart

You should now be able to open the following address in a browser on another PC: http:<ipaddress>901

Login as root at the prompt.

By default, the home folders are shared – if you created a non-root user-ID then the defaults may be fine. However, you’ll need to go to the SWAT password page and enter the user-ID and password that you login to Windows. Click the “Add New User” button when you’re done — it may be old to you, but it’s new to SWAT:

I’d also suggest you go to the Global page and change the workgroup to whatever your Windows PC uses, and enter something meaningful in the “netbios name” field.

After saving your changes you don’t need to restart Samba – it applies your changes automatically.

If you go to a Windows PC, you should now be able to see your Beagleboard in your network, with some shares underneath. If you are prompted for a user-ID and password, use the one you entered into SWAT’s password page.

11. One of the things I want to use my XM for is as a Tweet-A-Watt server. It’s a good match: software to monitor electricity usage running on an ultra low-power platform. This requires adding a couple of things to Angstrom: Python, and access to an XBee connected to a USB drive. Both of these turned out to be somewhat trickier than they are on Ubuntu.

12. The main package for Python is named python-core. However, Angstrom’s idea of a “core” lacks a lot of python libraries that are standard on pretty much any other Python installation. After playing hide-and-seek with various packages, I gave in and installed the full set of libraries, packaged as python-modules. I also included pyserial for access to the XBee

opkg install python python-modules python-pyserial

At this point, you can install the Python setuptools package, which includes the easy_install command. easy_install allows you to install many other python libraries that don’t (seem to) have Angstrom packages. In my case, I needed Tweepy, which can be used to provide Twitter access to tweetawatt (I wrote an earlier post about that):

opkg install python-setuptools
easy_install tweepy

13. Unlike Ubuntu for Beagleboard, Angstrom doesn’t include support for serial port USB devices in the kernel. If you plug in a USB serial I/O device like an XBee or Arduino, it will be recognized, but no port (e.g. /dev/ttyUSB0) will be assigned to it.

(On the other hand, USB storage devices are supported by default in the Angstrom kernel. And, unlike Ubuntu on the Beagleboard, some drives are automatically mounted, so after plugging in a USB drive you should be able to immediately access it under /media/sda. Kewl.)

To install support for USB serial I/O devices:

opkg install kernel-module-ftdi-sio

You can temporarily load the kernel module to confirm that it works by entering

/opt/bin/depmod -a
/opt/bin/modprobe ftdi-sio

If you plug in an XBee, it should now be assigned a port. (Probably /dev/ttyUSB0, but you can enter dmesg and look at the bottom of the output to confirm this).

To automatically load the kernel module at bootup:

echo usbserial > /etc/modutils/usbserial
echo ftdi_sio > /etc/modutils/ftdi_sio
update-modules

this will update the /etc/modules file.


Phew. In the unlikely even that you read this far before actually trying to get Angstrom working, you might be wondering if it is worth all the effort. The process of setting up the same configuration in Ubuntu is simpler and much better documented.

While its somewhat rewarding to use long-neglected command line skills (and learn a few new ones), I’d frankly have to admit that I’ll probably change back to Ubuntu when (if?) its performance on the XM is comparable to Angstrom’s. There are some packages that don’t seem to exist yet in Angstrom, and I don’t have the time/skillz to port them over. (Most notably, I would love to have Python’s Matlib package, and a browser administration tool along the lines of WebMin. Oddly, the standard Linux help command, man, is currently not available for the Beagleboard).

One other concern: Ubuntu is rock solid on my older Beagleboard, and Angstrom’s stability is, for me, unproven. However, Tweet-a-Watt is now up and running on my XM – if it’s still posting updates to www.twitter.com/gigamegabot when you read this, then maybe tiny Angstrom and the Big Dog really are a match.

Posted in Gadgets | Tagged , | 14 Comments

Surviving the Twitter OAuthcalypse with Tweet-A-Watt

When Twitter recently made the OAuth API mandatory for 3rd party applications, the resulting OAuthcalypse silenced the tweets of a lot of code.

Among those affected was Adafruit‘s innovative Tweet-A-Watt system, which interfaces with a low cost Kill-A-Watt monitor to track your electrical usage. After its original release the Tweet-A-Watt’s software support was expanded to include more sophisticated platforms, such as Google’s App Engine and GoogleMeter, but I found I missed the original Twitter functionality. What’s Tweet-A-Watt without the tweets?

Unfortunately, supporting OAuth isn’t as simple as updating a python script or two. OAuth is a far more sophisticated system than the basic user-ID/password authentication that Twitter used to support. (Though, as we learned this week, it seems that Twitter bolted the front door and left a back door open. Oops!)

A look through the “Beginner’s Guide to OAuth” might make you wonder how far down from a beginner you rank. A handy resource is this excellent blog post by Jeff Miller, which tells you all that you need to know to make Python software talk to OAuth.

The bad news is that the latest release of python-twitter API used by Tweet-A-Watt’s wattcher.py doesn’t yet support OAuth. That will change – eventually. OAuth support has been added in their development builds, and perhaps by the time you read this a new release might include it. You can check their download page here.

The other bad news is that, even with an API that supports OAuth, you’re going to have to get some authentication “tokens” of your own from Twitter – these replace your Twitter user-ID and password in the wattcher script. The process is relatively straightforward, and fully explained in Steps 2 and 3 of Jeff Miller’s post, but requires the somewhat intimidating process of registering “your application” with Twitter. Fortunately, Twitter is not at all choosy about who and what qualifies as an application, so don’t be shy about signing up.

If you followed the steps in Jeff Miller’s post, then you’ve become somewhat acquainted with the Tweepy API, an alternative to python-twitter which has supported OAuth for some time. Tweepy has excellent documentation and is quite easy to understand, so I decided that the easiest path would be to modify wattcher.py to use Tweepy instead of python-twitter. Here is what’s required:

1. Install Tweepy on the box which will run wattcher. I use a low-powered Linux “plug computer” (Tonido Plug) for this, since running a full desktop PC 24/7 to monitor your electrical usage is both ironic and dumb. TonidoPlug runs Ubuntu Linux, and installing Tweepy in Ubuntu is a snap:

$ 	aptitude install python-setuptools
$ 	easy_install tweepy

2. The first change to wattcher.py is to replace the python-twitter API with Tweepy:

#import twitter
import tweepy

3. The Twitter username and password constants are no longer needed – they are replaced with the 4 long hash strings that you’ll get from following the steps in Jeff Miller’s post:

# twitterusername = "username"
# twitterpassword = "password"
CONSUMER_KEY = 'your_consumer_key'
CONSUMER_SECRET = 'your_consumer_secret'
ACCESS_KEY = 'your_access_key'
ACCESS_SECRET = 'your_access_secret'

4. Your code needs to “login” to Twitter by calling a few Tweepy API functions. I put this code in the mainline of wattcher.py, right below the constants added in step 3. This means that wattcher logs in just once – if it somehow gets logged out or needs to re-authenticate, it is out of luck. It might be safer to login every time you tweet, as wattcher previously did in its TwitterIt function. However, I haven’t run into any problems with the approach I took so far (it has been running about 10 days with only the initial login).

auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_KEY, ACCESS_SECRET)
api = tweepy.API(auth)

5. Last is the code which sends the tweet, in the TwitterIt function. As explained in step 4, I no longer do the login here, so all that is required is a call to Tweepy’s update_status function:

 def TwitterIt(u, p, message):
    #api = twitter.Api(username=u, password=p)
    #print u, p
    try:
        #status = api.PostUpdate(message)
        api.update_status(message)
        #print "%s just posted: %s" % (status.user.name, status.text)
        print "twittered: ", message

If everything went well, your Tweet-A-Watt has got its tweet back. If you want some instant gratification (or instant disappointment if everything didn’t go well), you can add a test tweet to the startup code, and run wattcher in its debug mode (i.e. “python wattcher.py -d”)

if (sys.argv and len(sys.argv) > 1):
    if sys.argv[1] == "-d":
        DEBUG = True
        TwitterIt("", "", "Up and running with OAuth...")
Posted in Electronics, Programming | Tagged , , | Leave a comment

The iPhone and the clipboard: an uneasy alliance

I recently spent some time as a guest of the Ontario judicial system. No, not as a prisoner, something much more unpleasant: jury duty.

Recommended Reading
Recommended Reading

This is the second time in five years that I’ve “done my duty”, and in both instances the week consisted of 99% waiting around and 1% being rejected out-of-hand by defense counsel. I am, of course, crushed by the repeated rejections, but I managed to get some benefit from the other 99% of my week by reading a software development e-book.

When I went through this five years ago, I did my reading on a Dell Axim running Windows Mobile 5.0. You laugh, but let me tell you that one thing it did really well was copy and paste. You selected the text with the stylus (stop laughing), used a button to toggle to Word and pasted the text in. A task so simple even Windows Mobile could handle it.

Five years later, I’m using an iPhone. In technical terms, the iPhone is a Porsche and the Axim a Yugo. But when it comes to copy and paste, the Porsche is a freaking nightmare to drive.

The problem isn’t so much with the iPhone as with its apps. Very few of e-readers allow you to copy text into the clipboard. It’s bizarre! Do they think we don’t want that feature (we’re too dumb), or do they not trust us to use it responsibly (we’re too dishonest)? Either way, it’s insulting.

However, it’s not all doom and gloom. There are two very, very good apps that do support the clipboard, and a growing number of e-books that are compatible with them.

Stanza - march to a different drummer
Stanza - march to a different drummer

Stanza.   Stanza supports a long list of e-book formats, but unfortunately you are unlikely to encounter most of them when buying e-books. The 3 exceptions are:

  1. eReader – This is a “secure” (piracy-protected) format that is used by Fictionwise and Books on Board for most of their books, and by Barnes and Noble for many of them.
  2. ePub – The ePub format supported by Stanza is not “secure”, which unfortunately makes it harder to find (we’re dishonest, remember).  However, if you’re a computer geek then you’re in luck: O’Reilly and Microsoft Press offer ePub editions of almost all of their publications.
  3. PDF – Support for PDF format documents was added to Stanza last week.  Unfortunately, its copy support is pretty inconsistent, often copying gibberish to the clipboard.
GoodReader - not bad!
GoodReader - not bad!

GoodReader. GoodReader supports just one format, PDF, but that’s still the most commonly used format for technical publications. I’ve raved about this app before, and I’ve grown more and more impressed with it over time. They added clipboard support a couple of releases ago, and while its not the most convenient implementation (you can only copy a full page of text, not selected text), it is fast and reliable, just like the rest of GoodReader. I’ve yet to find a PDF that GoodReader couldn’t handle — amazingly, it loads 100 meg monsters faster than my desktop PC.

Once you’ve filled the iPhone’s clipboard, your options for pasting text are much better.

You might find that the built-in Notes app is good enough for this task. I use Pastebot – a lot of people rave about its advanced formatting features and automatic synchronization (Mac only), but I like the fact that it automatically saves the clipboard contents when you open the app. If you need to format the text after you paste it (change fonts, italics, etc.), you should consider Documents To Go. I’m really impressed with how quickly it manages to bring you back to your last spot in a document without multitasking — the app is ready to paste a couple of seconds after you click the launcher (on an iPhone 3GS).

The new iOS 4 will make the copy-and-paste process somewhat smoother with its support for multitasking and fast app switching: it might even be able to compete with Windows Mobile 5!

One last piece of good news: O’Reilly (and associated publisher, Microsoft Press) continue to make most of their books available as very inexpensive iPhone apps, generally $5 or $6. That’s the full book, pictures and all.

If you’ve bought one of these apps, you were probably disappointed to find that clipboard support was disabled. This seems to be related to the underlying version of Stanza (which their apps bundle with the e-book). For some reason Stanza quietly disabled clipboard support in an update to their app last year, then quietly re-enabled it in the next update. Hmm.

While O’Reilly continues to use the clipboard-disabled version of Stanza, it is easy enough to do the upgrade yourself. Just follow the process described in my earlier post to extract the ePub from the O’Reilly / Microsoft Press book app, then import the ePub into Stanza using their desktop app or a URL link.

I’m pretty sure that O’Reilly is OK with you doing so — as mentioned in my earlier post, it was O’Reilly themselves who originally documented the process. If not, maybe I’ll get yet another opportunity to read e-books as a guest of the Ontario judicial system!

Posted in Gadgets, Software Tools | Tagged , | Leave a comment

Book Review: SQL Cookbook by Anthony Molinaro

Like many developers, I’ve bounced between various relational databases on my coding projects, primarly DB2, SQL Server and Oracle.  If someone asked me if the SQL I write is compatible with all three, I’d sheepishly admit that most of it is.

Sheepishly, because generic, cross-platform SQL tends to be inefficient SQL.  You can get away with it when working with modest amounts of data and users, but it’s still lazy, shoddy work.

Although not obvious from its title, SQL Cookbook is actually a great way to hone your SQL-writing skills, taking advantage of the powerful, vendor-specific features available in each major RDBMS brand.  (The specific types of RDBMS covered by the book are DB2, SQL Server, Oracle, MySQL and PostgreSQL.)

The book’s format is to present a problem, such as creating a pivot table or generating a delimited list, then provide one or two solutions to the problem for each type of RDBMS.  Sometimes the same solution works for multiple RDBMS, but in most cases a particular RDBMS has proprietary functions that result in a shorter or faster piece of SQL.

A notable example are “window functions” (also known as “analytic functions” in Oracle).   They are a relatively new technology, having become an ISO SQL standard in 2003, but all recent versions of the major RDBMS products implement them to some extent.  An example from Oracle is the following, which returns each employee’s department and the # of employees in that department.

select ename,
deptno,
count(*) over (partition by deptno) as cnt
from emp

The generic/lazy way to implement that is with a subquery or joined query that returns the number of employees in each department, such as:

select e.ename,
e.deptno,
(select * from emp where deptno = e.deptno) as cnt
from
(select ename, deptno from emp)

The latter approach works, but it’s more complex and less efficient than using a window function.

If you’ve been writing SQL for more than 10 years, you learned to do it the long way and probably haven’t been forced to change your ways since.  To learn about window functions, Appendix A of SQL Cookbook is a great place to start.

Another example is Oracle’s ability to return an object from a subquery.  I had always thought it was a cardinal rule (and major pain-in-the-ass) of SQL: subqueries can return only 1 field.  I couldn’t tell you how many times I’ve pasted multiple copies of the same subquery into a SQL statement because I needed to return a few different fields from it.  I knew that it would hurt performance, but what option did I have?

It turns out that, when using Oracle, an option is to return those fields in an object.  For example, the following query returns a couple of fields from the department table for each employee:

select
x.deptno,
x.ename,
x.multival.val1 dname,
x.multival.val2 loc,
x.multival.val3 today
from
   (select e.deptno, e.ename, e.sal,
     (select generic_obj(d.dname,d.loc,sysdate+1)
     from dept d
     where e.deptno=d.deptno) multival
   from emp e
   ) x

I also liked the format of SQL Cookbook.  As the title suggests, one way of using the book is as a reference.  If stuck on a SQL coding problem, you can run through the table of contents, looking for a problem description that is similar to your own.

However, I found it worthwhile to read through the book chapter by chapter.  Every piece of SQL is accompanied by a detailed but easy-to-follow narrative describing how and why it works.  By reading these sections you’re sure to add a lot of new tools to your toolbox, even if you don’t need to solve the specific problem covered by that “recipe”.

Also, since the book separates the solutions by RDBMS, you’ll probably find that you can skip at least 50% of the book (for now),  focusing only on the one or two RDBMS systems that you are currently working with.  It’s refreshing to come across a software development book that can realistically be read in a week.

With the emergence of alternative ways of interfacing with databases, like LINQ, CouchDB and the various technologies gathering under the “NoSQL” banner, the day might come when crafting SQL statements is a dying art.  A parallel might be drawn between SQL today and assembly language in the past — increasingly less important as higher level languages and the underlying hardware grow stronger.  However, if this happens the remaining “artists” who’ve mastered SQL will be much in demand.  This book is a good step towards raising your SQL to a higher level — from Betty Crocker to Julia Child, if you will.

Posted in Programming | Leave a comment

An Audible Improvement in iTunes 9.1.1

I was pleasantly surprised to find that the latest iTunes patch, version 9.1.1, fixed a problem with Audible audiobooks on my iPhone and iPod Touch.

Previously, whenever I added new content from Audible I would lose my current place in whatever audiobook I was listening to. The next time I listened to the audiobook it would start playing at the beginning of the book. I know that it seems like a nit, but since I subscribe to a couple of daily Audible “podcasts” (Charlie Rose FTW, w00t!), it was a nit I encountered almost every day.

Nice to get some noticeable value for my 102 megabyte “patch”. Considering that the bug was introduced at about the time OS 3 was released, and OS 4 looms just around the corner, I wonder if I’ve seen the last of this one. Audible announced (well, twittered) a couple of months ago that they were working on supporting wireless downloads on the iPhone — will we finally have an app for that?

Posted in Gadgets | Tagged | Leave a comment