Cosm Gadget: Kissing the Pink Goodbye

One of the data logging services I use, Pachube, was rechristened as Cosm many months ago: a name that’s decidedly easier to pronounce, if only slightly easier to comprehend.  The new name and website design was making my Pachube Google Gadget look increasingly out of date, so I felt it’s well past time to redesign the gadget.

In addition to incorporating the new Cosm name and API, I’ve added a number of enhancements:

  1. Multiple datastreams per graph.
  2. Display datastreams with 2 different scales: left-axis and right-axis (e.g. temperature and humidity in the same graph)
  3. Mouse-over popups displaying the exact time and value of a datapoint
  4. Last, but not least, graph colors other than hot pink!

The new gadget can be installed from the Google Gadget Directory.  The old Pachube gadget will remain available, should you prefer it.

Settings Dialog

The gadget’s Settings dialog is shown below.

As a quickstart, you can just fill in the Feed ID field — everything else will work with default settings.

By default, the gadget will display 2 datastreams per chart, one plotted on the left axis and one on the right, for up to 3 charts. The default time frame is 1 day.

The default configuration is unlikely to be optimal for you — at a minimum, you’ll probably want to change to your local timezone and increase the gadget size to display all of the charts.

Here’s the full list of settings:

Feed ID – The numeric ID of a Cosm feed.  The feed needs to be public, even if it’s your own feed, since the API key used by the Gadget can only access public feeds.

Incidentally, Cosm’s current layout makes the feed ID a little hard to find.  It’s the last part of the feed’s URL, as shown in the screenshot below.

Chart 1 Datastreams – You can add as many datastreams as you like to each chart.  Datastreams entered in this particular field will be plotted using the left axis of the 1st chart.

Just type in the datastream ID, then click the Add button.  Each datastream you add will be listed under the field, as shown below.  You can click on the X button to remove the datastream from the chart.

Chart 1 Right Axis – Datastreams entered here will be plotted using the right axis of the 1st chart.  This field, and the other “right axis” fields are optional – if left blank, all datastreams in the chart will be plotted on the left axis only, freeing up a bit more space for the chart contents.

Adding a datastream to the right axis makes sense only if it is a completely different kind of measurement from the datastreams on the left axis (e.g. humidity on the right axis and temperature on the left axis), or if some datastreams have a significantly different range of values than others (e.g. Apple stock price vs. Microsoft stock price).

The screenshots below show the difference between plotting all of the data on the left axis, as opposed to making use of the right axis.

Personally, I prefer plotting different types of data in completely different charts.   If you need more than 3 charts, you can use multiple instances of the gadget.

Chart 2 and Chart 3 fields – The same as the above 2 fields, but for the second and third charts.  These are optional: you can have just 1 or 2 charts if you like.

Time Span and Time Span Units – The time range to be displayed in the charts.  Units are hours, days, weeks and months, and the Time Span can be any numeric value (e.g. 12 hours, or 10 days, or 3 months).

Note that Cosm currently “throttles” the API which retrieve datapoints to 100 values at a time: as a result, the datapoints will be evenly spaced over time so that only a maximum of 100 are used in the graph.  The longer the time interval, the more widely spaced the datapoints will be.

Gadget Height – The default height of 240 fits only a single chart: you have to scroll to see the others.  You can change this setting to make multiple charts visible.  Note that the height of each chart itself is fixed in the normal gadget view, but you can get a larger chart by maximizing the gadget.

Time zone – Cosm provides a large and somewhat eclectic list of time zones to select from: this affects the times that appear on the X axis of the chart.

Other features

Mouse-over Tooltip – Positioning the mouse over any part of a graph will display a tooltip displaying the time and value of the datapoint.  The time uses the time zone selected in the Settings dialog

Maximized View – Any gadget in iGoogle can be maximized using the dropdown menu.  This displays a much more conveniently sized version of the charts.

Browser Compatibility

The gadget works (for me) in Chrome, Firefox and Internet Explorer.  Being based on Google technology, it renders the graphs fastest in Chrome.  Firefox is almost as fast, and the appearance of the chart is pretty much identical to Chrome.

Troubleshooting

Error Codes: Cosm returns an error code when it is unable to return the data point values.  The most common error codes are:

  • 404 – The feed ID doesn’t exist, or isn’t public.  The gadget doesn’t support private feeds, even your own.
  • 405 – This is an intermittent error that occurs when Cosm isn’t able to retrieve your datapoints fast enough: basically a timeout.  I don’t think longer timespans will cause this error (i.e. a month of results takes about as long to retrieve as a day of results), but a large # of datastreams in a gadget will make this error more likely.  If you find it happens too often, try splitting the datastreams between multiple instances of the gadget: you can add as many copies of the gadget as you like to iGoogle.

Data Not Updating – The gadget will automatically refresh from time to time — using some inscrutable timetable controlled by Google — but you can manually refresh it using the browser refresh button.  However, Google automatically caches the data returned by the gadget, and the cache time for this gadget is set for 5 minutes.  This means that refreshing the browser more frequently than 5 minutes probably won’t display new results.

Wrapping Up

I wasn’t certain that Cosm would be fast enough to handle this new design of the gadget, where it needs to return 100s of individual datapoints rather than a prerendered graph, but I’m satisfied with the performance.  It’s proven to be similar in response time to the Nimbits gadget: slightly slower than the prerendered graph, but generally just a few seconds to display a chart.

Hopefully both Cosm and Nimbits will continue to thrive — I’ll certainly continue to use both.

The future of Google Gadgets is, on the other hand, rather murky.  Google hasn’t relented on its planned November 1 2013 shutdown of iGoogle, where the vast majority of users run their gadgets.  There doesn’t appear to be any plan to phase out the Gadget API, but there’s no clear option for hosting Google Gadgets after November 1.

I expect Google will step up with a solution, perhaps as part of their slick Google Now interface, but their indifference to users concerns thus far is evil disappointing.

If you have any suggestions or notice any bugs in my Cosm or Nimbits gadget, please leave a comment below or send me an e-mail.

Posted in Electronics, Software Tools | Tagged , , | 1 Comment

Beaglebone Coding 101: I2C

Updated 12/20/12 – Corrected instructions for installing python smbus on Angstrom.
Updated 1/6/13 – Another correction to the instructions for installing python smbus on Angstrom 

In previous articles in the Beaglebone 101 series, I covered digital GPIOanalog and serial pins, PWM and SPI.

One important coding interface remains: I2C.  I’ll cover it today, including I2C input and output, and level conversion, culminating in the Weatherbone, a basic LCD-based thermometer.

I2C Overview

I2C’s main advantage is simplicity.  It requires just 4 connections, 2 fewer than SPI: voltage, ground, clock and data.

The clock pin works pretty much the same way as with SPI: it is driven by the Beaglebone to control the timing of the data transfer, and works fine at 3.3V even when interfacing to a 5V device.  As with SPI, one option for controlling the state of the I2 clock pin is “bit banging”: using a plain old GPIO pin as the clock.  However, since the Beaglebone supports much higher and more reliable clock speeds through specialized hardware, we’ll make use of that.

Unlike SPI, I2C uses just 1 data pin, which transfers data in both directions.  This is great for simplicity and cost, not so great for speed.  As a result, I2C devices is most appropriate for devices that don’t send a lot of data, like sensors.

The fact that data travels in both directions on the same pin is an important consideration when interfacing with a 5V device: you must always use a level shifter on the data pin.  This is necessary even if using an output device like an LCD — there is no such thing as a “write only” connection when dealing with I2C.  I’ll give an example of level shifting in the I2C LCD circuit.

Like SPI, I2C supports multiple devices on the same I2C bus.  While SPI uses a latch pin to select the device, I2C handles the addressing with software.  This, again, makes wiring the circuit easier.

I2C and the Beaglebone

Most Beaglebone distros allow access to 2 I2C buses: /dev/i2c-1 and /dev/i2c-3.  For reasons that I don’t fully understand, /dev/i2c-1 doesn’t play nice with user applications, so we’ll stick with /dev/i2c-3.

The 2 pins used to interface with I2C bus 3 are pins 19 (clock) and 20 (data) on port 9.  Both pins default to I2C mode in Ubuntu and Angstrom, so no pinmuxing is required.

You may have seen references to the importance of pull-up resistors when using I2C.  You can ignore this when using the Beaglebone, since pull-up resistors are enabled by default on pins 19 and 20 in both Ubuntu and Angstrom.

When accessing SPI devices from Python, I used a C language extension (spimodule.c) as a go-between.  With I2C this isn’t necessary, since a Python-friendly interface exists: SMBus.   Technically, SMBus and I2C aren’t the same thing, but they both support the high-level read and write actions used by most applications.

Installing SMBus on Ubuntu is easy:

sudo apt-get install python-smbus

Updated 12/20/12 – I finally had a chance to try installing smbus on Angstrom.  It wasn’t as straightforward as the original text of this article suggested: here are the gory details..

Installing SMBus on Angstrom is somewhat more difficult.   There is no Angstrom package for python-smbus, so you’ll need to download it from source:

1/6/13: Added some corrections to the following instructions, as provided by the Hipstercircuit blog.  See Hipster’s instructions on getting an I2C PWM controller to work on Beaglebone Angstrom,  a good way of extending the Bone’s relatively limited PWM capabilities.

opkg install kernel-headers python-distutils
wget http://dl.lm-sensors.org/i2c-tools/releases/i2c-tools-3.1.0.tar.bz2
tar -xjf i2c-tools-3.1.0.tar.bz2
cd i2c-tools-3.1.0/py-smbus

Edit setup.py to add a new “include_dirs” line, as  highlighted in bold below:

...

 url="http://lm-sensors.org/",
 include_dirs=["../include", "/usr/include"],
 ext_modules=[Extension("smbus", ["smbusmodule.c"])])

Without that change, python setuptools won’t be able to find the “limits.h” file.

Run the standard python build command:

python setup.py install

You’ll get a new error “this linker was not configured to use sysroots”.  I don’t know how to properly fix this error, but a workaround is to run the same command without the sysroots parameter. The error message will include the full command, so just copy it to the clipboard and remove the part that begins with “–sysroot”.  On my PC, the resulting command was:

arm-angstrom-linux-gnueabi-gcc -march=armv7-a -fno-tree-vectorize
 -mthumb-interwork -mfloat-abi=softfp -mfpu=neon -mtune=cortex-a8
 -D__SOFTFP__  -shared -Wl,-O1 -Wl,--hash-style=gnu 
-Wl,--as-needed 
-I../include build/temp.linux-armv7l-2.7/smbusmodule.o 
-L/usr/lib -lpython2.7 -o build/lib.linux-armv7l-2.7/smbus.so

Lastly, run the python build command again to finish the process:

python setup.py install

The MCP23008 Port Expander

The MCP23008 is the I2C equivalent of the 74HC595 chip that I covered in my SPI article.  As mentioned then, port expanders aren’t terribly useful with the GPIO-rich Beaglebone, but when combined with buttons and LEDs the MCP23008 provides a quick and easy way of testing I2C circuits and code.

Compared to the Beaglebone’s GPIO pins, the MCP23008 does offer a few advantages:

1. It can interface directly with 5V devices.  Note that this requires powering the MCP23008 with 5V, so you still need to use a level shifter to connect the MCP23008′s data pin to the Beaglebone’s data pin.  But converting one pin is, of course, a lot easier than converting each GPIO pin separately.

2. It can source and sink a higher amount of current than the Beaglebone: 20 ma, compared to just 6 ma source and 8 ma sink for the Beaglebone.

3. It has registers to support interrupts.  This means that code doesn’t have to rely on polling to detect input — sort of.  I’ll show an example of that below.

To demonstrate the input and output interface to the MCP23008, I’ll use the circuit shown below:

On the Beaglebone, pin 19 on Port 9 is I2C Clock, and pin 20 is data.  Since I’m powering the MCP23008 at 3.3V, these can be directly connected to the chip’s pins 1 and 2.

The 3 address pins of the MCP23008 are all connected to ground.  These pins determine the address that is specified in our code to select the MCP23008, and are important when using multiple MCP23008s: since there is no latch pin, the address determines which chip gets the data.  With all 3 address pins connected to ground, we’ll use the MCP23008′s base address of 0×20.

Pins 10 (GP0) and 16 (GP6) are used as input pins, connected to a pushbutton that is connected to ground.  The MCP23008 has an optional pullup resistor, which we’ll enable on this pins.  So, the pin is normally high, but when the button is pressed it will change to low.

Pins 11 (GP1) and 17 (GP7) are connected to LEDs through resistors (anything in the range of 330 to 680 is fine).

The idea is that the buttons will toggle the state of the LEDs.  Yes, that does sound suspiciously like the same thing that we were doing back in the first Beaglebone Coding 101 article, but 2 LEDs makes it twice as exciting!

Before running the code, it’s a good idea to run the i2cdetect command to make sure that your MCP23008 is correctly connected.  This is part of the i2c-tools package, which is preinstalled in Beaglebone Ubuntu.  (If it’s not in your copy, run “sudo apt-get install i2c-tools.)

The MCP23008 should be detected at address x20, as shown below:

ubuntu@omap:~ sudo i2cdetect -r -y 3
 0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- UU -- -- -- --
20: 20 -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- UU UU UU UU -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- --

The Python code for this article can be downloaded from my Google Code page by running the following commands on your Beaglebone:

wget http://gigamega-micro.googlecode.com/files/WeatherBone.tgz
tar -xzvf WeatherBone.tgz
cd WeatherBone

The code for this circuit makes use of 2 Python classes:

  • Adafruit_I2C  - This is part of Adafruit’s Python library for the Raspberry Pi.  The code was almost 100% compatible with the Beaglebone: I only had to modify the bus number passed to SMBus (0 for the Pi, 3 for the Beaglebone)
  • gm_mcp23008.py – This is a wrapper I wrote for the MCP23008.

gm-mcp23008.py has a short bit of test code at the bottom, which is executed if you run it directly from python.

sudo python gm-mcp23008.py

You should find that the buttons faithfully toggle the LEDs’ state, with a slight pause.

The code begins with


mcp = mcp23008(3, 0x20, True)

The 3 parms being passed are the I2C bus number (3), the MCP23008 address (0×20), and a Debug flag that results in a message being written to the console each time I2C data is sent or received.

The code then initializes the 4 pins a la Arduino:


# button 1 connected to ground on pin 0 of MCP23008
mcp.pinMode(0, mcp.INPUT, True, mcp.INTERRUPT_PREVVAL, mcp.EDGE_LOW)
# LED 1 on pin 1
mcp.pinMode(1, mcp.OUTPUT)
# button 2 connected to ground on pin 6 of MCP23008
mcp.pinMode(6, mcp.INPUT, True, mcp.INTERRUPT_PREVVAL, mcp.EDGE_LOW)
# LED 2 on pin 7
mcp.pinMode(7, mcp.OUTPUT)

The INTERRUPT_PREVVAL and EDGE_LOW parameters require some explanation.

The MCP23008 supports interrupts, internally using some registers, and externally through pin 8 (which I’m not using).  The registers can be used to detect a case where the user pressed and released the button when the Python code wasn’t checking the pin.

The INTERRUPT_PREVVAL setting will trigger an interrupt when the pin state changes, and the EDGE_LOW parm will limit that to when it changes from high to low (remember, the pin goes from high to low when the button is pressed).

A good resource for learning more about the MCP23008′s registers is this article on embedded-lab.com.

The main part of the Python code loops indefinitely, checking the state of the buttons once per second — a leisurely rate that I’m using to test the interrupts:

# toggle LED when button is pressed
while 1 == 1:
   # NOTE!! - must get interrupts before reading pins, since reading pins clears the interrupts
   interrupt1 = mcp.wasInterrupt(0)
   interrupt2 = mcp.wasInterrupt(6)
   if interrupt1 or mcp.digitalRead(0) == mcp.LOW:
      led1State = not led1State
      mcp.digitalWrite(1, led1State)

   if interrupt2 or mcp.digitalRead(6) == mcp.LOW:
      led2State = not led2State
      mcp.digitalWrite(7, led2State)

   time.sleep(1)

As indicated by the comment, reading a pin’s state clears any interrupt for it. So, it’s necessary to call the wasInterrupt() method for both pins first.  This method returns True if the button was pressed since the last time the code checked (i.e. since the last call to digitalRead).

Writing to an LCD using the Adafruit I2C Backpack

As in my SPI article, I’ll leverage the GPIO extender code to write to a standard character LCD, using the Adafruit I2C/SPI LCD Backpack.

The backpack runs on 5V, and that voltage kills Beaglebones.  This wasn’t a concern when using SPI.  SPI uses a separate pin for input and output, so we were able to drive the output pin at 3.3V and leave the input pin disconnected.

With I2C, we need to use a level converter between the backpack and the Beaglebone’s I2C pins.  I used a $2 converter from Sparkfun.  I used the same Sparkfun board as a voltage divider in my article about analog input: the TX pins are level converters, while the RX pins are voltage dividers.  In this case, we want to use the TX pins.

I also made a small modification to the backpack to enable setting the backlight brightness via PWM.

The backpack normally feeds 5V into pin 15 of the LCD (the second from the end), resulting in maximum brightness.  As shown in the photo below, I’ve left pin 15 unsoldered in the backpack, connecting it instead to a NPN transistor.  One of the Beaglebone’s PWM pins is  connected to the base of the transistor.   A potentiometer attached to an analog pin is used to control the duty cycle setting of the PWM pin.

The PWM pin outputs 3.3V, which results in a noticeably dimmer LCD backlight, even though the transistor’s collector pin is connected to 5V.  So, I made use of the level converter to shift the PWM signal from 3.3V to 5V.

If all that seems way too complicated, you could actually avoid the need for level conversion by using the transistor with the LCD’s pin 16 instead of 15, connecting it to ground instead of 5V.  The only downside to that approach is you lose the API support for turning the backlight off and on, but the potentiometer can be used for this instead.

The circuit is shown below.  (The BMP085 part is explained in the next section.)

The Beaglebone pins are all on Port 9.  Along with ground, 3.3V and 5V, the pins are:

  • Pin 14 – PWM – white wire
  • Pin 19 – I2C Clock – brown wire (with no level conversion required)
  • Pin 20 – I2C Data – yellow wire (with level conversion)
  • Pin 32 – Analog 1.8V – orange wire
  • Pin 41 – Analog In – green wire

The Adafruit I2C backpack is (kind of) shown in the upper right of the diagram.  The wires leading to it are connected to:

  • Data – yellow wire1
  • Clock – brown wire
  • 5V – red wire
  • Ground – blue wire
  • Pin 15 of the LCD – Orange wire

I wrote a Python class, i2clcd to provide Arduino-like methods for writing to the LCD through the I2C port.  It’s included in the download file.

As a quick test, you can just run the Python class directly, which should turn on the backlight and write some text to the LCD:

sudo python i2clcd.py

The i2clcd class supports many of the same methods as the Arduino LiquidCrystal library.  So, to write text to all 4 lines of a 4×20 LCD:


# the LCD is connected to I2C bus 3

l = i2clcd(3)

l.begin(20, 4)
l.noBlink()
l.noCursor()
l.setBacklight(True)

l.clear()

for row in range(4):
   l.setCursor(0, row)
   l.Print("This is row " + str(row + 1))

Connecting Multiple I2C Devices

It’s quite easy to connect multiple devices to the same bus: just connect their clock pins and their data pins.  This is a lot simpler than controlling multiple SPI devices, which require a separate latch pin for each device.

I added a BMP085 to the circuit to demonstrate this.  It is powered by 3.3V, but coexists fine with the 5V LCD backpack provided that its data pin is connected to the 3.3V side of the level converter.

The code to read the BMP085 is taken from Adafruit’s Raspberry Pi Python library – it works on the Beaglebone without any changes at all.

To give all of the hardware something to do, I wrote a Python program, WeatherBone.py, to read the temperature and barometric pressure from the BMP085, and display that along with the sunrise and sunset time to the LCD.

Wrapping Up

The interfaces that I’ve covered in the Beaglebone Coding 101 series can be used to connect your Beaglebone to almost any type of device in the microcontroller world.

I have to admit that I’m not sure how big a gap the “almost” part leaves, since I tend to go down the path of least resistance.  If a particular sensor seems to be a little beyond the Beaglebone’s grasp (or beyond my grasp as a coder, anyway), I’ll switch to an equivalent sensor that uses a standard interface, or which others have already written code for.

The good news is that the paths of least resistance in embedded Linux coding are increasingly well marked, thanks to the Raspberry Pi.   It’s great to see so many new articles and sample code coming out of the Pi community.  The Pi’s code is generally easy to port to the Beaglebone, and the increased interest in embedded Linux is boosting both platforms.

Posted in Electronics, Programming | Tagged , , | 12 Comments

Beaglebone 101: Linux Tricks for Backing up and Resizing your microSD Card

I recently came across a couple of ideas on Raspberry Pi forums that made me think “wow, that’s so crazy it might just work!”

These are disk management commands that Linux allows, but probably shouldn’t.  Even Linux power users go pale (well, even paler) at the thought of trying them on their precious desktop PCs.  On SD-card systems like the Beaglebone, Beagleboard and Raspberry Pi, though, they’re considerably less scary, and they really do work.

Hot Backups

Let’s face it: backups are boring.  If I hadn’t put the word “hot” in the heading, you’d probably have skipped this section entirely.  Our aversion to backups stems from the PC world, where configuring backups is time-consuming and disk failures are increasingly infrequent.

You can’t be so complacent when working with a microSD card on a Beaglebone.  Systems can stop working for a number of reasons: package updates go awry, power outages and incorrect shutdowns can result in cryptic boot errors, and some microSD cards die young.

Fortunately, backups are relatively quick and painless on a Beaglebone.  Rather than decide which files to backup, you can easily and quickly do a “hot” backup of the entire microSD card while the system is still running.   A backup can be written to a new microSD card in about 15 minutes: a lot less time then it takes to configure a newly downloaded image.

The following work on both Angstrom and Ubuntu: actually, they’ll work on pretty much any Linux based system.

Hot Backups to a USB Drive

The fastest way to do this is by inserting a USB drive in your Beaglebone and copying the SD image onto it.

1. Determine the device ID of your USB drive.  If you’ve just inserted it, the last few lines of the output of the dmesg command will show it.  Otherwise, use fdisk:

root@beaglebone:~# fdisk -l
Disk /dev/mmcblk0: 8004 MB, 8004829184 bytes
4 heads, 16 sectors/track, 244288 cylinders, total 15634432 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/mmcblk0p1 * 63 144584 72261 c W95 FAT32 (LBA)
/dev/mmcblk0p2 144585 15634431 7744923+ 83 Linux
Disk /dev/sda: 15.6 GB, 15606349824 bytes
116 heads, 52 sectors/track, 5053 cylinders, total 30481152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xc3072e18
Device Boot Start End Blocks Id System
/dev/sda1 * 8064 30481151 15236544 c W95 FAT32 (LBA)

The last part of the output gives the size of the USB drive (15.6G), how it is formatted (FAT32), and the device ID (/dev/sda1).

2. Mount the USB drive.

mkdir /media/sda1
mount /dev/sda1 /media/sda1

3. Use the dd command to copy the full contents of the microSD card (partitions and all) to the USB drive, compressing it on the fly.  Double-check your typing before running any dd command — it’s nicknamed Disk Destroyer for a reason.  (The gzip parameter is a one, not an L.)

dd if=/dev/mmcblk0 | gzip -1 > /media/sda1/sd_backup.img.gz

A four gigabyte card will generally compress to under 1G.

The time required for this to complete will vary widely depending on the quality of your USB drive.

Among my motley crew of USB drives, the best performance I’ve seen is with a Patriot Xporter XT USB drive, 7.1Mb/s, resulting in a 19 minute backup for an 8G microSD card and 9 minutes for a 4G card.  The worst is with a grocery store brand (“President’s Choice”, but I’m pretty sure Obama had nothing to do with it):  I get just 1.6 MB/s, resulting in backup times of about 80 minutes and 42 minutes respectively.

If you’re in a hurry, you can speed things up by dropping the compression.

dd if=/dev/mmcblk0 of=/media/sda1/sd_backup.img

You’ll get a full byte-by-byte copy of the microSD card this way, so a 4G card will require (almost) 4G of free space on the USB drive.  This poses a problem if your microSD card is greater than 4G, since most USB drives are preformatted as FAT32, and FAT32 can’t handle files above 4G.

With my Patriot XT drive, an uncompressed backup runs at a 10M/s clip, completing the backup of a 4G card in under 7 minutes.

Monitoring Backup Progress

The dd command isn’t very chatty: it gives no indication of its progress whatsoever.  It does, however, respond to death threats.   Let me explain..

First, get the process ID of the dd command:

root@beaglebone:~# ps aux | grep "mmcblk0"
root 25437 4.6 0.2 3020 680 pts/2 S+ 18:17 0:01 dd if=/dev/mmcblk0
root 25446 0.0 0.2 2928 720 pts/0 S+ 18:17 0:00 grep --color=auto mmcblk0

The first line of the output shows the process ID of the dd command is 25437.

Next, enter the following command:

root@beaglebone:~# kill -USR1 25437

This won’t actually kill the dd process, but if you switch back to the session where the dd command is running, you’ll see some new lines of output, like this:

root@beaglebone:~# dd if=/dev/mmcblk0 | gzip -1 > /media/sda1/sd_backup.img.gz
175297+0 records in
175296+0 records out
89751552 bytes (90 MB) copied, 54.694 s, 1.9 MB/s

Updated 10/1: Thanks to kioan in the Comments for the following suggestion: use the Linux pv (pipe viewer) to get a progress bar for the backup.  The syntax is:

root@beaglebone:~# dd if=/dev/mmcblk0 | pv -s 4G -peta | gzip -1 > /media/sda1/sd_backup.img.gz

This displays a progress indicator like the following.

0:15:01 [1.57MB/s] [====================>               ] 51% ETA 0:14:06

A few notes on this:

  • The “-s 4G” parm specifies the amount of data being backed up (basically the size of the SD card).  If you have a bigger SD card, change that parm accordingly.
  • The “-peta” parm specifies that the average data transfer rate should be displayed, as well as being a fine tribute to the People For the Eating of Tasty Animals.  To display the (wildly fluctuating) current transfer rate, use “-petr” instead.
  • Ubuntu doesn’t include the pv command by default, but you can install it with apt-get install pv

Updated 10/2:  Thanks to mlitke and Chris Tyler for their comments with the following suggestions:

- Use killall instead of kill to save having to lookup the process ID:

killall -USR1 dd

- Use the watch command to have the output from dd update automatically.  For example, to update every 10 seconds:

watch -n10 killall -USR1 dd

Note that killall is included in Ubuntu’s psmisc package (i.e. apt-get install psmisc).  Also, since the “watch” command needs to be left running in the foreground, you’ll probably want to use a virtual terminal utility like screen for this (i.e. run screen, run the dd command in the first session, press Ctrl-A C to open a 2nd session, run the watch command, then Ctrl-A N to cycle back to the dd session.


Hot Backups Over the Network

If you don’t have a USB drive, or your Beaglebone is trapped in a burning house, it’s possible to run the backup over an SSH session.

From a Linux desktop PC, enter the following. This would be a good point at which to remind you of dd’s awesome ability to screw things up royally, so type carefully:

dan@ubuntu:~# ssh root@192.168.1.2 dd if=/dev/mmcblk0 | gzip -1 | dd of=sd_backup.img.gz

You should, of course, replace with IP address with your Beaglebone’s address.

The command will prompt you for your root password on the Beaglebone.  If you’re running Ubuntu on your Beaglebone this might stump you: there probably isn’t one.  You’ll have to set a password manually from the Beaglebone’s command line:

ubuntu@omap:~$ sudo -i
[sudo] password for ubuntu:
root@omap:~# passwd
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully

As far as I know, Windows users are out of luck here, since you need to run dd at both ends of the SSH session.

Naturally, your network speed comes into play, as does the speed of your PC, but it’s generally going to be much slower than the USB drive approach.  On a my wired network, backups run at about 2.5MB/s, resulting in over 50 minutes to backup an 8G card.

Restoring a Backup

The .img or img.gz file that you created with the previous commands has the same format as one you download from the Beaglebone demo files or CircuitCo sites.  So, you can write it to a microSD card using the same commands.

On a Windows PC, you can use Win32DiskImager.

On a Linux PC, run the following command as root, when /dev/sdX should be changed to the device ID of the microSD card:

zcat sd_backup.img.gz > /dev/sdX

Resizing a 4G Image on an 8G Card

Beaglebone image files tend to be sized for 4G microSD cards.  This is true for both the image files on the Beaglebone Demo Files page and the image files on the CircuitCo site.

You can install these on a 8G or larger microSD card, but the resulting root partition will always be a little under 4G in size, leaving the rest of your card unused.

If this is news to you (I’m embarrassed to admit that I only caught onto this recently), you can check how much of your microSD card is being used with the following command:

root@beaglebone:~# fdisk -l /dev/mmcblk0
Disk /dev/mmcblk0: 8004 MB, 8004829184 bytes
 255 heads, 63 sectors/track, 973 cylinders, total 15634432 sectors
 Units = sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes
 Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
 /dev/mmcblk0p1 * 63 144584 72261 c W95 FAT32 (LBA)
 /dev/mmcblk0p2 144585 7132859 3494137+ 83 Linux

The first line of the output shows the total MB on the card: 8004 MB, or 8G .  The last line shows us where the root partition ends, in 512 byte blocks.  So, this card is using 7132859 blocks, or about 3.5G of the card.

The safest way to extend the partition on the card is to take it over to a Linux PC and run parted or GParted.

However, you can actually extend the root partition while the Beaglebone is running off the card.  I wouldn’t have thought it possible, but it’s all the rage among Raspberry Pi users, where image files tend to be just 2G in size.  I’ve (belatedly) resized the partition on my 8G cards, and the process worked fine for me too.

Here are the steps:

1. Run fdisk

root@beaglebone:~# fdisk /dev/mmcblk0

Note that fdisk doesn’t touch the card until step 4, so if you make any mistakes before that point, it’s best to enter the q command to quit fdisk, then start over again.

2. At the fdisk command prompt, enter d to delete a partition, then select partition 2.

Command (m for help): d
 Partition number (1-4): 2
 Partition 2 is deleted

3. Still at the fdisk command prompt, enter n to create a new partition. You’ll then be prompted for various settings.  Enter

  • Partition Type: p
  • Partition Number: 2
  • First sector: Press Enter for the default
  • Last sector: Again, press Enter for the default.

Here’s the full text from one of my fdisk sessions:

Command (m for help): n
 Partition type:
 p primary (1 primary, 0 extended, 3 free)
 e extended
 Select (default p): p
 Partition number (1-4, default 2):
 Using default value 2
 First sector (144585-15634431, default 144585):
 Using default value 144585
 Last sector, +sectors or +size{K,M,G} (144585-15634431, default 15634431):
 Using default value 15634431
 Partition 2 of type Linux and of size 7.4 GiB is set

4. If everything went as expected, enter w to save your changes.  Again, if you’re unsure, it’s best to enter q at this point to abort the process then start over.

Command (m for help): w
 The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
 The kernel still uses the old table. The new table will be used at
 the next reboot or after you run partprobe(8) or kpartx(8)
 Syncing disks.

5. It’s important that you reboot before doing anything else:

 root@beaglebone:~# shutdown -r now

6. At this point, you have an 8G partition with a 4G file system.  Fortunately, Linux has a command to fix that:

root@beaglebone:~# resize2fs /dev/mmcblk0p2

This command will take quite awhile to run – at least 30 minutes on my system.  The output should look like this:

 resize2fs 1.42.1 (17-Feb-2012)
 Filesystem at /dev/mmcblk0p2 is mounted on /; on-line resizing required
 old_desc_blocks = 1, new_desc_blocks = 1
 Performing an on-line resize of /dev/mmcblk0p2 to 1936230 (4k) blocks.
 The filesystem on /dev/mmcblk0p2 is now 1936230 blocks long.

7. Another reboot is required at this point.

root@beaglebone:~# shutdown -r now

8. After this second reboot, you should have a nice big root partition.  You can confirm this with the df (disk free) command — the following shows a 7.6G root partition:

root@beaglebone:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
rootfs 7622928 1202920 6035060 17% /
Posted in Gadgets | Tagged , , | 15 Comments