Beaglebone 101: Linux Tricks for Backing up and Resizing your microSD Card

I recently came across a couple of ideas on Raspberry Pi forums that made me think “wow, that’s so crazy it might just work!”

These are disk management commands that Linux allows, but probably shouldn’t.  Even Linux power users go pale (well, even paler) at the thought of trying them on their precious desktop PCs.  On SD-card systems like the Beaglebone, Beagleboard and Raspberry Pi, though, they’re considerably less scary, and they really do work.

Hot Backups

Let’s face it: backups are boring.  If I hadn’t put the word “hot” in the heading, you’d probably have skipped this section entirely.  Our aversion to backups stems from the PC world, where configuring backups is time-consuming and disk failures are increasingly infrequent.

You can’t be so complacent when working with a microSD card on a Beaglebone.  Systems can stop working for a number of reasons: package updates go awry, power outages and incorrect shutdowns can result in cryptic boot errors, and some microSD cards die young.

Fortunately, backups are relatively quick and painless on a Beaglebone.  Rather than decide which files to backup, you can easily and quickly do a “hot” backup of the entire microSD card while the system is still running.   A backup can be written to a new microSD card in about 15 minutes: a lot less time then it takes to configure a newly downloaded image.

The following work on both Angstrom and Ubuntu: actually, they’ll work on pretty much any Linux based system.

Hot Backups to a USB Drive

The fastest way to do this is by inserting a USB drive in your Beaglebone and copying the SD image onto it.

1. Determine the device ID of your USB drive.  If you’ve just inserted it, the last few lines of the output of the dmesg command will show it.  Otherwise, use fdisk:

root@beaglebone:~# fdisk -l
Disk /dev/mmcblk0: 8004 MB, 8004829184 bytes
4 heads, 16 sectors/track, 244288 cylinders, total 15634432 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/mmcblk0p1 * 63 144584 72261 c W95 FAT32 (LBA)
/dev/mmcblk0p2 144585 15634431 7744923+ 83 Linux
Disk /dev/sda: 15.6 GB, 15606349824 bytes
116 heads, 52 sectors/track, 5053 cylinders, total 30481152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xc3072e18
Device Boot Start End Blocks Id System
/dev/sda1 * 8064 30481151 15236544 c W95 FAT32 (LBA)

The last part of the output gives the size of the USB drive (15.6G), how it is formatted (FAT32), and the device ID (/dev/sda1).

2. Mount the USB drive.

mkdir /media/sda1
mount /dev/sda1 /media/sda1

3. Use the dd command to copy the full contents of the microSD card (partitions and all) to the USB drive, compressing it on the fly.  Double-check your typing before running any dd command — it’s nicknamed Disk Destroyer for a reason.  (The gzip parameter is a one, not an L.)

dd if=/dev/mmcblk0 | gzip -1 > /media/sda1/sd_backup.img.gz

A four gigabyte card will generally compress to under 1G.

The time required for this to complete will vary widely depending on the quality of your USB drive.

Among my motley crew of USB drives, the best performance I’ve seen is with a Patriot Xporter XT USB drive, 7.1Mb/s, resulting in a 19 minute backup for an 8G microSD card and 9 minutes for a 4G card.  The worst is with a grocery store brand (“President’s Choice”, but I’m pretty sure Obama had nothing to do with it):  I get just 1.6 MB/s, resulting in backup times of about 80 minutes and 42 minutes respectively.

If you’re in a hurry, you can speed things up by dropping the compression.

dd if=/dev/mmcblk0 of=/media/sda1/sd_backup.img

You’ll get a full byte-by-byte copy of the microSD card this way, so a 4G card will require (almost) 4G of free space on the USB drive.  This poses a problem if your microSD card is greater than 4G, since most USB drives are preformatted as FAT32, and FAT32 can’t handle files above 4G.

With my Patriot XT drive, an uncompressed backup runs at a 10M/s clip, completing the backup of a 4G card in under 7 minutes.

Monitoring Backup Progress

The dd command isn’t very chatty: it gives no indication of its progress whatsoever.  It does, however, respond to death threats.   Let me explain..

First, get the process ID of the dd command:

root@beaglebone:~# ps aux | grep "mmcblk0"
root 25437 4.6 0.2 3020 680 pts/2 S+ 18:17 0:01 dd if=/dev/mmcblk0
root 25446 0.0 0.2 2928 720 pts/0 S+ 18:17 0:00 grep --color=auto mmcblk0

The first line of the output shows the process ID of the dd command is 25437.

Next, enter the following command:

root@beaglebone:~# kill -USR1 25437

This won’t actually kill the dd process, but if you switch back to the session where the dd command is running, you’ll see some new lines of output, like this:

root@beaglebone:~# dd if=/dev/mmcblk0 | gzip -1 > /media/sda1/sd_backup.img.gz
175297+0 records in
175296+0 records out
89751552 bytes (90 MB) copied, 54.694 s, 1.9 MB/s

Updated 10/1: Thanks to kioan in the Comments for the following suggestion: use the Linux pv (pipe viewer) to get a progress bar for the backup.  The syntax is:

root@beaglebone:~# dd if=/dev/mmcblk0 | pv -s 4G -peta | gzip -1 > /media/sda1/sd_backup.img.gz

This displays a progress indicator like the following.

0:15:01 [1.57MB/s] [====================>               ] 51% ETA 0:14:06

A few notes on this:

  • The “-s 4G” parm specifies the amount of data being backed up (basically the size of the SD card).  If you have a bigger SD card, change that parm accordingly.
  • The “-peta” parm specifies that the average data transfer rate should be displayed, as well as being a fine tribute to the People For the Eating of Tasty Animals.  To display the (wildly fluctuating) current transfer rate, use “-petr” instead.
  • Ubuntu doesn’t include the pv command by default, but you can install it with apt-get install pv

Updated 10/2:  Thanks to mlitke and Chris Tyler for their comments with the following suggestions:

- Use killall instead of kill to save having to lookup the process ID:

killall -USR1 dd

- Use the watch command to have the output from dd update automatically.  For example, to update every 10 seconds:

watch -n10 killall -USR1 dd

Note that killall is included in Ubuntu’s psmisc package (i.e. apt-get install psmisc).  Also, since the “watch” command needs to be left running in the foreground, you’ll probably want to use a virtual terminal utility like screen for this (i.e. run screen, run the dd command in the first session, press Ctrl-A C to open a 2nd session, run the watch command, then Ctrl-A N to cycle back to the dd session.


Hot Backups Over the Network

If you don’t have a USB drive, or your Beaglebone is trapped in a burning house, it’s possible to run the backup over an SSH session.

From a Linux desktop PC, enter the following. This would be a good point at which to remind you of dd’s awesome ability to screw things up royally, so type carefully:

dan@ubuntu:~# ssh root@192.168.1.2 dd if=/dev/mmcblk0 | gzip -1 | dd of=sd_backup.img.gz

You should, of course, replace with IP address with your Beaglebone’s address.

The command will prompt you for your root password on the Beaglebone.  If you’re running Ubuntu on your Beaglebone this might stump you: there probably isn’t one.  You’ll have to set a password manually from the Beaglebone’s command line:

ubuntu@omap:~$ sudo -i
[sudo] password for ubuntu:
root@omap:~# passwd
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully

As far as I know, Windows users are out of luck here, since you need to run dd at both ends of the SSH session.

Naturally, your network speed comes into play, as does the speed of your PC, but it’s generally going to be much slower than the USB drive approach.  On a my wired network, backups run at about 2.5MB/s, resulting in over 50 minutes to backup an 8G card.

Restoring a Backup

The .img or img.gz file that you created with the previous commands has the same format as one you download from the Beaglebone demo files or CircuitCo sites.  So, you can write it to a microSD card using the same commands.

On a Windows PC, you can use Win32DiskImager.

On a Linux PC, run the following command as root, when /dev/sdX should be changed to the device ID of the microSD card:

zcat sd_backup.img.gz > /dev/sdX

Resizing a 4G Image on an 8G Card

Beaglebone image files tend to be sized for 4G microSD cards.  This is true for both the image files on the Beaglebone Demo Files page and the image files on the CircuitCo site.

You can install these on a 8G or larger microSD card, but the resulting root partition will always be a little under 4G in size, leaving the rest of your card unused.

If this is news to you (I’m embarrassed to admit that I only caught onto this recently), you can check how much of your microSD card is being used with the following command:

root@beaglebone:~# fdisk -l /dev/mmcblk0
Disk /dev/mmcblk0: 8004 MB, 8004829184 bytes
 255 heads, 63 sectors/track, 973 cylinders, total 15634432 sectors
 Units = sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes
 Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
 /dev/mmcblk0p1 * 63 144584 72261 c W95 FAT32 (LBA)
 /dev/mmcblk0p2 144585 7132859 3494137+ 83 Linux

The first line of the output shows the total MB on the card: 8004 MB, or 8G .  The last line shows us where the root partition ends, in 512 byte blocks.  So, this card is using 7132859 blocks, or about 3.5G of the card.

The safest way to extend the partition on the card is to take it over to a Linux PC and run parted or GParted.

However, you can actually extend the root partition while the Beaglebone is running off the card.  I wouldn’t have thought it possible, but it’s all the rage among Raspberry Pi users, where image files tend to be just 2G in size.  I’ve (belatedly) resized the partition on my 8G cards, and the process worked fine for me too.

Here are the steps:

1. Run fdisk

root@beaglebone:~# fdisk /dev/mmcblk0

Note that fdisk doesn’t touch the card until step 4, so if you make any mistakes before that point, it’s best to enter the q command to quit fdisk, then start over again.

2. At the fdisk command prompt, enter d to delete a partition, then select partition 2.

Command (m for help): d
 Partition number (1-4): 2
 Partition 2 is deleted

3. Still at the fdisk command prompt, enter n to create a new partition. You’ll then be prompted for various settings.  Enter

  • Partition Type: p
  • Partition Number: 2
  • First sector: Press Enter for the default
  • Last sector: Again, press Enter for the default.

Here’s the full text from one of my fdisk sessions:

Command (m for help): n
 Partition type:
 p primary (1 primary, 0 extended, 3 free)
 e extended
 Select (default p): p
 Partition number (1-4, default 2):
 Using default value 2
 First sector (144585-15634431, default 144585):
 Using default value 144585
 Last sector, +sectors or +size{K,M,G} (144585-15634431, default 15634431):
 Using default value 15634431
 Partition 2 of type Linux and of size 7.4 GiB is set

4. If everything went as expected, enter w to save your changes.  Again, if you’re unsure, it’s best to enter q at this point to abort the process then start over.

Command (m for help): w
 The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
 The kernel still uses the old table. The new table will be used at
 the next reboot or after you run partprobe(8) or kpartx(8)
 Syncing disks.

5. It’s important that you reboot before doing anything else:

 root@beaglebone:~# shutdown -r now

6. At this point, you have an 8G partition with a 4G file system.  Fortunately, Linux has a command to fix that:

root@beaglebone:~# resize2fs /dev/mmcblk0p2

This command will take quite awhile to run – at least 30 minutes on my system.  The output should look like this:

 resize2fs 1.42.1 (17-Feb-2012)
 Filesystem at /dev/mmcblk0p2 is mounted on /; on-line resizing required
 old_desc_blocks = 1, new_desc_blocks = 1
 Performing an on-line resize of /dev/mmcblk0p2 to 1936230 (4k) blocks.
 The filesystem on /dev/mmcblk0p2 is now 1936230 blocks long.

7. Another reboot is required at this point.

root@beaglebone:~# shutdown -r now

8. After this second reboot, you should have a nice big root partition.  You can confirm this with the df (disk free) command — the following shows a 7.6G root partition:

root@beaglebone:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
rootfs 7622928 1202920 6035060 17% /
This entry was posted in Gadgets and tagged , , . Bookmark the permalink.

15 Responses to Beaglebone 101: Linux Tricks for Backing up and Resizing your microSD Card

  1. Pingback: Building Angstrom Linux for a Gumstix Overo 7/23/2012 « nickwideas

  2. Pingback: Beaglebone 101: Linux Tricks for Backing up and Resizing your microSD Card « adafruit industries blog

  3. tijs says:

    using bs=1024 on the dd command might speed up things consideratly.

    • dwatts says:

      Hi,

      Based on my limited testing, I didn’t get any improvement by setting the blocksize to something other than the default of 512. I tried various settings, when backing up to a USB drive and over an SSH connection, and the higher the block size, the slower it was.

      I just tried again, using a cheapo Kingston Data Traveler USB drive formatted with FAT32. With the default block size, it took 1797 seconds (about 30 minutes) at 2.2MB/s. With bs=1024, it took 2038 seconds (about 34 minutes) at 2.0MB/s.

      Do you know how to determine the optimal blocksize setting, other than trial and error?

      Dan.

  4. kioan says:

    In order to monitor the backup progress you could use the pv command. For example:

    dd if=/dev/mmcblk0 | pv -s 4G -petr | dd of=/media/sda1/sd_backup.img

  5. Chris Tyler says:

    Here’s a cheap & cheerful way to get dd to spit its progress out every two seconds: in another terminal, run “watch killall -USR1 dd”

  6. Alan says:

    FAT32 gave errors, when I tried to create files larger than 4GB (ISO images). I found out it was a limit of the FAT32 file system. Now my large (>8GB) USB sticks are all formatted with NTFS.

    When I get my Beaglebone, will I still be able to back up to these?
    Or should I re-format one as FAT32?

    • dwatts says:

      Yes, USB drives formatted with NTFS are supported on the Beaglebone.

      You need to install the ntfs-3g package in order to enable writing to the NTFS drives.

      For drives of 4G or less, I still use FAT32. I haven’t become 100% confident in the ntfs-3g Linux driver, though I haven’t had any problems recently that I’m sure were caused by the driver.

      Dan.

      • Douglas says:

        NTFS is not for Linux and reduce the life of flash drivers.
        I’m looking about exFAT, it is big size USB, this file system looks to have less fragmentation if compared to NTFS.

        The only problem looks to be the legal part!

  7. Cyril says:

    Hello Dan,

    I used your instructions yesterday on how to re-size SD card. I was running ubunt 12.04 on a 4GB and with your “How to” I was able to upgrade to 8GB without any issues. Thank you for this awesome article.

    Thank you.

  8. Alan says:

    Although I have Ubuntu in a virtual machine, I’m using the Beaglebone purely with Windows – and trying to document my fun. Thanks for the great articles!

    One piece of kit you may see on Ebay is – a USB adaptor for micro SDHC cards. Using one of these, you can make a backup image directly to another card.

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>