Backup Scripts: Keep your data safe for your peace of mind

Backup Scripts

You’ve worked hard to get your system in shape and worked out all of the bugs, but you know that things are not going to last, and you may be an update away from a broken browser, or you might want to experiment with a new kernel or beta package. To save yourself from future trouble, it’s always a good idea to make a backup! However, backups are usually a source of confusion on the forums, and lots of new users struggle with them. In this article, we’ll learn what needs to be done to keep your system safe.

Disks, partitions and file systems

Seasoned computer users have no problem distinguishing between disks, partitions and filesystems, but let’s analyze them to have a common starting point. A disk is generally a physical device that stores data into randomly-accessible blocks. In more complex setups, multiple physical disks can be combined as RAID arrays using either hardware or software, and exposing them to the operating system as virtual disks. Partitions are sections on disks that usually hold a filesystem. Filesystems manage how files and data are stored in order to be found later on. In order to make a backup of your ODROID system, you will need to preserve the partition information and the contents of the partitions.

Figure 1 - Partition layout of an Odroid C1 triple-boot image
Figure 1 – Partition layout of an ODROID-C1 triple-boot image

All disks start off with a 512 byte block of data that typically holds the bootloader (for x86 systems, 446 bytes) and the 64B Master Boot Record (MBR), which is explained at http://bit.ly/2bMCTUh. The MBR is a table with the start offset, length and partition type of your 4 primary partitions. These are the partitions mapped as 1-4 in the Linux kernel (e.g., sda1-sda4 for a disk called sda). The MBR is an old data structure, introduced in 1983, so it has some limitations. The need to use ever-larger disks (>2TB) led to the introduction of the GUID Partition Table (GPT) which replaces the MBR in newer systems and is detailed at http://bit.ly/2bvb4oL. ODROIDs can use both MBR and GPT, but the boot media is designed as a MBR volume because of its relatively small size and simplicity.

But, as shown in Figure 1, a disk may have more than 4 partitions. This is achieved by using a trick – one primary partition is marked as “extended” and it can contain any number of logical partitions. Linux represents them with numbers from 5 upwards (i.e., sda5, sda6 and so on). The partition information for the logical partitions is stored in structures similar to the MBR called Extended Boot Record (EBR) as explained at http://bit.ly/2bw47Re, which looks like a linked list as shown in Figure 2, but precedes the actual partition on disk.

Figure 2 - EBR position on disk
Figure 2 – EBR position on disk

The partitions you’ll usually see on ODROIDs are FAT16/FAT32 (seen as VFAT under the mount command) and Ext2/3/4. There are other partition types supported by Linux, such as NTFS, XFS, and ZFS, but they are usually not critical to the boot process, so they will be out of our scope. There are backup tools such as BackupPC (http://bit.ly/2bx3J6R) or Clonezilla (http://bit.ly/1Iq2mN7), which support more partition types or do backup on file level. These same tools should be used to backup your personal data, such as files, pictures or music. It’s also a good idea before starting a backup to do some “spring cleaning” and delete things you no longer need, such as temporary files or downloads, in order to reduce the time it takes to do the backup and the size of the backup file. For instance, you can delete the cache of downloaded apt packages with the following command:

$ sudo apt-get clean

Backup strategies

There are a few ways of making a backup of your eMMC/SD card. The simplest to implement is to make a 1:1 binary copy of your data to an image file. For this task, you can use a tool such as dd or Win32DiskImager. Note that all of the commands that follow expect to have the variable $backupDir replaced by the path to your desired backup directory, which can’t be on the same partition you’re trying to backup for obvious reasons.

$ sudo dd if=/dev/mmcblk0 of=$backupDir/backup.img bs=1M
In the command above, “if” represents “input file” and should point to the block device representing your disk, such as /dev/mmcblk0, and “of” represents the “output file” where data should be written to. The parameter “bs” represents “block size”, which signifies how much data is read and written at once. A variation of the dd command that shows progress uses the “pv” command (pipe viewer):
# apt-get install pv
# dd if=/dev/mmcblk0 bs=1M | pv | dd of=$backupDir/backup.img
Restoring the data is equally easy: just replace the values of “if” and “of”:
$ sudo dd if=$backupDir/backup.img of=/dev/mmcblk0 bs=1M
Note that dd makes a binary copy of your disk. This means that it will copy also the free space on your disk. The default output file will be as large as your disk, which means that copying a 64GB SD card of mostly empty space will take a long time and take up a lot of room. The advantage is that you can later run tools like PhotoRec (http://bit.ly/1jwXElB) on the free space and possibly recover deleted files, which is useful when doing data forensics or recovering from bad media. The disadvantage is that the image will be big and slow to copy. You can use dd together with gzip to shrink the image before writing it to reduce size a bit, but you won’t save time:
# dd if=/dev/mmcblk0 bs=1M | gzip -c > $backupDir/backup.img.gz
# gunzip -c $backupDir/backup.img.gz | dd of=/dev/mmcblk0 bs=1M
Also note that, in theory, you can do a backup with dd on a live system by copying it while the partitions are mounted, but there is a risk of inconsistencies if files are changed while doing the backup. It’s best to do an offline backup by pulling the eMMC/SD card, plug it into a different system, and do the backup without having mounted partitions. There’s also a disadvantage when copying between media of slightly different sizes. Since not all 16GB cards are exactly the same size, you might end up with a truncated partition on your destination.

The “dd” utility has the advantage that it is easy to use, but to gain backup/restore speed and minimize necessary backup space, you need to break up the backup operation into several steps and avoid backing up free space. For this, you’ll need to backup the MBR + EBR, bootloader, and individual partitions.

You can still cheat and use dd if you use gparted in order to shrink your largest/last partition to only the used size, dd up to that size, then resize the partition back to the original size after you restore it, but it involves some manual work.

MBR backup and restore

The MBR and EBR are small data structures and can be easily backed up with dd. But because the EBR’s position on disk can vary, you should rely on a partitioning tool to extract and restore the MBR/EBR data. Such a tool is sfdisk:

$ sudo apt-get install sfdisk
$ sudo sfdisk -d /dev/mmcblk0 > $backupDir/partition_table.txt
To restore it later, you need to supply the saved file to sfdisk like this:
$ sudo sfdisk /dev/mmcblk0 < $backupDir/partition_table.txt
Note that overwriting the MBR on a disk with existing partitions is equivalent to deleting the partitions since the operating system will not be able to find the offsets to the old partitions anymore, so use the restore step with extreme care! This backup can be performed on a live system without risks since partition tables are not usually changed during runtime.

Bootloader backup and restore

ODROIDs use U-Boot as a bootloader, as detailed in the November 2015 issue of ODROID Magazine November 2015 (http://bit.ly/2bA3P9g). U-Boot stores its code and data in the unallocated space after the MBR and at the beginning of the first partition. There is also some bootstrap code in the first 446 bytes in the first sector, before the partition table. Since the size and structure of U-Boot may differ between ODROID models, it’s safest to do a binary backup of this unallocated space with dd. First, you need to find out the start sector of the first partition with sfdisk:

$ sudo sfdisk -l /dev/mmcblk0
Figure 3 - Identify the start sector of the first partition with sfdisk and sector size
Figure 3 – Identify the start sector of the first partition with sfdisk and sector size

As indicated in Figure 3, the first partition (loop0p1) starts at offset 49152, so we’ll need to copy everything up to and including sector 49151. The bs (block size) parameter must match what sfdisk reported in the “Units” line:

$ sudo dd if=/dev/mmcblk0 of=$backupDir/bootloader.bin bs=512 count=49151
Note that the dd command will also copy over the MBR, which is sector 0). To restore the bootloader and skip restoring the partition table as well, you can use the following command:
$ sudo dd if=$backupDir/bootloader.bin of=/dev/mmcblk0 bs=512 skip=1 seek=1
You should also restore the bootstrap code from the first sector:
$ sudo dd if=$backupDir/bootloader.bin of=/dev/mmcblk0 bs=446 count=1
To restore the partition table as well, do not add the skip and seek parameters. This too can be done on a live system since the data is mostly read-only.

FAT partitions backup and restore

By default, Hardkernel’s images come with a FAT16/32 partition mounted under /media/boot that contains the kernel, initrd, device tree and boot.ini files. All of these are crucial to system startup. Android systems expose this partition as “sdcard” storage.

There are several tools for linux that backup FAT partitions. I used to use partimage, but it fails to verify the checksum of the partitions on C2, so I switched to partclone. Partclone can do a block backup of FAT partitions preserving data at the same offsets, but can skip empty space.

$ sudo apt-get install partclone
$ sudo partclone.vfat -c -s /dev/mmcblk0p1 -O $backupDir/partition_1.img
The “-c” specifies “clone”, “-s” is the source partition, which is the first partition in our case, and “-O” is the output file, which will get overwritten if it exists. Note that partclone cannot operate on mounted filesystems and will exit with an error. In order to back up from a running ODROID, you will need to unmount /media/boot, perform the backup and mount it back again.

To restore a FAT partition, you can run the following command:

$ sudo partclone.restore -s $backupDir/partition_1.img -o /dev/mmcblk0p1
Figure 4 - Partclone backup with prior unmounting of /media/boot
Figure 4 – Partclone backup with prior unmounting of /media/boot

Unfortunately, PartClone will not allow you to restore a partition to a smaller or larger target partition, so any size adjustment you will need to make after the restore is done. You can actually restore to a larger partition, but you will need to manually grow it in order to use the extra space.

Ext2/3/4 partitions backup and restore

In order to backup and restore Ext2/3/4 filesystems, we’ll need to use a different tool called FSArchiver. Unlike PartClone, FSArchiver creates a file level backup and reconstructs the filesystem upon restore. Unfortunately, because of certain particularities of FAT systems where Windows boot files need to be at specific offsets, the author of fsarchiver does not support backing up FAT filesystems as well, so we’re stuck to using two tools for the job. But with the help of external packages fsarchiver can support other filesystems as well, such as XFS, ReiserFS, JFS, BTRFS and NTFS. It usually backs up unmounted filesystems, but can be used on live filesystems as well with the “-A” flag, which may not always work. FSArchiver has the advantage that it can restore a filesystem in a bigger or smaller target partition while preserving UUIDs. In order to back up the second partition, you can run the following commands:

$ sudo apt-get install fsarchiver
$ sudo fsarchiver -o -v -A -j 4 savefs $backupDir/partition_2.fsa /dev/mmcblk0p2
The “-o” flag means overwrite the destination file if it exists, “-v” is verbose output, “-A” allows you to backup a mounted partition and “-j 4” allows it to use 4 cores for compression.

In order to restore a fsa backup you can run the following command:

$ sudo fsarchiver restfs $backupDir/partition_2.fsa id=0,dest=/dev/mmcblk0p2
Note that since FSArchiver supports multiple partitions inside an archive, it needs you to specify which partition id to restore. In our example, we store only one partition in an archive, so you’ll always specify id=0 when restoring.

SPI Flash

Newer boards, like the development Odroid N1 may feature a low capacity SPI NAND Flash chip designed to store the bootloader and kernel, so that it can boot from network, or from a SATA disk, without the need of a eMMC or SD card. Even if the layout of this chip hasn’t been decided fully at the time of this writing, we can still back it up and restore it as a block device with dd. You can get a list (and description) of MTD devices in your Odroid by running:

$ sudo cat /proc/mtd
 dev: size erasesize name
 mtd0: 01000000 00001000 "spi32766.0"
To back it up, you can simply use:
$ sudo dd if=/dev/mtd0 of=$backupDir/flash_mtd0.bin bs=4096
In order to write to a flash device, to restore it, you need to erase the block you’re going to write to. Fortunately, since we will write the whole device, we can erase it all before writing. For this we need mtd-utils which provides flash_erase:
$ sudo apt-get install mtd-utils
$ sudo flash_erase -q /dev/mtd0 0 0
$ sudo dd if=$backupDir/flash_mtd0.bin of=/dev/mtd0 bs=4096

ODROID backup tool

Now that you know how to do things manually, you may question why backup and restore operations are not simpler, using point and click operations. I agree that nobody has the time to remember all the command line arguments from various commands, so I hacked together a rudimentary GUI that can walk you through your backup and restore process.

The tool is descriptively called “odroid-backup”. It’s written in Perl and uses zenity and dialog to build a rudimentary GUI, because I’m too old to learn Python. To install the tool, you can download it from my GitHub repository:

$ sudo wget -O /usr/local/bin/odroid-backup.pl https://raw.githubusercontent.com/mad-ady/odroid-backup/master/odroid-backup.pl
$ sudo chmod a+x /usr/local/bin/odroid-backup.pl
The script depends on a bunch of non-standard Perl modules as well as some Linux utilities, and will display a list of missing dependencies and ways of fixing it when you first run it. To install all dependencies at once, run the following:
$ sudo apt-get install libui-dialog-perl zenity dialog libnumber-bytes-human-perl libjson-perl sfdisk fsarchiver udev util-linux coreutils partclone parted mtd-utils
The script is designed to run on Linux systems, such as a PC to which you’ve hooked up a SD card or eMMC module via a USB adapter, or directly on the ODROID (sorry Windows fans). Also, the script will create graphical windows if it detects that you’re running an X11 session, or will fall back to ncurses (display) if you’re connected via ssh or terminal. You can manually force this with the –text switch.

Figure 5 - Zenity vs display rendering
Figure 5 – Zenity vs display rendering

To perform a backup, start the tool in a terminal and select “Backup partitions”, then select OK (1):

$ sudo odroid-backup.pl
You will be presented with a list of removable drives in your system. You can start the program with the -a flag in order to display all drives, which is the case when running directly on the ODROID, since eMMC and SD are shown as non-removable. Select the desired one and click OK (2). You will then be presented with a list of partitions on that drive. Select the ones you wish to backup (3). Next, you will have to select a directory to which to save the backups. It’s best to have a clean directory (4). Press OK, and backup will start with a rudimentary progress bar to keep you company (5). When the backup is done, you will be presented with a status window with the backup results and possible errors (6). The backup files have the same naming convention used in this article. To backup a Flash NAND as well you need to re-run the tool and select it from the available disks. You can save the resulting file in the same directory as the partition backups.

Figure 6 - Backup steps
Figure 6 – Backup steps

To perform a restore, start the tool in a terminal, select “Restore partitions”, then select OK (1):

$ sudo odroid-backup.pl
You will have to select the directory holding your valuable backups and select OK (2). In the resulting window, select which partitions you wish to restore from the backup and select OK (3). Note that the partitions are restored in the same order as they were on the original disk, which means that partition 1 will be the first partition, and so on. In the last window, you will be asked on which drive to restore the data (4). Enjoy watching the progress bar progressing (5), and in the end you will have a status window with the restore results (6). The log file is also saved in /var/log/odroid-backup.log.

Figure 7 - Restore steps
Figure 7 – Restore steps

Known limitations

If you backup an eMMC for XU3/4, the hidden sectors (/dev/mmcblk0boot0, /dev/mmcblk0boot1) are not backed-up/restored. These blocks contain parts of the UBoot loader. When restoring a backup on an SD card or on a new eMMC, the board might boot with an older UBoot version (stored before the first partition). As a result of this the UBoot environment might be incomplete (e.g. there is no ${board_name} set), and booting might be different than normal (network might be missing). Once you do boot it is recommended that you reinstall uboot with this command on the new card:

$ sudo apt-get install --reinstall uboot
As you might suspect, no piece of software is free of bugs, but hopefully this six step script will have its uses. This script has some shortcomings, such as the zenity windows not always displaying the instruction text, which is why I added the title bar. There is also no validation of the backups or restores. You will have to review the log to verify that the backup or restore operation completed successfully. One other limitation is that FAT partitions need to be manually unmounted before backup, although Ext2/3/4 can be backed-up live. Finally, the sfdisk utility on Ubuntu 14.04 doesn’t support JSON output, so it will not work there, although I can add support if needed. The program was tested by backing up and restoring several official Hardkernel Linux and Android images, as well as triple-boot images, and so far everything seems to work. Ideas for improvement and patches are welcome as always on the support thread at http://bit.ly/2bEyFzl.

Be the first to comment

Leave a Reply