Thursday, June 30, 2011

Image storage "on the road"

As I've mentioned in the past (and as anyone who's lost files knows) it's essential to have reliable backup systems: fundamentally you need to have copies of any file you care about, and be able to find them when you need them! On each of my photography workshops and expeditions I usually end up fielding questions about my own system (which extends to backups in the field).

I currently use a MacBook Pro as my main computer. In the office (my desk office) it gets connected to a nice 24" LCD and an array of Firewire hard drives (as well as a NAS fileserver) and when on the road it just has its two internal drives plus one or two portable drives. With two high-speed internal drives, 8 GB RAM and a fast Core i7 processor it's not a sluggish machine at all.

The primary copies of all my files live on either the internal drives or on several of those external drives (some of them RAID disks for extra reliability). In the office a daily backup updates a secondary copy on other external drives, and a weekly backup (with some specific drives connected in SATA "docks") updates another secondary copy. Every month that "weekly" backup set gets swapped with another set of drives stored off-site.

In this way I've got from 2 to 4 copies of every file. By the way, the Time Machine disks for the office machines also get backed up offsite every month.

All this is well and fine when I'm near my desk, but when I'm working out of my "virtual offices" (whether it's a hotel room, a ship cabin, or the back of a car) I still need some level of secure storage. The system I use is just an extension of the above one.

I have an external portable Firewire drive split into two partitions: one for Time Machine, and the other to contain a secondary copy of the data on my laptop's internal drives. So while that drive is connected Time Machine does hourly backups of the operating system, applications, in fact everything other than my Lightroom catalogs, my photo/video storage, and the "scratch" folders. At the same time I can run a backup to update the mobile secondary copy of those files. The photo/video folders get synchronized to the external drive, along with the Lightroom catalogs (although without the review databases: they take too long). I can do this daily, or as often as I want. In fact I have things set up so a backup is automatically started after importing a memory card's contents. Because only new or changed files are copied, the backups are quite fast.

I also have a (physically) small USB drive for an extra secondary copy of the photos/videos and catalogs. I connect this at least once every few days to keep that copy updated. Depending on how much material I've generated this can take a while to update (especially given the performance penalty of USB compared to Firewire 800) but it either happens in the background or when I'm asleep.


Laptops at sea. That's me on the right.
Image by Forrest Brown
Altogether this gives me a lot of reliability, and a lot of peace of mind. If I need to access the backup of my photo folders, I just need to copy the files back from one of the secondaries (they have the same folder structure). If I need to go back to an earlier Lightroom catalog I can do the same. In fact the backups of those also include any snapshot backups Lightroom has made of the .lrcat file during its internal backups. I may need to regenerate Lightroom previews, but as long as the original images are safe Lightroom will take care of that for me.

In fact if my laptop burst into flames I would be able to restore the entire machine (except for the contents of the scratch folders) onto a new disk from that external backup drive. I have done this in the past (although without the bursting into flames bit!). The Time Machine partition on the external drive also includes a copy of the OS X install DVD so I can boot directly off it and then restore from the Time Machine archive. Copying back the other files is a simple manual process. If I was in the middle of nowhere and the laptop's internal drives failed, I could even re-format one of the external drives as a boot drive and get things back up and running.

I especially make sure that the USB copy is updated before heading off on significant travel such as flights, as I can swap each drive with a travelling partner and we can carry a backup for each other. I do try to ensure I don't have both secondaries connected at the same time (to avoid any unforseen accidents during the backup).

I have at times worked with just a single backup copy while on the road, but there was a time when that involved multiple drives. Now that I can fit a complete backup onto a single drive, it's easier to have dual backups. Currently I'm travelling with room for about 600 GB of new files on each trip (as long as I transfer files off to the office drives before leaving) and even on our 4-week March/April 2011 voyage to Antarctica, the Falkland Islands, and Patagonia I "only" generated ~450 GB (made up of RAW and video files, along with quite a few TIFF composite panoramas). If I needed more storage I could simply add another external drive (well, 3: one primary and two secondaries) but travel is much simpler without all those extra drives in my carry-on luggage! So far that's been plenty of storage for each trip I've been taking.


Sea freezing around tabular berg, Ross Sea (A2_076125)
Photographed from 11 km up (through a plane window)

RAID

I find it interesting to read articles by others including by my friend Michael Reichmann of portable drives such as the ToughTech Duo QR which is a portable Firewire RAID drive with two internal disks. That does look like a very interesting device. It provides peace of mind in that you're protected from the risk of a single drive failing. However, it doesn't provide complete protection and in fact introduces new opportunities for error. If the ToughTech chassis was to fail, it's not clear whether you could access the data by connecting the drives to ordinary SATA interfaces (either inside the laptop or in other USB/FW cases). There's probably additional RAID formatting that would get in the way. If so, to continue to access your data (e.g. while on expedition) you'd need to have an extra ToughTech chassis as a backup).

Michael points out that it doesn't protect against theft or physical damage, but it also doesn't protect against yourself (e.g. deleting the wrong file). For that there is no substitute for having a separate copy of the files (or fancy systems with versions/snapshots, but they have their own drawbacks for portable use and large files).
He correctly points out that you can take a snapshot by taking out one of the two drives and putting in a third. The ToughTech will rebuild all the RAID redundancy, and you can indeed keep working while it does this. But what he doesn't point out is that until it finishes the resync then the drive is not redundant. In these situations RAID devices typically treat the new drive as completely out of date: it will have to re-copy (or at least check) every block on the disk. Assuming an average throughput of 60 MB/s across the entire disk, for a 1 TB disk this would take over 4.5 hours. In any case unless the ToughTech does something special, it's a process that is going to take a very long time. And it's probably not a process that likes to be interrupted.

Even disregarding the $500 price of the ToughTech device (plus the costs of the disks to go in it) I'm much happier dealing with JBODs (Just Boring Old Disks) when travelling. As a technogeek I really want to love the device. In fact if price was no object I might travel with two of these, but there's a lot to be said for systems built out of simple and common components!
Continue reading "Image storage "on the road""...

Wednesday, June 29, 2011

Australian Professional Photography Awards 2011

I was very pleased when last weekend one of my printed images was awarded a Silver Distinction award in the 2011 Canon AIPP "APPAs"! This was in the Science, Nature, and Environment category.


Lonely Gentoo Penguin (A2_052751)

This photo was taken in Antarctica last November on our Antarctica 2010: A Photo Odyssey expedition (which I was the leader of: see LuminOdyssey.com for information about future expeditions). I got to watch the APPA judging live, and was pleased to also see some photos from another participant in that expedition get awarded!

On a related note, the Epson International Photography Pano Awards were judged last month, and two of my images from the March LuminOdyssey trip to Antarctica achieved Bronze Awards. These are only tiny web versions, but you'll get an idea:


Iceberg and Polar Pioneer (A2_066623)
Six frames, with lots of overlap


Point Hennequin, Antarctica (A2_067862)
Three frames
On that last voyage there was a bit of a panorama craze going on after one of my lectures covered the processing involved. An earlier article on this blog goes into some detail.
Continue reading "Australian Professional Photography Awards 2011"...