Sunday, March 20, 2011

Setting sail today

In case you haven't been watching, daily updates are happening over at the LuminOdyssey blog. Today we set sail for Antarctica again, leaving the sunny (!) port of Ushuaia.

There'll probably be no updates here until mid-April, other than on the map.



Continue reading "Setting sail today"...

Monday, March 14, 2011

On the road again...

As I mentioned over at LuminOdyssey.com, I'm about to head off on another trip across the globe, to teach and lead a photography expedition to Antarctica, the Falkland Islands, and Patagonia. I'll be back home in a month's time. While I'm away, the Where's David map should be updated daily, and there should also be regular (if brief) updates to the LuminOdyssey blog. I'm expecting to publish some material here while I'm travelling, but once we head off from South America to Antarctica (on the last voyage of the season) the Internet connectivity will be "somewhat limited"!

Of course even while I'm travelling, you can enquire about and book to join us on our forthcoming photography trips to Svalbard (think "Polar Bears"), Kenya (including the Masai Mara), and November's Aurora Expeditions South Georgia/Antarctica trip.

In a separate bit of news, the Autumn edition of Diabetes Connect magazine (from the Australian Diabetes Council) will be published later this month. The travel feature in this edition features photos and stories from me! In case you didn't know, I've been insulin-dependent since high school.
Continue reading "On the road again..."...

Friday, March 11, 2011

Measuring card speeds: not always simple!

After my February article where I measured the speed of various CompactFlash cards in my cameras, I've had some queries about my testing method. It turns out that while measuring the speed of the cards in my EOS 7D and 5DmkII cameras is easily achieved in the field, it's harder to do with some other models of camera!

Recently I met up with my friend Syd Low for coffee and he brought his Nikon cameras and some recent CF purchases along (Syd snapped this photo with his iPhone while we chatted). We made some interesting discoveries.

On my Canon cameras the camera seems to write out the data to the card as quickly as it can. If you hold down the shutter button for a continous burst, the card activity light comes on immediately: as soon as there's data in the buffer the camera's busy emptying it. When you eventually release the shutter the activity light remains solidly on for a while. Eventually the LCD lights up with a preview image, and a while afterwards the activity light goes off. I've been simply using a stopwatch to time the interval between the light going on and turning off, and then dividing the number of megabytes by the number of seconds. Conveniently the Format screen in the EOS menu system tells you how much data is on the card before formatting, making that measurement very easy in-camera. The fastest speed I've measured on my EOS 7D this way has been 55 MB/s using Lexar 600x cards, and the results are consistent between runs.

Syd and I did some experiments with the Nikon D3s, and the first difference is that it takes a while for the camera to start writing files to the card. Apparently the camera's willing to wait a while until there are a few shots in the buffer. No-one seems to complain about this: probably because the buffer's large enough to handle over 40 RAW frames without worrying about the card speed (the original D3 had half this buffer, although a buffer upgrade to match the D3s is available). Where D3s shooters care about the card speed is in how quickly the buffer has room for more shots. However, measuring this accurately requires a different method than I've been using on my EOS cameras.

The other significant difference is that once the Nikon camera's emptied out "enough" space in the buffer, it seems to slow down writing to the card. The activity light starts flickering instead of being solidly on. This means that if we just measure the total time, we'll only be determining the average writing speed. In light of this, I'm left wondering how sites like RobGalbraith.com have actually measured useful numbers for these cameras.

We do have a way of measuring the speed of these cards, but it needs a better setup than we had in the cafe! We can record audio of the camera firing, and after the buffer fills up we can accurately measure the frame rate. By examining the card on the computer we can determine the file sizes, and from these calculate the card writing speed. This should provide a valid indication of how quickly the camera will reclaim buffer space using each card model. This measurement method does also work with my EOS cameras, but the stopwatch technique is just more convenient.

For example, look at this graph of data from my earlier measurements of the Lexar 16GB 600x card in the EOS 5DmkII. It's a screen-grab from the Audacity software, and the dark area has just been selected for measurement:

That's 12 frames in 4.74 seconds (an average of 2.53 fps - remember this is after the buffer has filled). Those 12 images took up 214 MB (an average of around 18 MB: fairly small for the 5DmkII). 214 MB in 4.74 seconds comes to just over 45 MB/s, which matches the speed I measured for this card in this camera using the stopwatch method.

I expect that a similar method will allow us to reliably measure the throughput of cameras like the D3s. Being able to quantify the differences between cards can provide a better basis for purchase decisions than "they feel about the same".
Continue reading "Measuring card speeds: not always simple!"...

Friday, March 4, 2011

Memory card readers - pros and cons

More specifically, this post is about card readers for CompactFlash cards (as used in most DSLR cameras). It's a subject that comes up with every new group of students, so I'll just get it out of the way.

First some workflow recommendations, then an outline of my specific recommendations about current-model card readers to buy, and finally a story about my own experiences with a fast card reading setup.


Not all my photography is in cold weather!

To start with I recommend against using your camera as a reader. It can be tempting to plug your camera via USB into your computer and transfer the files that way, but try to resist. It's much better to use a standalone card reader. Drawbacks of using the camera:
    Get the damn flash off the camera (A2_043231) (Props to Rick!)
  • It can be very slow.
  • Depending on your camera model it may require special software.
  • It puts your camera at risk. The USB connector on the cameras is usually not designed to be connected/disconnected hundreds of times, and I have seen several cameras mysteriously fail when their USB cable is being connected. If something does go wrong it's much cheaper to replace a $30 card reader than a $2000 camera.
  • It uses your camera's battery. Battery life is getting better these days, but still it's unnecessary.
Advantages of using a standalone reader (usually connected via USB or Firewire, but integrated into some laptops):
  • It can be quite fast (depending on the combination of your card and reader models and the software you use).
  • It doesn't require any special software: the card appears to your computer as just another hard drive. At the least you can just copy the files off, but most people use "import" or "ingest" software to do this and put the files into the right destination, verify them, etc. Some cameras do work this way too.
  • If you have images spread across multiple cards it becomes a simple routine to sit down at your computer with a stack of cards and work your way through them. Using the camera it's actually easy to make the mistake of not downloading all your cards.
I do connect my cameras via USB to my laptop every now and then (e.g. when entering a new timezone) in order to set them all to exactly the same date/time (via the manufacturer's standard utility software) and sometimes to work in "tethered" mode, but that's unusual for me. If my card reader was broken and I had to transfer some files I might use the camera as a reader, but for me that's a step of last resort.


Contemplating his landscape... (A2_042860)
When selecting a card reader, there are two fundamental factors for me:
  • Speed. Many days I'm not especially fussed about this as I've only taken a small number of photos. But when I'm on a busy job and need to download several 8 GB and 16 GB cards before heading out again, speed is good. I'm not paying an assistant to download the cards for me while I shoot: I'm paying for my own time.
  • Reliability. It needs to Just Work. Every Time.


Shooting the sunset (A2_043248)
An important reliability issue with CompactFlash cards is not damaging the pins that slide into the holes on the end of the card. A bent pin in your card reader can stop the reader from working, but can also damage the card in a way that's likely to damage the pins in the next reader or camera the card is used in. The biggest cause of this damage is from the card not being perfectly aligned when it meets the pins, and this is a huge risk if the card reader you use has shallow slots. A good card reader will need the card inserted a long way (and thus be running straight) before it connects. Consider how deep the slot in your DSLR is!
SD/SDHC cards don't have this issue, but do have their own drawbacks of being much slower and of being small and easily lost (e.g. I've had one blown out of my hand by a strong wind).

Which model?

The speed champions of card readers are some of the ExpressCard models, although none of my machines have ExpressCard slots so those are out for me. But I am aware that many of these do have very shallow CF slots and I do hear repeated stories of damaged cards. There are a couple of models with deeper slots.

With external card readers the champions are the Firewire-800 readers such as those from SanDisk and Lexar (not the older Firewire-400 models) which have been measured [robgalbraith.com] with some cards at transferring over 80 MB/s (which is effectively filling the Firewire bus). Both of these devices have deep slots and I do recommend them, but they do have two drawbacks:
  • They seem to have been discontinued.
  • If you can find one, they're well over $100 each.
Note that most computers will struggle to do anything useful with 80 MB/s, unless they're using SSD storage. Some internal hard drives can write this fast, but most don't. If you're using external USB drives for storage then they become the bottleneck, and if you're using Firewire disks they'll be sharing the bus with the reader. On my dual-drive MacBook Pro I have a dedicated 750GB internal SATA drive for image storage, and can maintain around 60 MB/s writing to it (that's with test imports from a copy of a card on the other internal SATA drive, so including all the import software's overhead).

Getting down to business, the reader I currently use (since 2008) and recommend is the Lexar Professional Dual-Slot Card Reader. It connects via USB, has an SDHC slot as well as a deep CF slot (both cards can be connected at once: they appear as separate drives). Depending on the cards used, the SD throughput can be up to 20 MB/s or so, whereas many CF cards with UDMA can transfer over 30 MB/s in this reader.
At B&H Photo this reader is US$32 (plus shipping) but it is available at some Australian retailers for around the AU$60 mark.

There are also some other notable choices to mention, most of them in a good light...

I have an older SanDisk Extreme USB reader (shown on the right) which is a great device (although it was discontinued some years ago). Like the Lexar dual-slot reader it has both CF and SDHC slots, and the CF slot is deep. However as it came out before the days of UDMA it tops out at about 16 MB/s. I still have this reader although given the speed advantage of the Lexar reader it doesn't get a lot of use today.

Prior to that I used a SanDisk ImageMate CompactFlash reader. That version just had a single deep CF slot, but while it was a fast and reliable device for its day, today the speed just wouldn't compare.

Today SanDisk has the ImageMate All-in-One reader (shown above). This is comparable in speed to the Lexar dual-slot reader, and it has slots for many other card types. But its CF slot is very shallow, and for this reason I have to recommend against it (or at least suggest that users be very careful when inserting CF cards).

The only other current USB reader I recommend is the white Lexar Multi-Card 24-in-1 USB Reader. It's a similar physical design to the dual-slot UDMA reader, but is wider due to the extra slots. The CF slot is nice and deep, and the only drawback of this reader is the speed. The fastest I've got out of it is 16 MB/s, presumably due to the lack of UDMA. For many uses it's still a quite respectable speed, and if the rest of your computer system can't do much faster than this then it might not be much of an actual limit, and the variety of card types supported might make it worthwhile overall.

There are lots of other USB CF readers on the market, but most of them have very shallow CF slots, and many of them aren't particularly fast. The Lexar readers get my hearty recommendation. By the way, I've been using SanDisk cards for years and only recently got my first Lexar card, but the readers don't care: they'll work with all brands.

Overall Speed

As I mentioned above, the raw speed of the reader and card won't be that important to you if the rest of your system isn't going to keep up. Having fast disk storage is important, but is not the only factor. Some software will do multiple checks on each file as it's transferred, and this can slow things down. The worst case is something like Adobe Bridge's Downloader or Adobe Lightroom in the mode where they convert each file into DNG format. In this case a fast reader will just end up waiting for the CPU! Software that copies each file to two destinations (e.g. for backups) can also introduce delays.

My own workflow involves copying from the cards to just the internal storage drive, followed by a bulk copy (via synchronisation) to an external FW800 backup drive. I use a standalone program (part of my PteroFile tools) which copies the files from the memory card into a specified storage area, renaming the files as it goes and assigning unique IDs into the metadata.
When it's finished it starts the synchronisation (which internally uses the rsync tool included in OS X) in the background. While that's running I tell Lightroom to import all the new files in my storage area. As a final refinement, if I have multiple cards mounted on the machine I can run multiple copies concurrently.


Huddled King Penguins (A2_041495)
Experiences from my last Antarctic voyage

On my most-recent Antarctic trip (November 2010) I got to compare the speed of this system to that of just using Lightroom 3 to copy the files from the card reader. My cabin-mate was using similar cameras to me (EOS 5DmkII) but often shot a third to a half the number of frames as I did on any given outing. We were both shooting only in RAW (with an occasional video recording).
We'd get back to the cabin at about the same time, and started our imports not far apart. Our laptops were set up on the desk ready to go, so we chucked our cards into the readers and started the imports before we got out of our wet-weather gear.

Our laptops were both uni-body 15" MacBook Pros, and we were both using the Lexar dual-slot USB card readers. His primary image storage was on an external FW800 drive whereas mine was on an internal drive. I had helped him configure his machine, so I'm confident there weren't any hidden bottlenecks. He was using Lightroom's import function to copy files from the card reader (without concurrent backups) and then used the synchronisation component of PteroFile to backup to an external FW400 drive daisy-chained from the FW800 drive.

I usually had at least two CF cards to import (having been using two DSLRs) and sometimes some shots from another camera on an SD card. After the really big outings I might have a stack of 3 or 4 CF cards to import (usually they were 8 or 16 GB each). All my cards were SanDisk Extremes of various models, most with UDMA.
He was usually just downloading from one (sometimes two: he had a second camera) 32 GB SanDisk Extreme card.

Running OS X's Activity Monitor application allows us to easily monitor things like the aggregate disk I/O rates during the import. The Lightroom-only system could be seen to have bursts as it copied each file, but could never really keep the throughput up.

My system maintained from 16 to 29 MB/s throughput (only dependent on the cards being used) and when I started a concurrent read from an SD card (in the MBP's internal SD reader) the throughput rose by another 12-16 MB/s. On the occasion I borrowed his card reader and had my system downloading two CF cards concurrently, the throughput rose to a steady 45-50 MB/s (obviously at this point the USB bus was saturated, but it's pretty close to the maximum disk throughput anyway). Note that both readers were connected directly to USB ports without using a hub (which would drop the speed significantly by introducing latencies). The MBP only has two USB ports, but as my backup drive is on the Firewire bus this isn't an issue for me in the field.

Usually I was able to have all my hundreds (sometimes ~800 at a time) of photos downloaded and backed up to the external drive, and have the Lightroom preview-generation well underway long before my cabin-mate's system had even finished copying ~300 files from his CF cards. Once that had finished he was able to start his backup in the background (using the same synchronisation tool) while Lightroom was generating previews. But once my backups had finished I could disconnect the backup drive, suspend the laptop and carry it out to the common areas of the ship to talk to other people while the preview generation continued. Or just put the cards back into my cameras, reformat them, and go back outside to photograph.


That's not a mountain! (A2_045969)
Some people never shoot enough frames to worry much about the speed, but some of us do. It really was a joy to have the card imports and backups quickly dealt with so I could get on with the real work. Even on days where I took over three thousand ~25 MB RAW frames!
With the continual increases in RAW file size, card size, laptop processor power, affordability of SSD drives, and the recent introduction of the USB 3.0 and Thunderbolt I/O busses, it's all a moving target. I'm sure faster options will come along, but for now I'm quite happy.

If I want even more speed soon I could buy a second Lexar reader to use for concurrent downloads and still be ahead of the cost of a Firewire reader. I could use my existing SanDisk Extreme USB reader and the Lexar in parallel and almost saturate the USB bus, but if I had a stack of cards to download and ended up with a large card in the slow reader, the fast reader would finish first and then we'd be stuck waiting for the big card to finish. Workable, but easier if both readers are similar.

By the way, for people looking into dedicated "ingest" software for copying files off their flash cards, you can either wait for me to add a configuration GUI to PteroFile (at the moment it needs to be configured using commands in Terminal, and only works on OS X) or you can look into Marc Rochkind's ImageIngester.
Continue reading "Memory card readers - pros and cons"...