Finder Folder Actions not being triggered when files are added with rsync

A couple weeks ago I wrote about how I was automatically capturing the photos and videos my kids’ daycare emails to me and importing them into Photos.app. The major pieces of that script worked fine – parsing the emails, downloading the images, and then rsync’ing them down to my Mac every hour.

But what was failing was the Finder Folder Action I setup that was supposed to import the files into Photos.app whenever new ones were added to that folder.

For some reason, the Folder Action would only occasionally fire. Maybe for one out of every ten items. Sometimes, if I navigated to the folder in the Finder, the action would kick-in and import everything. But sometimes not.

All I can think is that because the files were being added to the folder via rsync – using some BSD-like filesystem APIs instead of the higher-level macOS ones – the Folder Action was never being triggered. Again, it occasionally worked, but mostly failed. So I could be entirely wrong about all of this.

Anyway, I rewrote the whole thing to just run an AppleScript every hour via cron, which handles the whole process itself. Since making that change it’s been working perfectly.

Maybe someone reading this will find it helpful.

Backing Up Shared iCloud Photo Albums and Where to Find Them on Disk

In my quest to backup ALL THE THINGS, I turned my attention earlier this week to the shared iCloud Photo Albums my friends and family use to pass around photos and videos of our kids.

All of the items in my iCloud library (and my wife’s library) are combined and backed up to Google Photos automatically. For better or worse, Google Photos is the “source of truth” that contains all of our archives and is sorted into albums. It’s the backup I’d use to restore if iCloud ever goes belly-up. (And I have a redundant backup of Google Photos itself in case Google ever loses my data.) And the actual Photos.app library on my iMac is backed up to Backblaze for good measure, too. So the photos we take are covered.

But there are a ton of great memories of our kids snapped by other people. Those only reside in the shared iCloud photo streams. How do I back those up?

Ideally, Photos.app on Mac (or iOS) would have a preference to automatically import shared items taken by other people – and then those would feed into Google Photos. But that doesn’t exist. I could manually save-to-my-library new items as they’re shared, but that’s error prone and not scalable.

Also, what about the 2,000+ previously shared photos? I thought I would be clever and just select-all on my Mac and drag them into my main library, but after doing a few quick tests I realized Photos.app isn’t smart enough to not duplicate the photos I took and shared when importing. (This is likely due to Apple scaling-down and stripping out metadata of shared items.) And there’s no way to sort by “other people” or build a smart album of “photos taken by other people” to filter out your own images when importing.

So, I decided to do some digging.

The first step was to locate the shared albums on disk. I searched my main Photos Library.photoslibrary bundle, but couldn’t find them inside. A quick glance through ~/Application Support/ didn’t turn up any obvious hiding places either. That’s when I fired up DaisyDisk to search for large (10GB+) folders.

Success!

For my own reference and for anyone else who comes across this post after googling unsuccessfully, iCloud’s shared photo albums are stored here:

~/Library/Containers/com.apple.cloudphotosd/Data/Library/Application Support/com.apple.cloudphotosd/services/com.apple.photo.icloud.sharedstreams/assets/

Each shared album is inside that folder and given a UUID-based folder name. And inside each album, every shared photo/video is itself inside its own UUID folder name. It’s quite impenetrable and obviously not meant for users to poke around, but the programmer in me understands why it is this way.

At the top level is a Core Data database. I thought I might get clever and explore that to see if I could extract out the metadata of the shared items and use it to help me write a “smart” backup script (that perhaps imports other people’s photos directly into Photos.app) instead of just taking the brute-force approach and backing up the entire album as a dumb blob, but I haven’t had enough time yet to investigate.

So until I find the time to build that “smart” approach, I’m going about it the dumb way and nightly syncing everything to B2. It’s not ideal, but it covers my needs for now.

Fixing a Broken Service With a Tiny Bit of Automation

This post is a nice, unintentional follow-up to yesterday’s one about backing up all of my family’s photos and home videos. Anyway…

My kids go to a fantastic daycare. My wife and I couldn’t be happier. The teachers are wonderful, they love our children, and our kids adore them, too. But, the third-party service the school uses to communicate with parents is absolute horseshit.

I won’t say what the service is because I don’t want to give them free publicity or maybe even alert them to what I’m doing, but if you have daycare-aged children, you probably know it. All the schools use use it.

All of the teachers carry around iPads in the classroom. They use this third-party app to check-in / check-out the children, capture photos and videos throughout the day, record what they ate for lunch and how long they napped, and (if your child is young enough) document their diaper changes. At the end of the day, after we sign them out of school, my wife and I get an automated email from the service with a summary of each kid’s day. But what we look forward to most are the photos/videos they take of our kids that get sent to us as they happen. When you’re slogging through a boring day at the office, seeing a happy picture of your kid on the playground with their friends is awesome.

Now, let me be clear. The service works. Mostly. I mean, it functions adequately. But it’s a horrorshow of app / website design.

It looks like something straight out of 2009-era iPhone development. It’s difficult to use. Crashes frequently. And from what the teachers have told me, the educator version isn’t any better.

Luckily, you don’t have to use their app. You can opt-in to get all the updates and photos sent to you via email, which is what my wife and I do. But, the HTML emails they send have never rendered properly in any email client – desktop or web – that I’ve tried. But that’s fine. They may not be pleasant to look at, but I can read the information in them.

My biggest gripe is that we often want to save any particularly good photos of our kids and share them with the grandparents. You can’t save the photo out of the email, because the embedded image is cropped to a square for some strange reason. You need to first tap on the image to load the full version in a browser and download it from there. Fine. But, any photo that contains any child in addition to your kid – like a group shot with a friend – is displayed with a transparent div on top of it so you can’t download it (at least on a mobile device) for privacy reasons. Look, I get it. Some parents might not want other parents unintentionally posting photos of their kids to social media. But it’s still annoying. It just forces us to take – and then crop – a screenshot. Also, the emails containing videos, which are often the best ones, can’t be downloaded at all.

Last night I got frustrated enough to finally do something about this.

I use Postmark to send all of my company‘s transactional emails. They’re fantastic for sending emails, but one feature they offer that I’ve never taken advantage of is handling inbound emails.

You can forward any email to a secret address they provide you, and they’ll parse the email and POST all of its information as a helpful JSON object to whatever URL you specify.

So, I setup a webhook in their control panel pointing to a PHP script on my web server. Then, I told Fastmail to forward all emails from the daycare service to my secret Postmark email address. You can see where this is going, can’t you?

When they send a new email to my server, the PHP script finds the link in the email’s HTML content that points to the full version on the service’s website. It then downloads that web page, parses out the URL to the full image, downloads that, and saves it into a folder on my server. This works for videos, too.

The PHP script I wrote is specific to the service our daycare uses, but if you’re curious, here it is…

That’s the first step.

Next, my iMac at home runs a script every hour to download any new photos or videos from my server and puts them in a folder inside my Mac’s “Pictures” folder. When that happens, a folder action I built with Automator automatically imports them into Apple’s Photos.app, where they’re synced to all of my mobile devices and iCloud. Soon after that, Google Photos on my iPhone will detect the new items and archive them in Google’s cloud, where they’re backed-up and made available on my wife’s phone as well.

Here’s a photo of the Automator action. It couldn’t be simpler – just one step…

The result? We get to see all of our kids’ photos as they happen, in the nice Photos app on our phones – rather than digging through the service’s crappy emails. And, sharing the pictures with the rest of our family is a one-tap process – even for the videos which previously weren’t available at all!

Backing Up Everything (Again)

This will take a while. Bear with me.

I’m obsessive about backing up my data. I don’t want to take the chance of ever losing anything important. But that doesn’t mean I’m a data hoarder. I like to think I’m pragmatic about it. And I don’t trust anyone else to do it for me.

From around 2006 to 2012, I kept a Mac mini attached to our TV with a Drobo hanging off the back. It had all our downloaded movies on it. And every night it would automatically download the latest releases of our favorite TV shows from Usenet so my wife and I could watch them with Plex the next day. It worked great, and all the media files were stored redundantly across multiple hard drives with tons of storage space. (Would it survive a house fire? No. But files like that weren’t critical.) But with the rise of streaming services and useful pay-to-watch stores like iTunes, now I’d rather just pay someone else to handle all of that for me. So, I don’t keep any media files like that locally any longer.

But my email? My financial and business documents? My family’s photo and home video archive? I’m really obsessive about that.

For most of my computing life, all of that data was small enough to fit on my laptop or desktop’s hard drive. In college, I remember burning a CD (not a DVD) every few months will all of my school work, source code, and photos on it for safe keeping. The internet wasn’t yet fast enough to make backing up to a cloud (were clouds even a thing back then?) feasible, so as my data grew I just cloned everything nightly to a spare drive using SuperDuper and Time Machine. It worked for the most part. Sure, I still worried about my house catching fire and destroying my backups, but there really wasn’t an alternative other than occasionally taking one of the backup drives to work or a friend’s house.

But then the internet got fast, really fast, and syncing everything to the cloud became easy and affordable. I was a beta user of Gmail back in 2004. I was an early paid subscriber of Dropbox since around 2008. All of my data was stored in their services and fully available on every computer and – eventually – mobile device. At the time, I thought I had reached peak-backup.

I was wrong.

Now we have too much data. My email is around 20GB. My family’s photo library is approaching 500GB. That’s more data than will fit on my laptop’s puny SSD. It will fit on my iMac, but it leaves precious available space for anything else. I could connect external drives, but that gets messy and further complicates my local backup routine. (Yes, Backblaze is a good, potential solution to that.)

Another problem is that most of our data now is either created directly in the cloud (email, Google Docs, etc) or is immediately sent to it (iPhone photos uploaded to iCloud and/or Google Photos), bypassing my local storage. If you trust Google (or Apple) to keep your data safe and backed up, that’s great. I don’t. I’ve heard too many horror stories about one of Google’s automated AI systems flagging an account and locking out the user. And with no way to contact an actual human, you’re dead in the water along with all your data. Especially if you lose access to your primary email account, which is the key to all your other online accounts.

So, I need a way to backup my newly created cloud data, too. This is getting complicated.

First step. My email. This is easy. Five years ago I setup new email addresses for my personal and business accounts with Fastmail. They’re amazing. I imported my 10+ years worth of email from Google (sadly, my pre-2004 college email and personal accounts are lost to the ether), setup a forwarding rule in Gmail, and with the help of 1Password, changed all of my online services to use my new email. It took about a month to switch everything over, but now the only email coming to my old Gmail address is spam. Fastmail keeps redundant backups of my email. And I have full IMAP copies available on multiple computers in case they don’t. And if something ever goes wrong, unlike Google where their advertisers are the customer – and I’m the product – I pay Fastmail every month and can call up a live human to talk to.

Source code. I’m a paying GitHub customer. Everything’s stored and backed up there. But still, what if they screw up. I ran a small, self-hosted server with GitLab on it for a while instead of GitHub and set it to backup all my code nightly to S3. That worked great. But, I like GitHub’s UI and feature set better. Plus, it’s one less server I have to manage. So, where do I mirror my code to? (Much of my code is checked out locally on my computer, but not all of it.)

Back in 2006, my boss at the web agency I was working at told me about rsync.net. They provide you with a non-interactive Unix shell account that you can pipe data to over SFTP, rysnc, or any other standard Unix tool. You pay by the GB/month, and they scale to petabyte sizes for customers who need that. So, I signed up and used them to backup all of my svn (remember svn?) repos. With the rise of git and switch to GitHub, I cancelled my account and mostly forgot about them.

But, aha!, I now have new data storage problems. Rsync.net could be a great solution again. So, I re-signed up and setup my primary web server to mirror all of my GitHub repos over to them each night. Here’s the script I’m using…

Next up, important documents. Traditionally, I’ve kept everything that would normally go in my Mac’s “Documents” folder in my Dropbox account. That worked great for a long time. But once I started paying Google for extra storage space for Google Photos (more on that later), it felt silly to keep paying Dropbox as well. So, after 10+ years as a paid subscriber, I downgraded to a free account and moved everything into Google Drive. Sure, it’s not as nice as Dropbox, but it works and saves me $10 a month.

Like I said above, I mostly trust Google, but not entirely. So, let’s sync my Google Drive’s contents to rsync.net, too. Edit your Mac’s crontab to add this line…

30 * * * * /usr/bin/rsync -avz /Users/thall/Google\ Drive/ user@server.com:google-drive

Also, I keep all of the really important paperwork that would normally be in a fire safe in my garage in a DEVONthink library so I can search the contents of my PDFs. It’s synced automatically with iCloud and available across my mobile devices. But still, better back that up, too.

45 * * * * /usr/bin/rsync -avz /Users/thall/FireSafe.dtBase2 user@server.com:

So, that’s all of my data except for the big one – my family’s photo and home video archives.

For a long time I kept all my family’s archives in Dropbox. I even made an iOS app dedicated to browsing your library. I could have stuck everything in Apple’s Photos.app where it’s available on my devices via iCloud, but that’s tied to my Apple ID. My wife wouldn’t be able to see those photos. Plus, any photos she took on her phone would get stored in her iCloud account and not synced with the main family archive. So, we used the Dropbox app, signed-in to my account, to backup our phones’ photos.

But, like I said earlier, our photo and video library become to big to comfortably fit in Dropbox. Plus, Google Photos had just been released and it was amazing. Do I like the thought of Google’s AI robots churning through my photos and possibly using that data to sell me advertisements? No. But, their machine-learning expertise and big-data solutions make it really hard to resist. So, I spent a week and moved everything out of Dropbox into Google Photos.

Now everything is sorted into albums, by date, and searchable on any device. I can literally type into their search box “all photos of my wife’s grandmother taken in front of the Golden Gate bridge” and Google returns exactly what I’m looking for. It’s wonderful.

My wife’s phone has the Google Photos app installed with my account on it so every photo she takes gets stored in a shared account we can both access and view on all our devices.

But what’s the recurring theme of this blog post? That’s right. I don’t fully trust any cloud provider to be the only source of my data. Someone clever said “the cloud is just someone else’s computer.” That’s exactly correct. If your data isn’t in at least two different places, it’s not really backed up.

But how do I backup my 500GB+ of photos that are already in Google’s cloud? And then how do I keep new items recently added synced as well?

As usual, I tried to find a way to make it work with rsync.net. I found a great open-source project called rclone. It’s a command line tool that shuffles your files between cloud providers or any SFTP server with lots of configurable options and granularity.

First off, even if rclone does do what I need, I can’t just run it on my Mac. My internet is too slow for the initial backup. I need to use it on one of my servers so I have a fast data center to data center connection between Google and rsync.net.

Getting it setup on one of my Ubuntu servers at Linode was a simple bash one-liner. Configuring it to then work with my Google and rsync.net accounts was just a matter of running their easy-to-use configuration wizard.

Note: rclone doesn’t support a connection to Google Photos. Instead, you need to login to Google Drive on the web and enable the “Automatically put your Google Photos into a folder in My Drive” option in Settings. (And also tell your Google Backup & Sync Mac app not to sync that folder locally – unless you have the space available – I don’t.) Then, rclone can access your Google Photos data via a special folder in your Drive account.

With everything configured, I ran a few connection tests and it all worked as expected. So, I naively ran this command thinking it would sync everything if I let it run long enough:

rclone copy -P "GoogleDrive:Google Photos" rsync:GooglePhotos

Things started out fine. But eventually, due to Google API rate limits, it was quickly throttled to 300KB/sec. That would have taken MONTHS to transfer my data. And, the connection entirely stalled out after about an hour. I even configured rclone to use my own, private Google OAuth keys, but with the same result. So, I needed a better way to do the initial import.

Google offers their Takeout service. It lets you download an archive of ALL your data from any of their services. I requested an archive of my Google Photos account and eight hours later they emailed me to let me know it was ready. Click the email link to their website, boom. Ten 50GB .tgz files. Now what to do with them?

I can’t download them to my Mac and re-upload them – that’s too slow. Instead, I’ll just grab the download URLs and use curl on my server to get them, extract them, and sync them over.

I don’t have enough room on my primary web server – plus I don’t want to saturate my traffic for any customers visiting my website. So, spin up a new Linode, attach a 500GB network volume, and we’re in business. Right? Nope.

The download links are protected behind my Google account (that’s great!) so I need a web browser to authenticate. Back on my Mac, fire up Charles Proxy and begin the downloads in Safari. Once they start, cancel them. Go to Charles, find the final GET connection, and right-click to copy the request as a curl command including all of the authentication headers and cookies. Paste that command into my server’s Terminal window and watch my 500GB archive download at 150MB(!!)/sec.

(Turns out, extracting all of those huge .tgz files took longer than actually downloading them.)

Finally, rsync everything over to my backup server.

And that’s where I currently am right now. Waiting on 500GB worth of photos and videos to stream across the internet from Linode in Atlanta to rsync.net in Denver. It looks like I have about six more hours to go. Once that’s done, the initial seed of my Google Photos backup will be complete. Next, I need a way to backup anything that gets added in the future.

Between the two of us, my wife and I take about 5 to 10 photos a day. Mostly of our kids. Holidays and special events may produce a bunch more at once, but that’s sporadic. All I need to do is sync the last 24 hours worth of new data once every night.

rclone is the perfect tool for this job. It supports a “–max-age=24h” option that will only grab the latest items, so it will comfortably fit within Google’s API rate limits. Once again, setup a cron job on my server like so:

0 0 * * * rclone copy --max-age=24h "GoogleDrive:Google Photos" rsync:GooglePhotos

And, that’s it. I think I’m done. Really, this time.

All of my important data – backed up to multiple storage providers – and available on all of my and my family’s devices. At least until the whole situation changes yet again.

A few more notes:

All of my web server configuration files are stored in git. As are all of my websites’ actual files. But, I still run an hourly cron job to backup all of “/var/www” and “/etc/apache2/sites-available” to rsync.net since it’s actually such a small amount of data. This lets me run one command to re-sync everything in the event I need to move to a new server, without having to clone a ton of individual git repos. (I know I need to learn a better devops technique with reproducible deployments like Ansible, Puppet, or whatever the cool tech is these days. But everything I do is just a standard LAMP stack (no containers, only one or two actual servers), so spinning up a new machine is really just a click in the Linode control panel and couple apt-get commands and dropping my PHP files into a directory.)

My databases are mysqldump’d every hour, versioned, and archived in S3.

All of the source code on my Mac is checked out into a single parent directory in my home folder. It gets rscyn’d offsite every hour, just in case. Think of it as a poor man’s Time Machine in case git fails me.

I do a lot of work in The Omni Group‘s apps – OmniFocus, OmniOutliner, and OmniGraffle. All of those documents are stored in their free WebDAV sync service and mirrored on my Mac and mobile devices.

All of my music purchases have gone through iTunes since that store debuted however many years ago. I can always re-download my purchases (probably?). Non-iTunes music ripped from CDs long ago, and my huge collection of live music, is stored in iTunes Match for a yearly fee. A few years ago when I made the switch to streaming music services and mostly stopped buying new albums, I archived all of my mp3s in Amazon S3 as a backup. I need to set a reminder to upload any new music I’ve acquired as a recurring task once a year or so.

Also, I have Backblaze running on my desktop and laptop doing its thing. So yeah. I guess that’s yet another layer of redundancy.

Switching to Google Photos from a Dropbox Photo Library

Five years ago I went all-in and migrated my ancient iPhoto library to generic files and folders on disk inside of Dropbox. I wanted something I could access from anywhere, and – perhaps more importantly – was future-proof. I liked this solution so much I started writing a book about it and even made an iPhone app to help me view my library on the go.

My library’s structure worked like this…

/Photos/
    /_Albums/
        /2017-12 Aaron's 4th Birthday Party/
        /2017-12 Christmas in Chattanooga/
        /2017-11 Thanksgiving in Nashville/
        etc...
    /2018-01/
        /2018-01-01 12:45:02.jpg
        /2018-01-02 02:38:15.jpg
        etc...
    /2017-12/
    /2017-11/
    /2017-10/
    etc...

That worked great. It allowed me to keep album-worthy photos separate from all of those one-off day-in-the-life photos we take. It also let me quickly find any photo just by knowing the album or month it was taken in.

The problem was that – especially with all of my home videos now in 4k/60fps – I was running out of disk space. My library was over 300GB. I had plenty of storage space left in my 1TB Dropbox paid account, but not on my hard drive.

I was facing the decision of not keeping all of my files locally or running Dropbox off a larger external drive. Neither option made me very happy.

But then came Google Photos.

If you know me then you might think I’ve gone crazy. I migrated ten years worth of Gmail to FastMail three years ago and never looked back. I wanted to be in control of my own data and domain name.

That said, I really did love Gmail for the ten years I used it. Most importantly, I trusted it. Many times I found myself referencing emails from a decade ago only to find them safely stored, not forgotten, just waiting to be read again. I’ve never lost a byte of data with any of Google’s product offerings – I trusted Photos would offer the same reliability. And sweetening the deal further, with my massive library I would be a paying customer. Google would have reason to keep my data safe versus my free Gmail account which came with no promises.

So I installed Google’s Mac uploader app on my Mac, pointed it at my Dropbox photo library, and waited. Three days later all of my photos and videos were in Google’s cloud. The only problem? I had no albums. Just a giant stream of 50,000 photos sorted (thankfully) by date.

So over the next few weeks I picked a couple albums each day from my old Dropbox library and recreated them in Photos. It was boring, monotonous, and not entirely pleasant work. But in the end it was worth the effort.

To keep things organized and easily searchable, each album follows the same naming convention as it did in Dropbox. “Year-Month Short Description” (2018-01 Aaron's Birthday Party). Here’s a screenshot.

IMG 4685FB06DD09 1

All the rest of my day-in-the-life photos are sorted individually by date under the “Photos” tab.

The Google Photos iPhone app is installed on my phone and takes care of backing everything up to their cloud. It’s also installed on my wife’s phone (and signed-in under my Google account) so it slurps her’s up as well.

Further, any SD camera cards we plug into my Mac are ingested by the Photos Mac app.

Every Monday, as part of my GTD weekly review, I do a search on the Photo’s website for “last 7 days”. That, predictably, shows the lasts seven days worth of photos, which I then go through and sort into albums and delete any pictures that aren’t worth keeping.

So that all takes care of getting my media into Google Photos, but once it’s all in there, then what?

Well, quite a lot actually.

You can search and filter by people. Here’s everyone in my library…

IMG 762E75E87362 1

Tapping on my wife’s grandmother filters down to only photos containing her…

IMG 714DA282CAE2 1

But Google’s AI is much smarter than just facial recognition. Watch what happens when I search for “Thelma Roberts bridge”…

IMG 60CAB60E7EF3 1

Amazing, right? But how clever is the AI, really? Well…

Search for “Inside House”….

IMG DE26B3947546 1

And then search for “Outside House”…

IMG 0B900B658B11 1

It’s truly astounding to be able to search, slice, and dice your photos this way. I can’t wait to see what features Google adds next.

How to Export Slow Motion Videos From iOS to Facebook

This was a puzzler.

I’ve had an iPhone 5S since last fall, but haven’t ever done much more than just play with the slow motion video capture feature. Yesterday though, I had a need to take a slow-mo video and post it to Facebook. I recorded the video easily enough, set the slow motion start and stop points, and then used OS X’s Image Capture app to export the video to my Mac. I figured I could then just upload the movie to Facebook.

Not so fast.

The video that Image Capture pulls off your phone is the raw 120FPS movie with no slow motion start/stop points. While this behavior actually makes sense, it’s not what I wanted. I need the version with the slow motion effect. I did some googling, and the only solution is to upload the video to Facebook using iOS’s native sharing feature. This gives the Photos app an opportunity to render the video at normal speed with the slo-mo effects in place. So I tried that, but the app complained the video was too large. Fine. I tried emailing it to myself – same problem. Too large. The Photos app wanted me to trim the video to a shorter length. I did some more googling but couldn’t find any way of exporting the full-res video with the slow motion effect in place.

Aha!

It dawned on me that Apple’s Photostream feature shares full-res videos to iPhoto on Mac. So here’s what I did…

  1. Create a new, shared Photostream that only you are a member of.
  2. Share the video to the new stream.
  3. Open up iPhoto on your Mac and navigate to the shared stream.
  4. Drag the video to your Desktop.
  5. Upload to Facebook/Instagram/wherever.

Boom.

Dropbox Photography Introduction

What follows is the first half of the introduction to the book I’m working on about organizing your photo library with Dropbox. I thought some of you might like to see what I’ve been working on as I get closer to launching in July.

I adamantly believe that our photos and home videos are the most important digital assets we have. They document our lives, remind us of our our best moments, and teach us who we were. Yet, most people rarely give a second thought to how they preserve, protect, and share those memories. They’re content to leave everything in a giant, digital shoebox that may or may not be backed up. Thousands of photos stored in your phone’s camera roll here, a smattering of memories uploaded to Facebook and Twitter there, and none of them organized, backed up, or protected in any way.

It didn’t use to be like this. For both my sister and I, from the day we were born until we turned eighteen, my mother kept a photo album for each of us for each year of our life. By the time my sister went off to college, our family room bookcase had amassed over forty albums – filled to the brim and overflowing with photos of our youth. My mother had a longstanding rule: if the house were to catch fire, my father’s first job was to get everyone out safely. He was then instructed to run back inside and save as many of those albums as he could. And in all seriousness, my mother wasn’t exactly kidding. Those memories meant the world to her. If she were to spill coffee on one of them she’d be sad. But if she were to lose all of them in a fire, she’d be devastated. Everything else in the house could be purchased a second time. But those albums were irreplaceable.

Today, things are much better. Sure we may print off and keep some of our family photos in actual albums, but most of our digital memories live on our hard drives, where we have plenty of opportunity to back them up and keep them safe. So why do so many of us go about our digital lives with our head in the sand? Every hard drive you own, whether in your laptop, your iMac, or even your super-expensive networked storage device, will fail. It’s a guarantee. One day, the photos and videos stored on it will be lost forever when the hard drive finally dies. So why wouldn’t you take a few simple steps and spend a little extra money to ensure that when it does happen, you have a backup in place?

I’m not sure when things changed, but I suspect it’s a two part problem brought on by our increasingly online lives. As digital cameras took over and people stopped paying to develop film, the actual price of each photo plummeted. And with it, the perceived value. Further, as the digital storage space available to us increased, people began taking more and more photos – eventually reaching the point where they could no longer keep track of their photo collections or find the tools to keep them organized and protected.

Instead of photo albums of family vacations and birthday parties sorted chronologically on a shelf, our digital photo libraries are a jumbled mess. Thousands of snapshots in a single camera roll on your phone, with no meaningful way to browse through them. The oldest ones haphazardly deleted and lost forever to make room for new ones. People mistakenly believe that they can always get their pictures back from Facebook if they ever need them. Or worse, they believe that Apple is somehow magically backing up all of their photos into iCloud. Neither one is true.

While researching this book, I had a conversation with two friends. Neither is technical at a developer level, but both are long-time Mac and iPhone users. They know their way around the platform.

My first friend, we’ll call him Ted, said he was having trouble with his iPhone. His photos and videos had grown to take up nearly 10GB of storage space. And with only a 16GB iPhone, his device was very nearly out of room. He wanted to know how to backup his photos off his device but still be able to access them when needed.

He assumed there was an Apple-centric way of doing this. He figured he could just buy more iCloud storage space. He was shocked when I explained to him that the only user-friendly solution I know of is Dropbox. I told him that even if he bought more than the default 5GB of iCloud space, he’d still be limited by his on-device storage limits. iCloud only backs up what’s currently on your phone. If you delete photos and videos from your device, they’re similarly deleted from your next iCloud backup.

I walked him through how the Dropbox app works. Install it on your phone, sign up for an account, and the app periodically uploads everything in your camera roll to your Camera Uploads folder in your Dropbox. Once uploaded, you’re safe to delete them from your device – freeing up space. Then, you can either use the Dropbox app to browse your photos, view them on your computer, or use a dedicated Dropbox photo app like the wonderful Unbound.

Ted thought that idea sounded great and was ready to sign up. But then I told him the catch – the free Dropbox plan only offers 2GB of space. He needed at least 10GB, which I said would cost $99 a year.

He was dumbfounded. In an age where seemingly every online service is free, why should he have to pay for Dropbox? I tried explaining the value it added and how much I loved the service, but he just couldn’t get past the idea that this was a service that Apple should be offering as part of owning the phone. I couldn’t really disagree with him. It does seem like a problem Apple should be solving, but they’re not.

My other friend chimed in. He said he wasn’t worried about his photos and videos because he was backed up. I asked if he were already using Dropbox or paying for one of the larger iCloud storage plans. He said “No, I bought the larger 32GB iPhone so all my data is backed up with Apple.”

This time I was flabbergasted.

He went on to explain that all of his data was safe because of iCloud. I let him finish and then had to break the bad news to him that iCloud only works for the first 5GB. Once you go beyond that, none of your data is backed up unless you’re a paying customer. Apple, through all of their flashy marketing around iCloud and promises of how it “just works”, has convinced a large set of non-technical users that their most precious data, their photos and videos, is safe when it’s actually anything but.

This is exactly the problem that I’m writing about in this book. I’m desperately trying to preach to users that the data you hold so dear isn’t being handled by Apple. If you lose your phone, in most cases, your photos and videos go with it. And as I talk to more and more people about this problem and explain to them how things actually work, so often I’m met with incredulous stares. Not that they’re surprised or outraged that their data isn’t safe, but at the sheer audacity that someone would suggest they need to pay for their data’s safekeeping.

In person and in this book, my goal is to make a solid case that your data is most definitely worth the measly $9.99 per month Dropbox charges. My sister was absolutely devastated when her iMac’s hard drive failed without a backup in 2005. She mistakenly thought that her iPod’s photo cache served as a full backup of her photos. I can still remember the awful look on her face when I explained that her four years of high school photos along with the first year of her college photos were reduced to nothing more than 160 pixel wide thumbnails.

Apple never marketed to her that her iPod Photo backed up her memories, but she still mistakenly assumed that when it asked to sync her photos. But now we’re in an even worse situation where Apple really is marketing to users that all of their data is safe. I can only cringe at the thought of how many customers go to a genius bar to get a replacement phone and are told “just sign into iCloud and all your data will be restored” only to find out that’s not the case.

My mother meticulously archived and organized our family photos into dozens of albums when we were growing up. All those albums and film development cost real money. Every roll of film, every trip to the camera shop, every album, and don’t forget all the time put into organizing and pasting photos into the albums. It was an investment. The idea that everything online should be free has created a pernicious entitlement culture where users are unwilling to pay a little money to safeguard their most precious digital memories.

It’s a terrible situation and one I’m trying to correct. This book is the antidote to most people’s digital negligence. It will teach you a simple method for organizing, sharing, enjoying, and protecting your most valuable memories.

Who this book is for

I should say up front that this book is for consumers. While the sharing and backup solutions I present may be of use to professional photographers, the organization and review strategies will likely clash with the workflows of people who take photographs for a living. I also won’t be covering how to handle and process RAW format photographs. Everything I write about will assume we’re dealing with JPG files from your camera phone, point and shoot camera, or your prosumer DSLR.

I should also say that the examples I present and the software I recommend are definitely Apple-centric. We’ll be covering ways to archive photos from your iPhone and iPad, ways to share those photos with Photostreams and Apple TV, and we’ll be using tools that, to my knowledge, are only available on Mac. Most of the ideas and tactics I write about can be used on Windows or Linux platforms, but the automation techniques are Mac only.

Why listen to me?

Hopefully everything sounds great so far. But you may be wondering why you should listen to me. What makes me the expert? The simple answer is that I’ve been thinking about today’s digital photography challenges constantly for the last few years as I’ve watched my own collection become more and more unwieldy. I’ve honed my workflow to be as simple and, most importantly, automatic as possible. I’ve learned that I lose focus and things grow unorganized as the complexity of my workflow increases. But, by keeping things simple, repeatable, and efficient I’ve found a happy medium where I’m able to keep my photo library under control without devoting too much time to curating it. It’s the techniques I’ve settled on over the last few years that I plan to share in this book.

Those techniques are modeled after the principles I’ve learned practicing David Allen’s Getting Things Done methodology for the last ten years. In his book, he teaches a series of concrete steps that can be taken to reign in any complex system. I’ve simply adapted his teachings into a workflow for managing your photo library.

Further, my background as a long-time Mac app developer provides me with additional insight into finding the best software tools and automation techniques for the job.

If you follow the steps I outline, you’ll quickly find yourself with an organized photo library that’s accessible on all your devices, backed-up in case of disaster, and shareable to all your friends and family.

iCloud Doesn’t Backup Your Photos

I had a conversation yesterday with two friends. Neither one is technical at a developer level, but both are long-time Mac and iPhone users. They know their way around the platform.

My first friend, we’ll call him Ted, said he was having trouble with his iPhone. His photos and videos had grown to take up nearly 10GB of storage space. And with a only a 16GB iPhone, his phone was very nearly out of room. He wanted to know how to backup his photos off his device but still be able to access them when needed.

He assumed there was an Apple-centric way of doing this. He figured he could just buy more iCloud storage space. He was shocked when I explained to him that the only user-friendly solution I know of is Dropbox. I told him that even if he bought more than the default 5GB of iCloud space, he’d still be limited by his on-device storage limits. iCloud only backs up what’s currently on your phone. If you delete photos and videos from your device, they’re similarly deleted from your next iCloud backup.

I walked him through how the Dropbox app works. Install it on your phone, sign up for an account, and the app periodically uploads everything in your camera roll to your Camera Uploads folder in your Dropbox. Once uploaded, you’re safe to delete them from your device – freeing up space. Then, you can either use the Dropbox app to browse your photos or a dedicated Dropbox photo app like the wonderful Unbound.

Ted thought that idea sounded like a great one and was all ready to sign up. But then I told him the catch – the free Dropbox plan only offers 2GB of space. He needed at least 10GB, which I said would cost $99 a year.

He was dumbfounded. In an age where seemingly every online service is free, why should he have to pay for Dropbox? I tried explaining the value it added and how much I loved the service, but he just couldn’t get past the idea that this was a service that Apple should be offering as part of owning the phone. I couldn’t really disagree with him. It does seem like a problem Apple should be solving, but they’re not.

With all that said, my other friend Larry chimed in. He said he wasn’t worried about his photos and videos because he was backed up. I asked if he were already using Dropbox or paying for one of the larger iCloud storage plans. He said “no, I bought the larger 32GB iPhone so all my data is backed up with Apple.”

This time I was flabbergasted.

He went on to explain that all of his data was safe because of iCloud. I let him finish and then had to break the bad news to him that iCloud only works for the first 5GB. Once you go beyond that, none of your data is backed up unless you’re a paying customer. Apple, through all of their flashy marketing around iCloud and promises of it “just works”, has convinced a large set of non-technical users that their most precious data, their photos and videos, is safe when it’s actually anything but.

This is exactly that problem that I’m writing about in my book. I’m desperately trying to preach to users that the data you hold so dear isn’t being handled by Apple. If you lose your phone, in most cases, your photos and videos go with it. And as I talk to more and more people about this problem and explain to them how things actually work, so often I’m met with incredulous stares. Not that they’re surprised or outraged about their data not being safe, but at the sheer audacity that someone would suggest they need to pay for their data’s safekeeping.

In person and in my upcoming book, my goal is to make a solid case that your data is most definitely worth the measly $9.99 a month Dropbox charges. My sister was absolutely devastated when her iMac’s hard drive failed without a backup. She mistakenly thought that her (at the time) iPod’s photo cache served as a full backup of her photos. I can still remember the awful look on her face when I explained that her four years of high school photos along with the first year of her college photos were reduced to nothing more than 160 pixel wide thumbnails.

Apple never marketed to her that her iPod Photo backed up her memories, but she still mistakenly assumed that. But now we’re in an even worse situation where Apple really is marketing to users that all of their data is safe. I can only cringe at the thought of how many customers go to a genius bar to get a replacement phone and are told “just sign into iCloud and all your data will be restored” only to find out that’s not the case.

My mom meticulously archived and organized our family photos into dozens of albums when we were growing up. All those albums and film development cost real money. It was an investment. The idea that everything online should be free has created a pernicious entitlement culture where users are unwilling to pay a little money to safeguard their most precious digital memories.

It’s a terrible situation and one I’m trying to correct with my book this Summer.

Printing and Mailing Photos to Your Grandparents

We have a three month old kid. That means we take a lot of photos. I’ve done the math, and in the last three months we’ve taken 1,202 photos of him. As I’ll write about in my upcoming book on Dropbox photography, all of those photos are stored and sorted in a shared Dropbox folder that both my wife and I have access to. For other family members and friends, we share the best of those photos via a shared iOS Photostream. I’m really a big fan of this feature. With just a few taps I can share as many photos as I want and have them near-instantly delivered to the twelve people who subscribe to our photostream. Everyone in our family and circle of friends has an iOS device, so no one’s left out. I never have to fumble with emailing attachments, or posting links to Flickr. All the photos are available in the native iOS Photos app. Best of all, I can post comments with the photos I add, other people can like and add their own comments, and they can even share their own photos, too. It all works splendidly.

The only downside is for our grandparents. They don’t have iDevices. Sure, our parents are always showing photos to them on their phones and iPads, but our grandparents miss out on the personal connection they’d get from having their own collection of photos. To try and fix this, I’ve started physically printing and mailing batches of photos to them every ten days or so. They love getting photos they can touch and display in the mail. It really is like their own real-world photostream.

The problem is this is manually intensive. Printing batches of photos, keeping up with ink and photo paper, finding sturdy enough envelopes to handle twenty photos at a time, and then dealing with postage slows the whole process down. So I’ve been experimenting with three online photo delivery services to handle all of this for me.

Over the last few months I’ve tested Shutterfly, iPhoto, and PicPlum. Ideally, I’m looking for a service that I can quickly upload the latest photos – from both mine and my wife’s iPhone and from our good camera – and have them sent to multiple addresses without having to re-type the address each time. The photos need to be delivered fairly quickly and, most importantly, arrive in good condition.

I’ve given each of the above services multiple tries, and they all have their good and bad points. For those of you who like to skip ahead, the winner was Shutterfly, followed by PicPlum, and then iPhoto.

iPhoto

I really wanted to like iPhoto, as it’s Apple’s recommended service. But there are a few negatives that keep me from going this route. First of all, the iPhoto iPhone app is nearly impossible to figure out. I’m an app developer by trade, so I like to think I can understand most apps without much instruction, but the iPhoto UI baffles me. Selecting multiple photos and preparing an order for delivery was a beast of a process. Add to that extraordinarily long ship times and they were a clear no go. The actual photos were of middle of the road quality and arrived in a plain white cardboard envelope that seemed to protect them well enough.

PicPlum

PicPlum is an interesting service. Unlike iPhoto and Shutterfly, which are really designed for printing and delivering photos to yourself, PicPlum bills itself as a service designed for printing and mailing photos for other people. Everything is done through their lovely web interface. You can drag and drop your photos directly into the web browser. Then it’s just a matter of choosing the recipients from your previously saved addresses.

PicPlum loses points for not having an iOS app. Typically, I want to only send my best photos to be printed. All of the best ones are already handily organized and available in our shared photostream. If they had an iOS app, I could choose them directly from that album. But, as they only support desktop uploading, I have to find and gather them from the various photo albums in my Dropbox. This isn’t a huge deal-breaker, but it is slightly less convenient.

The biggest downside to PicPlum, and ultimately the reason I no longer use them, is the photos arrive in a flimsy paper envelope. The kind of thing you’d mail a birthday card in. I used PicPlum to send twenty photos three times. Twice, the envelope arrived torn with the photos sticking out. In one case, the adhesive sealing the envelope was barely affixed and everything was in danger of spilling out. And while I didn’t encounter this problem in my testing, with such a flimsy delivery method, there’s absolutely no protection against water damage.

I’m actually quite sad that I can’t use PicPlum. They make it easy to send to multiple recipients and their photos were by far the highest quality of the three services.

Shutterfly

As I said above, Shutterfly is who I decided to go with. Their iOS app is a little long in the tooth, but it’s serviceable and easy enough to use. I’m able to choose photos from my Photostream and upload them quickly. I can pick from a list of previously saved addresses. The price is the cheapest of the three services, and the photo quality is good. Unlike PicPlum, Shutterfly’s prints arrive in a sturdy cardboard envelope inside an even larger cardboard sleeve. I’ve mailed six batches of photos so far and none have arrived damaged. The double envelopes even protected the photos against our rain soaked mailbox.

The only problem I’ve encountered with Shutterfly is their shipping time. Using their default shipping option, which is about three dollars, the photos arrive anywhere from five to twelve days later. For three bucks, I’m not sure what I expect, but two weeks is way too long to wait for a delivery. So I usually just pony up the extra cash and pay for the $10 two-day delivery method instead. It’s faster, and comes with a tracking number, too.

Overall, our grandparents have been thrilled with the service. They absolutely love getting their bi-weekly photo surprise in the mail. The physicality of holding real photos in your hands makes them feel connected to our son in a way that FaceTime and flipping through photos on an iPad just can’t. I highly recommend keeping your non-technical friends and family in the loop this way.

Nostalgia – Rename Your Photos

I have a problem. Half of my photos come from my iPhone (via the Dropbox uploader), which creates filenames based on the date they were taken. But all the photos from my awesome DSLR are named BLAHBLAH_7001.jpg and BLAHBLAH_7002.jpg. That annoys me. I want all of my filenames to be date based so I can quickly sort them and find the photos I’m looking for.

Much to my dismay, Hazel and Automator can’t solve that problem.

This Mac app does.

Launch the app, drag and drop a bunch of photos onto the window (or the Dock icon), and boom! Everything gets renamed appropriately based on the EXIF date.

Currently, there’s no way to customize the output date format. It’s set to yyyy-MM-dd HH.mm.ss. I make no apologies for that. If you’d like something different, feel free to modify the Xcode project or submit a pull request with the ability to customize the date :) Also, it only looks for the standards based yyyy:MM:dd HH:mm:ss EXIF format. If your camera doesn’t follow along, Nostalgia won’t be able to parse it. Again, pull requests are very much welcome.

You can download Nostalgia for your Mac from here.