Month: August 2020

As I’ve written about previously, for better or worse, Google Photos is the initial destination of all the family photos and videos we take as well as the source of truth for the albums I sort them into. I’ve tried every consumer photo organization tool/website/app on the market, and nothing comes as close to hitting my feature requirements as Google Photos. (Well, short of building my own solution, but that’s…uh…not yet.)

Photos is the only Google product I use. (Besides search – sigh). I gave up Gmail years ago because even with full backups of all my messages, my email address itself is the key to nearly every other online account. The chance of getting locked out due to an automated flag is too high – even if I am a paying customer. But if I lost access to my photo library stored with Google? That would be bad, but not the end of the world since I have all that data backed up.

However, I’m always playing the long game and thinking about contingency plans with data this important. Chief among them is my looming monthly price increase apocalypse. I’m currently paying Google $99/year for 2TB of storage space. When I hit that limit, the next tier is 10TB for $600/year. That’s a hell of a jump for that next byte. And while I totally get the business reasons behind that pricing, sheesh.

So I’ve been thinking about my eventual exit strategy. The obvious next and most comparable choice is Amazon Photos. (I’m keeping a close eye on PhotoPrism.) They solve the storage pricing problem because they’re Amazon and just charge you an extra 1TB at a time as your needs increase.

I’m perfectly willing to pay for what I use, so that’s great. And Amazon’s website and apps are actually better than Google’s in many ways. But they do fall on their face as soon as you start dealing with videos larger than 2GB (easy to do with kids and a modern iPhone shooting 4K video) or over 20 minutes long.

So, I’m keeping a very close watch on Amazon and hoping they improve enough to be the right solution in the future.

Anyway, the point of this blog post is to say that I’m preparing for an eventual move to another photo cloud service. I’m also trying to keep my local backups neatly organized. So, I wrote a small command-line tool to specifically deal with the Google Photos backup format that you’ll receive if you request a dump of your data.

It takes Google’s directory structure and all their duplicated files, merges, sorts, and deduplicates your photos and videos into a sane folder structure – the one I’ve been using for over a decade.

You can request a dump of all of your Photos or just specific albums/dates using Google Takeout. (Kudos to Google for making this sort of stuff so easy.) It works great, and you’ll get everything. The problem is the backups are structured as if you’ll only ever do a single backup in your life – as opposed to incremental ones. And they also don’t deduplicate your data before sending it to you. (I understand why.)

Once your backup is ready, you’ll get everything split into 50GB .tgz files. Each one will extract into the following directory structure:

You’ll get a folder for every item’s capture date. If you took 50 photos on August 15, 2020, you’d get a folder named “2020-08-15”. Except, for reasons I don’t understand, you’ll occasionally get a folder named “2020-08-15 #2”, too. Same day, another folder.

Google will also create folders for every album included in your backup. Perfect. But any items in those albums (folders) will be duplicated in their respective date folders, too. So, if you have a 2GB video included in two albums, it’ll be in three backup folders, which means 6GB of space.

And this is totally fine. It’s definitely the most flexible and complete solution and leaves it to the end-user to figure out what to do with all this data. And what I want to do is convert everything into a structure that looks like…

…along with all of my items deduplicated. So, any items that are included in an album are not duplicated in a year-month folder. Those date-based folders only contain items that are not sorted into albums.

I spent a few hours messing around with various bash scripts and some StackOverflow posts but realized I needed something better than any shell script I could write. (I’m sure someone could write it, though.)

I came up with a tiny Swift command-line tool that scans all of your files and stores a hash of each into a sqlite database. (Thanks, Gus!) Then, figures out where each file belongs, moves it there, and ignores any duplicates it finds.

As you might imagine, it’s not the fastest process, but the results were worth it for me. My iMac Pro was able to scan and process my 1.4TB library (100k+ files) on an external USB3 spinning drive overnight. (Sorry, I should have timed it. I think it was between 4 – 8 hours.)

I’m quite pleased with the results. Not only did it clean up all of the per-day folders into a more sane by-month directory structure, but removing duplicates cut the total file size by 30%.

Running the tool is a two-step process.

First, you need to import your Google Photos’ backups into your library – a shared folder where all of your photos and videos are kept. The reason for this step is that Google’s backup structure will often contain duplicate folder names in addition to filenames. And since merging folders on macOS is a delicate process, the import step will do that for you.

Just run this

photoz import /path/to/google/photos/backup /path/to/library

on each of the .tgz files that Takeout gives you.

After everything is merged in, run

photoz organize /path/to/library

and wait.

If all goes well, you’ll end up with a sane by-month/album folder structure that uses considerably less space.

The code is open source. Fixes and improvements are welcome. I haven’t cleaned it up much since first writing it, but it’s only three Swift files, so it should be easy to dive into and customize to your liking.

And while it works for me and I did a crap ton of dry-runs and testing building this tool, please, please, please make backups of your data before running an internet stranger’s code over something as important as your photo library.

Oliver Reichenstein, founder of iA Writer, had this to say about his post on Apple and modern software monopolies:

This wasn’t easy to write it to publish. But, today, I just had to hit that publish button and get it over with. I have another hot potato post on subscriptions I’ve been juggling with for some time now. Some thoughts are sketched out in the monopoly post. I’ll drop that one soon, too. These topics are complex and dangerous. Writing about them publicly is simply terrifying.

For me, two highlights were:

Apple are not treating Netflix, Nintendo, and Spotify like iA Writer to be fair with us small devs. Think about it. Apple doesn’t profit much from taking 30% of iA Writer’s sales. It would cost them a smile to cut us off. And, given that from the hundreds of thousands of app, only very few reach our position, it wouldn’t be a big deal to cut us all loose. The benefit of having small passionate devs is not in the money they make from them. The benefit of having us is that we enrich the platform. For years, people have been telling us that they buy Apple gear because of iA Writer and similar apps. The benefit of cutting 30% from the small ones is that they can say “we treat everyone the same” and cut off a major chunk of the big Netflix, Spotify and Fortnite cakes.

and

Apple, a company with California Hippie roots that encourages to think different, has a developer community that is afraid to speak their mind in public.

The entire post is well worth reading. And whether you side with him and the $17.86 billion corporation or the $1.97 trillion corporation, you gotta admit it takes guts to lay out that argument on your company blog when the future of your business depends on the kind of day your next anonymous App Store reviewer is having.

Like I said the other day, Jigsaw is one of those ridiculously fun (dumb?) ideas that come along and smack you upside the head one day and you can’t help but take an afternoon to build.

It’s also such a silly idea that I’d normally open source the code so other folks can tinker around with it. Unfortunately, my CloudKit syncing stack is based on a closed source framework that I can’t distribute. (Ensembles, if you’re curious. It’s basically the greatest thing ever and makes my life as a Mac/iOS developer so easy.)

Instead, Jigsaw is free to download and have fun with.

In case you missed my post about why this app even exists…

Apple already lets you sync the contents of your Desktop using iCloud. But, if you’re a visual person like me who often arranges their Desktop icons in meaningful ways, not having the positions of your files on screen also stay in sync is frustrating as I move between my laptop and desktop throughout the day.

Jigsaw solves that by syncing the positions of your Desktop icons over iCloud. Move a folder on your iMac, and a few seconds later it mirrors itself on your laptop. Here’s a demo…

Caveats

I’ve tested Jigsaw on Mojave and Catalina. I don’t know of any reason why it wouldn’t also work on Big Sur, but it’s untested for now.

I also haven’t even attempted to think about or solve any of the myriad problems that might arise from using multiple monitors.

Along those lines…

During testing this evening, I thought my sync code had gone haywire. I would move an icon on my iMac and almost immediately it would reposition itself into a random spot somewhere else rather than syncing across to my laptop.

What I finally realized was happening was this:

The position I happened to be moving that one icon on my large iMac screen was beyond the bounds of my laptop’s smaller Desktop window. The icon position was syncing correctly, however Finder on my laptop immediately moved it into a new (random) location onscreen. That new location then synced back to my iMac, which made me think my syncing was off. Turns out, iCloud sync was so fast I didn’t even notice it had round-tripped on me. (Nice work CloudKit folks!)

I mentioned in my previous Jigsaw post that I experimented with syncing the relative positions of icons instead of their absolute positions. It was as interesting solution in that it guaranteed icons would always remain on screen, but it led to odd behavior, so I took out that option.

Instead, Jigsaw’s one and only Preferences setting is to enable debug mode. When turned on, Jigsaw will sync all of your connected Mac’s screen sizes to iCloud, compute the smallest width and height among them, and then draw a red rectangle on your Desktop to show (in theory) the valid area you can safely arrange your icons within. (Again, I’ve tested this feature among my three computers – and only very briefly. YMMV.)

To support various screen sizes, I’m wondering about just ignoring icons that fall outside those bounds. But that presents its own sync challenges for another day.

Download Jigsaw

In any case, that’s Jigsaw. Probably not very practical, but it was fun to build on a rainy day. Download the app and let me know what you think.

Earlier today, I had a short twitter conversation with Lauren Herda (she illustrated my first three Mac app icons).

I tweeted about symlinking my Mac’s Desktop folder into Dropbox to work around some iCloud Drive problems that had been plaguing me. It was a quick, off the cuff conversation, but for some reason it kept bouncing around in my head. A little while later when I was washing dishes, I jotted down the following note into Drafts…

Could we sync desktop icon positions over CloudKit??

Here’s the thing.

For me, my Mac’s Desktop is my staging ground, my active workspace, the digital representation of my mental RAM. I’ll typically have all of the files related to the task I’m currently working on stored on my Desktop. Once it’s complete, I’ll either file them away or delete them and move on to the next thing.

Having the Desktop on my iMac at home stay in-sync with my work laptop eases the transition and context switching as I move between locations. Dropbox has been doing this for years, but actually getting into the correct folder in Dropbox always has just enough friction to keep me from using it with active files the way I do my Desktop. When Apple added the option to sync your Documents and Desktop folders into iCloud Drive a number of years ago, it was a perfect fit for me…

…almost.

I’m a visual learner, and that way of thinking translates to how I work, too. That means, not only do I keep my active files on my Desktop, but I also arrange them visually in meaningful ways.

I’ll often shuffle files around into little groups to denote that they’re related to each other. Sometimes I’ll line a queue of files up on the left side of my Desktop and then move each one to the other side as I process them. My point is, in my workflow, the position of those Desktop items is a special bit of metadata that is often just as important as the contents of the files themselves.

So, it’s always seemed to me like a missed opportunity to not sync that metadata, too. And I realize that keeping the visual positions of files in sync comes with all sorts of thorny problems around different sized screens, multiple monitors, etc. But it seemed like a fun problem to think about on a rainy Saturday evening.

As you might guess, I did more than think about it.

The video above shows a macOS Mojave and Catalina virtual machine side-by-side, signed into my iCloud account. I wrote a small helper app that runs in the background and watches your Desktop for changes, which are then synced over CloudKit. The Mac on the other end, subscribes to those updates and repositions your Desktop icons accordingly.

For no particular reason other than I had to pick a name of some kind, this ridiculous little app is called Jigsaw.

When arranging your icons, Jigsaw has two modes of operation.

1. It will sync the literal pixel coordinates of your Desktop icons from one machine to the next. This is mostly fine until you get into a scenario where one Mac has vastly more (or less) screen space than the other. In that situation, an icon can get accidentally repositioned offscreen. When that happens, the Finder steps in and moves it back onto the visible portion of your Desktop.
2. Another way that I briefly toyed with was instead of syncing actual coordinates, I kept all of the icons synced proportionally / relative to each other and the screen. If you have an 800px by 600px monitor, with a folder at (400px, 300px) – that translates to 50% on each axis. When Jigsaw repositions the icon on your 1200px by 900px monitor, it will be placed at (600px, 450px).

I’m not sure which method is better. (Honestly, this whole idea is dumb ????) But that’s Jigsaw. Likely nothing more than an experiment but a fun, nerdy way to spend a Saturday evening.

If anyone is interested in trying it, you can download Jigsaw here.