Missing Rdio and Making the Best of Apple Music with Shortcuts

Man, I miss Rdio. I mean, I really miss it. I loved that service.

When I was a teenager, I’d spend hours on the weekend and get lost in new and used music stores (CD’s) just digging through stacks of beautiful album artwork and unfamiliar band names. I’d talk with other customers and ask the clerk to let me sample a few tracks when something caught my eye. The joy was in the discovery as much as the actual purchase and listening that came later.

Rdio was the first streaming music service I used. It was like walking into an infinitely large music store. And it was all free! (Well, $10/month.) Their UI was wonderful. Websites today are walled gardens designed to keep you on the property for as long as possible. But Rdio, like the web of the late 90’s and early 2000’s, was overflowing with links leading through a maze designed to get lost in. Each album page – in addition to the artwork and song listing – had detailed info about the band, their other music, related artists, genres, etc. (The only similar mainstream experience I can think of today is when you fall down a Wikipedia rabbit hole.) And the majority of the pages had an in-depth critic’s take on the music in addition to listener submitted reviews that were generally well written and free of the awfulness we see in YouTube’s and Facebook’s comments section today.

What I’m trying to say is Rdio came very close to recreating the record store experience in digital form. Apple tried and failed with Ping and has since made additional social attempts inside Apple Music. I can’t really speak about what Spotify is like now. I was a subscriber for a few months after Rdio died, but it never really stuck for me – and Apple Music’s tight integration with Mac and iOS have kept me tied to their service instead.

So using Apple Music the last few years has been fine I guess. I can play what’s in my own collection (usually), and search their streaming library, but I find it extremely difficult to organically discover new music. It’s sort-of possible on the desktop with iTunes, but the iOS app (and I took a look just now to double-check) only shows “other albums by this artist.” The “For You” tab makes an attempt by showing other genres similar to what you already listen to, but I find their algorithmic recommendations lacking. And, again, if you do tap on one of the suggested albums, that’s about as far as you can go. You can’t further explore beyond that artist. And don’t even get me started on the “Browse” section or whatever the hell Beats 1 is doing. That’s a dumpster fire of shitty editorialized content that I can only assume is mass promoted by the record labels for the masses. (Yes, I might just be snobby and elitist about my music, but I really do have a lot of pop music in my collection that I enjoy. I just find most of Apple’s selections…shallow.)

Anyway, like I said, it’s fine. Not anything special, but fine.

But over the last few months I’ve made a conscious effort to start listening to more music again. I used to always have something playing in my bedroom, dorm, various apartments, and later houses. But I think once my kids were born, their needs and noise took over and music fell to the wayside. But now I’m using the wonderful Anesidora app to keep Pandora shuffling through songs in my office where I sit all day. I need to stay focused on my work, and having to think about and choose something to play takes me out of the zone. I like that I can just tell Pandora to play something it thinks I’ll enjoy and it will take care of the rest. It’s mindless and exactly what I want.

But Pandora typically only plays music I’ve already listened to and given a thumbs-up. It rarely surfaces new music. That’s what I still have to try and use Apple Music for. And I typically try and do that when I’m in the car.

Using your phone for anything while driving is stupid. So if I want to queue up some music, I have to do it when I’m still in the driveway or if I think I have time and it’s safe while stopped at a traffic light. But I need to be fast about it. And that’s where Marvis, Launch Center Pro, and Apple’s Shortcuts app come into play.

I discovered Marvis last month from Ryan Christoffel at MacStories. It’s a highly-customizable client for Apple Music. All of your songs, playlists, and the entire Apple Music catalog in a gorgeous, functional UI that you can design around your own needs. Here’s what my setup looks like:

Marvis Pro Screenshot

I’ve got the Home screen organized so that I can tap and play my most listened to playlists and albums without scrolling or having to dig through Music.app’s tabs and navigation stacks.

Specifically, at the top I can start any music I’ve recently added to my library. I’ll often go on a music adding binge and add a ton of stuff at once then finally listen to it days or a week or so later. This section collects all those albums in one spot so I don’t forget to try something new that looked interesting to me.

Beneath that are three of Apple Music’s main auto-generate playlists. Again, I have one-tap access to my Favorites when I want to hear something familiar and New Music that Apple thinks I might like (which is often hit or miss).

Further down the screen is “New For You”, which is a stream of new releases from artists already in your library. I’ve wanted this feature in iTunes for years, and I’m thrilled Apple Music delivered.

Of note: I’ve used the display settings in Marvis to pack as much music into as small a space as possible. This puts as many tap targets within “thumb reach” as possible and minimizes any scrolling I need to do. Very important when that red light could turn green at any moment.

Next up, if I want even faster access to my most common playlists, I’ve created three shortcuts in Shortcuts.app to play Apple’s “New Music for You”, “Your Favorites”, and their top Alternative songs and added them to Launch Center Pro’s excellent Today widget. From my phone’s home screen, I can swipe left and tap to start playing.

Launch Center Pro Screenshot

And going a bit further with Shortcuts, I’ve added two as icons on my home screen:

  • “Play Album”, which starts playing the full album that the current song belongs to. This is super useful when I’m listening to a suggested music playlist and it plays a new artist I’d like to hear more of.
  • And “Bookmark Song”. This adds the current song to a playlist I made called “Bookmarks”. I treat it like an Instapaper for music that I can come back to later when I have time to explore.

Shortcuts Screenshot

So, that’s my music setup at the moment. I achingly miss Rdio but am trying to make the best of Apple Music by making it as easy as possible to listen to the music I love and explore the new songs it thinks I’ll enjoy.

Starting and Finishing More Long-Form Writing in Drafts.app

I’m writing this blog post in Drafts.app on my Mac. But before I publish it, I’ll also probably do some light editing of it on my iPad before bed. And I’ve been capturing ideas, short thoughts, and building a basic outline on my phone over the past few days as things occur to me or I find myself with a bit of downtime.

I’ve been a heavy Drafts.app user on iOS for years. But this workflow is relatively new for me and came about with the release of its Mac counterpart two months ago.

Previously, for going on I don’t know how many years, all of my long-form writing, which typically means blog posts, has lived exclusively in MarsEdit. (Really long-form stuff like magazine articles and a few aborted attempts at writing a book are done in Scrivener.) The app syncs perfectly with WordPress and allows me to write in Markdown and preview my words in a live, pixel-perfect preview of my blog’s template.

But MarsEdit doesn’t sync between my desktop and laptop. And there’s no iOS counterpart for writing on-the-go. So that forced me to write the whole post from first draft to published version on the same machine. It also meant the outlines for my longer posts and short scribblings lived in other apps – sometimes OmniOutliner, other times Apple Notes, and occasionally OmniFocus and Drafts.app. I didn’t have a single unified workspace for all my thoughts.

Drafts for Mac has changed that. With seamless sync, I can begin writing anywhere I want, come to a stopping point, and pick back up later exactly where I left off on any device.

Dr. Drang wrote last week

Drafts has become the place “where text is.”

I couldn’t agree more. Looking through my Drafts Archive section, it’s now overflowing not just with short beginnings and snippets that got quickly shuttled off to other apps, but real writing. It’s my one place for everything in-progress as well as a history of final outcomes.

As for the specifics of my new blogging workflow, all of my posts are assigned a tag of blog. (It’s currently the only tag I use for anything.) Then, I have two custom Workspaces.

One called “Blog Posts” which only shows drafts tagged blog, defaults to Markdown syntax highlighting, and automatically switches to my Markdown helper keyboard on iOS.

And then there’s another workspace titled “Everything”. It doesn’t actually show everything. Rather, it’s everything that’s not tagged with blog. I’ve currently got about twenty unfinished posts in some form or another in my Inbox. I don’t want those clogging up and crowding all my other notes which tend to be more “action oriented”.

These two workspaces let me quickly switch between notes that I might need to do something with or reference and those that stick around longer and require a more focused, creative mood to tackle.

And as an added bonus, the Mac app can now show a live preview of your post rendered in your website’s theme, too – just like MarsEdit. (The iOS app has been able to do this for a while as well, but it doesn’t let you see the preview and edit text at the same time.)

Anyway, that’s all a long way of saying that I couldn’t be happier with this new setup. Drafts truly is where all of my text starts.

Why Many of my Apps Failed And What Comes Next

I’ve started building something new.

I’m about four weeks in and already finding it incredibly useful in my day-to-day.

I’ve built many different apps over the years, thrown them against the wall, and excitedly watched which ones developed a following and which ones failed miserably.

Most of my apps have fallen into two categories. There are the ones that solve a personal need I face, that I can inform and direct with my own experiences. And there are those where I saw a market opportunity or just thought they might be fun to build.

Almost all of the ones in that second category have failed. Incoming!, Nottingham, Highwire. They each had their share of a few passionate and engaged users, but mostly were ignored. (The jury is still out on Triage.)

But that first category of apps? Those have flourished.

I was a web developer for ten years before I started building Mac apps. I know the industry well. And working at an agency meant I was juggling many different websites at once. I needed a faster/easier way to spin up local development environments, and that led to me creating VirtualHostX. Which, in turn, led to Hostbuddy and Hobo years later.

My core set of apps, as I think of them, found a wonderful niche among solo web designers and developers and the small companies they work for. And the apps have succeeded beyond my wildest dreams. From 2012 to 2014 they were my full time and only job.

But since their heyday in 2014, sales have steadily declined. A big part of that, I think, is because the world has moved on from the LAMP-based standards of 2004 – 2012. Nginx, NodeJS, etc. have led to a sea change. If it weren’t for WordPress’s continued dominance, I’m not sure if I’d have any more sales at all.

So, I’ve seen the writing on the wall for a few years now, and have been on the lookout for ways to branch out and diversify my app portfolio.

I’ve done some freelance jobs here and there, and tried a few new things on my own, but nothing has taken off. And as I talked about earlier, I think a lot of that is due to none of those new ventures being true passion projects that I could bring my own experiences as a user with specific needs.

But four weeks ago I finally became fed-up with the awfulness of Mint.com. I know they’re able to provide a free service because they plaster the site with advertisements and sell my data, and I’d be ok with that if they weren’t so intrusive. But they’re taking over nearly the entire browser window now. Add to that the cumbersome Web 2.0 UI and lack of any real reporting capabilities, and I’d had enough.

What I really want is a fast, powerful, native Mac app that automatically imports my financial data and gives me the ability to slice, dice, filter, organize, and export my data in every possible way. The flexibility and power of an Excel sheet with the learning curve of iOS and the familiar paradigms of a real Mac app. Something that keeps me in control of my data, respects my privacy, and syncs to all my devices.

So I built it.

Preview1

Most of it. It’s not done yet, but all the major pieces are in place, and I’m using it to track my family’s budget every day.

I’ve taken a heavy dose of inspiration from one of my favorites apps – OmniFocus – both from a UI perspective (a clean, modern, attractive Mac interface) and from the power they afford users over their data by way of custom perspectives.

My app is smart. It auto-categorizes your transactions. You can just let the app do its thing and everything will end up in the (mostly) appropriate place. Or you can categorize each transaction manually. Or you can setup smart rules to do it for you automatically. Whatever fits your workflow.

Each transaction can belong to a single category. But categories can be nested. With one click you can see all the dining out you did last month. Or just the fast food orders. Or just what you spent at McDonalds.

And then there are tags. Assign multiple tags to a transaction. Do it manually, or create a smart rule to assign them automatically. Then use the app’s powerful search feature to find any combination of AND / OR.

Take search further by combining tags with categories and date based filtering. Find every Uber ride tagged #business during the last quarter and export the results to a CSV you can send to your boss to get reimbursed. Snapped a picture of the receipt with your iPhone? Yep, you can attach files, too.

And once you’ve got that perfect search query and set of filters in place, save it as a new Report that you can recall at any time. Just like an OmniFocus perspective.

And, just for fun, why not view all of your transactions plotted on a map?

Preview2

I’ve got all of this working (albeit without the necessary UI/UX polish) and it’s totally opened my eyes to some aspects of our financial situation that I’d overlooked. The last major piece is adding a budgeting component. I know the system I want to adhere to, but I’m still working through how it will fit in the interface.

Of course, I’m leaving out the iOS counterpart. But I’m saving that for later. The model layer is still a bit too much in flux. But I’ve written all the business logic to be platform agnostic. So the plan is to reimplement the appropriate features in UIKit (not everything the Mac app can do would make sense or be needed on a mobile device) and reuse as much of the existing codebase as possible.

But don’t forget syncing. As I said in my requirements, I want this app to be privacy-focused and have the user be in charge of their own data. So, no financial data will ever touch my server. Unless they enable syncing, no data will ever leave their device. But if they do want to sync, that’s all privately handled by CloudKit so I, the developer, can’t see their info even if I wanted to.

So, that’s it. That’s what I’ve built and am working towards completing. I have no idea when it will be ready. CloudKit could easily throw a big wrench into everything. And I’ve also got major updates to VirtualHostX and Hostbuddy underway and due out this Fall.

But I’m excited. It’s an app I’m passionate about and I have a clear direction and feature set in mind. It caters to a broader audience than my developer-focused products and could potentially save my tiny company if they continue to trend downwards.

iTunes Match Failed Me

Remember iTunes Match? It's great. But Apple stopped promoting it (probably rightly so) a couple years ago when they realized they could make more money charging $10/month for Apple Music than Match's $25/year.

Anyway, I loved it and still do. It uploads all of your digital music to Apple's cloud and makes it streamable on all of your devices. And I mean all of it – especially your ripped mp3s, live albums, or anything else not in the iTunes Store. And if they can "match" any of those unofficial mp3s to a song from the store, they'll "upgrade" you to the higher-quality AAC file for free. I had thousands of low quality mp3s ripped in the early 2000's and late 90's. Now they're metadata tagged appropriately, with artwork, and sound better.

When I first joined iTunes Match (seven?) years ago, I uploaded all my music, made a backup of my local library, deleted everything, and just streamed from then on to save hard drive space.

As I acquired new music, it went to Apple's cloud, but I never got around to backing it up offline.

As I've been writing about this month, I'm re-evaluating my backups strategy. This week I got around to looking at my music collection and decided it was time to retire that old external drive and put everything in B2 – around 300GB.

Knowing that drive was out of date, I figured I'd just download a fresh, complete copy from Apple.

I selected everything in iTunes, and clicked the download button. And waited.

The next morning I found my nearly 25,000 tracks stored locally – and a ton of errors.

Out of the 25,000, nearly 1,500 had failed to download and reported all sorts of various network errors.

I made a quick smart playlist showing all songs in the cloud but not available locally. This made it easy to isolate the problem items.

I tried downloading all the missing songs, but each one failed again. So I tried downloading a few individually with no luck. You can see where this is going.

After much testing and troubleshooting these 1,500 songs (all from various albums, some from ripped CD's, some purchased from the iTunes Store) are seemingly gone.

It's not the end of the world. If I really want some of them, I'm sure I could just re-purchase or stream from Apple Music. But others, especially some amazing live albums I collected in college are gone.

So I uploaded what I had to B2. And then made a csv export of the playlist of missing songs for good measure.

And while I love and take advantage of the cloud's convenience, this is why I don't trust my data to be in only one place. It's my fault for not backing up this part of my data if it was important to me. It's the first real data loss I've experienced in years – maybe since 2010. Maybe it's just a bug that Apple will eventually fix.

But I've learned my lesson.

Fixing a Broken Service With a Tiny Bit of Automation

This post is a nice, unintentional follow-up to yesterday’s one about backing up all of my family’s photos and home videos. Anyway…

My kids go to a fantastic daycare. My wife and I couldn’t be happier. The teachers are wonderful, they love our children, and our kids adore them, too. But, the third-party service the school uses to communicate with parents is absolute horseshit.

I won’t say what the service is because I don’t want to give them free publicity or maybe even alert them to what I’m doing, but if you have daycare-aged children, you probably know it. All the schools use use it.

All of the teachers carry around iPads in the classroom. They use this third-party app to check-in / check-out the children, capture photos and videos throughout the day, record what they ate for lunch and how long they napped, and (if your child is young enough) document their diaper changes. At the end of the day, after we sign them out of school, my wife and I get an automated email from the service with a summary of each kid’s day. But what we look forward to most are the photos/videos they take of our kids that get sent to us as they happen. When you’re slogging through a boring day at the office, seeing a happy picture of your kid on the playground with their friends is awesome.

Now, let me be clear. The service works. Mostly. I mean, it functions adequately. But it’s a horrorshow of app / website design.

It looks like something straight out of 2009-era iPhone development. It’s difficult to use. Crashes frequently. And from what the teachers have told me, the educator version isn’t any better.

Luckily, you don’t have to use their app. You can opt-in to get all the updates and photos sent to you via email, which is what my wife and I do. But, the HTML emails they send have never rendered properly in any email client – desktop or web – that I’ve tried. But that’s fine. They may not be pleasant to look at, but I can read the information in them.

My biggest gripe is that we often want to save any particularly good photos of our kids and share them with the grandparents. You can’t save the photo out of the email, because the embedded image is cropped to a square for some strange reason. You need to first tap on the image to load the full version in a browser and download it from there. Fine. But, any photo that contains any child in addition to your kid – like a group shot with a friend – is displayed with a transparent div on top of it so you can’t download it (at least on a mobile device) for privacy reasons. Look, I get it. Some parents might not want other parents unintentionally posting photos of their kids to social media. But it’s still annoying. It just forces us to take – and then crop – a screenshot. Also, the emails containing videos, which are often the best ones, can’t be downloaded at all.

Last night I got frustrated enough to finally do something about this.

I use Postmark to send all of my company‘s transactional emails. They’re fantastic for sending emails, but one feature they offer that I’ve never taken advantage of is handling inbound emails.

You can forward any email to a secret address they provide you, and they’ll parse the email and POST all of its information as a helpful JSON object to whatever URL you specify.

So, I setup a webhook in their control panel pointing to a PHP script on my web server. Then, I told Fastmail to forward all emails from the daycare service to my secret Postmark email address. You can see where this is going, can’t you?

When they send a new email to my server, the PHP script finds the link in the email’s HTML content that points to the full version on the service’s website. It then downloads that web page, parses out the URL to the full image, downloads that, and saves it into a folder on my server. This works for videos, too.

The PHP script I wrote is specific to the service our daycare uses, but if you’re curious, here it is…

That’s the first step.

Next, my iMac at home runs a script every hour to download any new photos or videos from my server and puts them in a folder inside my Mac’s “Pictures” folder. When that happens, a folder action I built with Automator automatically imports them into Apple’s Photos.app, where they’re synced to all of my mobile devices and iCloud. Soon after that, Google Photos on my iPhone will detect the new items and archive them in Google’s cloud, where they’re backed-up and made available on my wife’s phone as well.

Here’s a photo of the Automator action. It couldn’t be simpler – just one step…

The result? We get to see all of our kids’ photos as they happen, in the nice Photos app on our phones – rather than digging through the service’s crappy emails. And, sharing the pictures with the rest of our family is a one-tap process – even for the videos which previously weren’t available at all!

Backing Up Everything (Again)

This will take a while. Bear with me.

I’m obsessive about backing up my data. I don’t want to take the chance of ever losing anything important. But that doesn’t mean I’m a data hoarder. I like to think I’m pragmatic about it. And I don’t trust anyone else to do it for me.

From around 2006 to 2012, I kept a Mac mini attached to our TV with a Drobo hanging off the back. It had all our downloaded movies on it. And every night it would automatically download the latest releases of our favorite TV shows from Usenet so my wife and I could watch them with Plex the next day. It worked great, and all the media files were stored redundantly across multiple hard drives with tons of storage space. (Would it survive a house fire? No. But files like that weren’t critical.) But with the rise of streaming services and useful pay-to-watch stores like iTunes, now I’d rather just pay someone else to handle all of that for me. So, I don’t keep any media files like that locally any longer.

But my email? My financial and business documents? My family’s photo and home video archive? I’m really obsessive about that.

For most of my computing life, all of that data was small enough to fit on my laptop or desktop’s hard drive. In college, I remember burning a CD (not a DVD) every few months will all of my school work, source code, and photos on it for safe keeping. The internet wasn’t yet fast enough to make backing up to a cloud (were clouds even a thing back then?) feasible, so as my data grew I just cloned everything nightly to a spare drive using SuperDuper and Time Machine. It worked for the most part. Sure, I still worried about my house catching fire and destroying my backups, but there really wasn’t an alternative other than occasionally taking one of the backup drives to work or a friend’s house.

But then the internet got fast, really fast, and syncing everything to the cloud became easy and affordable. I was a beta user of Gmail back in 2004. I was an early paid subscriber of Dropbox since around 2008. All of my data was stored in their services and fully available on every computer and – eventually – mobile device. At the time, I thought I had reached peak-backup.

I was wrong.

Now we have too much data. My email is around 20GB. My family’s photo library is approaching 500GB. That’s more data than will fit on my laptop’s puny SSD. It will fit on my iMac, but it leaves precious available space for anything else. I could connect external drives, but that gets messy and further complicates my local backup routine. (Yes, Backblaze is a good, potential solution to that.)

Another problem is that most of our data now is either created directly in the cloud (email, Google Docs, etc) or is immediately sent to it (iPhone photos uploaded to iCloud and/or Google Photos), bypassing my local storage. If you trust Google (or Apple) to keep your data safe and backed up, that’s great. I don’t. I’ve heard too many horror stories about one of Google’s automated AI systems flagging an account and locking out the user. And with no way to contact an actual human, you’re dead in the water along with all your data. Especially if you lose access to your primary email account, which is the key to all your other online accounts.

So, I need a way to backup my newly created cloud data, too. This is getting complicated.

First step. My email. This is easy. Five years ago I setup new email addresses for my personal and business accounts with Fastmail. They’re amazing. I imported my 10+ years worth of email from Google (sadly, my pre-2004 college email and personal accounts are lost to the ether), setup a forwarding rule in Gmail, and with the help of 1Password, changed all of my online services to use my new email. It took about a month to switch everything over, but now the only email coming to my old Gmail address is spam. Fastmail keeps redundant backups of my email. And I have full IMAP copies available on multiple computers in case they don’t. And if something ever goes wrong, unlike Google where their advertisers are the customer – and I’m the product – I pay Fastmail every month and can call up a live human to talk to.

Source code. I’m a paying GitHub customer. Everything’s stored and backed up there. But still, what if they screw up. I ran a small, self-hosted server with GitLab on it for a while instead of GitHub and set it to backup all my code nightly to S3. That worked great. But, I like GitHub’s UI and feature set better. Plus, it’s one less server I have to manage. So, where do I mirror my code to? (Much of my code is checked out locally on my computer, but not all of it.)

Back in 2006, my boss at the web agency I was working at told me about rsync.net. They provide you with a non-interactive Unix shell account that you can pipe data to over SFTP, rysnc, or any other standard Unix tool. You pay by the GB/month, and they scale to petabyte sizes for customers who need that. So, I signed up and used them to backup all of my svn (remember svn?) repos. With the rise of git and switch to GitHub, I cancelled my account and mostly forgot about them.

But, aha!, I now have new data storage problems. Rsync.net could be a great solution again. So, I re-signed up and setup my primary web server to mirror all of my GitHub repos over to them each night. Here’s the script I’m using…

Next up, important documents. Traditionally, I’ve kept everything that would normally go in my Mac’s “Documents” folder in my Dropbox account. That worked great for a long time. But once I started paying Google for extra storage space for Google Photos (more on that later), it felt silly to keep paying Dropbox as well. So, after 10+ years as a paid subscriber, I downgraded to a free account and moved everything into Google Drive. Sure, it’s not as nice as Dropbox, but it works and saves me $10 a month.

Like I said above, I mostly trust Google, but not entirely. So, let’s sync my Google Drive’s contents to rsync.net, too. Edit your Mac’s crontab to add this line…

30 * * * * /usr/bin/rsync -avz /Users/thall/Google\ Drive/ user@server.com:google-drive

Also, I keep all of the really important paperwork that would normally be in a fire safe in my garage in a DEVONthink library so I can search the contents of my PDFs. It’s synced automatically with iCloud and available across my mobile devices. But still, better back that up, too.

45 * * * * /usr/bin/rsync -avz /Users/thall/FireSafe.dtBase2 user@server.com:

So, that’s all of my data except for the big one – my family’s photo and home video archives.

For a long time I kept all my family’s archives in Dropbox. I even made an iOS app dedicated to browsing your library. I could have stuck everything in Apple’s Photos.app where it’s available on my devices via iCloud, but that’s tied to my Apple ID. My wife wouldn’t be able to see those photos. Plus, any photos she took on her phone would get stored in her iCloud account and not synced with the main family archive. So, we used the Dropbox app, signed-in to my account, to backup our phones’ photos.

But, like I said earlier, our photo and video library become to big to comfortably fit in Dropbox. Plus, Google Photos had just been released and it was amazing. Do I like the thought of Google’s AI robots churning through my photos and possibly using that data to sell me advertisements? No. But, their machine-learning expertise and big-data solutions make it really hard to resist. So, I spent a week and moved everything out of Dropbox into Google Photos.

Now everything is sorted into albums, by date, and searchable on any device. I can literally type into their search box “all photos of my wife’s grandmother taken in front of the Golden Gate bridge” and Google returns exactly what I’m looking for. It’s wonderful.

My wife’s phone has the Google Photos app installed with my account on it so every photo she takes gets stored in a shared account we can both access and view on all our devices.

But what’s the recurring theme of this blog post? That’s right. I don’t fully trust any cloud provider to be the only source of my data. Someone clever said “the cloud is just someone else’s computer.” That’s exactly correct. If your data isn’t in at least two different places, it’s not really backed up.

But how do I backup my 500GB+ of photos that are already in Google’s cloud? And then how do I keep new items recently added synced as well?

As usual, I tried to find a way to make it work with rsync.net. I found a great open-source project called rclone. It’s a command line tool that shuffles your files between cloud providers or any SFTP server with lots of configurable options and granularity.

First off, even if rclone does do what I need, I can’t just run it on my Mac. My internet is too slow for the initial backup. I need to use it on one of my servers so I have a fast data center to data center connection between Google and rsync.net.

Getting it setup on one of my Ubuntu servers at Linode was a simple bash one-liner. Configuring it to then work with my Google and rsync.net accounts was just a matter of running their easy-to-use configuration wizard.

Note: rclone doesn’t support a connection to Google Photos. Instead, you need to login to Google Drive on the web and enable the “Automatically put your Google Photos into a folder in My Drive” option in Settings. (And also tell your Google Backup & Sync Mac app not to sync that folder locally – unless you have the space available – I don’t.) Then, rclone can access your Google Photos data via a special folder in your Drive account.

With everything configured, I ran a few connection tests and it all worked as expected. So, I naively ran this command thinking it would sync everything if I let it run long enough:

rclone copy -P "GoogleDrive:Google Photos" rsync:GooglePhotos

Things started out fine. But eventually, due to Google API rate limits, it was quickly throttled to 300KB/sec. That would have taken MONTHS to transfer my data. And, the connection entirely stalled out after about an hour. I even configured rclone to use my own, private Google OAuth keys, but with the same result. So, I needed a better way to do the initial import.

Google offers their Takeout service. It lets you download an archive of ALL your data from any of their services. I requested an archive of my Google Photos account and eight hours later they emailed me to let me know it was ready. Click the email link to their website, boom. Ten 50GB .tgz files. Now what to do with them?

I can’t download them to my Mac and re-upload them – that’s too slow. Instead, I’ll just grab the download URLs and use curl on my server to get them, extract them, and sync them over.

I don’t have enough room on my primary web server – plus I don’t want to saturate my traffic for any customers visiting my website. So, spin up a new Linode, attach a 500GB network volume, and we’re in business. Right? Nope.

The download links are protected behind my Google account (that’s great!) so I need a web browser to authenticate. Back on my Mac, fire up Charles Proxy and begin the downloads in Safari. Once they start, cancel them. Go to Charles, find the final GET connection, and right-click to copy the request as a curl command including all of the authentication headers and cookies. Paste that command into my server’s Terminal window and watch my 500GB archive download at 150MB(!!)/sec.

(Turns out, extracting all of those huge .tgz files took longer than actually downloading them.)

Finally, rsync everything over to my backup server.

And that’s where I currently am right now. Waiting on 500GB worth of photos and videos to stream across the internet from Linode in Atlanta to rsync.net in Denver. It looks like I have about six more hours to go. Once that’s done, the initial seed of my Google Photos backup will be complete. Next, I need a way to backup anything that gets added in the future.

Between the two of us, my wife and I take about 5 to 10 photos a day. Mostly of our kids. Holidays and special events may produce a bunch more at once, but that’s sporadic. All I need to do is sync the last 24 hours worth of new data once every night.

rclone is the perfect tool for this job. It supports a “–max-age=24h” option that will only grab the latest items, so it will comfortably fit within Google’s API rate limits. Once again, setup a cron job on my server like so:

0 0 * * * rclone copy --max-age=24h "GoogleDrive:Google Photos" rsync:GooglePhotos

And, that’s it. I think I’m done. Really, this time.

All of my important data – backed up to multiple storage providers – and available on all of my and my family’s devices. At least until the whole situation changes yet again.

A few more notes:

All of my web server configuration files are stored in git. As are all of my websites’ actual files. But, I still run an hourly cron job to backup all of “/var/www” and “/etc/apache2/sites-available” to rsync.net since it’s actually such a small amount of data. This lets me run one command to re-sync everything in the event I need to move to a new server, without having to clone a ton of individual git repos. (I know I need to learn a better devops technique with reproducible deployments like Ansible, Puppet, or whatever the cool tech is these days. But everything I do is just a standard LAMP stack (no containers, only one or two actual servers), so spinning up a new machine is really just a click in the Linode control panel and couple apt-get commands and dropping my PHP files into a directory.)

My databases are mysqldump’d every hour, versioned, and archived in S3.

All of the source code on my Mac is checked out into a single parent directory in my home folder. It gets rscyn’d offsite every hour, just in case. Think of it as a poor man’s Time Machine in case git fails me.

I do a lot of work in The Omni Group‘s apps – OmniFocus, OmniOutliner, and OmniGraffle. All of those documents are stored in their free WebDAV sync service and mirrored on my Mac and mobile devices.

All of my music purchases have gone through iTunes since that store debuted however many years ago. I can always re-download my purchases (probably?). Non-iTunes music ripped from CDs long ago, and my huge collection of live music, is stored in iTunes Match for a yearly fee. A few years ago when I made the switch to streaming music services and mostly stopped buying new albums, I archived all of my mp3s in Amazon S3 as a backup. I need to set a reminder to upload any new music I’ve acquired as a recurring task once a year or so.

Also, I have Backblaze running on my desktop and laptop doing its thing. So yeah. I guess that’s yet another layer of redundancy.

Coding on My iPad Pro

Last month, my 9-5 job was kind enough to gift me an iPad Pro and its new keyboard. I’ve had a few iPads in the past, but they’ve always ended up stashed away, unused, in a drawer somewhere. I simply never got hooked on their utility. I never found that killer app, which, for me, would be the ability to code anywhere. This Pro model, however, has changed all of that.

I’ve always had two Macs. One to take places and another to get “real work” done. In the past that meant a spec’d out iMac and an 11″ MacBook Air. More recently, it’s been a work-issued 15″ MacBook Pro that stays plugged into my cinema display 99% of the time and a MacBook (One) when I travel. The new MacBook is certainly the most portable Mac I’ve ever owned, but it’s slow and lacks the screen space to do any UI intensive work.

Now that I have an iPad Pro, I’ve sold my MacBook and only touch my MacBook Pro when I have serious work to do. The iPad has replaced nearly everything I use my laptop for. That may not be so unbelievable. Lots of folks like Viticci have moved to an iOS only way of life. As I do more and more tasks on my phone, I’ve been tempted to try going iOS primarily, but I could never make that jump because I code for a living.

Until now.

I was screen sharing from my iPad to another machine on my local network, when it dawned on me how great it could be if this particular Mac were always available to me – even from outside my house. So, I splurged and ordered a datacenter-hosted Mac Mini from MacStadium. Ten minutes later I was connected to my new Mac in the cloud. And ten minutes after that, I had Xcode open and started testing the waters.

I’m using Screens.app to connect. And with a good internet connection there’s virtually no lag when screen sharing with my new Mac Mini. I’m able to run a native Mac resolution of 1920×1200 on my iPad in full screen. That gives me plenty of room to run Xcode and the iOS Simulator. With Apple’s new external keyboard, all of my usual Xcode and OS X keyboard shortcuts work just fine. And since coding is primarily a keyboard driven activity, my arm doesn’t get tired from reaching out and touching the screen like a designer’s might.

All in all I’m thrilled with my new setup. It gives me the simplicity and benefits of iOS, while still allowing me to do real work outside of the house or from the couch.

ipad-pro-xcode

Constraints

I often find that constraints, real or artificial, can be a huge motivation and productivity boost when I find myself stalled on a project or piece of work. Forcing yourself to work within a specific limitation can cause you to find a creative solution in a direction you might otherwise never consider.

In my own work, I find time constraints the most useful. When I don’t know where to start developing or writing, I’ll often force myself to do something – no matter how small or tangential – to move the work along for thirty minutes or an hour. The artificial time limit frees my mind to attempt starting points I might not typically choose because I know that even in a worst case scenario, I’ll only lose an hour worth of work if my chosen path is unsuccessful. Similarly, when I find myself lacking motivation and energy to work, I’ll use the pomodoro technique. That gives me the kick in the pants I need to break the ice and get moving, which can often turn into real energy and sometimes even that magical “flow” state.

A few weeks ago on a Friday night I found myself burnt out from my day job. It had been weeks since I had done any recreational programming or development on my side projects. I desperately wanted to move them forward, but simply couldn’t find the energy.

So I tried an experiment.

I decided to give myself thirty minutes to brainstorm and come up with a product idea that I could then build to a shippable state within the next two hours. Further constraining myself, I decided to forgo my two 27″ monitors and do everything exclusively on my 11″ MacBook Air.

I knew the easiest way to come up with a new product idea was to examine my typical day and find a small pain point that I could solve with software. After mentally going through my usual day in my head, I began to focus on my calendar. I realized that I was often checking multiple times throughout the day to see how much time I had until my next appointment, which would influence which task I could take on before I have to be somewhere. (As many developers will know, a programming assignment that will take an hour, often takes longer as there’s a definite lead-up time before you really begin coding where you work to get yourself into the zone.)

I thought “Wouldn’t it be easier if I could just glance up and see how much time I had remaining?”

With that insight, I had my product idea.

Over the next ninety minutes I built a simple Mac menubar app that looks at your calendar and simply displays how long you have until your next appointment. I call it Up Next. I’ve been using it as part of my daily routine for the last few weeks and have found it works great. Without losing place in my current work, I can glance at my menu bar and see “Oh, I’ve got two hours until I need to be somewhere” or “Crap, I’ve only got twenty minutes to wrap this up”.

That little bit of awareness doesn’t distract me from getting in the flow, yet keeps me persistently time boxed and motivated.

If you find yourself similarly lacking motivation throughout the day, I encourage you to try placing a constraint on your work. And if you think Up Next might be useful to you, you can give the app a try.

Minion for Mac

In case you missed it, I’ve been working on a new Mac app for the past month or so. I spent the last few days polishing things and getting it into an MVP state – not quite as full featured as I’d like, but complete enough that I felt comfortable sharing with the world. So, yesterday evening I launched Minion.

What is Minion?

It’s a simple concept. Minion watches for events to happen on your Mac and then notifies you when they occur. Right now, those notifications go to your email or your phone via SMS or the Minion iOS app.

The initial idea for Minion came from my good friend Mike over at Divergent Media. Back in his video editing days, he’d spend hours waiting for export jobs to finish. I have a few friends who do animation for a living, and they confirmed that, yes, waiting for a render to complete is boring and can take hours. But with Minion, you can point the app at whatever job you’re waiting on and go someplace else. Minion will buzz your phone when the job finishes.

How does Minion know when the job is done? It watches for what I call “triggers”. Things such as whether or not a file exists, if a file has reached a certain size, or if a file’s size hasn’t changed in the last five minutes. Minion can also monitor the titles of your windows or the color of a pixel on screen. Does the app you’re watching throw up a progress bar? Point Minion at a pixel at the end of the bar, and when it fills in, Minion will notify you.

But this is just the start. My long term goal for Minion is to create a local IFTTT service for your Mac. Choose from a long list of triggers to watch, and then pick a task for Minion to carry out when that trigger fires. Put the two together and you’ve got recipes you can save, reuse, and even share with other power users.

This is just the beginning.

Nostalgia – Rename Your Photos

I have a problem. Half of my photos come from my iPhone (via the Dropbox uploader), which creates filenames based on the date they were taken. But all the photos from my awesome DSLR are named BLAHBLAH_7001.jpg and BLAHBLAH_7002.jpg. That annoys me. I want all of my filenames to be date based so I can quickly sort them and find the photos I’m looking for.

Much to my dismay, Hazel and Automator can’t solve that problem.

This Mac app does.

Launch the app, drag and drop a bunch of photos onto the window (or the Dock icon), and boom! Everything gets renamed appropriately based on the EXIF date.

Currently, there’s no way to customize the output date format. It’s set to yyyy-MM-dd HH.mm.ss. I make no apologies for that. If you’d like something different, feel free to modify the Xcode project or submit a pull request with the ability to customize the date :) Also, it only looks for the standards based yyyy:MM:dd HH:mm:ss EXIF format. If your camera doesn’t follow along, Nostalgia won’t be able to parse it. Again, pull requests are very much welcome.

You can download Nostalgia for your Mac from here.