Triage Your Email in the Car with Siri

Installing a CarPlay system into my ten year-old Subaru is easily one of the best purchases I’ve made in years in terms of how it’s changed the way I go about my day. Right up there with the original iPhone. I know lots of folks who swear by their iPad, and, sure, I like mine. But iPad has always seemed like a product in search of a problem. CarPlay feels like a product that is a solution.

CarPlay has made my morning and afternoon commutes far more enjoyable. And with so much dead time between home and work, I’m now trying to see if I can make some of that a bit more productive, too.

I’ve got a big ass post about using Siri Shortcuts with OmniFocus coming soon (I hope), but for now I want to talk about email.

I understand that Apple wants to limit what’s possible while driving for safety reasons, but Siri’s constant “I’m sorry, I can’t help you with that while you’re driving” responses are slightly maddening. All I want is for Siri to read my unread emails.

If I cheat and leave my phone unlocked while barreling down the interstate at 75mph, Siri will read my recent email. But that’s kind of it. I can’t perform common actions like archiving, deleting, marking as read/unread, etc. I can’t even reply. (Or if you can, I haven’t discovered the correct voice incantation to make Siri do so.) You can compose and send a new email though, so there’s that at least.

Nonetheless, what I’m really after during my morning commute is the ability to triage my inbox and prepare for my daily standup call that happens as soon as I arrive at work. Between the time I wake up and when I finish dropping my kids off at school and start my drive in, I simply don’t have time to read everything that came in overnight or from my coworkers in earlier timezones.

What I want is to process and clear out all the junk (not in the literal spam email sense) and get a handle on any messages that need action before the people who sent them blindside me about their status during our phone call.

So, I built something to do that. I call it Voxmail, and it’s free on GitHub.

There are two components.

  1. A 49-step Siri Shortcut that you install on your iPhone.
  2. A small PHP script that you can throw onto most any web server that the Shortcut communicates with. Other than two Composer requirements that deal with speaking to email servers, it’s plain, vanilla PHP. It doesn’t even need a database.

When you run the Shortcut, it connects to the PHP script, which fetches your email and returns it in a format optimized for the Shortcut to read, parse, and let you take action on.

Siri will speak a summary of your unread messages and then allow you to take action on each individual email. You can listen to the full email body. Or, you can archive, delete, mark as spam, mark as read or unread, and send a reply.

I’ve been using the script for a week or two and it’s fast, reliable, and awesome. At least with Fastmail. I have no idea how it will perform against a Frankensteined IMAP implementation like Gmail. But I’d love feedback! Also, it doesn’t support Exchange. So sorry. Pull requests are welcome, though!

How does it work in practice? Here’s a demo video…

And here’s another showing CarPlay in action…

Note: In this example I’m propping my iPhone precariously on top of my gear shift against the air vents. I’m just doing this so you can see the status of the emails updating live from the Siri Shortcut, to the server, and back down to my phone’s mail client. My voice commands and Siri’s output are going through CarPlay – not the phone. This all works with your phone happily locked and in your pocket. (And also on your watch and AirPods, too!)

And here’s the ridiculous Shortcut in all its glory.

Of course, you may be thinking: “Hey, you’re an app developer! Why didn’t you write an app to do this?”

I thought about it. I’ve actually written an iOS email client before, so I knew MailCore would be up to the task. And I’ve seen amazingly deep Siri integrations from other iOS developers like OmniFocus.

But I’ve done some basic Siri programming for my clients, and it was a royal pain in the ass to setup and debug. There’s no way I could have prototyped and built this project to the point it’s at in 7h 56m over the course of a weekend like I did. (I know the time down to the minute because Timing.app is amazing.)

And that’s not because of Swift or Xcode, etc. Swift’s static typing would have actually saved my butt a few times when I did some dumb PHP things. It’s all the damn hoops Apple makes you jump through just to run a bespoke app on your own device.

I wanted to build this for myself – not to make money or distribute through the App Store. And being an open source project, I sure wasn’t going to make pro-but-not-actually-developer-users who want to use it deal with certificates, provisioning profiles, etc. This whole Siri Shortcut / PHP script solution is a complete hack, but it’s easier than the alternative. Ok. Rant over.

So, go visit Voxmail on GitHub and give it a try. I’d love your feedback.

And if all of this was somehow already possible natively with Siri and I just wasn’t asking the right questions, boy do I feel dumb.

Rebecca Stand and the phone reason

If you saw my tweet from earlier today then you’ll already know the punchline to this particular bug report. But for those of you who don’t follow my every dumb online comment, I present to you the strangest corner case I’ve come across in fifteen years of professional development.

The app I work on during my day job was originally written in Swift in 2016, but some of the larger framework components are older Objective-C code dating back to 2012. The major development focus since I joined the project earlier this year has been on rewriting the UI layer. And given all the upheaval that in itself would cause, we’ve left the majority of the lower level code that talks to our API alone. No reason to throw out working, battle-tested code, right?

Anyway, a week or so ago we received a bug report from a customer who said the app crashed shortly after launch every time. The report they submitted was verbatim as follows…

Change the phone reason to Rebecca Stand and the app will crash

That didn’t give us much to go on, and QA wasn’t seeing any crashes on launch in their testing. And the logs coming to us from our crash reporting service didn’t show any bugs consistently crashing early in the app’s lifecycle. So, we filed the ticket away and moved on with the sprint.

Not to be deterred (and I don’t blame them), the customer got in touch with our 800 number and eventually got escalated to the head of QA.

Fast forward to today. He (the QA head) drives two hours outside Boston to visit the customer in person and install a special TestFlight build with extra logging onto their device.

He reproduces the crash. Every. Single. Time.

Being there in person with a debug build, QA is able to email me the log files directly, and every crash points to the same code…

let df = DateFormatter()
let someDate = df.date(from: dateStringFromServer)!

First off, yes. The problem is obviously that the code is force-unwrapping a string coming back from an API request. That’s a Bad Idea™.

In mine and my coworkers’ defense, this comes from that older layer of code we inherited from another company. This is nothing we’ve ever really touched because, previously, it had always Just Worked™.

Also, in defense of whoever did write the code originally, there was a contract in place with the API that guaranteed the string coming from the server was in the correct format. Sure, it’s still playing fast and loose, but, meh. I’ve certainly done worse things.

But, still. Why wasn’t it crashing for us in our testing? The customer graciously even let us log into the app as them, and we still couldn’t reproduce it on our devices.

I thought back to their original bug report…

Change the phone reason to Rebecca Stand and the app will crash

Who is Rebecca Stand? What is “the phone reason”?

And then I stopped thinking about what line of code was crashing and started thinking more closely about how it could crash.

Apple’s documentation for that method says…

Returns a date representation of a given string interpreted using the receiver’s current settings.

The “receiver’s current settings”.

“Rebecca Stand”.

I called the QA guy still at the customer’s home. “Open up Settings.app and check their date and time settings, language, etc.”

30 seconds of silence and then…

“Phone is set to English. They’re using military time. And, oh. Huh. His phone’s region is…Uzbekistan.”

Phone region. Phone reason.

Uzbekistan. Rebecca Stand.

The customer dictated their bug report to us via email using Siri. They speak English, live outside Boston, but their phone’s region was set to Uzbekistan instead of United States (or anything else more common).

The string coming from the server was the same format it had always been. But this was the first time (to our knowledge) our NSDateFormatter had tried to parse it as an Uzbekistanian format, couldn’t do it, returned nil, and crashed.

I’m not sure what the moral of this bug report is. Maybe to always program defensively? To take the time to really listen to your customers?

Anyway. That was my Monday.

Gone

So, uhhhh. This isn’t good. I really hope I’m just confused and not sounding a false alarm, but…

Back in April I wrote a quick post about how I was backing up the shared iCloud Photo albums that my friends and family all use to send pictures and videos of our kids back and forth.

The conclusion of the post was that all of those shared pictures are not stored inside your primary Photos.app library. Instead, they were squirreled away deep inside here…

~/Library/Containers/com.apple.cloudphotosd/Data/Library/Application Support/com.apple.cloudphotosd/services/com.apple.photo.icloud.sharedstreams/assets/

Fair enough. And for someone who is crazy paranoid about backing up all of their data like myself, that was fine. I just pointed Arq at that folder and rested comfortably knowing all the photos of my kids and my nephew and my best friend’s kids were safely being backed up.

Not so much with Catalina.

A reader emailed me today to ask if I knew where that folder had been moved to after upgrading to 10.15. I’ll admit – I had no idea it had been moved. I didn’t even think to check.

So, I looked, and, sure enough, that sharedstreams folder is gone. But where? All said, the shared albums had previously been taking up 112 GB, so I turned to DaisyDisk again to look for another large folder that size, and…

Nothing.

But here’s the thing. Photos.app still had the Shared Albums preference enabled. And I can still see all of those shared photos and videos in the app.

I put on my developer hat and started watching the filesystem while I browsed my shared albums. I found that when I double-clicked a photo to open it to its full size, Photos.app downloaded the file and cached it locally here:

~/Pictures/Photos Library.photoslibrary/resources/cloudsharing/data/<PERSON ID>/<ALBUM UUID>/

(Where PERSON ID is I think an identifier assigned to your iCloud account? Maybe? Not sure. And ALBUM UUID is the unique identifier for a shared album.)

Ok. That’s great. It actually makes more sense to have shared photos also live inside your Photos.app library instead of somewhere random in ~/Library.

But here’s the shitty part I’m trying to figure out. Of the 112 GB my shared albums previously took up, only 250 MB (MB!!!) are cached locally.

Don’t get me wrong. This is likely an awesome thing for folks low on hard disk space. Keeping possibly hundreds of gigs locally for shared photos was probably a silly idea to begin with. But…

and that’s a big but…

I don’t trust iCloud with my photos. And I’m not trying to single out Apple. I don’t trust Google Photos either. I don’t trust any cloud with my data. I love the convenience of iCloud photo sharing with friends and family and using Google Photos as a source of truth for my own family archives. But I want a backup of my data – just in case – that I’m in control of.

And now I don’t know how to do that. Here’s the Preferences window for Photos.app on Catalina…

There’s an option to download the originals of my own photos/videos, but not for shared albums? And, try as I might, I couldn’t find a way to “select all” and tell Photos.app to download everything like you can with your music stored in iTunes Match.

So, uh, Apple? How can I get the full-res versions of my five years worth of shared photos and videos? I obviously have the originals of the pictures that my wife and myself took. But there are hundreds if not thousands of images of my kids taken by their grandparents as well as all the photos of my sister’s kid, our friends’ kids, etc. that I would very much like to keep safe.

And, just to make sure I’m not absolutely crazy or mis-remembering something, I took a look at my historical Arq backups. I upgraded my iMac to Catalina on October 7th. On October 6th, that folder of shared albums was 112 GB…

By October 8th, it was gone…

And they certainly weren’t moved inside my Photos.app library because that hasn’t grown in an equivalent size.

Just 112 GB of memories.

Gone.

(Note for the HN crowd: Obviously, as this post talks about, I have backups. I haven’t technically lost anything. But this new caching strategy does my backups no good going forward as I won’t reliably have the full-res content automatically cached locally and available to be backed up.)

Followup…