Uncategorized

I occassioanlly need to scan a folder and all of its subdirectories to see if any of them DO NOT contain files of a certain type.

I’m fully aware you can do this with some combination of shell commands, but I always spent 20 minutes googling for how to do it again every time I needed to. It was faster just to write this small utility myself.

I call it dnc, which stands for “does not contain”. You can download the source or a pre-built binary on GitHub. The builds aren’t notarized. So be sure to ask Apple if it’s OK to run this on your Mac.

USAGE: dnc --directory <directory> --query <string> [--invert]

OPTIONS:
  -d, --directory <directory>
                          The directory to recursively scan.
  -q, --query <string>    A string to search for in the filename of the directory's files.
  --invert                Invert the query. i.e., show directories that DO contain the query.
  -h, --help              Show help information.

My primary use-case is finding which albums in my music collection (folders on disk) are in a lossy format. This gives me a quick shopping list of used CDs to buy when I need something mindless to do during quarantine.

A typical command would be

dnc -d /path/to/music -q flac

That will output something along the lines of

/path/to/music/Violent Femmes/1999-11-23 - Viva Wisconsin
/path/to/music/Weezer/1994-05-10 - Weezer
/path/to/music/White Denim/2013-10-29 - Corsicana Lemonade
/path/to/music/Yonder Mountain String Band/A Decade of Yonder Live, Vol 9_ 6_29_2006 Apple Valley, MN
/path/to/music/Zero 7/The Garden

which means of all the folders recursively inside /path/to/music, those directories DO NOT contain any flac files.

A key difference of this script than many of the solutions that Google gives me, is I’m not interested in every file that does not match query. I just want to know about any folders that do not contain matching files.

You can use the full command line arguments listed above with --help, or you can quickly run the command on your current directory by only passing a query string like this:

dnc flac

In line with my affinity for backing up and owning my data, one component of that strategy is avoiding proprietary file formats and databases whenever possible and reasonable. That’s why I prefer plain text (and more recently .textbundle) for all of my notes, and why I’m so meticulous about how I organize my family’s photo archives.

It gives me the agility to move from app to app or even (heaven forbid, it may eventually happen) to a new platform as my needs change. If you own your own data, there’s no lock-in.

So along those lines, here’s a very tiny optimization (is that the right word for this?) that I’ve been doing for years that helps keep my reference material organized and more easily searchable and filterable.

(Oh, and also a quick story about how I effed up someone else’s data.)

My primary “database” is the Finder – files on disk. And while Spotlight is amazing and wonderful and all that (go purchase HoudahSpot), I find its results too fuzzy and noisy unless I’ve really lost a file. It’s way faster for me to find what I’m looking for using the combination of how I naturally organize files into folders based on the client / app / subject / etc. and then sorting by date.

And it’s sorting by date that this simple trick is all about.

I gave up long ago relying on file creation and modification dates as a reliable way to sort files. They’re too easily corrupted.

Some modern Mac apps that adopt the post-Snow Leopard policy of automatically saving will save files behind your back just by viewing them. That can easily blow away a file’s timestamp. Or, if you restore from a backup, it’s very likely the original dates won’t be preserved. And now that all our data is flying across the network at any given moment with Dropbox, iCloud, Google Drive, etc. it’s easy for a bug in their sync code or your own mistake to propagate everywhere instantly.

My solution is basically what every Microsoft Office Warrior has already been doing ever since email became a thing and people needed to keep track of multiple revisions.

I stick the date in the filename itself.

Ok, I know this is ridiculously obvious, but it works, it’s flexible and has saved my bacon many times.

Rather than relying on filesystem dates, which can be mangled at any time, I sort by filename and voilà.

Here are my archived photo albums:

It’s especially awesome for this purpose because, for me, I want my photos / videos stored chronologically. After all, I mentally see them as a timeline of my life. And if every album of a birthday party were named “Birthday Party Blah Blah,” it would be a nightmare to find what I was looking for among 1,000+ folders.

This strategy can really shine when you go beyond Finder and use an app that can filter, search, and sort by dates this way.

Here’s my note-taking app focused on my daily standup meeting notes. In addition to the full-text search, I can quickly filter to meetings that took place in a given month or day.

This whole thing works for me because my memory is naturally oriented around dates and timelines. When I’m trying to reference a piece of information, I can almost always pin it down to the specific month it was created, at a minimum, and then use that date range as a launching point for more refined searching if needed.

But how do you make adding these dates faster? Text snippets, of course.

I’m a terrible typist, and I can barely remember the current date as is. So I’ve been using Typinator for years to convert short abbreviations into longer snippets, including variables and dynamic data.

(Lots of apps that do this sort of thing. I feel like TextExpander is the default choice for most people, but I’ve always preferred Typinator for many reasons that warrant their own blog post.)

So when I need to insert the current date, in any app, I type .dt

…and it expands into 2020-10-09. You’ll also see in that gif that typing .ts inserts the current time, too.

Neither of these snippets is amazing or original or unique. Folks have been raving online for years (rightly so) about all the amazing and clever uses you can find for apps that automate typing for you.

(If you’re curious about the sorts of things you might use a text expansion app for, go visit David Sparks. I’ve stolen a ton of great ideas reading his blog over the years.)

Anyway, this blog post’s point is just a long-winded way to say that inserting the current date into filenames or note titles is a small habit I’ve learned that makes me so much more organized. It lets me find the content I’m looking for much faster and get on with my work.

Story Time

I wrote above about the delicate nature of filesystem dates and how, if you rely on them, losing that part of your reference material can be a Bad Thing™?. That’s why I use the date-inside-the-filename trick.

But what clued me into this?

In 2009, when I was building the first version of Nottingham over Thanksgiving weekend, I sent an early build to John Gruber. I knew he was a big fan of Simplenote, and I was excited to show that my new Mac app would sync with their backend.

It didn’t go well.

(I know it’s not cool to publish emails with another person without their permission first, but I hope he’ll let this one slide as it’s been 11 years and I was clearly the idiot in the exchange.)

So, yeah. That was my first time writing any sort of network synchronization code, and I probably shouldn’t have used him as a guinea pig.

That said, a few years later, I think it was WWDC 2012, I struck up a conversation with him at Bourbon & Branch.

I basically said, “Hi, I’m the guy who fucked up your note timestamps. Sorry about that.”

I don’t remember his exact reply other than being very nice about it and not punching me in the face, but I do remember during the entire short conversation he was scribbling into a Field Notes journal. I can only assume he had learned his lesson and swore off digital note-taking.

I don’t quite understand how it’s been five years already. Nevertheless, happy birthday to the most fearless and persistent young lady I’ve ever known.

If you need something to listen to this weekend or during your next commute, here’s a link to me prattling on for forty-six minutes about how and why I build the software I do.

Brett Terpstra was kind enough to invite me on as the guest of his Systematic podcast this week. And when he says I’m “on about the same level of a geek frequency as” he is, well, that’s a heck of a compliment in my book.

(Sorry in advance for the SEO-keyword heavy blog post. I want to make sure Google can do its job.)

After my post last week about the updates I made to my audio app Ears, longtime internet buddy @macrael asked:

@tylerhall Do you use libraries for the hotkey-setting UI or the start-on-launch stuff? I’m putting together my first Mac app and am looking into those parts rn.

I do, actually. In addition to replying to MacRae on the nightmare birdsite, I thought I’d post the two helper projects here for anyone else searching.

(It saddens me that so many of the Mac development resources I used to learn from and rely on have disappeared from Google (and Apple’s developer website) since Mobile swallowed the industry. Ah, well. Here’s my contribution…)

Creating Custom Global Keyboard Shortcuts in your macOS App

I’ve used the excellent MASSShortcut framework for years to handle global hotkeys in my Mac apps. Here’s the original blog post about the project from Vadim Shpakovski. (By the way, this new app from Vadim looks super helpful.)

When I rebuilt my app CommandQ, I needed a way to trap keyUp events in addition to the keyDown events that MASShortcut offered. I modified the framework to allow that, and I’m sorry to say I’m a terrible open-source citizen and never contributed that patch back to the original project. But you can find my fork here.

How to Create a macOS Login Item For Your Mac App

Here’s a small sample Xcode project I extracted from my existing apps. It shows how to create a helper app inside your main application to launch your app at login. (Hopefully, only if the user consents first!)

I’ll be up-front and say I have no idea if this sample code is the “right way” to do this, if it ever was the right way, or if it’s even the modern way any longer. I cribbed it together from a blog post years ago, and it’s worked fine for me thus far.

If anyone would like to point out a better way, I’m all ears. And if anyone knows the original blog post author I should give credit to, I’d love to know that, too.

One of the best things that have come out of the pandemic for me has been my little Mac app, Ears. I had the idea for it and built it about a month into quarantine because I was in so many remote meetings throughout the day. And depending on the time of day, how much notice I had before the call, if my kids were around, all sorts of reasons – I found myself frequently switching my Mac’s audio between speakers, AirPods, headphones, etc. It was a pain, so I built Ears to make that easier.

Since that first release in June, I’ve been refining the app to fit my workflow even better. And tonight, I’m delighted to push out a new release with additional features for all the work-from-home-warriors out there jumping between calls.

First up, if you don’t know the purpose of Ears or why I think it’s helpful, here’s a quick walkthrough FAQ I wrote that explains the why’s and how’s of the app.

InstaMute

InstaMute is a terrible name for a feature, but I needed something to put in my git commit message, and that’s what popped into my head.

The client I work with daily uses RingCentral for all their scheduled and recurring meetings. But then, they use Microsoft Teams for ad-hoc meetings and for employees low enough on the food chain that management doesn’t want to pay for their RingCentral account.

But within the company I work for, the developers bounce around between Google Meet and Discord.

I’m saying all this so I can point out that each service – in the browser or their desktop apps – use a different keyboard shortcut to mute/unmute the microphone. Worse, even if you do remember the correct shortcut per app, you have to switch focus to the app to invoke it because it’s not a global hotkey in every case. (And don’t get me started that they all hide the on-screen mute button until you hover over it, so you don’t initially know where to try and click.)

And it drives me crazy because when working from home – or even at the office – I don’t want coworkers or clients hearing my kids screaming, me eating potato chips, or whatever else. But having to jump back to some random app and quickly unmute to speak and then go back on mute was a thousand paper cuts every workday.

I needed a global hotkey that mutes the system audio input, so it works in every app. I’m sure there’s probably a native way to do this or some other app that will solve it for me, but it seemed like a good fit to add to Ears.

InstaMute has three modes of operation:

1. Toggle On / Off Like it says, press your selected keyboard shortcut, and Ears will toggle your mic’s mute on and off.
2. Push to Talk Ears will keep your microphone muted. Only when you press and hold the hotkey will it turn on. As soon as you release the keys, your mic will go back on mute.
3. Push to Mute The opposite of PTT. A cough button. Ears keeps your mic live except when you press and hold your keyboard shortcut. Let go, and folks will hear you again.

And to make sure you always know if you’re on or off mute, the Ears menu bar icon will update based on your mic’s current status to show a cross-line when muted.

Default Volumes

A customer suggested this next feature.

You can assign a default volume on a per-device basis. Speakers, headphones, or even your microphones. Do you want your iMac speakers to be quiet? But don’t mind blasting your AirPods. Give them each a preferred volume, and when you switch devices with Ears, the app will automatically set the correct volume.

(I know macOS will, in theory, do something like this for you, too. But it’s never worked consistently for me. Hence why the user suggested this feature, and I wanted to build it.)

Ears has a straightforward UI. I’m trying not to clutter up the interface beyond its core functionality. So forgive me for hiding this feature behind a context menu. I don’t expect most users will need it frequently – or at all – so it seemed a fine enough compromise.

Right-click on an audio source to choose a value and enable or disable the setting per device.

Notifications

This feature was added shortly after Ears’ first release, but I never mentioned it in my earlier blog post.

When your system audio changes – whether Ears makes the change or some other app – Ears will display a notification letting you know the new active devices.

I think it’s a great way to confirm what you think happened really did happen and prevent surprises.

Also, you can choose to have your Mac speak your new audio device selections to you. I’ve added a slight delay between when your device changes and when the audio speaks. This delay gives your new device a quick moment to fully come online. So, in the case of switching to a different speaker or headphones, you’ll hear the change announcement with your new choice to confirm it’s active.

Download and Enjoy

So that’s the new version of Ears.

The app is free to download and use. But you can purchase an optional, pay-what-you-want license to remove the nag screen that appears when you open the app.

A Few Nice Things

I made Ears on a lark because I really just wanted it for myself. But it’s turned into one of the most well received apps I’ve ever built. So if you’ll forgive me for bragging, I thought I’d share a few comments I’ve gotten from customers this Summer, and also say thanks to the thousands of folks who have given it a try. I get so much satisfaction everytime someone takes time to email and say it made their workflow during this crazy time just a little better.

I didn’t know I needed this until today, but this is seriously a great app. I’ve spent so much time fiddling with audio devices the past few months 🙁

I do online teaching for a University (in Australia) and have a stream deck set up with some apple scripts to mute/enable the mic. Your software was the missing link in getting it all together.

Ears is great! Thanks for making it! I use it every day to switch among several devices. I wanted to show my support with more than just words, so I purchased a license a few minutes ago.

As an aside: I did notice that Ears posts notifications for audio source changes done in Ears, which is perfect for my use case. It can give that extra piece of confidence when switching.

I just wanted to say – thanks for Ears! I tried it out after reading about it on your website, and it’s exactly what I wanted to help smooth over many audio devices during Work From Home and otherwise. This is especially helpful in an environment where you don’t want to accidentally have your Mac switch to speakers, like in an open office environment. Anyways, thank you again for a great app! Looking forward to see how Ears evolves 🙂

Ears has been a lifesaver – thanks so much for the program!

A coworker and I have been working crazy hours since March on a huge new product feature – him on Android and myself on iOS. Quite frankly, it’s maybe the best work we’ve done in our careers. And work I, at least, wasn’t sure we were even skilled enough to pull off. When we first pitched it to the client, we asked for eight weeks of uninterrupted dev time to build an MVP. They gave us five.

But we did it. And since then, it’s been layering on more complex interactions, polishing, and performance improvements.

Yesterday we were wrapping up code review of a bunch of new math in my office. He was surprised when I mentioned that on my todo list was to go back and update some earlier code to use the formulas we had just gone over. He thought I had always been using this more performant method.

I told him:

I cheated back in April when I first wrote this. I was too scared of the math to try and do it the right way.

He replied:

I didn’t know enough about it to be scared.

It was a throwaway comment – more of a joke, I think. But it struck me and stuck with me the rest of the day, and now I’m writing this post.

Is this what the old cliché “ignorance is bliss” means?

I don’t know, but it made me think back to past projects I’ve built. Some that succeeded and others that failed spectacularly.

For many of the successful ones, I think his observation was spot-on. I don’t know if I would even have attempted the work if I had had enough experience to know what lay ahead.

On the other side, I might have avoided some of the lowest points in my life if I had been more attuned to knowing what I didn’t know.

I can hardly believe I’m quoting this man, but Donald Rumsfeld had a not-completely-ridiculous moment of clarity in 2002 when he popularized unknown unknowns:

Reports that say that something hasn’t happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don’t know we don’t know. And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones.

When we started on this project months ago, there were plenty of known unknowns we knew we’d have to face, and we’ve worked with the design and product teams ever since to resolve.

But there have also been many staggering unknown unknowns that we never considered. Some we’ve gotten past, while others have morphed from tickets, to sprints, to their own epics.

If he and I had predicted these challenges upfront, I’m not sure if we would have pitched the work at all. But we missed them. And now I think that’s a good thing. Because, as I said above, unfinished loose ends or not, we now find ourselves mere weeks from shipping the best work we’ve ever done. (We called our feature work this Summer “pure resumé fodder.”)

So now, I see the two sides as a balancing act.

Early in my career (and even before I had a career), I’d jump into a new project without any hesitation or thought given to how big of a challenge it might be. Or if I was even capable of doing it.

That’s a good thing, right? Challenging yourself is the best way to grow. And how many incredible leaps forward in our industry would never have happened if everyone let self-doubt and worries block them from even starting?

But now that I’m into the middle(?) of my career and leading a team of (mostly) younger developers, I find myself looking at the new features and challenges on our roadmap differently than I did before. Am I more cautious? Timid? Wary of wasting time?

Whatever the reasons, as I wrote about in Fear and Light, I need to remind myself to get out of my head, out of my own way, and look to my coworkers to remind me to hold onto that optimistic approach to difficult problems where anything is possible.

As I’ve written about previously, for better or worse, Google Photos is the initial destination of all the family photos and videos we take as well as the source of truth for the albums I sort them into. I’ve tried every consumer photo organization tool/website/app on the market, and nothing comes as close to hitting my feature requirements as Google Photos. (Well, short of building my own solution, but that’s…uh…not yet.)

Photos is the only Google product I use. (Besides search – sigh). I gave up Gmail years ago because even with full backups of all my messages, my email address itself is the key to nearly every other online account. The chance of getting locked out due to an automated flag is too high – even if I am a paying customer. But if I lost access to my photo library stored with Google? That would be bad, but not the end of the world since I have all that data backed up.

However, I’m always playing the long game and thinking about contingency plans with data this important. Chief among them is my looming monthly price increase apocalypse. I’m currently paying Google $99/year for 2TB of storage space. When I hit that limit, the next tier is 10TB for $600/year. That’s a hell of a jump for that next byte. And while I totally get the business reasons behind that pricing, sheesh.

So I’ve been thinking about my eventual exit strategy. The obvious next and most comparable choice is Amazon Photos. (I’m keeping a close eye on PhotoPrism.) They solve the storage pricing problem because they’re Amazon and just charge you an extra 1TB at a time as your needs increase.

I’m perfectly willing to pay for what I use, so that’s great. And Amazon’s website and apps are actually better than Google’s in many ways. But they do fall on their face as soon as you start dealing with videos larger than 2GB (easy to do with kids and a modern iPhone shooting 4K video) or over 20 minutes long.

So, I’m keeping a very close watch on Amazon and hoping they improve enough to be the right solution in the future.

Anyway, the point of this blog post is to say that I’m preparing for an eventual move to another photo cloud service. I’m also trying to keep my local backups neatly organized. So, I wrote a small command-line tool to specifically deal with the Google Photos backup format that you’ll receive if you request a dump of your data.

It takes Google’s directory structure and all their duplicated files, merges, sorts, and deduplicates your photos and videos into a sane folder structure – the one I’ve been using for over a decade.

You can request a dump of all of your Photos or just specific albums/dates using Google Takeout. (Kudos to Google for making this sort of stuff so easy.) It works great, and you’ll get everything. The problem is the backups are structured as if you’ll only ever do a single backup in your life – as opposed to incremental ones. And they also don’t deduplicate your data before sending it to you. (I understand why.)

Once your backup is ready, you’ll get everything split into 50GB .tgz files. Each one will extract into the following directory structure:

You’ll get a folder for every item’s capture date. If you took 50 photos on August 15, 2020, you’d get a folder named “2020-08-15”. Except, for reasons I don’t understand, you’ll occasionally get a folder named “2020-08-15 #2”, too. Same day, another folder.

Google will also create folders for every album included in your backup. Perfect. But any items in those albums (folders) will be duplicated in their respective date folders, too. So, if you have a 2GB video included in two albums, it’ll be in three backup folders, which means 6GB of space.

And this is totally fine. It’s definitely the most flexible and complete solution and leaves it to the end-user to figure out what to do with all this data. And what I want to do is convert everything into a structure that looks like…

…along with all of my items deduplicated. So, any items that are included in an album are not duplicated in a year-month folder. Those date-based folders only contain items that are not sorted into albums.

I spent a few hours messing around with various bash scripts and some StackOverflow posts but realized I needed something better than any shell script I could write. (I’m sure someone could write it, though.)

I came up with a tiny Swift command-line tool that scans all of your files and stores a hash of each into a sqlite database. (Thanks, Gus!) Then, figures out where each file belongs, moves it there, and ignores any duplicates it finds.

As you might imagine, it’s not the fastest process, but the results were worth it for me. My iMac Pro was able to scan and process my 1.4TB library (100k+ files) on an external USB3 spinning drive overnight. (Sorry, I should have timed it. I think it was between 4 – 8 hours.)

I’m quite pleased with the results. Not only did it clean up all of the per-day folders into a more sane by-month directory structure, but removing duplicates cut the total file size by 30%.

Running the tool is a two-step process.

First, you need to import your Google Photos’ backups into your library – a shared folder where all of your photos and videos are kept. The reason for this step is that Google’s backup structure will often contain duplicate folder names in addition to filenames. And since merging folders on macOS is a delicate process, the import step will do that for you.

Just run this

photoz import /path/to/google/photos/backup /path/to/library

on each of the .tgz files that Takeout gives you.

After everything is merged in, run

photoz organize /path/to/library

and wait.

If all goes well, you’ll end up with a sane by-month/album folder structure that uses considerably less space.

The code is open source. Fixes and improvements are welcome. I haven’t cleaned it up much since first writing it, but it’s only three Swift files, so it should be easy to dive into and customize to your liking.

And while it works for me and I did a crap ton of dry-runs and testing building this tool, please, please, please make backups of your data before running an internet stranger’s code over something as important as your photo library.

Oliver Reichenstein, founder of iA Writer, had this to say about his post on Apple and modern software monopolies:

This wasn’t easy to write it to publish. But, today, I just had to hit that publish button and get it over with. I have another hot potato post on subscriptions I’ve been juggling with for some time now. Some thoughts are sketched out in the monopoly post. I’ll drop that one soon, too. These topics are complex and dangerous. Writing about them publicly is simply terrifying.

For me, two highlights were:

Apple are not treating Netflix, Nintendo, and Spotify like iA Writer to be fair with us small devs. Think about it. Apple doesn’t profit much from taking 30% of iA Writer’s sales. It would cost them a smile to cut us off. And, given that from the hundreds of thousands of app, only very few reach our position, it wouldn’t be a big deal to cut us all loose. The benefit of having small passionate devs is not in the money they make from them. The benefit of having us is that we enrich the platform. For years, people have been telling us that they buy Apple gear because of iA Writer and similar apps. The benefit of cutting 30% from the small ones is that they can say “we treat everyone the same” and cut off a major chunk of the big Netflix, Spotify and Fortnite cakes.

and

Apple, a company with California Hippie roots that encourages to think different, has a developer community that is afraid to speak their mind in public.

The entire post is well worth reading. And whether you side with him and the $17.86 billion corporation or the $1.97 trillion corporation, you gotta admit it takes guts to lay out that argument on your company blog when the future of your business depends on the kind of day your next anonymous App Store reviewer is having.