I was completely at my wit’s end and feeling like I had lost my mind until about a half hour ago. Let me start from the beginning…
I don’t have an exact date, but within the last week I realized that I was unable to ssh into my primary web server – the one that runs my business website, activation server, etc. It’s sort of the linchpin for my tiny software company. When it goes down, I get worried.
At first I thought maybe the server was down? I hadn’t received any alerts, so I did a quick check. And, yes, it was still up and running and serving web traffic. Ok, did sshd somehow become unresponsive? I login through the Linode control panel and restart the service. Still can’t login.
It’s odd. I don’t get a connection refused. Not even a timeout. It just…hangs.
That’s the ssh output with the verbose flag. Nothing. I waited 10+ minutes and it never timed out or produced any other output.
I reboot the server itself and the problem persists.
Then, I notice some more oddities. I’m able to connect using ForkLift – my FTP client, which connects via SFTP. Also, SequelPro is able to connect to MySQL via ssh as well.
And then things get even stranger. This is all happening on my iMac. I try connecting from my laptop, and it works. My MacBook Pro is at home right next to my iMac, which is refusing to login. They’re both on the same wifi and thus the same IP. So, it can’t be that my home IP address got mistakenly banned somehow.
Next, I ssh into a different server and then hop to the problematic one. It connects without any trouble. At this point I’m thinking maybe the permissions on my local private key got screwed up. So, I blow away ~/.ssh and recreate all of my keys from a backup. Still can’t login.
Ok. I think about it for a few minutes and then – aha! – I have an Ubuntu virtual machine running on this iMac inside Parallels. I’ll ssh into it and then try and connect. That will rule out if there’s just something odd about my iMac’s LAN IP. (To be clear, my home network is perfectly ordinary. Just a cable modem and a router.) So, I login to the VM, try and connect, and it works fine.
At this point here’s what I’ve found:
- My iMac is the only machine that cannot login.
- I’ve connected successfully from behind the same public IP using a laptop, a virtual machine, and my iPhone and iPad.
- I’ve verified my
sshkeys are correct and have the appropriate permissions. - I can connect to other servers from the problematic machine – both at the same hosting provider (Linode) and others (AWS and DigitalOcean).
- I can connect from my iMac if I jump through any other server, first.
I start trying to think what could possibly be different about this one machine. And then it dawns on me. This all started around the time I updated my iMac to 10.15.4. My laptop is still on 10.15.3 – and, of course, the virtual machine isn’t macOS at all.
Totally grasping for straws I google for “10.15.4 ssh” and find this top result on the Apple discussion forums:
Catalina 10.15.4 SSH port > 8192 does not work when using server name instead of IP
This issue started just after upgrading to macOS Catalina 10.15.4.
After that update I am no longer able to open a SSH connection to a port greater than 8192 using server name (instead of IP). Yes, I do change the port on the server side prior to every test.
That can’t possibly be real?
Up until this point, I was connecting via a saved hostname defined in my ~/.ssh/config, which let me login simply by tying ssh some-server. So, I tried ssh ip-address -p9944 and it worked! (That server runs on an alternate ssh port.)
Ok. Time to narrow this down a bit further. I changed the server to listen on standard port 22 and tried connecting via the hostname once again.
Holy crap, it worked.
The user in the Apple forums was right. At least in my case, my one server that happened to be running on a non-standard ssh port above 8192 will not connect from Catalina 10.15.4 when using the hostname instead of the IP address.
Just to verify, I boot up a Mojave and Catalina (10.15.3) VM on the same iMac. They both connect fine, while the host machine continues to fail.
The internals of this is all so incredibly above my head I have no idea what the underlying problem might be. Am I and this one other forum poster just doing something totally bizarre yet the same? This ssh setup has been working for years for me until just the last week. I would love to be proven wrong and told I’m an idiot. But I don’t know what difference connecting via the hostname versus the IP address would make when specifically using a non-standard port above a certain threshold.
It just….sigh.
I’m not even going to go into it. I don’t want to end up on Hacker News again bitching about Catalina. I just hope I’ve stuffed this post with enough keywords so that anyone else searching on Google might come across the answer.