Here’s one of those ideas you have when you are not sleeping: why don’t we use frequency hopping to make it hard for attackers to find listeners to attack?

In scenarios where you want to keep the port number a secret,  you could randomly vary it’s location.  You could use TOPT, so both sides can rendezvous.  Seems this wouldn’t be that hard to add to ssh.  The sshd_config file might look something like this:

# Enable dynamic port listening, and the TOPT secret
Port dynamic 6000 16000
PortSecret 12345678901234567890

And the user’s ~/.ssh/config file would then have something like this in it

Host crazy.example.com
   Port dynamic
   PortSecret 12345678901234567890

You could let the PortSecret default to something derived from host key.