Here’s one of those ideas you have when you are not sleeping: why don’t we use frequency hopping to make it hard for attackers to find listeners to attack?
In scenarios where you want to keep the port number a secret, you could randomly vary it’s location. You could use TOPT, so both sides can rendezvous. Seems this wouldn’t be that hard to add to ssh. The sshd_config file might look something like this:
# Enable dynamic port listening, and the TOPT secret Port dynamic 6000 16000 PortSecret 12345678901234567890
And the user’s ~/.ssh/config file would then have something like this in it
Host crazy.example.com Port dynamic PortSecret 12345678901234567890
You could let the PortSecret default to something derived from host key.