How to Protect Your SSH Keys. Why Not Certificates?

[Man]

In March 2023, a popular code hosting site Github experienced an incident involving a leak of a secret RSA SSH key used for host operations github.com.

Such incidents are not uncommon. Most often, they are classified as “human error,” that is, an oversight by a specific employee who accidentally posted secret data in the public domain. In this case, the employee accidentally committed the secret key to an open repository.

Of course, it’s easiest to blame an individual. But if a security system allows for such errors, then you should think about changing the system first.

What happened on Github​

According to the official version, in mid-March 2023, the company suddenly discovered its RSA SSH secret key in its public repository. The security service immediately removed it and began an investigation. The secret key was changed within thirty minutes. After that, Git users saw a warning about a possible hack, as in the screenshot above:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s.
Please contact your system administrator.
Add correct host key in ~/.ssh/known_hosts to get rid of this message.
Host key for github.com has changed and you have requested strict checking.
Host key verification failed.

This warning scared many, although in fact the key change was not the result of an attack, but in normal mode. To continue working in normal mode, it is enough to delete the old key with the following command:

$ ssh-keygen -R github.com

Or manually update the file ~/.ssh/known_hostsby deleting the old public key and adding a new one:

github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=

Automatic key update can be performed using the following command:

$ ssh-keygen -R github.com
$ curl -L https://api.github.com/meta | jq -r '.ssh_keys | .[]' | sed -e 's/^/github.com /' >> ~/.ssh/known_hosts

After a correct update you will see the correct fingerprint:

SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s

SSH Key Infrastructure​

Let's figure out what happened.

Everyone knows how public key infrastructure (PKI) and asymmetric cryptography work. For example, the TLS protocol. Each server has a public/private key pair. When you visit a website, it presents a TLS certificate with a public key, and the browser performs a series of cryptographic operations to verify the certificate and generate an ephemeral key to encrypt the current session.

This infrastructure is used for almost all protocols using TLS, including SMTP and IMAP. But SSH does not use TLS and does not participate in the traditional PKI infrastructure.

Instead, SSH uses the Trust on First Use (TOFU) model. The first time you visit a server, you are asked if you trust its public key, you usually click "Yes" and move on with your life.

All this works fine until the key changes - and SSH starts complaining about a mismatch and something terrible could have happened, like MiTM (someone is intercepting your traffic).

You should check the new keys and make sure they are legitimate. This is a rather tedious procedure for regular users, so many just click "Yes". In the case of a real MiTM attack, the attacker counts on exactly this attitude.

What's wrong with current SSH certificates​

The immediate question is: why not use certificates for SSH hosts?

OpenSSH does support certificates, but in a slightly odd way. It has its own format that is much simpler than the X509 certificate format in TLS.

Essentially, an SSH certificate contains the following:

• public key;
• list of host names it matches;
• signature of the certification authority.

There is no pre-established set of trusted authorities (CAs). So anyone can generate a certificate on behalf of github.com. In theory, this is a security issue. But in reality, it is not a problem, because almost no one checks SSH host certificates.

In principle, you can use regular PKI certificates for SSH, but this is a bad idea, because it contradicts existing standards and practices for their use.

There is only one way: always use a private key. Github is a mass service with thousands of servers, and everyone needs access to one private key. DevOps is obviously responsible for deploying this private key to new systems as they are put into production. Hence the risk that the key accidentally ends up in the wrong place (in a public repository).

What to do?​

In the current situation, there is only one recommendation for server/domain owners: protect their private SSH keys as best as possible.

For simple situations (non-commercial use), there are tools like the melt utility for secure backup and recovery of SSH keys.

For commercial use, it is better to secure the key storage and isolate it from the Internet on a separate machine. There are hardware modules (HSM) specifically designed to perform cryptographic operations. They guarantee that no one will ever be able to get to the private key.

The problem is that HSMs are not fast enough to handle the millions of requests that Github handles. In such a situation, some experts recommend deploying an HSM service that will issue a new SSH host certificate to servers every minute, signed by the private key in the HSM. If clients are configured to trust the key that signs the certificates, then it does not matter what private key is on the servers - the client will see that there is a valid certificate and will trust the key, even if it changes. Limiting the certificate validity period protects against key leakage.

To make this possible, we need to slightly change the client toolkit, because at the moment OpenSSH does not have the ability to do TOFU for certification authorities, only for the keys themselves. That is, you cannot make a request ssh://git@github.com/whateverand receive a request for trust in the Github CA. You can only manually add a line @cert-authority github.com (ключ)to the file known_hosts, and almost no users are ready for such operations, so the benefit of implementing this infrastructure will be insignificant.

The most important thing we can do to improve the security of the SSH ecosystem is to simplify the use of certificates, and this means improving the behavior of client software, writes security specialist and Linux developer Matthew Garrett. He draws attention to the fact that OpenSSH supports a protocol for key rotation. That is, the server can have a set of several trusted keys that the client can cache and then cancel the old keys. But still, the approach with certificates looks preferable as a universal solution to the problem.

As a result of the incident, you need to remember an important piece of advice: never store passwords, keys, and other secret parameters in the same directory as the code. But ideally, a security system should be built that does not allow such incidents in principle.

Source