“Change is the only constant in life.”

Heraclitus

Old dog
new tricks

Things you might not know about openssh

Created by Marcus Rückert or just darix ;)

Conventions

All options are just single line options. If you see something like this example, remove the backslash and the following newline.

# /etc/ssh/ssh_config || ~/.ssh/config
SomeOption somevalue, \
  anothervalue

So we get:

# /etc/ssh/ssh_config || ~/.ssh/config
SomeOption somevalue,anothervalue

The comment on top of each config snippet notes which file the setting can go into. In the example it can go in either /etc/ssh/ssh_config or ~/.ssh/config.

The ssh client has a config file?

The initial problem

For openSUSE VMs we use ssh forwarding with xinetd and have a lot of connections to non standard ports. This leads to:

ssh -p 1234 root@proxyhost.example.com

Especially annoying. scp uses a different commandline argument:

scp -P 1234 somefile root@proxyhost.example.com:

Making our life easier

# ~/.ssh/config
Host myvm
   Port 1234
   User root
   Hostname proxyhost.example.com

ssh myvm

Warning for ansible users

It ignores the port, but the hostname alias works when using the ssh transport. You have to configure it in ansible again or patch ansible.

Advancing crypto

The future is now

Things might not be state of the art anymore

• DSA
• 1024 bit keys
• NIST curves?
• SHA1
• MD5
• RC4

Key exchange protocols

1. curve25519-sha256: ECDH over Curve25519 with SHA2
2. diffie-hellman-group1-sha1: 1024 bit DH with SHA1
3. diffie-hellman-group14-sha1: 2048 bit DH with SHA1
4. diffie-hellman-group-exchange-sha1: Custom DH with SHA1
5. diffie-hellman-group-exchange-sha256: Custom DH with SHA2
6. ecdh-sha2-nistp256: ECDH over NIST P-256 with SHA2
7. ecdh-sha2-nistp384: ECDH over NIST P-384 with SHA2
8. ecdh-sha2-nistp521: ECDH over NIST P-521 with SHA2

Key exchange protocols - considerations

We have to look at 3 things here:

• ECDH curve choice: This eliminates 6-8 because NIST curves suck. They leak secrets through timing side channels and off-curve inputs. Also, NIST is considered harmful and cannot be trusted.
• Bit size of the DH modulus: This eliminates 2 because the NSA has supercomputers and possibly unknown attacks. 1024 bits simply don’t offer sufficient security margin.
• Security of the hash function: This eliminates 2-4 because SHA1 is broken. We don’t have to wait for a second preimage attack that takes 10 minutes on a cellphone to disable it right now.

Key exchange protocols - conclusion

We are left with 1 and 5. 1 is better and it’s perfectly OK to only support that but for interoperability (with Eclipse, WinSCP), 5 can be included.

Key exchange protocols - recommended config

# /etc/ssh/sshd_config
KexAlgorithms curve25519-sha256@libssh.org, \
  diffie-hellman-group-exchange-sha256

# /etc/ssh/ssh_config || ~/.ssh/config
# Github needs diffie-hellman-group-exchange-sha1
# some of the time but not always.
#Host github.com
#    KexAlgorithms curve25519-sha256@libssh.org, \
#       diffie-hellman-group-exchange-sha256, \
#       diffie-hellman-group-exchange-sha1, \
#       diffie-hellman-group14-sha1
Host *
     KexAlgorithms curve25519-sha256@libssh.org, \
       diffie-hellman-group-exchange-sha256

Server authentication

Disable SSH1. Disable DSA. Enable ED25519. (If your server supports it)

# /etc/ssh/sshd_config
Protocol 2
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key

Client authentication

# /etc/ssh/sshd_config
PubkeyAuthentication yes
PasswordAuthentication no
ChallengeResponseAuthentication no

# /etc/ssh/ssh_config || ~/.ssh/config
Host *
    PasswordAuthentication no
    ChallengeResponseAuthentication no
    PubkeyAuthentication yes
    HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com, \
      ssh-rsa-cert-v01@openssh.com, \
      ssh-rsa-cert-v00@openssh.com, \
      ssh-ed25519,ssh-rsa

Symmetric ciphers

1. 3des-cbc
2. aes128-cbc
3. aes192-cbc
4. aes256-cbc
5. aes128-ctr
6. aes192-ctr
7. aes256-ctr
8. aes128-gcm@openssh.com
9. aes256-gcm@openssh.com
10. arcfour
11. arcfour128
12. arcfour256
13. blowfish-cbc
14. cast128-cbc
15. chacha20-poly1305@openssh.com

Symmetric ciphers - considerations

• Security of the cipher algorithm: This eliminates 1 and 10-12 - both DES and RC4 are broken. Again, no need to wait for them to become even weaker, disable them now.
• Key size: At least 128 bits, the more the better.
• Block size: Does not apply to stream ciphers. At least 128 bits. This eliminates 14 because CAST has a 64 bit block size.
• Cipher mode: The recommended approach here is to prefer AE modes and optionally allow CTR for compatibility. CTR with Encrypt-then-MAC is provably secure.

Symmetric ciphers - conclusions

# /etc/ssh/sshd_config
Ciphers chacha20-poly1305@openssh.com, \
  aes256-gcm@openssh.com, \
  aes128-gcm@openssh.com, \
  aes256-ctr,aes192-ctr,aes128-ctr

# /etc/ssh/ssh_config || ~/.ssh/config
Host *
    Ciphers chacha20-poly1305@openssh.com, \
      aes256-gcm@openssh.com, \
      aes128-gcm@openssh.com, \
      aes256-ctr,aes192-ctr,aes128-ctr

Message authentication codes

• Encrypt-then-MAC: encrypt the message, then attach the MAC of the ciphertext.
• MAC-then-encrypt: attach the MAC of the plaintext, then encrypt everything. (TLS)
• Encrypt-and-MAC: encrypt the message, then attach the MAC of the plaintext. (SSH)

Available MAC

1. hmac-md5
2. hmac-md5-96
3. hmac-ripemd160
4. hmac-sha1
5. hmac-sha1-96
6. hmac-sha2-256
7. hmac-sha2-512
8. umac-64
9. umac-128
10. hmac-md5-etm@openssh.com
11. hmac-md5-96-etm@openssh.com
12. hmac-ripemd160-etm@openssh.com
13. hmac-sha1-etm@openssh.com
14. hmac-sha1-96-etm@openssh.com
15. hmac-sha2-256-etm@openssh.com
16. hmac-sha2-512-etm@openssh.com
17. umac-64-etm@openssh.com
18. umac-128-etm@openssh.com

MAC considerations

• Security of the hash algorithm: No MD5 and SHA1. Yes, I know that HMAC-SHA1 does not need collision resistance but why wait? Disable weak crypto today.
• Encrypt-then-MAC: I am not aware of a security proof for CTR-and-HMAC but I also don’t think CTR decryption can fail. Since there are no downgrade attacks, you can add them to the end of the list. You can also do this on a host by host basis so you know which ones are less safe.
• Tag size: At least 128 bits. This eliminates umac-64-etm.
• Key size: At least 128 bits. This doesn’t eliminate anything at this point.

MAC - conclusions

# /etc/ssh/sshd_config
MACs hmac-sha2-512-etm@openssh.com, \
  hmac-sha2-256-etm@openssh.com, \
  hmac-ripemd160-etm@openssh.com, \
  umac-128-etm@openssh.com, \
  hmac-sha2-512,hmac-sha2-256, \
  hmac-ripemd160,umac-128@openssh.com

# /etc/ssh/ssh_config || ~/.ssh/config
Host *
  MACs hmac-sha2-512-etm@openssh.com, \
    hmac-sha2-256-etm@openssh.com, \
    hmac-ripemd160-etm@openssh.com, \
    umac-128-etm@openssh.com, \
    hmac-sha2-512,hmac-sha2-256, \
    hmac-ripemd160,umac-128@openssh.com

The complete client config (openssl >= 1)

# /etc/ssh/ssh_config || ~/.ssh/config
Host *
   KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
   HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-ed25519,ssh-rsa
   Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
   # workaround for bug
   # Ciphers chacha20-poly1305@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
   MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com
   PubkeyAuthentication yes
   PasswordAuthentication no
   ChallengeResponseAuthentication no

The complete client config (openssl < 1)

# /etc/ssh/ssh_config || ~/.ssh/config
Host *
   KexAlgorithms diffie-hellman-group-exchange-sha256
   HostKeyAlgorithms ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa
   Ciphers aes256-ctr,aes192-ctr,aes128-ctr
   MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com
   PubkeyAuthentication yes
   PasswordAuthentication no
   ChallengeResponseAuthentication no

The complete server config (openssl >= 1)

# /etc/ssh/sshd_config
Protocol 2
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com

PubkeyAuthentication yes
PasswordAuthentication no
# reenable if you want to use OTP together with ssh keys
ChallengeResponseAuthentication no

The complete server config (openssl < 1)

# /etc/ssh/sshd_config
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
KexAlgorithms diffie-hellman-group-exchange-sha256
Ciphers aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com

PubkeyAuthentication yes
PasswordAuthentication no
# reenable if you want to use OTP together with ssh keys
ChallengeResponseAuthentication no

Generating ssh keys

ssh-keygen -t ed25519 -o -a 100
ssh-keygen -t rsa -b 4096 -o -a 100

"-o -a 100" makes it harder to brute force the passphrase in case the key gets stolen. (openssh >= 6.5)

How to verify?

$ ssh -vv hostname
[snip]
debug1: kex: server->client chacha20-poly1305@openssh.com <implicit> none
debug1: kex: client->server chacha20-poly1305@openssh.com <implicit> none
[snip]
debug1: Server host key: ED25519 cf:c0:.... [MD5]
[snip]

General advices

• Protect your keys
• Install only really required software
• Install updates timely
• Do dist upgrades
• Harden your system (Apparmor/selinux)

Signing

Based on:
"OpenSSH certificates" by Thomas Habets

Requires at least openssh >= 5.5 or better >= 6.0

Server side

Trust on first contact

• Requires side channel
• Accumlates a huge list of server keys in ~/.ssh/known_hosts
• Made even worse when you change your client config or the server ip

Changing keys

• People forgetting to back up the keys
• Host key changed on purpose
• Requires editing ~/.ssh/known_hosts. only host part is covered by ssh-keygen -R.
• At least 2 failed connections until openssh stops complaining.

Key rollover

• Some really old keys might not be state of the art anymore
• Applying the same principles as we do for SSL keys
• Proposed Key Rotation Support in 6.8+ doesn't really solve it either

Changing SSH server keys shouldn't suck

The user side of things - setup

# ~/.ssh/known_hosts
@cert-authority *.example.com ssh-rsa AAAAB3[...]== Comment

Still a side channel - but only once for many hosts

The user side of things - connecting

darix@mylaptop:~ $ ssh unknownhost.example.com
darix@unknownhost:~ $

The user side of things - visiting old friend

darix@mylaptop:~ $ ssh oldmachinenewkey.example.com
darix@oldmachinenewkey:~ $

The magic

ssh-keygen -f host_ca

cat host_ca.pub

ssh-keygen -s host_ca -I 'host:unknownhost.example.com' -h \
  -n unknownhost.example.com -V +52w \
  unknownhost.example.com_ssh_host_rsa_key.pub

scp unknownhost.example.com_ssh_host_rsa_key.pub \
  root@unknownhost.example.com:/etc/ssh/ssh_host_rsa_key.pub

# /etc/ssh/sshd_config
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub

Rinse, repeat for the other key types (DSA, ED25519)

Reload ssh server

Nothing comes without a price

• Time limited certs require monitoring and timely refresh
• Hostnames are locked down in the certificate
• Adding a new alias to a server requires redoing the cert
• You can have host certs without the hostnames listed
  • Allows impersonation/MITM if such a cert gets in the wrong hands
  • Always lock down the hostname on certs. Always!

Key Rotation Plan - initial deployment

1. Establish CA
2. Fetch all keys from the servers
3. Sign them
4. Upload certs to servers
5. Adapt configuration
6. Restart sshd
7. Communicate the pub key of the CA to the users.
8. Give users time to adapt their config

Key Rotation Plan - real roll over

1. Generate new keys for each server
2. Sign them
3. Upload new keys + new certs to the server
4. Restart sshd
5. User impact:

   User with CA config

   Would not notice anything

   User without CA config

   Will get lots of warnings about changed keys

Signing a client

~/.ssh/authorized_keys

• Works in principle
• How to remove someone who left the company?
• How to prevent someone putting a key into authorized_keys for someone, who shouldnt be there?
• If you manage to compromise the ~/.ssh/authorized_keys file, you can elevate what you can do. A global authorized_keys file makes this harder.

The Magic

ssh-keygen -f user_ca

scp user_ca.pub root@somehost.example.com:/etc/ssh/user_ca.pub

# /etc/ssh/sshd_config
TrustedUserCAKeys /etc/ssh/user_ca.pub

Reload ssh server

ssh-keygen -s user_ca -I user_darix \
  -n darix,root -V +52w darix_id_rsa.pub

Rinse, repeat for the other key types (DSA, ED25519)

scp darix_id_*-cert.pub darix@mylaptop.example.com:.ssh/

How does it look like

ssh -i ~/.ssh/id_25519.pub somehost.example.com

sshd[10219]: Accepted publickey for darix from 1.2.3.4 port 39190 \
  ssh2: ED25519 b4:3d:... [MD5]

ssh -i ~/.ssh/id_25519-cert.pub somehost.example.com

sshd[10241]: Accepted publickey for darix from 1.2.3.4 port 39194 \
  ssh2: ED25519-CERT ID user_darix (serial 0) CA RSA 58:5a:... [MD5]

ssh -i ~/.ssh/id_25519-cert.pub nobody@somehost.example.com

sshd[6377]: error: Certificate invalid: name is not a listed principal

None Shall Pass

ssh-keygen -k -f /etc/ssh/revoke.crl darix_id_rsa.pub

ssh-keygen -ku -f /etc/ssh/revoke.crl darix_id_ed25519.pub

# /etc/ssh/sshd_config
RevokedKeys /etc/ssh/revoke.crl

ansible all -m copy \
  -a 'src=/etc/ssh/revoke.crl dest=/etc/ssh/revoke.crl'

sshd[11462]: error: WARNING: authentication attempt with a revoked \
  RSA key 99:41... [MD5]
sshd[11462]: error: WARNING: authentication attempt with a revoked \
  RSA-CERT key 99:41... [MD5]

Useful things to know

• TrustedUserCAKeys requires to lock down the user with "-n [listofusernames]"
• Client certs can also be used with ~/.ssh/authorized_key
  • With authorized_keys you dont have to lock down the users with "-n [listofusernames]"
• Recommendation: Disable authorized_keys in the openssh config - Avoids problems mentioned before
• All options available in authorized_keys are also supported when signing a client cert
• To see the options:
  ssh-keygen -L -f id_25519-cert.pub

Q & A

Thank you for flying with openSUSE