In my previous post I described how to set up an SSH certificate authority using a YubiKey.

I’ve been experimenting with NixOS again recently — having had several failed attempts before due to the utterly impenetrable documentation — but I’m curious enough to spend a bit more effort on it.

Even though I’m only running it in test VMs right now I still want to get a good feel for running it on a server, so I want to be able to log in using a signed certificate.

Initially I did it the “easy” way by adding a cert-authority key for my user in configuration.nix:

{ config, pkgs, ... }:

{
  users.users.jamesog = {
    # Other user options omitted
    openssh.authorizedKeys.keys = [
      ''cert-authority,principals="jamesog" ssh-rsa ...''
    ];
  };
}

But ideally I wanted to set it up system-wide, which requires putting the CA public key in a file and adding TrustedUserCAKeys to sshd_config.

I hoped NixOS would have a knob for this already, but searching the options for services.openssh revealed nothing for that, but it does have services.openssh.extraConfig to put any arbitrary configuration. That’ll do, I suppose.

I wasn’t sure how to get the CA key into a file at first. Some digging made me come across pkgs.writeText for building derivations. Even though it’s in pkgs, which I wasn’t sure about, I figured that it’s already passed to configuration.nix so maybe it’ll work. I tried this:

{ config, pkgs, ... }:

let
  trustedCAUserKeys = pkgs.writeText "/etc/ssh/ca.pub"
    ''
    ssh-rsa ...
    '';
in
{
  services.openssh.extraConfig =
    ''
    TrustedUserCAKeys ${trustedCAUserKeys}
    '';
}

But this was very unhappy. It turns out — and this should have been obvious from how it’s used in environment.packages — that this pkgs is just a list, so it doesn’t have writeText.

After some more digging I came across environment.etc whose description says:

Set of files that have to be linked in /etc.

Well, that’s exactly what I want in this case. I don’t know what I’d do if I wanted a file outside of /etc though. And it’s kind of weird it’s called environment. Let’s give this a go then.

{ config, pkgs, ... }:

{
  environment.etc = {
    "ssh/ca.pub".text = ''
      ssh-rsa ...
    '';
  };

  services.openssh.extraConfig =
    ''
    TrustedUserCAKeys ${environment.etc."ssh/ca.pub"}
    '';
}

This didn’t quite work due to the variable expansion I tried to use in extraConfig. I was hoping environment.etc would somewhere inject the full path to the file it created like writeText does. But then I realised I don’t need to: I’m literally telling environment.etc that I want a file in /etc and the path. I can just hard-code it:

{ config, pkgs, ... }:

{
  environment.etc = {
    "ssh/ca.pub".text = ''
      ssh-rsa ...
    '';
  };

  services.openssh.extraConfig =
    ''
    TrustedUserCAKeys /etc/ssh/ca.pub
    '';

  # Note that nixos-rebuild will complain if no users have a password nor authorizedKeys set, so add a backup key
  users.users.jamesog = {
    # Other user options omitted
    openssh.authorizedKeys.keys = [
      "ecdsa-sha2-nistp256 ..."
    ];
  };
}

Note the last block: nixos-rebuild will refuse to build if no users have a password or SSH keys defined as it doesn’t want to leave you with a system you can’t log in to. Fair enough. There’s probably some option I could tweak to bypass that, but it seemed just as safe to set at least one normal SSH public key on my user as a backup in case I nerf the sshd config.

Now it works. Nice!

This is a guide to setting up a YubiKey for use as an offline SSH certificate authority.

This assumes a brand new YubiKey with no prior configuration on it, to be used solely as a CA.

Why?

Typically a CA should be on a secured, isolated machine. Using a dedicated YubiKey means you can isolate your CA and keep it in a drawer so that it can’t be accessed. YubiKeys offer protections such as requiring a PIN and/or touching the key for PIV operations.

Using an SSH CA might sound like overkill for personal use, but it provides a lot of conveniences: you only need to configure remote machines once, with the CA public key. Any new key pairs you create don’t need distributing to those machines. As long as the client presents a certificate signed by the CA, the remote machine will accept any new keys. This removes the need for manually copying public keys or adding them to configuration management.

It also provides automatic verification of your servers: you don’t need to accept the server’s public key and add it to the known_hosts file. As with the client keys, as long as the server presents a certificate signed by your CA, you can automatically verify it.

Prerequisites

• YubiKey Manager (aka ykman)
• A password manager or other tool to generate random passwords and store your CA management key, PIN and PUK
• A PKCS#11 library for OpenSSH, one of:
  • yubico-piv-tool, which provides libykcs11.so; or
  • OpenSC, which provides opensc-pkcs11.so

I’ll assume the use of libykcs11.so as I found it more straightforward to use. OpenSC works, but depending how it’s installed it might not work out of the box with SSH (looking at you, Homebrew), which requires libraries to be in a location that’s hard-coded at compile-time.

(If you have multiple YubiKeys) Identify the YubiKey to use

List all connected YubiKeys

$ ykman list
YubiKey 5 NFC [OTP+FIDO+CCID] Serial: xxxxx
YubiKey 5 NFC [OTP+FIDO+CCID] Serial: yyyyyyyy

The first one is used as my current SSH key. The second is the new one to set up as the CA. We can check this by looking at the PIV application on each.

$ ykman --device yyyyyyyy piv info
PIV version: 5.2.6
PIN tries remaining: 3
CHUID:	No data available.
CCC: 	No data available.

$ ykman --device xxxxx piv info
PIV version: 5.1.2
PIN tries remaining: 3
CHUID:	No data available.
CCC: 	No data available.
Slot 9a:
	Algorithm:	ECCP256
	Subject DN:	CN=SSH key
	Issuer DN:	O=yubikey-agent,OU=v0.1.1
	Serial:		xxxxx
	Fingerprint:	abcdef
	Not before:	2020-05-10 11:50:33
	Not after:	2062-05-10 12:50:33

We can see the second one has PIV configured by yubikey-agent and the first has nothing configured in any slots.

Setup

Let’s set up the new YubiKey by changing the management key, PIN and PUK.

Here we change the management key (hit Enter at the first prompt to use the default key) and tell ykman to generate a random key (--generate), and require touching the YubiKey whenever the management key is used (--touch).

Be sure to save the generated management key somewhere safe! You will need it any time you want to sign.

$ ykman --device yyyyyyyy piv access change-management-key --touch --generate
Enter your current management key [blank to use default key]:
Generated management key: <some string of characters>

123456 is the default PIN and 12345678 is the default PUK. You can use pwgen or a password manager to generate a new PIN and PUK. See Yubico’s PIN and Management Key documentation for PIN limitations.

$ pwgen -sy 8 1
$ ykman --device yyyyyyyy piv access change-pin -P 123456
Enter your new PIN:
Repeat for confirmation:
New PIN set.
$ ykman --device yyyyyyyy piv access change-puk -p 12345678
Enter your new PUK:
Repeat for confirmation:
New PUK set.

We only need CCID mode on the CA YubiKey, OTP and FIDO won’t be used, so let’s disable them.

$ ykman --device yyyyyyyy mode CCID
Set mode of YubiKey to CCID? [y/N]: y

Creating the CA

First we create a private key. We’ll use slot 9c.

The ca-key.pub argument tells ykman to write the public key for the generated key to the file ca-key.pub. This will be used when we generate the certificate in the next step.

$ ykman --device yyyyyyyy piv keys generate 9c ca-key.pem
Enter a management key [blank to use default key]:
Touch your YubiKey...

Now generate a certificate whose subject is “SSH CA” and is valid for ~10 years, using the public key from the last step.

$ ykman --device yyyyyyyy piv certificates generate -s "SSH CA" -d 3650 9c ca-key.pem
Enter PIN:
Enter a management key [blank to use default key]:
Touch your YubiKey...

At this point we have a private key and self-signed certificate on the YubiKey.

$ ykman --device yyyyyyyy piv info
PIV version: 5.2.6
PIN tries remaining: 3
CHUID:	aaaa
CCC: 	bbbb
Slot 9d:
	Algorithm:	RSA2048
	Subject DN:	CN=SSH CA
	Issuer DN:	CN=SSH CA
	Serial:		abcde
	Fingerprint:	abcde
	Not before:	2020-10-04 15:21:26
	Not after:	2030-10-02 15:21:26

Let’s get the SSH public key for this. You’ll need the path to the shared library for the PKCS#11 library as mentioned in the prerequisites.

ssh-keygen -D /opt/local/lib/libykcs11.dylib

This will output two or three ssh-rsa keys. Take the second one with comment “Public key for Digital Signature” and put it in a file called ca.pub. This is the signing key used by ssh-keygen to sign keys, and by sshd and ssh to validate certificates.

Signing SSH public keys

The part you’ve probably been waiting for! This section is what you’ll do any time you create a new SSH key pair.

I keep all public keys and certificates stored on the computer I primarily use my CA key with, mostly for convenience. I structure it as:

/Users/jamesog/ssh-ca
├── host
└── user

I copy all host keys into ~/ssh-ca/host/ and user keys to ~/ssh-ca/user/. The signed certificates then live alongside the public key.

When signing keys you have to provide a “key ID”. For user keys, this is what’s logged on the server when you authenticate with a certificate. I use my email address for the ID.

Host keys

The certificate needs an ID (-I), which is usually the machine’s hostname, and a principal (-n) of the machine’s hostname (usually a DNS name).

ssh-keygen -s ca.pub -D /opt/local/lib/libykcs11.dylib -I <hostname> -h -n <hostname> host/somehost_ed_25519.pub

If the machine can be accessed using several hostnames, add each of them to the -n flag separated by commas. If you ever need to access the machine by IP address instead of a hostname, add that to the principals too, otherwise it won’t verify.

User keys

The certificate needs an ID (-I) which should be something to identify you — I use my email address — and your remote username as a principal (-n).

$ ssh-keygen -s ca.pub -D /opt/local/lib/libykcs11.dylib -I <email address> -n jamesog user/id_ed25519.pub
Enter PIN for 'YubiKey PIV #yyyyyyyy':
Signed user key user/id_ed25519-cert.pub: id "<email address>" serial 0 valid forever

You can inspect the generated certificate with ssh-keygen:

$ ssh-keygen -L -f user/id_ed25519-cert.pub
user/id_ed25519-cert.pub:
        Type: ssh-ed25519-cert-v01@openssh.com user certificate
        Public key: ED25519-CERT SHA256:xxx
        Signing CA: RSA SHA256:xxx (using rsa-sha2-512)
        Key ID: "<email address>"
        Serial: 0
        Valid: forever
        Principals:
                jamesog
        Critical Options: (none)
        Extensions:
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

Signing another YubiKey

If you use another YubiKey SSH private key (for example using yubikey-agent), you can also sign the public key for this.

Export the SSH public key from the YubiKey:

ssh-keygen -D /opt/local/lib/libykcs11.dylib > user/yubikey.pub

And then sign the public key as above.

Setting up servers

Presenting the signed host certificate

Copy the signed host certificate to each server, e.g. host/somehost-cert.pub, and place it alongside the host’s private keys, usually in /etc/ssh, ensuring the name matches the private key name. Add the following directive to sshd_config:

HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub

Restart sshd after adding this directive (or wait until you’ve configured client keys, below).

Authenticating client certificates

sshd can be configured with the CA certificate so that any user presenting a certificate signed by this CA can be authenticated with no other configuration, assuming the certificate has a valid principal for the user they’re trying to authenticate as.

Take the ca.pub from earlier and install it on each server, e.g. at /etc/ssh/ca.pub and add the following directive to sshd_config:

TrustedUserCAKeys /etc/ssh/ca.pub

Restart sshd after adding this directive. See sshd_config(5) for more information, and also the AuthorizedPrincipalsFile in case you want to do more advanced things with user principals.

Alternatively, each user can add the CA to their ~/.ssh/authorized_keys file:

cert-authority,principals="jamesog" ssh-rsa ...

The principals= here ensures only certificates with the correct usernames can authenticate.

Setting up clients to use the certificate

In your ~/.ssh/config use the CertificateFile directive to use the certificate. If your key is loaded in ssh-agent, ssh will match the certificate to the private key, otherwise add IdentityFile to your SSH config too.

If your private key is not a file on disk (i.e. you’re using a YubiKey or a Mac’s Secure Enclave) you should set the IdentityFile directive to be the path to your public key. It’s nonsensical, I know, but somehow it works.

Setting up clients to authenticate server host keys

Add to ~/.ssh/known_hosts the DNS name(s) or a wildcard of remote servers that have the signed certificate, followed by the contents of ca.pub:

@cert-authority *.example.com ssh-rsa ... SSH CA

With this you no longer need TOFU or to accept individual host keys.

Further reading

• If you’re not using SSH certificates you’re doing SSH wrong

Secure hardware-backed private keys

Using a regular key pair generated with ssh-keygen is just fine, however for added security you can make the private key completely inaccessible by using a hardware device.

YubiKey

As we’ve already discused, the YubiKey is great for this. yubikey-agent makes setting up the key and your computer very straightforward.

The bonus of a YubiKey is it is transportable and can be used across computers.

Mac Secure Enclave

Every modern (since 2016) Mac comes with a built-in security device called a Secure Enclave. This is where things like your Touch ID fingerprints are safely stored.

Secretive uses the Secure Enclave to generate and store SSH keys. The private key can never be extracted so you can’t transfer the key to aonther Mac.

Secretive has a couple of great usability features: you can get notifications whenever the key is used, in case some nefarious process is using your key in the background, and you can require biometrics (Touch ID or Apple Watch) every time the key is used for maximum security. This is similar to requiring a tap of the YubiKey when the key is used, but only you can authenticate it.

Many people have a misapprehension about me; they believe I don’t like Christmas. This even manifested itself in a colleague buying me a T-shirt. Really, it’s simply that I don’t enjoy Christmas.

Some examples:

Enforced / organised “fun” at work.

The insistence that you have to spend it with your family.

The idea that you must buy people gifts, but often end up getting them any old rubbish or novelty item just to get them something. And likewise, receiving things you don’t want or need, just because somebody felt obliged because they’re a relative. (I’m discounting the thoughtful gifts here - those are always appreciated, at any time of year.)

The food. Oh, the (generally terrible) food. As a child I was made to eat dry, overcooked turkey along with Brussels sprouts, the most evil thing ever devised.

The fact that many people use this time of year as an excuse to get absolutely plastered. I’m not a big drinker. I enjoy a drink from time to time, but I’ve never seen the point of daytime drinking, let alone full-on sessions that leave you completely insensible. This then gets inflicted on those around them as they have to endure the drunken antics, shouting, vomiting, and in the worst cases, fights, and so on.

The insane lights and decorations and fairs that exist only to empty people’s wallets.

I like the idea of the “modern Christmas” - that is, (ignoring the religious background of it) spending time with people you like and having a nice time. Having pretty lights and a few decorations is nice, too - a way to cheer things up during the bleak winter months of the northern hemisphere, at least for a little while. I simply get tired of the gaudiness of it all, and people insisting you simply must enjoy this thing, and do this and that.

And on that note, today is Christmas day and it’s nearly noon. Merry Christmas, everyone. Have a lovely day doing things you like with people you love.

I posted to my Tumblr blog that I had relocated to here.

Forgetting that I’d set Tumblr to automatically share new posts to Twitter, that post got tweeted and a few people picked it up. That’s not necessarily a bad thing, but there’s not much interesting here yet. :-)

Over time I might start to post some technical things here. I haven’t really decided yet. Largely I just wanted to play with Hugo.

So, hello, Hugo!

FreeBSD 8.4 was recently released. I noticed in the release notes that virtio is now enabled by default. Hurrah!

With this in mind, I decided to give Bytemark’s BigV another go. Previously I’d had to use a custom CD image and semi-manual installation process to use FreeBSD on BigV, or otherwise face the performance penalties for using KVM’s IDE emulation. (It’s really slow.)

The good news is that with 8.4-RELEASE you don’t have to do a thing to get virtio to work. It detected the disk and network adapter fine. The VM is flying along. Very happy with that.

Bytemark have had IPv6 support on their physical machine for some time so I thought I’d give that a go too. The documentation for setting this up was a bit sparse (what’s the default router?). I eventually found some documentation on Bytemark’s main support site which seems to work for BigV too.

Herein lies the problem: Bytemark’s network uses fe80::1 for the default router so that you can use the same setting anywhere on their network. This is a nice touch, however FreeBSD already creates this address on lo0 by default so it gave me some pause for thought.

After a short while I remembered that with IPv6 you can specify the interface in addresses, so I set my inet6 default route to fe80::1%vtnet0 and away it went.

I’m pretty happy with this setup now. BigV seems to give a bit more than most VPS providers do for the same money.

It’d be nice if they could provide a ready-made FreeBSD image now. :-)

I recently decided to try SmartOS as it looks to be a very interesting virtualization platform. I didn’t have a machine to spare to trial it, so I decided to rent a Hetzner EX4S with a 16GB flash drive.

For the initial install I asked Hetzner to set up the server to boot to their rescue environment, rather than preinstall one of their OS images. From here I was able to wget the SmartOS USB image and dd it to the flash drive.

The next bit is where it got “interesting”. I knew I’d need console access to the server to set up SmartOS, so I filed a support request with Hetzner. Once I had access to the IP-KVM (or LARA in their terminology) I rebooted the server to boot from the USB stick. However I started seeing a lot of error messages about disk timeouts as SmartOS tried to do its initial configuration. Once I got to the disk selection for creating the zpool, none were available! On a hunch, I rebooted the machine and entered the BIOS setup utility. They’ve hidden the message that tells you which key does this, so I basically mashed the keyboard a bit until I found the right one :-) I think it was Delete in the end. So, once inside the BIOS setup, I found they’d set the server’s hard drives to run in IDE emulation mode rather than SATA AHCI. I changed this, rebooted, and lo! SmartOS found the disks, with no timeout messages!

Once the SmartOS configuration was done, I could access the machine remotely. As I only had a single IPv4 from Hetzner, I decided to “rent” another (for €1/month) so that I could create a router zone. All of my other VMs are NATed behind this. Bit of a hack, but it does the job for now.

I’ve yet to configure it for IPv6 yet as SmartOS doesn’t yet have a way of enabling this from their configuration so you end up having to write custom SMF manifests. There’s also an issue with KVM guests and IPv6, due to the way that all VMs, whether SmartOS or KVM, are run inside Zones. The Zone’s VNIC has a MAC address which can either be defined by the VM config, or generated randomly. When you’re using KVM, the VM’s VNIC also gets the same MAC address, which leads the guest to throw DAD errors for the link-local address.