The first stage of the RFD77 soft-token was fairly straightforward, all things considered: copy-pasting the agent.c code from OpenSSH; making a forking libsysevent(3LIB) subscriber that watched for zones coming and going and forked it off; and writing a PIV client to sit in the middle and decrypt the keys with a hardware token. When the agent code wanted to use a key, it would send a simple fixed-length request struct on a pipe up to the PIV client, which would decrypt the key and write it into a shared memory segment then reply back down the pipe. This meant no locks between processes and no variable length data being exchanged, which makes writing a robust implementation a lot simpler.

Certificates, however, involve a fair bit more back-and-forth. They need to be periodically renewed, and in the case of X.509 certificates, they have a potentially variable length chain back to their CA. Parsing and generating them is also vastly more complex, involving a lot of ASN.1 and complex structures.

My initial strawman design was to handle renewal entirely in the "supervisor" (PIV) layer and have the agent just shovel whatever cert chain was loaded into a second shared memory segment out to clients. This runs quickly into a problem, however: if the supervisor is updating the shared memory whenever it likes, there needs to be a lock so that the agent doesn't read partially updated data, and interprocess locking is somewhat easy to mess up (and previous to this I had avoided it successfully in the design).

So, instead, I decided to make certificate renewal happen only at the behest of the agent process. It sends another fixed-length request to the supervisor asking for the cert chain for a given key to be renewed, and waits for a reply from the supervisor before going to use the shared memory segment again.

The shared memory segments each look like this:

struct token_slot_data {
    volatile uint32_t tsd_len;
    volatile char tsd_data[1];
};

A simple length-prefixed data segment. Every time we use it we VERIFY the tsd_len to fit within the bounds of the memory region before doing any work. Also, the supervisor layer is carefully written to never ever read any of the fields in the shared memory segment. It only ever blindly overwrites them. This enforces the trust hierarchy between the supervisor and the agent child (which runs with very few privileges). The token_slot_data structs are pointed to by the struct token_slot:

struct token_slot {
    uint8_t ts_id;
    enum slot_type ts_type;
    enum slot_algo ts_algo;
    const char *ts_name;
    struct token_slot *ts_next;
    struct token_slot_data *ts_data;
    struct token_slot_data *ts_certdata;
    struct token_slot_data *ts_chaindata;
    size_t ts_datasize;
    struct sshkey *ts_public;
    nvlist_t *ts_nvl;
    struct agent_slot *ts_agent;
};

As you can see, we have 3 data segments associated with each "slot" -- one for the key itself (ts_data), one for the direct certificate (ts_certdata) and one for the certificate chain (ts_chaindata).

The certificate and chain shared segments are of a fixed maximum length: MAX_CERT_LEN (8kb) and MAX_CHAIN_LEN (16kb) respectively. Given that most real certificates are under about 2kb each and we expect to have a max chain length of about 4 certificates, this seems quite reasonable. The pages allocated for these segments don't have the guard pages at the start and end of the mapping -- since they don't contain any private key material it's not as important to protect them from memory overread.

Currently the implementation just assumes that the chain segment contains exactly one certificate. This works fine for the simple case where our cert is signed directly by our own host node only and the chain goes no further. (In the zone soft-token we have the cert.key ou=instances cert in the cert slot, and then the ou=nodes cert for the global zone soft-token cert.key in the chain slot. In the global soft-token the chain slot is empty.)

The elephant in the room, then, is how to expand this approach to deal with chains back to a head node from an ordinary CN in Triton. Currently my thoughts for this centre around making a "CertAPI" headnode service that the CNs can talk to to request signing of their certificates. The client to talk to this from the CN global zone though and how to integrate it into the soft-token is still something to be considered.

One option I've considered is making the global soft-token agent code support writing a new chain through the normal socket interface. Then a secondary process (perhaps written in node) can sit off to the side to manage conversing with CertAPI, obtaining the new chain, and uploading it into the soft-token. This has the notable advantage of not requiring much complex code to run inside the soft-token process, which is something I like. It does, however, make the cert renewal process partly push- and partly pull-based, which seems prone to racey errors in implementation. Another thing I've considered is making another fork of the soft-token process for this communication, with a shared memory segment to write into like the agent process. Then it could still be pull-based, as the PIV supervisor requests the renewal from the certAPI client process when the agent asks for a cert renewal itself.

In any case, I'm going to continue prototyping and testing with certificates directly on each node for the time being, and start work on the gossip service. Certificates chaining back to the headnode from a CN is an end-user feature and not required for the internal use of RFD77 for Triton headnode services, so there's plenty of work that can be done before having to come back and solve this problem.