A key with one job

The fleet was fine. The keyring was the problem.

I keep a small number of machines running. Caspar does most of the work, tied up here in the harbour where I can see it. Three more sit further out: a lighthouse box on a public address, and two production hosts I can only reach by rowing through a jump box first. Between them they run the things that are actually live and earning their keep.

For a long time, “what is running out on those three” was a job I did by hand. Open a terminal, connect to the lighthouse, list the containers, read the list, close it. Connect to the jump, hop through to the first production host, list again. Then the second one. Every time I wanted a straight answer about the state of the fleet I rowed the whole circuit myself. It is not hard. It is just something you stop bothering to do, and then you are running blind on the hosts that matter most.

So I built bosun-x, a dashboard that keeps an eye on the whole mesh, and I wanted it to row that circuit for me. One page that shows every container on every host, refreshing quietly while I drink my coffee, instead of me connecting to three machines in turn.

For the dashboard to ask a host what it is running, it has to log in over SSH. The obvious way to make that work is to give it my own key. I did, for about an hour. It worked on the first try, which should have been the warning.

Here is what my key actually is. It opens every box I own, and on each one it can become root. It is the master key to the entire harbour. The job I was about to hand it is “run one docker ps and read a few text files.” And that master key would then be sitting on disk inside a container, mounted in and readable, behind a web application. Web applications have bugs. One of those bugs, one day, plus my key right there for the taking, and I have not lost a container list. I have lost the mesh.

That was the moment to stop and think about what the dashboard genuinely needed. Not “log in as the captain.” It needed to ask three small, harmless questions and get three answers back. That is a far smaller thing to give away. So I cut a key that can do exactly that and refuses everything else, on any host it touches.

So I cut a key with one job

It is a brand new ed25519 SSH keypair that exists for this feature and nothing else. On each of the far hosts it goes into authorized_keys as a single extra line, with my own key left untouched beside it. That one line does three things, and the rest of this note is those three things, and how I checked that each of them holds when someone leans on it.

The three things: the key can only run one script, that script can only tunnel to one place, and the script itself only answers a fixed list of questions.

The key can only run one script

Why it is here. This is the whole point. A normal SSH key is a general permission: log in, then do anything the account can do. I do not want the dashboard to be able to do anything. I want it to be able to run one specific, harmless program and then be shown the door.

How it works. The key goes into the host’s authorized_keys as one line, with options in front of it that constrain what presenting that key is allowed to do:

restrict,command="/usr/local/bin/bosun-x-ro.sh" ssh-ed25519 AAAA... discovery

Think of a bank teller behind glass. You write what you want on a slip and push it through the slot. Whatever you wrote, they carry out one fixed procedure. You cannot climb through the slot, you cannot walk around the counter, and you never see the vault. Mechanically:

restrict is the glass. It is an allowlist, not a blocklist: it switches off pty allocation, port forwarding, agent forwarding, X11, and ~/.ssh/rc in one word, and it keeps switching off anything new that OpenSSH adds in future versions. The forced command= is the one procedure. Whatever command the client actually sends, sshd ignores it and runs that script instead. bosun-x’s SSH call passes the string bosun-x-ro, and the server pays it no attention at all.

The script can only tunnel to one place

Why it is here. One of the three hosts is not reachable straight from Caspar. I get to it by jumping through another box on the way, which SSH does with a ProxyJump. A ProxyJump needs to open a forwarded channel through the jump host, and restrict in the last step just switched forwarding off. So I need to hand back exactly one piece of what restrict took away, and not a scrap more.

How it works. Back to the teller: the glass has one small pass-through drawer, and that drawer only reaches one desk in the back office. Not the office. One desk. That is permitopen. It re-enables forwarding to a single host and port, and refuses every other target:

restrict,command="...",permitopen="192.168.6.180:22" ssh-ed25519 AAAA... discovery

The script only answers a fixed list

Why it is here. The forced command closes off the shell, but now the whole security of the arrangement rests on that one script. If the script can be argued into reading an arbitrary file, or running an arbitrary command, then restrict bought me nothing. So the script has to be as narrow as the key.

How it works. The script never trusts its input. SSH_ORIGINAL_COMMAND, the string the client asked for, is compared against exact literal patterns and is never handed to a shell:

case "$SSH_ORIGINAL_COMMAND" in
  doc:/opt/gp-forms:SPEC.md)   DOC=/opt/gp-forms/SPEC.md ;;
  doc:/opt/gp-forms:AGENTS.md) DOC=/opt/gp-forms/AGENTS.md ;;
  # ...one line per file it is allowed to read...
  doc:*) echo "===REJECTED==="; exit 1 ;;
esac

Even the teller’s one procedure has a checklist taped to the counter. Ask for something that is not on it and the slip comes straight back. Everything else the script does is a fixed sequence with no inputs at all: uname, nproc, free, df, the load average, docker ps -a, docker stats. The handful of file reads it does allow are capped with head -c 65536 and refuse symlinks and anything that is not a regular file, so the script cannot be talked into being a general file reader or into returning something unbounded.

I pulled on the handle

You do not trust a lock because the locksmith says so. You pull on it. After switching the forced command over, I tried the three things anyone who had lifted this key would try:

• Asked it to run id. It ran the pinned script instead and did not so much as acknowledge the request.
• Asked for an interactive shell with ssh -tt. Refused: PTY allocation request failed.
• Opened a tunnel to a host that was not the one permitted destination. Refused, administratively prohibited. Then relayed an SSH banner through the one tunnel that is allowed and got bytes back, so the exception works and only the exception works.

Verified vs assumed

Verified. The three tests above, run against the live hosts, after the change. Also that bosun-x still gets its snapshot from each host in 2.4 to 2.9 seconds, cached for five minutes, and that a failed poll is only remembered for thirty seconds rather than pinning an empty result for the full five.

Assumed, and worth another look. That the pinned script has no argument-handling path I have missed. That head -c 65536 is a sane cap and not one I will regret the first time a STATUS.md grows past it. Neither has bitten yet, which is not the same as neither being real.

The shape here carries over. Any time something inside a container has to reach a host outside it: make a credential that exists for nothing else, cut it down to the one capability the feature actually needs, allow one narrow exception only if it genuinely cannot work without one, and then run the tests that prove the wall is where you think it is.