Workloads¶

A workload is a program that logs in to devices without a person: a configuration backup, a compliance check, a deployment job. It needs an identity of its own so that its access is granted, logged and revoked on its own, not borrowed from a person's. port49 knows two kinds:

Identity	Proof	Principal	Use it when
A workload certificate	an x509 certificate with a SPIFFE ID in a URI SAN, from a CA you run	`spiffe://<trust domain>/<path>`	the job runs on a host you manage, such as a cron job or a backup server
A Kubernetes service account token	the pod's projected token, checked against the cluster's published keys	`system:serviceaccount:<namespace>:<name>`	the job runs in a cluster, and you want no CA and no secret at rest

Either is a peer of the other, and neither is required. The concepts are on Identities and Trust domains; how a workload becomes a subject, and why it gets its own device login, is on Roles and grants.

The workload anchor and the people's anchor must be in unrelated hierarchies, so that nothing a workload CA signs can be read as a person.

Workload certificates and step-ca¶

The worked example is Oxidized, the configuration backup tool, with step-ca as the workload CA. The same shape fits any job on a host: the host holds a certificate valid for a few hours, a renewal timer replaces it, and the broker verifies it against the CA you name.

Four rules carry the design:

The workload CA is the trust root, and it is yours. The broker's broker.identity.machines entry names the CA's certificate (trust_anchor_file) and the SPIFFE trust domains it may speak for (spiffe_trust_domains). Anyone who can get that CA to sign a certificate with a SPIFFE ID inside those domains is that workload.
Give each host its own provisioner, with the SPIFFE ID fixed in the provisioner's template (uris), so a host can only ever obtain its own name. A shared provisioner lets any enrolled host ask for another's.
The certificate must name the CA's CRL. port49 applies step-ca's CRL, which carries a critical issuing distribution point, only to a certificate whose CRL distribution points name it exactly. A certificate without that name is undecided, and the broker refuses it with 503 certificate_status_unavailable (under broker.certificate_status.on_unavailable: refuse, the default; allow would admit it and say so).

The template below adds the extension. Replace its URL with the name your CA's CRL carries (crl.idpURL when set, otherwise https://<the first of ca.json's dnsNames>/1.0/crl) and check it with step crl inspect. This was measured on step-ca (L48).
For an immediate stop, use port49 revoke (Revocation) rather than waiting for the CA's CRL to be republished and read.

The provisioner's template, step-ca-workload.tpl, is step's default leaf with clientAuth alone and the one addition:

{{- /*
  step-ca x509 template for workload certificates (the provisioner's
  options.x509.templateFile). It is step's default leaf, with clientAuth alone
  and one addition: a CRL distribution point naming the CA's CRL.

  step-ca's CRL names its own distribution point in a critical issuing
  distribution point extension, and port49 applies that CRL only to a
  certificate whose CRL distribution points name it, byte for byte (RFC 5280
  §5.2.5; L48). A certificate without the name is undecided, and the broker
  refuses it (503 certificate_status_unavailable).

  Replace the URL below with the name in your CA's CRL: crl.idpURL when it is
  set, otherwise https://<the first of ca.json's dnsNames>/1.0/crl. Check it
  with `step crl inspect` (Issuing Distribution Point, Full Name).
*/ -}}
{
    "subject": {{ toJson .Subject }},
    "sans": {{ toJson .SANs }},
{{- if typeIs "*rsa.PublicKey" .Insecure.CR.PublicKey }}
    "keyUsage": ["keyEncipherment", "digitalSignature"],
{{- else }}
    "keyUsage": ["digitalSignature"],
{{- end }}
    "extKeyUsage": ["clientAuth"],
    "crlDistributionPoints": ["https://ca.corp.example.com/1.0/crl"]
}

The broker side is the verifier below; its keys are in the port49d reference. Nothing about the broker's TLS is special: workloads reach it over the same listener as everyone (TLS).

broker:
  identity:
    machines:
      - type: x509
        trust_anchor_file: /etc/port49/workload-ca.pem
        spiffe_trust_domains: ["example.com"]

Enrolling, renewing, the broker policy snippet, what port49 policy test says about it and what to do on a bad day are in the Oxidized guide at the end of this page. Its section 7a covers host keys: they are OpenSSH's job, and the guide uses accept-new and known_hosts.

The workload directory¶

By default a workload's groups come only from the policy: a subject names the SPIFFE ID in identity: and the grant follows. At fleet scale you may want the other thing, a directory group of workloads, so that adding a host to cn=ci-runners grants it what the group has, with no policy edit. That is the machine_group: subject:

subjects:
  - name: ci_runners
    machine_group: "cn=ci-runners,ou=groups,dc=example,dc=com"
  - name: svc_backup
    identity: ["spiffe://example.com/host/backup-01"]
    username: svc-backup

A machine_group: subject matches workloads only. It takes no username: (the name comes from the entry), and it never sits beside group: or identity: in one subject. port49 policy test says so when you write one (output re-run for the policy above, with a third subject, svc_mapped, that names a SPIFFE ID and no username):

report: group_subject: subject "ci_runners" is a machine_group: subject: it mints only with broker.identity.machines[].directory configured (a workload certificate's directory mapping), through which the broker matches a workload to the group; explain and policy tests evaluate it by name
report: workload_no_username: subject "svc_mapped" declares no username: it mints only under broker.identity.machines[].directory, where its entry's username_attribute names it

Opting in¶

The broker finds a workload's entry through a directory: block on the machine verifier. It is off until you write it, and all four keys are required:

broker:
  identity:
    machines:
      - type: x509
        trust_anchor_file: /etc/port49/workload-ca.pem
        spiffe_trust_domains: ["example.com"]
        directory:
          kind: spiffe_id
          attribute: uid
          allow_weak: true
          issuers: ["CN=Example Workload CA,O=Example"]

kind: spiffe_id is the only kind. The entry holds the workload's SPIFFE ID in attribute.
attribute has no default, and it must not be the attribute broker.directory uses for username_attribute or principal_attribute: the config refuses it. Use uid or description. Never wWWHomePage or url on Active Directory: the entry itself can write those whatever the OU's ACL says, so a workload could put another workload's SPIFFE ID on itself. This was measured on a service account's entry (L50), and port49d says it at start if you choose one of them.
allow_weak: true acknowledges what the mapping is: a SPIFFE ID is matched by a lookup, not a signature, so it is only as strong as who can write attribute. It is required, with issuers, the RFC 4514 DNs of the CAs whose certificates this mapping accepts.

What it costs¶

Every workload this verifier accepts then needs exactly one entry, those whose subject names them in identity: included. port49d prints it at start:

directory: workload certificates map by spiffe_id (weak, pinned to <issuers>) through <attribute> under <filter>; every workload this verifier accepts must have exactly one entry holding its SPIFFE ID: none is refused identity_unmapped, two or more identity_ambiguous, and while the directory is down past grace it is refused directory_unavailable

(Shortened here: the line goes on to name where the username comes from.) In the operator's terms:

Situation	The workload sees	What to do
No entry holds its SPIFFE ID	refused `identity_unmapped`	add the entry, or the value to it
Two entries hold it	refused `identity_ambiguous`	remove one
The directory is down past `grace`	refused `directory_unavailable`, and `port49 workload` exits 75	restore the directory; pre-warm keeps recently mapped workloads working until then
The directory is down longer than `session.staleness_ceiling`	live device sessions deny `context_stale`	the same

Keep pre-warm on, since it is what lets a mapped workload mint through a short directory outage. A workload whose entry changed identity (a host rebuilt with a new directory entry, say) is released from its pin with port49 directory forget spiffe://example.com/host/backup-01. It is run through your agent and needs the forget_principals right and a press of your security key.

The workload's login name comes from the subject that lists its SPIFFE ID (username:), else from the entry's username_attribute. The entry needs a unique canonical one either way.

Whoever writes `attribute` chooses the groups¶

Anyone who can write attribute on an entry decides which workload takes that entry's group memberships. So it must be writable only by directory administrators, and you should alert on writes to it: Windows event 5136 or 4662, and on Samba dsdb_json_audit at level 5. port49 prescribes no deny ACL; the measured behaviour is in the L50 note for Samba.

A person's entry is never a workload. An entry that holds a SPIFFE ID in attribute is refused as a person, and the reverse.

There is one window. Writing a SPIFFE ID onto a person's entry takes effect at the next directory refresh, whereas port49 revoke --principal user:<email> is immediate, so until the refresh the entry could serve both domains. Only a directory administrator can open that window, and that is the same person who could add the user to any group. If it happens, revoke the person, then fix the entry (Revocation).

The directory itself is configured under broker.directory; see Identity providers.

Kubernetes service accounts¶

Nothing here makes a cluster necessary or preferred: it is the path for a job that already runs in one. The pod's projected token is the identity, with no CA to run and no secret at rest. The kubelet writes and rotates the token, and the broker verifies it against the cluster's published keys.

Unmeasured

Managed Kubernetes services (GKE, EKS, AKS) are unmeasured: whether their token issuers, discovery and key endpoints behave as the kind cluster did is unknown until they are measured (lab register entry L70, open). The Kubernetes guide below is measured on kind only; read each claim for the cell that backs it.

The guides in full¶

Everything from here to the end of the page is two example READMEs verbatim, the Oxidized one first and the Kubernetes one after it. Each numbers its own sections from 1, so the second "1." below is the Kubernetes guide's "What it is". Section headings of the Oxidized guide run from "What runs where" to "What L49 measured, and what it did not".

Oxidized backs up network devices over SSH with a username and password. port49 gives it a short-lived token per device as that password, so no standing credential exists on the Oxidized host. Three files fit together:

port49-oxidized-run runs port49 workload token, writes Oxidized's router.db, and asks Oxidized to back each node up now.
config is Oxidized's configuration. It reads router.db, and its own interval is set a week out so that its schedule tries as few nodes as possible with an expired token. A try with a spent token cost one authentication, refused token_spent, on each of the five platforms (L49, phase B); how often Oxidized's own schedule still tries is not measured.
policy.yaml is the port49 policy: a config_backup role that confines the Oxidized host to show commands where port49 authorizes commands (see §5a).

1. What runs where¶

Oxidized and port49-oxidized-run run on one host, under one user. The workload certificate and its key are that user's files. Cron runs the wrapper once per backup window:

17 2 * * *  PORT49_BROKER=https://port49-broker.example.com:8443 PORT49_BROKER_CA=... \
            PORT49_CERT=... PORT49_KEY=... /usr/local/bin/port49-oxidized-run

The wrapper's environment is documented at the top of the script. OXIDIZED_NODES is a file of device names, one a line. PORT49_SCOPE names a policy scope in its place: the wrapper then asks port49 for every device the scope selects that this host holds a grant on, page by page, and reads no nodes file (policy engine spec §13 decision 42). A scope that selects nothing for this host, and a scope name that does not exist, both exit 77 with one message. L49 ran with a nodes file; a sweep was not part of it.

2. Enrollment, once¶

Get a certificate whose one URI SAN is the host's SPIFFE ID (for example spiffe://corp.example.com/host/oxidized-01) from your workload CA. step-ca 0.30.2 is one such CA: a certificate requested with --san spiffe://… carries that ID as its one URI SAN, and a renewal keeps the ID and the key (L48, cells 1 and 2). Any CA that puts one URI SAN on a client certificate works. Its key is readable by the Oxidized user alone. On step-ca, issue it under step-ca-workload.tpl (§3 and §5).

3. The workload CA is the trust root¶

Whoever can get the workload CA to issue a certificate naming a SPIFFE ID is that workload to port49. spiffe_trust_domains does not stop one workload impersonating another within a domain. Restrict each provisioner to the host's own ID. On step-ca 0.30.2 a name policy cannot do it (L48, cell 6):

an x509 allow on a URI in the provisioner's options was ignored: another trust domain's ID was issued as asked;
the authority's x509 allow takes a URI domain, not an ID: given the full SPIFFE ID, step-ca refused to start ("contains scheme (not supported yet)").

What does pin it is a provisioner x509 template with the ID fixed in uris in place of the requested sans. It does not refuse another request: it issues the pinned ID instead. So give each host its own provisioner, whose copy of step-ca-workload.tpl replaces the sans line with "uris": ["spiffe://corp.example.com/host/oxidized-01"], and keep the provisioner's credential the host's alone. L48 measured the fixed uris in a template without the CRL distribution point; the two in one template are not measured.

4. Renewal¶

Renewal is your CA's renewer's job (step's step ca renew --daemon, or its systemd timer; step ca renew --daemon renewed a 15m certificate at 0.66 of its lifetime, L48 cell 3). port49 needs nothing at a renewal. With the CA down, once under a quarter of the lifetime is left the broker logs "its renewal is late" and port49 workload warns (first seen at 0.79 of it, polling every 30s), and minting stops at expiry with exit 77 (L48 cell 4). A late one shows in cron mail and in port49_workload_certificates_renewal_late_total, which each broker replica counts once per certificate a day. Alert on its increase.

5. The broker¶

broker:
  identity:
    machines:
      - type: x509
        trust_anchor_file: /etc/port49/workload-ca.pem
        spiffe_trust_domains: [corp.example.com]

The anchor is the CA that issues the workload certificates (on step-ca, its intermediate, L48).

Its hierarchy. The workload anchor and the human anchor must be in unrelated hierarchies. The start refuses one that signs the other directly. One further down is refused at every request whose chain shows it.
Its status. certificate_status must cover it: name the CA's CRL in crls. An https CRL endpoint under a private root needs that root in the broker's trust store (SSL_CERT_FILE, L48), or sync the CRL to a file: source. With no source, every certificate is refused, and port49 workload exits 77, not 75. Turning status checking off for everyone is not the fix.
step-ca's CRL names a distribution point. Its issuing distribution point names the CRL's own URL (in the lab, https://<the CA's first DNS name>/1.0/crl; step crl inspect shows yours), and port49 applies such a list only to a certificate whose CRL distribution points name that URL (spec, "Certificate status, as built"). step-ca's default leaf names none, so with the CRL configured it is refused 503 certificate_status_unavailable and port49 workload exits 77, the detail saying which name it lacks. Issue workload certificates under step-ca-workload.tpl, with its crlDistributionPoints set to the name step crl inspect shows. Measured: a default leaf was refused; a templated one minted, and was refused credential_revoked 1m after step ca revoke (L48 cell 5).

5a. What `port49 policy test` says about this snippet¶

Merged as shipped, port49 policy test prints two reports. Neither fails the run, and both are true.

fallback_exposure on arista_eos and cisco_nxos. config_backup is at admin, which those platforms render as network-admin. port49's rules confine the session to the permits in policy.yaml. But if port49 stops answering while a session is open, and the device's command method list falls back to local, the device authorizes that session by its role alone. L22 measured that such a session could change the configuration on vEOS and the N9Kv (tests/lab/results/L22-arista_eos.md, L22-cisco_nxos.md). admin is the level spec §2 gives a backup account so show running-config is reachable. Whether a lower level serves each Oxidized model is unmeasured: L49 ran at admin. Until it is, the exposure belongs to whoever holds a session opened with this run's tokens. Keep router.db and the workload key the Oxidized user's alone, and keep PORT49_TTL to one sweep.
unenforced_rules on juniper_junos. A policy that pins no platform_set renders with set 1.0.0. Under 1.0.0, Junos neither asks port49 about a command nor receives the role's rules in its session reply, so on Junos the permits above decide nothing: the session gets what the device's own login class gives the port49 username. Set 1.1.0 renders them into the session reply (spec §8), but the permits here name no Junos command. L49 measured it on vJunos-router 23.4R2-S2.1 and vSRX: a full backup under set 1.0.0 sent port49 no command at all, so the device's login class for svc-oxidized is what limits a Junos backup. Which permits a Junos backup would need under set 1.1.0 is not measured.

6. Check before cron¶

port49 workload identity --broker https://port49-broker.example.com:8443 \
  --broker-ca ca.pem --cert w.pem --key w-key.pem

prints the SPIFFE ID, the subject and the grants.

7. Under cron¶

Always --out, as the wrapper does: cron mails a job's stdout, tokens included. The wrapper gives --out a fresh file each run, so a run that could not write it leaves the last router.db alone. Size PORT49_TTL to the whole sweep, since the wrapper queues every node at once. A fleet larger than one sweep's TTL mints in waves.

The wrapper never prints a token. If you edit it, do not add set -x or an echo of one.

7a. Host keys¶

Oxidized runs with secure: true and never secure: false. It has no accept-new of its own, and the one setting it reads, secure, is a boolean (lab L90, vEOS 4.31.2F, Oxidized 0.37.0): with secure: true and no known_hosts entry it raises Net::SSH::HostKeyUnknown and sends no token; with a changed key, HostKeyMismatch and no token; any other value of secure, accept_new included, turns verification off, so a changed key is accepted. An entry OpenSSH's accept-new wrote, hashed or not, is read and the backup runs.

So the wrapper saves a key for each node absent from known_hosts (~/.ssh/known_hosts of the user Oxidized runs as; OXIDIZED_KNOWN_HOSTS moves it), once, with the line L90 measured: ssh -F /dev/null -o UserKnownHostsFile=<known_hosts> -o StrictHostKeyChecking=accept-new -o BatchMode=yes -o PreferredAuthentications=none -o PubkeyAuthentication=no -o IdentityAgent=none -o IdentitiesOnly=yes -o ConnectTimeout=15 -- <address> exit. The key is saved at key exchange, before authentication, and the flags stop every method that would sign or send, so no credential leaves the host. -F /dev/null means the sweep does not read your ssh configuration: a device reached only through a jump host or a ProxyCommand, or one saved under a HostKeyAlias, is not reached by it. Add that device's key with your own tooling (ssh-keyscan, or ssh with your config); the wrapper then finds it listed and leaves it alone. Only IP literals are visited.

A listed node is never visited. A changed key stays refused until you remove its line (ssh-keygen -R <address>) after checking the change. The wrapper stays green for it: the symptom is one node whose backups stop, and Oxidized's log says Net::SSH::HostKeyMismatch for it. Look there.
The sweep is bounded. The visits run OXIDIZED_HOSTKEY_JOBS at a time (default 32) and none starts after OXIDIZED_HOSTKEY_BUDGET seconds (default 600), nor runs past it. A node not reached is named on stderr, the run exits 1, and the node is tried again at the next window; this window its backup fails at the host key, with no token sent, and every node with a known key is backed up. A node whose key could not be saved (unreachable, blackholed) is named the same way and costs up to the budget each window until it is fixed.
Tokens do not age through it. The addresses come from the mint, so the sweep follows a first mint and a run that visited any node mints again after it: the tokens Oxidized gets are minted after the sweep. A window with nothing to visit mints once. The first mint's tokens are never used and lapse at their TTL. A node that stays unreachable makes every window mint the whole fleet twice, which doubles the broker's mint load and audit records for that window: fix or remove the node. A node that first appears in the second mint is visited too.
Concurrency is unmeasured. L90 swept one device; up to 32 visits append to known_hosts at once, which OpenSSH does with one small append each. A lost line fails closed: the node is named and tried again at the next window.
Needs ssh, ssh-keygen and timeout on the wrapper's host. A missing one is named and the run exits 2 with router.db untouched.
Measured on vEOS only (Oxidized 0.37.0). That Oxidized reads what OpenSSH saved for the host-key type it negotiated is unmeasured on NX-OS, IOS-XE and Junos.

8. On a bad day¶

Exit 75: the broker or its store did not answer within the deadline, or the broker could not establish the certificate's revocation status (a CRL past its next update, an OCSP responder down: look at the CA, not the broker), the output could not be written, files could not be made beside router.db, another run holds the lock (router.db.lock), or Oxidized refused a request. The nodes minted before it are queued; with none, Oxidized kept its last router.db and was not reloaded. Try the next window.
- A partial run replaces router.db with the minted nodes alone, so the unminted nodes are absent from Oxidized's list until a later run mints them. Their backup history in git is untouched.
- Oxidized refusing (each request has a 10 second limit): a failed node prints its name and the rest are still queued. A failed reload prints that router.db was replaced but Oxidized was not reloaded.
- Just after a broker start: a port49-broker's first start, or a start after a policy edit, refuses every mint with a retryable 503 store_unavailable until its runner's takeover is done.
  
  How long that takes: up to the session store's etcd.hold_ttl/3 (stores.session.etcd.hold_ttl, or stores.directory.etcd.hold_ttl when stores.session is omitted), which is 5s at the default 15s and 20s at 60s. Add the takeover itself, plus up to hold_ttl more when the last run's lease must expire first (a crash or a SIGKILL of the only broker: 80s or more at 60s).
  
  The takeover purges every unspent token, so the broker holds the mint rather than hand out a token that is about to be deleted. The cause is this start gate, not a store fault, so do not chase etcd.
  - port49 workload retries it within its deadline, so a run normally just takes longer; exit 75 means the window outlasted the deadline.
  - port49 token, through the agent, does not retry: it exits saying to retry in 5s, and running it again is the answer.
  The broker's stderr says mints wait for a takeover to judge bundle ... and then mints open: ..., and port49_broker_mint_gate_closed reads 1 meanwhile. /readyz stays 200. A rolling restart on the same policy is never held.
  
  A replica whose policy file differs from the runner's stays held until the files agree (its port49_policy_bundle_matches_runner is 0). When the runner's takeover could not judge every context (the directory or the IdP unreachable), the runner says N live context(s) left unjudged at the takeover. Its own broker mints, and the other replicas stay held until its settle pass records the bundle. - Exit 77: nothing was minted and running again will not change it: an expired certificate (said before anything connects), a revoked principal, every node denied, a certificate status nothing covers, or a TLS trust failure (the broker's certificate is not under --broker-ca, or the broker refused the workload's certificate: wrong CA, a clock off). Oxidized kept its last router.db. The stderr line says which. - Exit 1: a node was not permitted, was not minted after a final refusal, or was left out because its output line would hold a control character. Its line on stderr says why; the rest are queued. A node whose platform has no Oxidized model in the script, or whose name is outside A-Z a-z 0-9 . _ -, is skipped with a line and exits 1 the same way. A run that has both a node of this kind (1) and a failed queue or reload (75) exits 75: 75 takes precedence over 1, and the stderr lines name every node either way. - To stop the workload: port49 revoke --principal spiffe://..., and revoke the certificate at the CA.

The config sets auth_methods with keyboard-interactive for the eos model alone (models: eos: vars). Arista EOS offers only publickey and keyboard-interactive over SSH, and Oxidized's default list does not include the second, so without this line Oxidized never sends the token to an EOS device (L49, L49-oxidized-arista_eos.md). Do not change the EOS side to suit it. It is not global: on NX-OS and IOS-XE, which also offer password, an L49 run with it global (at 0a6d546, superseded and not kept as a note) measured a stale token costing two authentications (both denied as spent) instead of one, so those models keep Oxidized's default. After a refused token net-ssh's keyboard-interactive asks a tty for a second password; under a daemon or cron Oxidized logs Errno::ENOTTY for it (measured in the L49 lab, which has no tty). Run from a terminal in the foreground, it may stop at a password prompt instead; not measured.

9. What L49 measured, and what it did not¶

Measured on all five platforms (tests/lab/results/L49-oxidized-<platform>.md):

the wrapper's REST calls answered 200, and the models eos, ios, nxos and junos loaded their nodes and backed them up;
one authentication per run, and a stale token refused in one (retries: 0);
the commands each model sends: policy.yaml permits every one port49 was asked about (show vtp status on IOS-XE and show inventory all on NX-OS were added from it);
no token in Oxidized's log.

Not measured: whether a lower level than admin serves each model (§5a), how often Oxidized's own schedule tries a node with an expired token, and models other than these four.

1. What it is¶

The pod's projected service account token is the workload's identity. No CA, no secret at rest: the kubelet writes and rotates the token, the broker verifies its signature against the cluster's published keys.

2. The cluster¶

Set --service-account-issuer per cluster. kubeadm's default is the same everywhere, and port49 refuses one issuer twice. This is the main onboarding cost for a self-managed fleet.

The safe sequence when changing an issuer is below. It is not measured: it is reasoned from decision 49 of the policy engine spec (rulings 3 and 4), and the two-verifier config is unit-tested. No L69 cell changed a real cluster's issuer; cell (d) rotated a signing key under one issuer.
1. Add the new issuer as the first --service-account-issuer flag, and keep the old one as a second. The API server signs with the first and accepts both.
2. In port49, list both issuers as two k8s verifiers on the same jwks: two verifiers with one jwks and different issuers are how one cluster's issuer migration is written.
3. Roll the pods.
4. After the longest expirationSeconds in use has passed, remove the old flag and the old verifier.
Apply issuer-discovery.yaml. Measured on kind (L69 (c)); managed clusters are unmeasured (L70).

Know what it exposes. The binding is to system:unauthenticated, so anyone who can reach the API server's address can read the issuer discovery document and the JWKS without authenticating. They are public keys only, no secret, and the binding grants no other anonymous access.

Where that reach is too wide, use a narrower route instead of the binding:
- The API server's structured authentication configuration (1.32+), with AuthenticationConfiguration's anonymous.conditions limited to the two discovery paths, /.well-known/openid-configuration and /openid/v1/jwks. The exact form is not measured: L69 (c) measured the binding alone.
- Publish a mirror of the keys and point jwks at it, with no binding at all (see the jwks item).
Issuer discovery needs --anonymous-auth on (not measured; kind's default had it on). A CIS-hardened cluster turns it off.
jwks is the API server's own address, not the issuer's host. Discovery's jwks_uri names the issuer's host, which nothing outside the cluster may resolve. That is why decision 49 ruling 2 reads jwks directly:
```
echo "$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')/openid/v1/jwks"
```
It may be any https mirror of the keys (a bucket, a reverse proxy) for a cluster whose API server the broker must not reach.

The cluster CA for ca_file:

kubectl config view --raw --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d

3. The broker¶

port49d-identity.yaml is a fragment for port49-broker.yaml. Per verifier:

Key	Meaning
`type: k8s`	a machine verifier; a cluster's service accounts are workloads, not humans
`cluster`	port49's name for the cluster, in records and log lines; the cluster's manifests do not change
`issuer`	the cluster's `--service-account-issuer`, as its tokens carry it in `iss`
`jwks`	where the broker reads the keys (section 2)
`ca_file`	the CA the broker trusts for `jwks`
`audience`	what the pods' token volume asks for (`port49` here)
`namespaces`	the namespaces whose service accounts this cluster may name

One verifier per cluster, each with its own cluster name.

Several clusters may list one namespace. The same CronJob deployed to every region is then one identity, system:serviceaccount:netops:oxidized, and one policy subject, with no per-cluster namespace or overlay.

Each record's auth.k8s.cluster says which region minted.
The start's line identity: k8s namespaces: netops: eu_west, us_east says which clusters vouch for each namespace. Check it after adding a cluster.
Where a namespace means different tenants in different clusters, list different namespaces per verifier.

The overlap is allowed because a compromised cluster already holds a genuine token for its own copy of the workload. Refusing the overlap would buy nothing.

4. The workload¶

backup-cronjob.yaml:

automountServiceAccountToken: false, on the ServiceAccount and the pod. The default API token's aud is the API server's and port49 refuses it (measured on kind, L69 (h)); only the projected volume is mounted.
expirationSeconds: 3600. The kubelet refreshes a projected token at about 80% of its lifetime (measured on kind, L69 (b)), and the broker refuses a token issued longer ago than broker.identity.max_token_age (2h by default), so keep it at or under 3600.
Least privilege: the pod sets runAsNonRoot: true (the image runs as 65532) and seccompProfile: RuntimeDefault; the container sets allowPrivilegeEscalation: false, readOnlyRootFilesystem: true (/work is the writable emptyDir) and drops all capabilities.
--token-file is read at every request, so a rotated token is picked up without a restart.
The example stops at the mint. /work/tokens.tsv is on an emptyDir; the consumer (your backup tool) runs in the same pod after this container, sharing the work volume. It is not shown.
The image is the repository's Dockerfile target port49: docker build --target port49 -t port49:local .. The broker address and CA in the manifest, and the port49-backup ConfigMap (devices, broker-ca.pem), are placeholders for your own.

5. The policy¶

policy.yaml maps the principal system:serviceaccount:netops:oxidized to the subject svc_backup. username: is required: a service account has no login name of its own. Check a policy offline with port49 policy test --policy policy.yaml --inventory <file>.

6. 3am¶

What the operator sees. <cluster> is the verifier's cluster name with its issuer.

Case	The workload's stderr	The broker's log and metric
First start, JWKS unreachable	503 `identity_provider_unavailable`; retries within `--deadline`, exits 75	`identity: <cluster>: no keys fetched yet (<error>)…`; `port49_oidc_keys_fetch_failures_total{issuer}` climbs; record `deny trust_domain_unresolved`
JWKS unreachable later	mints go on with the keys held for `jwks_max_age` (24h); then as above	`keys not refreshed since <t>…`, hourly; alert on `time() - port49_oidc_keys_last_success_timestamp_seconds > jwks_max_age/2`
Key rotated at the cluster	none: one refetch finds the new key (measured on kind, L69 (d))	none; rotated wrongly: 401 "no one key of matches its key id", exit 77
Clock skew	401 "issued at , ahead of the broker's clock by more than clock_skew"	fix the broker host's clock
Token from an unlisted cluster	401 "no configured verifier accepts its issuer", exit 77	metric under `issuer="unknown"`
Unlisted namespace	401 `identity_unmapped` "its namespace is not one lists"	record `auth.method: k8s`
Wrong audience, or the default API token	401 "its aud is not exactly [port49]", exit 77	fix the volume's `audience` (L69 (g), (h))
Token past `max_token_age`	401 "issued ago, past broker.identity.max_token_age"	the kubelet stopped rotating, or `expirationSeconds` is too long
Discovery names another issuer	each token refused until `issuer` matches	`identity: <cluster>: the cluster's discovery … names issuer <X>`, once

A deleted service account or pod is not a refusal: see section 7.

7. Stopping a workload¶

port49 revoke --principal system:serviceaccount:netops:oxidized

Deleting the service account alone is not immediate. The broker does not call the cluster's TokenReview: doing so would put every cluster's API server on the mint path, and needs a broker credential on each cluster. A token already in hand mints until its exp or iat plus max_token_age, at most 2h. L69 (e) measured that TokenReview answers unauthenticated after the pod and then the service account are deleted and port49 still mints, and that revoke then refuses at once (the 2h bound is the configured limit, not measured). revoke closes the window at once; the mark also stands for a service account re-created under that name until port49 revoke --lift.

8. What is not measured¶

Managed clusters (EKS, GKE, AKS): L70, open.
A CronJob run in a cluster against a broker outside it: not measured.
Changing a real cluster's --service-account-issuer (section 2): not measured.