When it will not start¶

port49d and port49-broker refuse to start rather than start wrong. Each refusal is one line that names the file, the setting or the key. Read that line first.

The sections below cover the three deployment shapes, which are peers. Each says where to find the line and what to do.

The fastest check on any shape is the metrics port:

curl <host>:9049/readyz for port49d
curl <host>:9050/readyz for the broker

serving with a 200 means the process is up. Anything else is a 503 with a line that says why (starting, waiting for the store, draining).

systemd¶

journalctl -u port49d -b (or -u port49-broker) holds the line. Fix the cause and the next try starts.

The units set no start limit, so a start that fails is tried again every ten seconds. A port49d whose etcd is down at boot comes back on its own when etcd does. journald rate-limits a repeated line.

Refusal	The line	Where	Fix
A secret file missing or unreadable	`config: encryption.current.secret_ref (id "e1") file:/etc/port49/secrets/encryption.key: open ...` naming the file; a key file: `config: keys.k version 1 secret_ref file:...: ...`	`journalctl -u port49d -b`	create it, owned and moded as the table above
The fence refuses a directory	`...is mode drwxrwxr-x, writable by others than its owner; whoever can write it decides who gets access - chmod go-w it ...`	same	`chmod go-w` the directory the line names, or an ancestor of it
A key readable by a group the service is not in	`...is group-readable by gid N, which this process is not in (gid M); members of that group could read the secret too - chgrp it to the service's group ...`	same	`chgrp port49d` (or `port49-broker`) the file the line names. `chmod 0600` only if the service's user owns it: on the shipped `root:port49d 0440` file it leaves the key readable by root alone. A group such as Debian's `ssl-cert` is refused
A secret file's POSIX ACL names someone else	`...carries the ACL entry user:N:r--, a user that is neither this process's (uid M) nor root; it could read the secret too - remove it with setfacl -x u:N on the file (setfacl -b clears every extended entry)`	same	run the `setfacl` the line prints (`-x u:N` or `g:N` for a named entry; `-m g::-` or `-m o::-` for a base one) on the file it names, or `setfacl -b` it. port49d reads the ACL, not the mode's group bits (which are the ACL mask): a file with `group::---` and only port49d named is accepted
The issuing CA differs from the store's	`issuance: this broker's issuing CA (...) is not the one the store records (...)`	`journalctl -u port49-broker -b`	copy `issuing-ca.pem` and `.key` from a broker that holds it
etcd below the floor	`etcd at <url> is version <v>, below the floor port49 needs (3.4.24, 3.5.8 or 3.6): ...`	`journalctl -u port49d -b`	upgrade the member
etcd unreachable at boot	a store refusal after `start_timeout` (30 s unless you set it), then the unit restarts	`journalctl -u port49d -b`	nothing: it retries every ten seconds; start etcd

Containers (compose)¶

docker compose logs port49d (or port49-broker) holds the line.

restart: unless-stopped has no limit. A start that fails is tried again, with a restart delay that grows to a minute, until you fix the cause or docker compose stop it. So docker compose ps shows a restarting container, and the log repeats the line.

A port49d whose etcd is down at boot comes back on its own when etcd does.

Refusal	The line	Where	Fix
A secret file missing or unreadable	`config: encryption.current.secret_ref (id "c1") file:/etc/port49/secrets/encryption.key: open ...: permission denied` (or the file it names); a key file: `config: keys.k version 1 secret_ref file:/run/port49/keys/tacacs_v1: ...`	`docker compose logs port49d`	write the file as the table above says (`setup` installs a key that is absent, and refuses one that differs; it never rewrites one)
The fence refuses a directory	`...is mode drwxrwxr-x, writable by others than its owner; whoever can write it decides who gets access - chmod go-w it ...`	same	fix the mode the line names in the volume (`docker run --rm -v <volume>:/v alpine:3.22 chmod go-w /v/<dir>`)
A key readable by a group the container is not in	`...is group-readable by gid N, which this process is not in (gid M); ...`	same	the container's group is 65532: `chgrp 65532` the file in the volume (`chmod 0600` only if the container's user owns the file: on a `root:65532` file it makes the key unreadable)
The issuing CA differs from the store's	`issuance: this broker's issuing CA (...) is not the one the store records (...)`	`docker compose logs port49-broker`	copy `issuing-ca.pem` and `.key` from a broker that holds it (above)
etcd below the floor	`etcd at <url> is version <v>, below the floor port49 needs (3.4.24, 3.5.8 or 3.6): ...`	`docker compose logs port49d`	upgrade the member
etcd unreachable at boot	a store refusal after `start_timeout` (30 s unless you set it), then the container restarts	`docker compose logs port49d`	nothing: it retries; start etcd

Kubernetes (Helm)¶

kubectl logs --previous <pod> holds the line of a pod in CrashLoopBackOff. The chart README's table of what you see:

You see	Look at	Cause and action
Pod `CrashLoopBackOff`	`kubectl logs --previous <pod>`	The config or a Secret is wrong; the log names the field or reference, never its value.
Pod `ContainerCreating` for minutes	`kubectl describe pod <pod>` (the Events)	A named Secret or ConfigMap is missing: a kubelet event, never a port49 line.
Pod `Pending`	`kubectl describe pod <pod>`	A named PVC is missing or unbound, or nothing can be scheduled.
port49d refuses to start naming etcd and a version	`kubectl logs --previous <pod>`	A member is below the floor (3.4.24, 3.5.8 or 3.6); upgrade etcd. Every member is listed and checked.
Not Ready, running	`curl :9049/readyz` through a port-forward	Not serving yet: the body says why (`starting`, `waiting for the store`, `draining`). It does not name the store; `port49 store status` does.
No pod is ever created; the StatefulSet's events say `violates PodSecurity`	`kubectl -n <ns> describe statefulset <name>`	The namespace enforces `restricted`. The data-dir init runs as root, so the namespace must be `baseline`: `kubectl label --overwrite ns <ns> pod-security.kubernetes.io/enforce=baseline`.
Pod `Init:Error` or `Init:CrashLoopBackOff`	`kubectl logs <pod> -c data-dir` and `-c data-dir-owner`	`data-dir-owner` naming a path: a file in the data directory belongs to another uid; run the `chown -R 65532:65532 ...` it prints. Naming a directory that may be mode 0000: `chmod u+rwx` it as root on the volume, then restart. `data-dir` saying the data directory is a symlink: someone replaced it; inspect and remove the link.
`kubectl get externalsecret` shows `SecretSyncedError`	`kubectl describe externalsecret <name>`	The external secrets operator cannot reach its store or a remote key. The Secret keeps its last data, so a new key version never appears and port49d names the missing `secret_ref`.

A broker refused "behind a promotion"¶

After a CA rotation, a broker that missed the copy refuses to start and the pod crash-loops with:

issuance: this broker's issuing CA (<fingerprint>) is behind a promotion: the store records <fingerprint>, promoted from it by <broker> at <time>. Copy issuing-ca.pem, issuing-ca.key and issuing-ca-previous.pem from that broker into broker.state_dir, then start

The chart's scripts/copy-issuer.sh cannot fix it: it never overwrites a broker's CA, and this broker holds the old one. Copy the three files by hand:

Find the broker the line names, and the two PVCs. The chart's StatefulSet is <release>-broker. It makes one claim per ordinal, data-<release>-broker-<ordinal>, and the files live in state/ inside it.
Start a throwaway debug pod for each PVC.
- The claims are ReadWriteOnce, which only one node can mount. Set each debug pod's nodeName to the node of the broker pod that holds that claim, or the debug pod stays pending.
- The port49 image has no shell, so use an image that has one. The chart README's backup note uses a throwaway alpine pod the same way.
Read the three files off the source broker's claim, mounted readOnly: true. kubectl exec ... -- tar c -C /s/state issuing-ca.pem issuing-ca.key issuing-ca-previous.pem streams them. Write them into state/ on the crash-looping broker's claim, with owner 65532:65532 and mode 0600.
Delete the debug pods and let the broker start. Compare the issuing as ... (sha256:<fingerprint>) line on every broker; the fingerprints must be identical.

Two commands look like fixes and are not:

Do not run init-issuer on this broker. It would make a different CA.
Do not run forget-issuer. Every certificate the old CA issued would stop working.

The key rotation page has the other two refusals.

Which healthcheck to use where¶

Both port49d and port49-broker have a healthcheck subcommand, for an image with no shell and no curl. port49-inventory healthcheck takes the same flags. The subcommand asks the process's own metrics_listen, and it exits 0 or 1.

Command	Asks	Means
`port49d healthcheck`	`/healthz`	Liveness: the process runs and its metrics listener is bound.
`port49d healthcheck --ready`	`/readyz`	Readiness: the TACACS+ listener is serving. 503 while it waits for the store or drains.

The broker's /readyz is its own: 503 starting until its API is up, then 200. The default and --ready mean the same for it as for port49d.

Use the default, /healthz, for anything that restarts a container or holds something back when it is unhealthy: Docker Swarm, an autoheal sidecar, a Nomad check_restart, an ECS container health check that replaces the task, and a Compose depends_on with condition: service_healthy.

Use --ready (/readyz) only to route or to display: Compose's own healthy in docker compose ps, a load balancer's member check.

The reason is the 3am case. Whenever port49d waits for its store it keeps its listener closed, and /readyz answers 503 waiting for the store: the TACACS+ listener stays closed until etcd answers. Two cases differ in how long it waits:

A store lost while port49d runs. It waits with no limit and comes back by itself when etcd does. Devices find TCP 49 refused and fall to their local method meanwhile.
A cold start while the store is down. It waits start_timeout (30 seconds by default; 0 waits as long as it takes), then exits 1, and systemd or compose starts it again.

A supervisor that restarts on --ready makes both cases worse. It turns the first into a restart loop. In the second, it adds restarts of its own to the ones the exit already causes.

Each restart throws away what the process holds in memory: its cached directory membership and pins. The restarted process serves from its stored snapshot only up to session.staleness_ceiling. A store outage must never restart-loop port49d.

The shipped compose.yaml probes with --ready, and docker stack deploy acts on it

compose.yaml runs healthcheck --ready for port49d and the broker, which is right for docker compose: it only reports healthy and restarts nothing on it. Docker Swarm (docker stack deploy) restarts a container that turns unhealthy. Deployed to Swarm as shipped, port49d is restarted again and again for as long as the store is down. If you adapt the file to Swarm, do two things to both healthcheck blocks: remove --ready so they probe /healthz, and raise retries from 3 to 6. The shipped interval: 10s and retries: 3 are a 30-second window, which is not enough (next paragraph).

Give a check that restarts on failure a window of more than 30 seconds. Even /healthz has a gap: between one run of port49d and the next after a store loss, /healthz and /readyz are unbound for seconds.

The chart meets this: /healthz every 10 seconds with six failures is 60 seconds.
Compose's shipped healthcheck (10 seconds, three retries, a 30-second start period) does not. It does not need to, because compose restarts nothing on it.

On Kubernetes the chart already does this: /readyz is the readiness probe, so a not-ready pod leaves the Service, and /healthz is the liveness probe.