Failure matrix

For each part of port49 that can fail, this matrix gives:

• what the device does
• what you see
• what to do
• the test or lab note that shows it

Start with "At 3am", the table of the failures that cost a login. If you begin from a report, an alert or a log line instead, use the 3am index; it links here for the device behaviour.

What a device does with an ERROR differs by platform. Some fall to their local method at once. Some treat the ERROR as final and refuse break-glass until port49d closes its listener. The rows say which. A platform a row does not name is unmeasured, not safe.

Every row names its evidence. A row that cites a lab note is measured behaviour. A row that says a case is open or unmeasured means the behaviour there is unknown.

Each way port49 can fail, as an operator reads it: what the device does, what you see, what you do, and the evidence that says so. Every row names a subtest of tests/compose/failmatrix_test.go (TestFailureMatrix/<row>, run by make failure-matrix and the failmatrix CI job), an existing test, or a closed lab row. TestEveryMatrixRowHasEvidence fails when a row of the design's failure table is missing here or has no evidence.

At 3am

You see	What the devices do	Do this	Page
port49d is up and its store is failing (port49_store_health_failures rising)	port49d answers a token login ERROR store_unavailable and every command is denied, for up to 5.5 s (close_after 5 x interval 1 s + call_timeout), then closes TCP 49. Cisco IOS XE treats that ERROR as final, so break-glass is refused for those 5.5 s. Junos under the reference authentication-order tacplus (Decision 3) does the same. Cisco IOS XR over TLS does the same (L73, L73-cisco_xrd.md); over TCP it is unmeasured (L93, open). After the close, devices find the connection refused and fall to their local method	Wait the 5.5 s, or stop port49d to get there at once. Then fix the store	port49_store_health_failures_close
SR Linux, same case	Break-glass logs in at once: with exit-on-reject true an ERROR is not final for the local method (it logged in and ran a command in 2.4 s, L87 a), and once port49d is dark it does the same in 2.3 s (L87 b). A port49 user's token login is refused for the 5.5 s. Measured in tests/lab/results/L87-nokia_srlinux.md	Nothing to wait for. Fix the store	port49_store_health_failures_close
port49_store_restore_detected_timestamp_seconds above 0, and the drop line on port49d's stderr (the session store was restored from a snapshot)	A token minted before the restore is refused FAIL token_spent once (port49d compares the token's store revision with the revision it read the restore at); the user mints again and the new token logs in. Break-glass is as without the restore. A restore taken without --bump-revision also refuses, for 20 minutes, a token minted within 5 minutes after port49d's read whose revision is at or below the old timeline's; the runbook's own --bump-revision restore needs no such band, but a Velero restore cannot use it, so after one port49d usually arms the band and every token minted in the 5 minutes after its read is refused token_spent (users mint a new token after it: one minted inside the window stays refused until the band ends, 20 minutes after the read; break-glass is unaffected). port49d does not refuse a token only when all of these hold: a non-bumped restore; a second old-timeline restore that no read tells, within 20 minutes; a broker clock more than 5 minutes ahead of port49d's; and a token redeemed between the second snapshot and the first restore	Start a broker, whose purge removes what is left, and follow the restore runbook (Restoring etcd from a snapshot); there is no need to stop port49d. After a restore that also restored the spool or broker.state_dir (Velero, a volume snapshot), the token delete of the runbook's step 3 is required: port49d sees that restore only if its <accounting.spool_path>.restore.json is newer than the etcd copy (usual on a busy store, not on an idle one), and a broker sees it on the same terms (its witness.json is then newer than the restored witness) and purges once more at its first takeover. Revocations made after the snapshot are lost by any restore: re-issue them	port49_store_restore_detected (ticket)
port49_redemption_restore_unjudged at 1 on port49d	port49d cannot read meta/witness, so it cannot tell whether its store was restored, and refuses every token login with an ERROR store_unavailable until a read answers. IOS XE and Junos under tacplus alone treat that ERROR as final, so break-glass is refused too until a read answers, until go-dark closes the listener (about 5.5 s) when the store itself is failing, or until port49d is stopped (devices then find TCP 49 refused and fall to their local method)	Fix the store, or the daemon role's read of meta/witness (the log line carries the store's own error). Stop port49d only to let devices fall to break-glass at once	port49_redemption_restore_unjudged (page, 2m)
After a restore, port49_broker_mint_gate_closed at 1 on a broker, POST /v1/token answered 503 store_unavailable with Retry-After: 5	A broker that owes the restore's purge mints nothing until the purge is done, so a token minted in that window is never deleted by it. Token logins of tokens already held are unaffected; port49 token retries	Wait: the runner's purge line (every unspent token is purged) ends it on the runner, and the other replicas open once they find a runner running the keyspace. A broker with no witness.json (a new or lost broker.state_dir) that has seen no runner write a witness since it started also purges once at its first takeover, so every unspent token is minted again: expected on a cold start of the whole fleet on new state_dirs. If the gauge stays at 1 for more than a minute, no runner is purging: check that one holds the keyspace (port49_store_keyspace_held summed over the brokers is 1) and read the runner's stderr for the takeover stopped or a purge that failed; it is tried again by the next takeover	port49_broker_mint_gate_closed (ticket, 1m)
TCP 49 refused at once	The device treats port49 as down and uses its local method	Start port49d, or leave it stopped until the store is back

Rows

Failure	Device sees	Operator sees	Fix	Evidence
Directory unavailable	Logins in the grace window pass on the stored membership; past it a new session is refused 503 directory_unavailable	port49_directory_server_down, port49_directory_refresh_stale	Restore the directory servers; the grace is broker.directory.grace	TestGraceReadsTheStoreAfterARestart, TestTheDirectoryDownSignInSaysSoAndHoldersMint
Enrollment store lost	Held certificates, tokens and IdP sign-in serve; a password sign-in is refused 503 enrollment_store_lost	port49_directory_enrollment_store_ok 0, port49_directory_enrollment_store_lost	port49-broker init-enrollments on a new install, else restore the file or port49 directory import	TestALostEnrollmentStoreRefusesOnlyTheSignIn
Identity provider unreachable	A token held is accepted until its exp or max_token_age; new OIDC logins fail	port49_oidc_keys_stale, port49_oidc_tokens_unavailable	Restore the provider, or the broker's route to it	TestAClusterJWKSOutageKeepsTheHeldKeys, TestATokenOlderThanMaxTokenAgeIsRefused
Token store unavailable	A token login is answered ERROR store_unavailable and every command FAIL store_unavailable, until port49d closes its listener; then the connection is refused and the device falls to local. While port49 is dark, on EOS and NX-OS commands in sessions already open ran under local authorization (L41 cell 3); a command port49 would deny is then decided locally. Junos asks port49 for no command authorization (decision 21), dark or not. IOS XE refuses them % Authorization failed. (L35, L41); XR and SR Linux are unmeasured	port49_store_health_failures, then the listener closing	Restore the store; port49d listens again after open_after passes	TestFailureMatrix/store_stopped, IOS XE: tests/lab/results/L4-error-cisco_iosxe.md, tests/lab/results/L41-cisco_iosxe.md; EOS and NX-OS (cell 3): tests/lab/results/L41-arista_eos.md, tests/lab/results/L41-cisco_nxos.md
Store hangs or loses quorum	A token login is answered ERROR store_unavailable and every command FAIL store_unavailable (an authorization is never ERROR), within the packet deadline, then the listener closes (5.5 s at the defaults), and opens again once the store answers, an attempt that succeeds (they are 1 s apart for the first ten, then back off to every 10 s, each with 20% jitter, so up to 1.2 s after etcd answers in the first ten) and open_after health passes (3 s at the defaults), about 9 s after the members resume when the first attempt fails (store_hangs' 9.2 s bound); one member dark, measured in CI run 41769703556 (commit 3ec4349f, six attempts, all passed). A follower dark refuses no login: 0 of 30 logins refused in every attempt, the slowest 271 ms (one_member_dark logs the leader and pauses a follower). The redemption and the context swap are re-sent on another member, and a member whose call just timed out is not chosen for 15 s, so a mint (which is not re-sent: a second copy would write a second token) does not land on it: no mint needed a retry, the slowest took 144 ms. ExtendReceipt and DeleteToken are not re-sent either. A raft leader dark costs a refusal window while the others elect one (leader_dark, 6 rounds per attempt, a new leader each time): over 36 rounds the last refused login came at most 8.2 s after the pause, and most under 3 s. One round in about 54 (this run plus the one before it, d61a41dc) refused for 17.5 s, past the bound below, with the redeem denied store_unavailable within milliseconds for several seconds and no circuit-open line; its cause is not established and it is under investigation (port49_store_fast_fail_total and the matrix's counter dump are there to name it). The context feed and the marks feed each run as two watches on two different members, merged by revision, so a leader or follower dark leaves them following; a watch quiet for 2 s while the store answers is opened again on a member neither is on. The subtest fails if a refusal outlasts the sum of the election timeout, close_after x health.interval, call_timeout, open_after x health.interval, open_for and 2 s of slack (16.5 s at the stack's settings), and logs each round's timeline from the pause: the first failed health check, the listeners' close and reopen from port49d's log, and the first login that passed. A device retries a refused login. Whether a device falls back to its local account during the window is per platform: see L88 cell (c). Whether the breaker's open_for can stretch a short election into a longer refusal is an open question	port49_store_health_failures >= 5, port49_store_circuit_open (also port49_store_marks_feed_down, the same outage)	Restore quorum; the listener returns on its own	TestFailureMatrix/store_hangs, TestFailureMatrix/one_member_dark, TestFailureMatrix/leader_dark
Store unreachable at start, or after a store gone	Nothing listens while port49d waits, as with port49d stopped	/readyz 503, no listener	Restore the store; port49d listens and logins pass	TestFailureMatrix/store_absent_at_start, tests/lab/results/L35-cisco_iosxe.md
port49d restarted on etcd, or listening again after a store gone	A token minted before a restart still redeems on the split deployment; an open session's next command is answered from the same context	A restart in the log; no alarm	None	TestFailureMatrix/port49d_restarted, tests/lab/results/L37-cisco_iosxe.md
Broker unavailable	An open session runs on, a token minted before the stop logs in; no new token can be minted	Mint fails with a connection error	Start the broker	TestFailureMatrix/broker_stopped, tests/lab/results/L37-cisco_iosxe.md
All daemons unavailable	The connection is refused at once; the device falls to local	TCP 49 refused	Start port49d	TestFailureMatrix/port49d_stopped, tests/lab/results/quarantine-cisco_iosxe.md
One region unavailable	The device fails over to the next server	Per-region listener metrics	None	L74 (c): the cut region keeps serving, tests/lab/results/L74-regions.md; L37 (2): a connection to the stopped first server was refused after 0 s and the second answered, on vEOS, the N9Kv and the C8000V (tests/lab/results/L37-cisco_iosxe.md); on both Junos images a fresh login with the first server stopped was refused after 0 s there and the second accepted it, and the login passed (tests/lab/results/L37-juniper_vjunos-router.md, tests/lab/results/L37-juniper_vsrx.md)
Device→daemon partition	Silence: no reply and no reset; the device waits out its own timeout, then falls to local	No connection from the device reaches port49d while the path is cut	Restore the path	TestFailureMatrix/partition, IOS XE: tests/lab/results/L4-cisco_iosxe.md
Operator credential lost	Existing logins persist until port49 revoke ends them	port49_admin_touch_refused on a refused call	Enroll another key; revoke the lost one with port49 revoke	TestRevokeOthersMarksAPrincipalAndEndsAKey, TestARevokedSessionIsDeniedInsideTheTTL
Inventory service unavailable	Logins pass on the last good snapshot, persisted across a restart, never expiring	port49_inventory_snapshot_stale, port49_inventory_sync_failing	Restore the service	TestRunStartsFromThePersistedCopyWithTheServiceDown, TestTheSnapshotNeverExpiresWithTheServiceDown
Device presents a retired or wrong key	The connection is reset with no reply; the device falls to its local method as fast as with port49d stopped	An unvalidated record (matches_retired) and a strike for a new connection; none for an open one	Give the device the current key (port49 keys show)	TestFailureMatrix/retired_key, TestAReloadThatWithdrawsTheVersionEndsItsConnectionWithoutAStrike, tests/lab/results/L28-cisco_iosxe.md
Policy file refused	Logins and commands pass on the policy in force	port49_policy_file_refused 1	Fix the file or its directory's mode	TestFailureMatrix/policy_refused
Revocations file missing or refused	port49d does not start: no listener, so the device falls to local	port49d exits naming the file	port49d init-revocations once; fix the file's mode	TestARevocationsFileOthersCouldWriteIsRefused
Certificate status unavailable	A new token for the issuers it covers is refused 503 with on_unavailable: refuse; the agent retries	port49_tls_cert_status_unavailable, port49_crl_next_update_passed	Restore the CRL or OCSP responder	TestAnAgentConnectingDuringAStatusOutageIsToldToTryAgain, TestRunRefusesOnAStaleCRL
Policy refresh incomplete	A context not yet reached keeps its previous policy until the retry (30 s) reaches it	port49_policy_refresh_pending	Restore the store	TestAScheduledRefreshTheStoreStopsIsPending, TestARefreshTheStoreStopsIsRetried
Store restored from a snapshot (not a §12 row; Decision 2)	See the 3am table: a token minted before the restore is refused token_spent once; the broker's purge follows	port49_store_restore_detected_timestamp_seconds above 0 (ticket port49_store_restore_detected)	Start a broker and follow the runbook; after a restore of the spool or broker.state_dir too, run its step 3's token delete	TestFailureMatrix/restore_floor