Alerts¶
port49.rules.yaml holds one Prometheus alerting rule for each alarm port49 defines. Load it in
one of two ways:
- into Prometheus, with
rule_files: - through the Helm chart, with
prometheusRule.enabled: true. The chart renders the same file as a Prometheus OperatorPrometheusRule.deploy/helm/port49/files/port49.rules.yamlis a byte copy, because Helm cannot read outside the chart, andtests/alertsfails if the two differ.
make alerts-check runs promtool check rules and promtool test rules on it. go test
./tests/alerts/ checks that every metric an expression names is one port49 registers.
There are two severities; route on the severity label.
page: a fault that is refusing devices or losing records now.ticket: one that costs nothing yet, or that a person fixes in working hours.
The job names are the rules' parameter. Three rules read Prometheus's own up series for
job="port49d" and job="port49-broker": port49_target_down, port49_target_absent and
port49_broker_target_absent. The chart's PodMonitors (rendered with prometheusRule.enabled) set
those names. If you scrape another way, give the jobs those names or edit the rules. Without them,
port49_target_absent and port49_broker_target_absent fire, which is the point of them.
The absent rules are fleet-wide: absent() stays quiet while any scrape of that job exists. So use
one Prometheus per port49 fleet, or add a namespace matcher to the two expressions.
No expression names a device, and none groups by one. The series carry no device identity, and the
decision log answers "which devices". The labels used are bounded: instance,
key_ref, version, server, issuer, outcome, feed, pool, component.
Each alert below carries a runbook_url annotation that points at its section here. The metrics each rule reads are in the metrics reference. An alert that has just fired: start at the 3am index, which gives a short do and don't for each family of alerts.
Thresholds and defaults¶
Each row pairs what an alert covers with the rule that answers it.
A threshold that follows a setting assumes that setting's default. The defaults are close_after
5, broker.refresh_interval 5m, jwks_max_age 24h, expiry_warning_fraction 0.33 and
staleness_ceiling 4h. Edit the rule if you set another. The hold_ttl rules are the exception: they
use multiples of the 15s hold_ttl default that hold at any allowed setting (10s to 1m), so they need
no edit.
Edit in the safe direction. A setting lowered below the default can silence its alarm:
- With
close_after: 3, the page onport49_store_health_failures >= 5never fires. - With
jwks_max_age: 6h, the 12h keys rule fires after tokens are already refused.
Recompute each such threshold when you change the setting it follows.
Notes¶
The component label. port49_cert_status_checks_total, port49_crl_loads_total,
port49_crl_next_update_timestamp_seconds and port49_cert_status_unavailable_total carry
component (broker, tls_listener), and the store metrics carry component and pool. A rule
that divides or compares two such series one-to-one fails to match across components: aggregate with
sum by (...) or match with ignoring(component). Every rule here names its component or needs
no match.
A dark etcd member costs up to 12 s. gRPC raises a keepalive below 10 s to 10 s, plus a 2 s
timeout, so a dropped link to one member stalls the marks watch (events resume on another member)
and an unhedged call for that long. Reads and the opted-in mark writes are hedged, other writes are
not, so no alert can fire faster than that; the for: of the store rules is longer than 12 s
for this reason.
Nothing warns before the listener closes. close_after x interval (5 s at the defaults) is
shorter than any scrape interval, so no alert can fire between the first failed health check and
the close. port49_store_health_failures_close pages once the listeners are closed: the gauge is
held by the process, not the run, so the page stays raised through the whole outage and resolves
when the listeners reopen.
Series that exist only where the thing does. port49_store_directory_reachable,
port49_store_directory_confirmed_timestamp_seconds, port49_store_keyspace_held and
port49_directory_refresh_last_complete_timestamp_seconds are created by the code that starts what
they watch (the etcd store, the keyspace runner, the scheduled refresh), at 0. On the memory backend,
the default, they do not exist, so no alert on == 0 or on age has a series to fire on. Where a gauge
cannot work that way, a *_configured gauge says whether to expect it: port49_issuing_ca_configured,
port49_directory_password_configured.
Where the rules differ from the statement they answer.
- port49_oidc_keys_stale drops the statement's or absent(): every configured issuer's series is
created at 0, so an issuer that never fetched already shows time() - 0, and the series is absent
only where OIDC is not configured, where absent() would fire forever.
- port49_directory_enrollment_store_lost keeps it, as port49_directory_password_configured == 1
unless on(instance) port49_directory_enrollment_store_ok: the gauge is absent until the store
first answers, and a broker that never reached it is the case to catch.
- port49_issuing_ca_expiring also requires the CA gauge above 0, so the page for a missing CA is
not doubled by a ticket for a CA "expiring" in 1970.
- port49_store_directory_stale requires its timestamp above 0 (a process that has not yet confirmed
is port49_store_directory_unreachable and port49_store_feed_down).
- port49_directory_refresh_stale and port49_directory_refresh_half_ceiling take the freshest
refresh across all runner-capable replicas, so a replica that lost the hold, whose gauge stops
moving, does not alarm; and they require the newest such process to have been up as long as the
threshold, so a restart is quiet but a restart in the middle of a directory outage (the gauge reads 0
until a pass completes) still fires. Both wait 15 minutes (for:), which covers a runner failover:
the new runner's series sits at 0 until its first pass completes. They read last_over_time(...[1d]),
so they keep firing for a day after every broker is gone, when no series is left to read.
- port49_inventory_snapshot_stale is the spec's expression with its NaN arm and no gate: a process
that never watches the inventory reads 0, and one waiting for its first snapshot reads NaN before
its listener opens, which is the case the NaN arm exists for.
- port49_store_keyspace_unheld sums over the processes that run a keyspace runner, and fires for a
day after the last one disappears (a crashed fleet exports no 0): count(last_over_time(...[1d]))
unless count(...). A memory-backend install never had the series, so it never fires.
- The key-retirement rules leave out the spec's uptime gate (spec, key rotation: line 5578). That only makes them
fire more. A restart blanks port49_key_version_devices (it counts devices whose latest validated
connection since start used the version), so a rule that resolves after a restart is not evidence
the devices have moved: read the decision log.
- enrolled sign-ins outside a rollout are not alerted: "outside a rollout" is yours to define.
The rules¶
port49_target_down¶
Severity: page. For: 2m.
A port49d or broker target is not answering its scrape: up is 0 for it.
up{job=~"port49d|port49-broker"} == 0
Source: A port49d or broker that is down raises no alarm of its own, because every other rule reads series it no longer exports
Why this threshold: up == 0 for a scrape target of either job; two minutes outlasts a restart's missed scrapes.
port49_target_absent¶
Severity: page. For: 5m.
Prometheus has no up series for port49d: no port49d target exists at all, so every device has lost port49 and falls to local authentication, and no other port49d rule can fire.
absent(up{job="port49d"})
Source: A port49d target Prometheus is not scraping at all has no up series, so == 0 never matches
Why this threshold: absent() of port49d's up. Prometheus's pod discovery keeps a target only while its pod exists, so a StatefulSet with every pod Pending or deleted leaves no series for == 0 to match, and every device has lost port49: a page. Five minutes outlasts a rollout. A mislabelled scrape config shows here too.
port49_broker_target_absent¶
Severity: ticket. For: 10m.
Prometheus has no up series for the broker: no broker target exists at all, so no token can be minted and no other broker rule can fire.
absent(up{job="port49-broker"})
Source: The same for the broker: no up series for port49-broker at all
Why this threshold: absent() of the broker's up: no token can be minted, but held tokens still log in. On the memory-store trial, which runs no separate broker, the broker rule fires; drop it there or silence it.
port49_global_cap_shedding¶
Severity: page. For: none.
port49d reset TACACS+ connections at accept because the global connection cap was full; devices fall to local authentication.
increase(port49_connections_shed_total{reason="global_cap"}[5m]) > 0
Source: Connections reset at accept because the global connection cap is full: port49_connections_shed_total{reason="global_cap"} increasing
Why this threshold: Any shed connection is a device refused, so any increase pages; five minutes smooths one scrape.
port49_connections_near_cap¶
Severity: ticket. For: 10m.
A port49d is holding more than 80% of limits.max_connections open; raise the cap or add replicas before it sheds.
sum by (instance) (port49_connections{state="open"}) / on(instance) port49_connections_max > 0.8
Source: A warning before shedding starts: port49_connections near the cap
Why this threshold: Ten minutes at 80% is a trend, not a burst; the page above covers the cap itself.
port49_key_secret_files_changed¶
Severity: ticket. For: 10m.
A key version's secret file changed or cannot be read; the held secret stays in force until the next start.
port49_key_secret_files_changed > 0
Source: A key version's secret file that changed or cannot be read: port49_key_secret_files_changed > 0 for 10m
Why this threshold: A rotated file is expected for minutes while the rollout restarts; ten minutes is a forgotten restart.
port49_issuing_ca_missing¶
Severity: page. For: 5m.
A broker is configured to issue certificates and has no usable issuing CA, so every OIDC sign-in is refused issuance_unavailable; run port49d init-issuer or restore the CA.
port49_issuing_ca_not_after_timestamp_seconds == 0 and on(instance) port49_issuing_ca_configured == 1
Source: A broker set to issue certificates that has no usable issuing CA: port49_issuing_ca_not_after_timestamp_seconds == 0 while broker.identity.issuance is configured
Why this threshold: A broker started without its CA says so and keeps serving; five minutes outlasts a rollout and nothing else clears it.
port49_issuing_ca_expiring¶
Severity: ticket. For: none.
The issuing CA expires within 90 days; make its successor and distribute the new bundle before sign-ins stop.
(port49_issuing_ca_not_after_timestamp_seconds - time() < 90 * 86400) and on(instance) (port49_issuing_ca_not_after_timestamp_seconds > 0) and on(instance) port49_issuing_ca_configured == 1
Source: The issuing CA expires within 90 days
Why this threshold: Ninety days is a CA rollover's lead time; a CA already at 0 is the page above, not this ticket.
port49_key_retirement_within_7d¶
Severity: ticket. For: none.
A key version with devices still on it passes its retire_after within 7 days; move the devices (the decision log names them) before it does.
(port49_key_version_devices > 0) and on(key_ref, version) (port49_key_retirement_seconds < 7 * 86400)
Source: A key version that devices still present nears its retire_after: the first step of an alarm whose severity rises as the date approaches
Why this threshold: A week is a change window; the series carry no device identity, the log names the devices.
port49_key_retirement_within_1d¶
Severity: page. For: none.
A key version with devices still on it passes its retire_after within a day, or already has; those devices fail authentication once it does.
(port49_key_version_devices > 0) and on(key_ref, version) (port49_key_retirement_seconds < 86400)
Source: The same key version within a day of its retire_after, or past it: the alarm's last step
Why this threshold: A day is the last working shift; a version past retire_after with devices on it is refusing them now.
port49_admin_touch_refused¶
Severity: page. For: none.
An admin call was refused its hardware-key touch proof: a stolen token or a broken key.
increase(port49_admin_touch_total{outcome="refused"}[10m]) > 0
Source: An admin call refused its hardware-key touch proof: port49_admin_touch_total{outcome="refused"}
Why this threshold: Someone holds an admin credential without the touch; no benign cause repeats.
port49_policy_fail_closed_compositions¶
Severity: ticket. For: none.
The policy in force has compositions answered fail-closed: those users are denied all but the allows; port49 policy test names them.
port49_policy_fail_closed_compositions > 0
Source: A policy in force that answers some compositions fail-closed: port49_policy_fail_closed_compositions > 0
Why this threshold: A standing condition of the policy, not an incident; a ticket the author sees.
port49_policy_file_refused¶
Severity: ticket. For: 5m.
The policy file on disk was refused or will not settle; an older policy is serving.
port49_policy_file_refused == 1
Source: A policy file on disk that is refused or will not settle: port49_policy_file_refused == 1
Why this threshold: An edit in progress is refused for seconds; five minutes is a file that stayed wrong.
port49_policy_refresh_pending¶
Severity: ticket. For: 2m.
A policy change has not yet reached every live login context.
port49_policy_refresh_pending == 1
Source: A policy swap or grant expiry that has not yet reached every live login context: 1 from the change until its settle pass
Why this threshold: A pass takes seconds; two minutes is a stuck settle pass.
port49_policy_bundle_mismatch¶
Severity: ticket. For: 5m.
A broker or port49d serves a policy bundle other than the runner's; its tokens or admissions carry another policy until it reloads.
port49_policy_bundle_matches_runner == 0
Source: A broker serving a policy bundle other than the runner's: port49_policy_bundle_matches_runner 0 for longer than twice broker.policy_watch_interval
Why this threshold: Five minutes outlasts twice the watch interval and the settle delay at their defaults; raise it if the intervals are raised.
port49_broker_mint_gate_closed¶
Severity: ticket. For: 1m.
A broker mints nothing: its mint gate has been closed for over a minute; the store does not answer, or no runner judges its bundle.
port49_broker_mint_gate_closed == 1
Source: A broker that mints nothing: port49_broker_mint_gate_closed at 1 for a minute
Why this threshold: The gate is closed from a start until a takeover is seen, so a minute is where a restart ends and a fault begins; a ticket, since other replicas keep minting.
port49_devices_generic_platform¶
Severity: ticket. For: 5m.
Admitted devices are served by the generic adapter: their platform has no adapter, and their commands are judged without its knowledge.
port49_devices_generic_platform > 0
Source: Admitted devices served by the generic adapter: port49_devices_generic_platform, naming no device
Why this threshold: A standing condition of the inventory, not an incident; the start line names the platforms and counts.
port49_context_refreshes_failed¶
Severity: ticket. For: none.
A policy refresh could not carry a login context onto the policy in force: a context revoked for a compile fault, or one that would not open.
increase(port49_context_refreshes_total{outcome="failed"}[10m]) > 0
Source: A login context a policy refresh could not carry onto the policy in force: counted failed in port49_context_refreshes_total
Why this threshold: Each failure is one context left on an older policy or revoked.
port49_per_login_mfa_refused¶
Severity: page. For: none.
A run of per-login MFA proofs did not verify: someone is trying another token or another person's sign-in.
increase(port49_per_login_mfa_total{outcome="refused"}[10m]) > 2
Source: A run of per-login MFA proofs that did not verify: per_login_mfa_refused, alerted on its own, because a run of them is someone trying another token or another person's sign-in
Why this threshold: More than two in ten minutes is a run; a mistyped touch is one.
port49_store_directory_unreachable¶
Severity: page. For: 1m.
This process cannot read the directory store: revocations and password sign-ins from other regions do not arrive.
port49_store_directory_reachable == 0
Source: A process that cannot read the directory store: port49_store_directory_reachable == 0 for 1m
Why this threshold: A cut region keeps logging devices in on stale marks, so one minute is a real partition.
port49_store_directory_stale¶
Severity: ticket. For: 1m.
This process last confirmed its copy of the directory store over a minute ago, whatever the reason; its revocation marks may be stale.
(time() - port49_store_directory_confirmed_timestamp_seconds > 60) and on(instance) port49_store_directory_confirmed_timestamp_seconds > 0
Source: A process whose copy of the directory store is unconfirmed for over a minute: time() - port49_store_directory_confirmed_timestamp_seconds > 60
Why this threshold: The series exists only under etcd, at 0 until the first confirmation, so a 0 is skipped here: port49_store_directory_unreachable and port49_store_feed_down cover a process that never confirmed.
port49_store_logouts_pending¶
Severity: ticket. For: 5m.
Back-channel logouts were accepted while the directory store did not answer and have not reached it.
port49_store_logouts_pending > 0
Source: Back-channel logouts accepted but not yet in the directory store: port49_store_logouts_pending > 0 for 5m
Why this threshold: A cut region holds logouts the others have not seen; five minutes is longer than a blip.
port49_store_marks_feed_down¶
Severity: page. For: 1m.
The marks feed is down: revocation marks written elsewhere do not reach this process.
port49_store_feed_up{feed="marks"} == 0
Source: The marks feed down, so revocation marks written elsewhere do not arrive: port49_store_feed_up{feed="marks"} == 0
Why this threshold: Revocation rides this feed; a minute is four times the 15s hold_ttl default and the longest hold_ttl allows (10s to 1m), so at any setting the feed has been down for a full hold.
port49_store_feed_down¶
Severity: ticket. For: 1m.
A store feed other than marks is down (contexts, or a split broker's pins or logouts): this process misses other replicas' writes, and its directory copy is not confirmed.
port49_store_feed_up{feed!="marks"} == 0
Source: A store feed other than marks down: port49_store_feed_up{feed} 0 for longer than hold_ttl
Why this threshold: A minute is four times the 15s hold_ttl default and the longest hold_ttl allows (10s to 1m). Every feed but marks, which pages, so a feed added later is covered.
port49_store_feed_legs_colocated¶
Severity: ticket. For: 10m.
Both legs of the {{ $labels.feed }} feed's watch are on one etcd member: port49 reaches only one etcd member for this feed, so check the other members; that member going dark stalls the feed.
port49_store_feed_legs_colocated == 1
Source: Both legs of a feed's watch on one etcd member: port49_store_feed_legs_colocated at 1 for 10m
Why this threshold: Both legs of a feed's watch on one member is no redundancy against that member going dark; the move apart is tried from 2 s doubling to 30 s, so ten minutes is a cluster where no other member is reachable for this feed. A client with one etcd endpoint never moves a leg and holds the gauge at 0 (colocation is not a fault there); port49d logs once at start that the feeds share one member.
port49_store_circuit_open¶
Severity: ticket. For: 1m.
A store pool's circuit is open: its calls fail at once until a probe succeeds.
port49_store_circuit_open == 1
Source: A store pool's circuit open, so its calls fail at once
Why this threshold: The rule fires for every {component, pool} series, and what an open circuit costs depends on which. On port49d (component="daemon"), a token or context pool open for hold_ttl (the 15s hold_ttl default) closes the listeners, so a minute is four times that and this ticket follows the close; it does not warn before it. A bulk pool never closes them: logins and commands are unaffected and the scans (the refresh, a revocation by principal, a start's purge) fail until a probe succeeds. On port49-broker (component="broker") no circuit closes anything, because its guards never stop; mints, refreshes and revocations through the open pool fail. The component broker_enrollments refuses password sign-in and second-factor enrollment through its open pool, and closes nothing.
port49_contexts_reindexed¶
Severity: ticket. For: none.
Live login contexts were found missing from the index: an index record was deleted in the store.
increase(port49_contexts_reindexed_total[10m]) > 0
Source: Live login contexts found missing from the index: any increase of port49_contexts_reindexed_total
Why this threshold: No benign path deletes an index record.
port49_contexts_reindex_failures¶
Severity: ticket. For: none.
A login context could not be re-indexed or judged; one sealed under a key port49d does not hold is denied, never revoked (check PORT49_ENC_V1).
increase(port49_contexts_reindex_failures_total[10m]) > 0
Source: A login context that could not be re-indexed or judged: any increase of port49_contexts_reindex_failures_total
Why this threshold: Each failure leaves a context for the next pass; a mis-set key repeats it.
port49_store_calls_refused¶
Severity: page. For: none.
etcd's roles are refusing port49's store calls; port49d answers store_unavailable.
increase(port49_store_calls_total{outcome="refused"}[10m]) > 0
Source: etcd's roles refusing port49's store calls: any increase of port49_store_calls_total{outcome="refused"}
Why this threshold: A refused call is a role or credential fault that retries cannot fix.
port49_store_keyspace_unheld¶
Severity: ticket. For: 2m.
No port49-broker replica holds the store keyspace: no runner is judging bundles, purging or refreshing contexts.
(sum(port49_store_keyspace_held) == 0) or (count(last_over_time(port49_store_keyspace_held[1d])) unless count(port49_store_keyspace_held))
Source: No broker replica holding the store keyspace: the sum of port49_store_keyspace_held over port49-broker's replicas 0 for twice hold_ttl
Why this threshold: Two minutes is eight times the 15s hold_ttl default, and exactly twice the longest hold_ttl allows (1m), so it holds at any setting. The series exists only on a process that runs a keyspace runner, so the memory backend, with none, raises nothing. The second arm covers every broker gone (a crashed fleet exports no 0): series seen in the last day and none now, so it fires for a day after the brokers vanish, then stops.
port49_store_client_cert_expiring¶
Severity: ticket. For: none.
A component's etcd client certificate expires within 30 days.
port49_store_client_cert_expiry_timestamp_seconds - time() < 30 * 86400
Source: A store client certificate within 30 days of expiry
Why this threshold: Thirty days is the stated lead time.
port49_crypt_store_key_unknown¶
Severity: page. For: none.
port49d started on a store last written under a current key its encryption window does not hold (a mis-set PORT49_ENC_V1).
port49_crypt_store_key_unknown == 1
Source: port49d started on a store written under a key its encryption window does not hold: port49_crypt_store_key_unknown at 1
Why this threshold: Records written under the missing key cannot open; fix the key and start again.
port49_crypt_token_open_failures¶
Severity: ticket. For: none.
A stored token would not decrypt: a rotation may have retired a key while tokens it sealed were live.
increase(port49_crypt_open_failures_total{record="token"}[10m]) > 0
Source: A stored token that would not decrypt: any increase of port49_crypt_open_failures_total{record="token"}
Why this threshold: The counter is the only witness; the token is dropped from its candidates with no error.
port49_directory_pins_unopenable¶
Severity: ticket. For: none.
A directory pin opens under neither encryption key: its principal cannot bind, mint or refresh.
port49_directory_pins_unopenable > 0 and on(instance) port49_store_keyspace_held == 1
Source: A directory pin that opens under neither encryption key: port49_directory_pins_unopenable above 0 on the replica holding the keyspace
Why this threshold: One principal is locked out until the key that sealed the pin is back as encryption.previous or the pin is forgotten; the pass never deletes it. Gated on port49_store_keyspace_held == 1, since a replica that handed over keeps its last value.
port49_store_feed_rescans¶
Severity: ticket. For: none.
A feed's watch was compacted and rescanned the store; the feed lagged.
increase(port49_store_feed_rescans_total[10m]) > 0
Source: A feed whose watch was compacted and rescanned the store: any increase of port49_store_feed_rescans_total
Why this threshold: A rescan means a watch fell behind compaction; repeated, the feed is too slow.
port49_ledger_pending¶
Severity: ticket. For: 1m.
The runner's ledger record is behind: a context put back at one would be taken after a restart.
port49_ledger_pending > 0
Source: The runner's ledger record behind: port49_ledger_pending above 0 for a minute
Why this threshold: A write takes a round trip; a minute is a record that cannot be written.
port49_ledger_behind¶
Severity: ticket. For: 1m.
The runner's ledger keeper runs with its context feed down; the record does not follow other processes' logins.
port49_ledger_behind == 1
Source: The runner's ledger keeper running with its context feed down: port49_ledger_behind at 1
Why this threshold: One minute rides out a feed reconnect.
port49_store_health_failures_close¶
Severity: page. For: none.
port49d closed its TACACS+ listeners: its store health checks failed close_after times in a row. Break-glass logins use local passwords until it reopens.
port49_store_health_failures >= 5
Source: port49d closing its TACACS+ listeners after its store health checks fail: port49_store_health_failures >= <close_after>
Why this threshold: 5 is close_after at its default; set it to yours. The gauge is held through the outage, so the page stays up and resolves when the listeners reopen. Not above 0: a defragmenting member fails an occasional check. close_after x interval is shorter than any scrape, so nothing warns first.
port49_store_health_check_failing¶
Severity: ticket. For: none.
port49d's store health checks are failing often, short of closing the listeners.
increase(port49_store_health_checks_total{outcome="fail"}[10m]) > 60
Source: Store health checks failing often, short of closing the listeners: increase(port49_store_health_checks_total{outcome="fail"}[10m]) above a threshold the operator picks
Why this threshold: At the default 1s interval 60 is a tenth of the checks in ten minutes; a defragmenting member measured 4 to 7 of 18. Set your own.
port49_context_cache_stale¶
Severity: ticket. For: none.
A replica is answering commands from login contexts the store did not confirm; revocation does not reach them until the store answers or the listeners close.
increase(port49_context_cache_lookups_total{outcome="stale"}[5m]) > 0
Source: Commands answered from login contexts the store did not confirm: any increase of port49_context_cache_lookups_total{outcome="stale"}
Why this threshold: Every stale read is a command authorized on unconfirmed state.
port49_tls_handshake_failures¶
Severity: ticket. For: 5m.
RFC 9887 listener handshakes are failing; the outcome label says why.
sum by (outcome) (rate(port49_tls_handshakes_total{outcome!~"ok|resumed"}[5m])) > 0
Source: TLS listener handshakes failing, by outcome: rate(port49_tls_handshakes_total{outcome!~"ok\|resumed"}[5m])
Why this threshold: Scanners cause the odd failure; five minutes of them is a device or a certificate.
port49_tls_device_certificate_renewal_late¶
Severity: ticket. For: none.
A device certificate is past two thirds of its own lifetime: its renewal is late. The ratio is the least over certificates verified in the last 24h, so it clears up to a day after the device renews.
port49_tls_device_certificate_lifetime_left_ratio < 0.33
Source: A device certificate whose renewal is late: port49_tls_device_certificate_lifetime_left_ratio < 0.33
Why this threshold: 0.33 is the default expiry_warning_fraction; a ratio holds for any lifetime, a fixed number of days would fire forever on short-lived certificates.
port49_tls_server_certificate_renewal_late¶
Severity: ticket. For: none.
The RFC 9887 listener's certificate is past two thirds of its own lifetime: its renewal is late.
port49_tls_server_certificate_lifetime_left_ratio < 0.33
Source: The TLS listener's certificate, its renewal late: port49_tls_server_certificate_lifetime_left_ratio < 0.33
Why this threshold: As for the device certificates.
port49_tls_server_certificate_expired¶
Severity: page. For: none.
The RFC 9887 listener's certificate has expired; TLS devices cannot connect.
port49_tls_server_certificate_lifetime_left_ratio < 0
Source: The TLS listener's certificate expired: port49_tls_server_certificate_lifetime_left_ratio < 0 pages
Why this threshold: Below 0 is expired.
port49_tls_cert_status_unavailable¶
Severity: ticket. For: none.
The TLS listener could not establish a certificate's revocation status; with on_unavailable: refuse its devices are refused.
increase(port49_cert_status_unavailable_total{component="tls_listener"}[10m]) > 0
Source: The TLS listener unable to establish a certificate's revocation status: increase(port49_cert_status_unavailable_total{component="tls_listener"}[10m]) > 0 while TLS devices are admitted
Why this threshold: The series carries only the component; with no TLS devices admitted it does not move.
port49_crl_next_update_passed¶
Severity: ticket. For: none.
A CRL the TLS listener relies on passes its next update within an hour, or has.
time() - port49_crl_next_update_timestamp_seconds{component="tls_listener"} > -3600
Source: A CRL the TLS listener relies on within an hour of its next update, or past it: time() - port49_crl_next_update_timestamp_seconds{component="tls_listener"} > -3600
Why this threshold: An hour is time to fix a failed fetch before the list stops serving.
port49_inventory_transport_changed¶
Severity: ticket. For: none.
A sync changed a device's transport, which may be a downgrade from tls to tcp; port49 records it and does not block it.
increase(port49_inventory_changes_total{kind="transport_changed"}[1h]) > 0
Source: A sync that changed a device's transport: increase(port49_inventory_changes_total{kind="transport_changed"}[1h]) > 0
Why this threshold: A downgrade is silent otherwise.
port49_oidc_keys_stale¶
Severity: ticket. For: 5m.
An OIDC issuer's keys were never fetched since this process started, or last fetched over 12h ago, half of jwks_max_age at its default; tokens are refused once they are 24.
time() - port49_oidc_keys_last_success_timestamp_seconds > 12 * 3600
Source: An OIDC provider outage that keeps its keys from being fetched: time() - port49_oidc_keys_last_success_timestamp_seconds > jwks_max_age / 2 or absent(...), with a for: of a few minutes
Why this threshold: Half of jwks_max_age (24h by default): one more failed cycle and every token for that issuer is refused. Set it to yours. Not or absent(): the series exists only for configured issuers, so on a deployment with no OIDC it would fire forever.
port49_oidc_logouts_failing¶
Severity: ticket. For: none.
OIDC back-channel logouts were not fully applied (store_error or partial); sessions may outlive their sign-out.
increase(port49_oidc_logouts_total{outcome=~"store_error|partial"}[10m]) > 0
Source: OIDC back-channel logouts not fully applied: logouts store_error\|partial
Why this threshold: Each is a logout the store did not take whole.
port49_oidc_tokens_unavailable¶
Severity: ticket. For: none.
Bearer tokens are refused because their issuer's keys are unavailable.
increase(port49_oidc_tokens_total{outcome="unavailable"}[10m]) > 0
Source: Bearer tokens refused because their issuer's keys are unavailable: tokens unavailable
Why this threshold: The identity provider is unreachable and the held keys have expired.
port49_workload_certificates_renewal_late¶
Severity: ticket. For: none.
Workload certificates are being presented with under a quarter of their validity left: their renewal is late.
sum(increase(port49_workload_certificates_renewal_late_total[1h])) > 0
Source: Workload certificates presented with their renewal late: increase(port49_workload_certificates_renewal_late_total[1h]) > 0, summed over replicas
Why this threshold: Each replica counts a certificate once per 24h, so any increase of the sum is a late renewer.
port49_redemption_restore_unjudged¶
Severity: page. For: 2m.
port49d cannot read meta/witness, so it cannot tell whether its store was restored, and refuses every token login (store_unavailable) until it can; check the store and the daemon role's read of meta/witness; stopping port49d lets devices fall to break-glass
port49_redemption_restore_unjudged == 1
Source: A port49d that cannot tell whether its store was restored: port49_redemption_restore_unjudged at 1
Why this threshold: The one refusal state the restore floor keeps: port49d cannot read meta/witness, so it refuses every token login with an ERROR, which IOS XE and Junos under tacplus alone treat as final. A failed read clears in seconds; two minutes is a store or a role that stayed wrong.
port49_store_restore_detected¶
Severity: ticket. For: none.
port49d found its session store restored from a snapshot: tokens minted before the restore are refused token_spent once (users mint again); revocations and marks made after the snapshot are lost: follow the restore runbook, Restoring etcd from a snapshot
(port49_store_restore_detected_timestamp_seconds > 0) and (time() - port49_store_restore_detected_timestamp_seconds < 86400)
Source: A session store restored from a snapshot: port49_store_restore_detected_timestamp_seconds above 0
Why this threshold: Fires for a day after port49d finds its store restored (the gauge holds the time); the restore runbook on the backup and restore page is the work, and revocations made after the snapshot are lost until it is followed. The 0 of a port49d that never found one is skipped.
port49_directory_refresh_stale¶
Severity: ticket. For: 15m.
No scheduled directory refresh has verified every principal for over three refresh intervals; contexts keep their last verification and will deny context_stale at the ceiling.
(time() - max(last_over_time(port49_directory_refresh_last_complete_timestamp_seconds[1d])) > 3 * 300) and on() (time() - max(last_over_time(process_start_time_seconds[1d]) and on(instance) last_over_time(port49_directory_refresh_last_complete_timestamp_seconds[1d])) > 3 * 300)
Source: No scheduled directory refresh completed for three intervals: time() - port49_directory_refresh_last_complete_timestamp_seconds > 3 × refresh_interval
Why this threshold: 300 is broker.refresh_interval at its default (5m); set 3x yours. The freshest refresh anywhere, so a runner that lost the hold does not alarm on its old value; and only once the newest process with the series is also that old, so a restart does not, but a restart in the middle of an outage does (the gauge reads 0 until a pass completes). 15 minutes covers a failover: the new runner's series sits at 0 until its first pass completes. last_over_time keeps the rule firing for a day after every broker is gone.
port49_directory_refresh_half_ceiling¶
Severity: page. For: 15m.
No directory refresh has verified every principal for half of session.staleness_ceiling (4h at its default); at the ceiling every live session denies context_stale. Escapes: session.staleness_ceiling: 0s with a port49d restart, or stop port49d so break-glass applies.
(time() - max(last_over_time(port49_directory_refresh_last_complete_timestamp_seconds[1d])) > 2 * 3600) and on() (time() - max(last_over_time(process_start_time_seconds[1d]) and on(instance) last_over_time(port49_directory_refresh_last_complete_timestamp_seconds[1d])) > 2 * 3600)
Source: No directory refresh completed for half the staleness ceiling: time() - port49_directory_refresh_last_complete_timestamp_seconds > session.staleness_ceiling / 2
Why this threshold: 7200 is half the default ceiling; set half yours. Same shape and 15m for: as port49_directory_refresh_stale. A page: at the ceiling sessions are denied.
port49_directory_lookups_ambiguous¶
Severity: ticket. For: none.
A directory lookup matched more than one principal and was refused identity_ambiguous.
increase(port49_directory_lookups_total{outcome="ambiguous"}[10m]) > 0
Source: Directory lookups that match more than one principal: port49_directory_lookups_total{outcome="ambiguous"} increasing
Why this threshold: Each is a user refused; the directory holds two entries for one name.
port49_directory_server_down¶
Severity: ticket. For: 10m.
A directory server's breaker is open; queries go to another server, or fail if there is none.
port49_directory_server_up == 0
Source: A directory server down: port49_directory_server_up == 0 for a while
Why this threshold: Ten minutes: one server down is a degraded pool, not an outage.
port49_directory_group_names_ambiguous¶
Severity: ticket. For: 5m.
Group names match no one because two groups under groups_base_dn share them.
port49_directory_group_names_ambiguous > 0
Source: Group names that two directory groups share: port49_directory_group_names_ambiguous > 0
Why this threshold: A gauge over the last hour; the grants naming those groups are not given.
port49_directory_enrollment_store_lost¶
Severity: page. For: 2m.
The broker's password enrollment store is lost, not initialised, or was never reached; password sign-in is refused.
port49_directory_enrollment_store_ok == 0 or (port49_directory_password_configured == 1 unless on(instance) port49_directory_enrollment_store_ok)
Source: The broker's password enrollment store lost or never reached: port49_directory_enrollment_store_ok == 0 or absent()
Why this threshold: The gauge is absent until the store first answers, and a broker that has never reached it is the case to catch. port49_directory_password_configured is 1 only on a broker with the verifier, so a broker without one never matches the absent arm.
port49_sign_ins_throttled¶
Severity: ticket. For: none.
Directory-password sign-ins are being throttled: failed attempts passed failures_per_principal or failures_per_source.
increase(port49_sign_ins_total{outcome="throttled"}[10m]) > 0
Source: Password sign-ins being throttled: a rise of throttled
Why this threshold: A throttle needs failures_per_* failed attempts: a guessing run, or one user's stale password on a script.
port49_sign_ins_enrollment_store_lost¶
Severity: page. For: none.
A password sign-in was refused because the enrollment store is lost.
increase(port49_sign_ins_total{outcome="enrollment_store_lost"}[10m]) > 0
Source: Password sign-ins refused because the enrollment store is lost: any enrollment_store_lost
Why this threshold: Any one is an operator or user refused; the gauge alert above covers a store that is lost with no sign-in in flight.
port49_accounting_spool_drops¶
Severity: page. For: none.
The accounting spool discarded records; the stage label says where.
increase(port49_accounting_spool_drops_total[10m]) > 0
Source: An accounting spool dropping records
Why this threshold: Accounting records are an audit trail; no drop is acceptable.
port49_accounting_spool_disk_slow¶
Severity: ticket. For: 10m.
port49d's accounting spool disk cannot keep up; every accounting record waits on its fsync, so devices see slow or failed accounting. Move the spool directory to a disk with sustained fsync p99 under 10 ms.
histogram_quantile(0.99, rate(port49_accounting_spool_sync_seconds_bucket[5m])) > 0.010
Source: An accounting spool disk too slow, by port49's own threshold: p99 of port49_accounting_spool_sync_seconds over 10 ms for 10m
Why this threshold: Every accounting ack waits on the spool's fsync; with the session etcd and the spool on a CT1000P3PSSD8 drive, account p99 was 152 ms at 170/s (tests/lab/results/load-beehive.md, cell c, step_1_account_p99_ms); with both in RAM the spool's capacity was 1291/s (tests/lab/results/load-beehive-ram.md, cell c), and on a GCE pd-ssd (operator-declared) 2905/s with account p99 3.67 ms at 170/s (tests/lab/results/load-p49-gce-n2s16.us-central1-a.c.port49.internal.md, cell c).
port49_decision_log_write_failures¶
Severity: page. For: none.
The decision log sink refused writes; decisions are not being recorded.
increase(port49_decision_log_write_failures_total[10m]) > 0
Source: Decision log writes refused: the log must be complete, never sampled
Why this threshold: The log is the audit trail; a refused write counts per attempt.
port49_accounting_foreign¶
Severity: page. For: none.
Accounting records arrived for a session port49 never saw begin: a local fallback or console login on a device, or a device bypassing port49.
increase(port49_accounting_attributions_total{attribution="foreign"}[10m]) > 0
Source: Accounting for sessions port49 never saw begin: the foreign series, not rate-limited, because a burst of foreign is itself the signal
Why this threshold: Any foreign record is the detection the spec names for a local login the policy did not see; it pages.
port49_inventory_sync_failing¶
Severity: ticket. For: 10m.
The last inventory sync failed: a source, the policy file or the state directory did not answer.
port49_inventory_sync_failing == 1
Source: An inventory sync that failed: the service retains the previous bundle and alarms
Why this threshold: Syncs repeat; ten minutes is several failed ones.
port49_inventory_sync_refused¶
Severity: ticket. For: 5m.
The last inventory sync was refused and an older bundle is published.
port49_inventory_sync_refused == 1
Source: An inventory sync that was refused: the service keeps the last good snapshot and alarms
Why this threshold: A refused sync is a bad source document; five minutes is not a race with its fix.
port49_inventory_bundle_refused¶
Severity: ticket. For: 5m.
port49d was given an inventory document it refused; the bundle in force keeps serving.
port49_inventory_bundle_refused == 1
Source: An inventory document port49d refused: the fleet keeps the last good snapshot and alarms
Why this threshold: The old bundle serves, so a ticket; five minutes outlasts a rollout.
port49_inventory_mirror_behind¶
Severity: ticket. For: 10m.
The inventory mirror's copy differs from the bundle published.
port49_inventory_mirror_behind == 1
Source: The inventory mirror behind the published bundle
Why this threshold: A write is quick; ten minutes behind is a mirror that fails.
port49_inventory_persist_failed¶
Severity: ticket. For: 5m.
The inventory document in force could not be written to inventory.service.state_dir; a restart would start without it.
port49_inventory_persist_failed == 1
Source: The inventory document in force not written to the state directory
Why this threshold: The state directory is full or unwritable.
port49_inventory_snapshot_stale¶
Severity: ticket. For: 5m.
The inventory snapshot in force was last confirmed over a day ago, or there is none; the fleet still authenticates on it, and nothing fails closed.
port49_inventory_snapshot_age_seconds > 86400 or port49_inventory_snapshot_age_seconds != port49_inventory_snapshot_age_seconds
Source: A stale inventory snapshot: port49_inventory_snapshot_age_seconds raises an alarm, never a fail-closed
Why this threshold: 86400 is a day; set your sync cadence times a few. The NaN arm (no snapshot yet) is the spec's own expression. Not gated: a process that never watches the inventory reads 0, and one that waits for its first snapshot reads NaN from before its listener opens, which is the case the NaN arm is for.
port49_inventory_policy_mismatch¶
Severity: ticket. For: 5m.
The inventory service built the bundle in force against a policy other than port49d's own; port49d applies the inventory against its own policy.
port49_inventory_policy_mismatch == 1
Source: An inventory bundle built against a policy other than port49d's: the hash mismatch is logged and alarmed
Why this threshold: Five minutes outlasts a policy rollout reaching both.
port49_inventory_privilege_changes¶
Severity: ticket. For: none.
A source-of-truth edit moved privilege: a device changed key or platform, was admitted or added, gained an address, or entered or left a security_log scope; the inventory_change record names the device.
increase(port49_inventory_changes_total{kind=~"key_ref_changed|platform_changed|admission_changed|address_added|scope_joined|scope_left|device_added"}[1h]) > 0
Source: Source-of-truth edits that move privilege: port49_inventory_changes_total{kind}
Why this threshold: Every kind but transport_changed, which has its own alert. A device_added during onboarding raises it too: expected, and it is the only place a new capture target shows.
etcd_no_leader¶
Severity: page. For: 1m.
An etcd member has no leader; a dark member costs up to 12 s, a lost quorum stops the store.
etcd_server_has_leader == 0
Source: An etcd member with no leader: etcd_server_has_leader == 0
Why this threshold: A minute rides out an election.
etcd_db_nearly_full¶
Severity: ticket. For: 10m.
An etcd member's database is over 80% of its quota; at the quota etcd refuses writes.
etcd_mvcc_db_total_size_in_bytes / etcd_server_quota_backend_bytes > 0.8
Source: An etcd database nearly at its quota: etcd_mvcc_db_total_size_in_bytes / etcd_server_quota_backend_bytes > 0.8
Why this threshold: Defragment or compact before the quota: ten minutes is not a spike.
etcd_disk_slow¶
Severity: ticket. For: 10m.
etcd's disk cannot keep up; port49 store calls will time out in bursts. Move etcd to a disk with sustained fsync p99 under 10 ms (datacenter NVMe or a cloud SSD volume).
histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) > 0.010 or histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) > 0.025
Source: An etcd disk too slow, by port49's own threshold: p99 of etcd_disk_wal_fsync_duration_seconds over 10 ms, or of etcd_disk_backend_commit_duration_seconds over 25 ms
Why this threshold: etcd's own guidance is a WAL fsync p99 under 10 ms; a drive that stalls can push store calls past the 500 ms call timeout in bursts: on a CT1000P3PSSD8 drive the session etcd logged 7974 to 9244 slow applies in cells d and e (tests/lab/results/load-beehive.md), against 1 and 0 in RAM (tests/lab/results/load-beehive-ram.md) and at most 74 on a GCE pd-ssd (operator-declared; a 44, c 74, d 1, e 0) with a WAL fsync mean of 0.72 to 0.86 ms (tests/lab/results/load-p49-gce-n2s16.us-central1-a.c.port49.internal.md). Ten minutes over a 5m rate is a drive, not a compaction.
etcd_leader_changes¶
Severity: ticket. For: none.
etcd changed leader more than 3 times in an hour: a flapping network or a slow disk.
increase(etcd_server_leader_changes_seen_total[1h]) > 3
Source: etcd changing leader too often: increase(etcd_server_leader_changes_seen_total[1h]) > 3
Why this threshold: Occasional changes are rollouts; four in an hour is a fault.