Skip to content

Qualification

A platform is qualified when the lab has run a real firmware image against port49 and the measured behavior is recorded. Two tiers use that record:

  • The mock tier (make qual) replays each vetted firmware's measured behavior against a real in-process port49d. It needs no image, no lab and no special runner, so it runs on every pull request.
  • The lab tier boots the real image. It is make lab-qualify, and make qual-vet wraps it for the vetting procedure below. It needs the lab, so only a maintainer with the lab runs it.

The commands you will use:

Command What it does
make qual The mock tier: the device-mock packages, tests/qual, and the port49d qualification tests.
make qual-vet P=<selector> Vets one firmware in the lab and generates its mock for review. It commits nothing.
make qual-update Rewrites the goldens after a deliberate change to port49's decisions. Review the diff.
make qual-fleet Linux only: many mock devices, each from its own loopback address, against one port49d.
make qualified-platforms Regenerates docs/qualified-platforms.md from the profiles, goldens and notes.

Vendor images are never hosted

port49 does not host, mirror or redistribute a vendor's device image, and no pull request adds one. You obtain an image from the vendor under your own license and keep it on your own lab host. What the repository holds is the measured behavior: the notes, the profiles and the goldens.

The procedures

The rest of this section is the contributor guide from tests/qual/README.md, included as it stands. engine_gaps are the behaviors of a device that port49's engine cannot yet express; each one needs Go in internal/devicemock and a test.

The mock tier (make qual) replays each vetted firmware's measured behavior against a real in-process port49d: no images, no lab, any runner. A mock exists only for a firmware the lab has run. The lab run is make lab-qualify; make qual-vet wraps it. Details of the run itself are in tests/lab/README.md, "Qualification".

Layout: profiles/<platform>/<slug>.yaml (the mock), golden/<platform>/<slug>.json (the expected port49 decisions), qual-policy.yaml and qual-daemon.yaml (what every run is played under). docs/qualified-platforms.md is generated from these (make qualified-platforms).

Vet a new firmware and add its mock

  1. Put the image where the lab host keeps its images (for example /mnt/images).
  2. Add its platforms row in tests/lab/harness_test.go: selector, topology, image, version (as the device reports it) and a measured boot.

  3. Run make qual-vet P=<sel>, for example make qual-vet P=nxos106.

    It refuses when any of these holds:

    • /tmp/port49-STOP-LAB exists
    • the lab is occupied
    • tests/qual, tests/lab/results, tests/lab/evidence or docs/qualified-platforms.md has uncommitted changes

    It then runs make lab-qualify, genprofile, make qual and make qualified-platforms, and commits nothing. If any step fails, it lists the paths the run left changed. Restore them before committing anything, or a failed run's note or evidence goes in with the next commit.

  4. Review the profile against the note. genprofile prints a checklist. Before you commit, confirm every line of it against tests/lab/results/QUAL-<slug>-<lp>.md:

    • each reaction it pre-filled as measured
    • each acceptance rule and its status
    • the review-left defaults, client_chain and server_identity
    • each engine gap id
    • each device fact

    A claim is measured only when a note for this firmware measured it (decision 52, ruling 4).

    On Junos the command cells are not_applicable (L18: tests/lab/results/L18-juniper_vjunos-router.md). So genprofile records the engine gap command_authorization_not_sent; that is expected, not a failure.

    TLS is played in two modes. The TLS login is played the way the device defaults, per-request (tls_token_login), and again with single-connection on (tls_single_connection). Each verdict line names its mode. The harness never turns single-connection on to make a per-request cell pass.

    Where per-request TLS sends port49 no command authorization (NX-OS 10.6(2)F, L73 cell a):

    • genprofile records the device fact tls_per_request_no_command_authorization.
    • The mock plays that login with no command step.
    • The list's "command authorization over TLS" column reads "no" or "single-connection only".
    • The tls_denied_command cell then types the denied command alone over per-request TLS and writes tests/lab/results/L80-<lp>.md (spec ยง15 L80). Its verdict is observed, never pass. genprofile records its outcome as the device fact tls_per_request_denied_command_runs or ..._refused, citing that note.
    • The list's TLS column then reads "no: over per-request TLS, a command port49 denies runs (L80)" or "no command authorization over per-request TLS; denied commands refused locally (L80)".

    Single-connect offered over TLS. port49 offers single-connect over TLS to some platforms (platform.TLSServerSingleConnect; today cisco_nxos). For those:

    • run.json's tls_server_single_connect_offered is true.
    • The run waits until no connection is open before and at tls_token_login's capture (tls_capture_started_clean).
    • genprofile grades tls_token_login from its header-only *.tls.json and port49's records. It writes tls_server_single_connect_honored when a connection carried a session after a completed one that was not accounting, and tls_server_single_connect_ignored when every connection carried one session.
    • It writes nothing if the capture did not start clean, any two of the cell's connections overlap, or a record falls in no connection.

    Neither is written where port49 never offered: absent is no claim. The mock reuses an offered connection where the profile says honored. The drift test (tests/qual/drift_test.go) fails when the table and the profiles disagree either way. So a firmware that ignores the offer fails before a release ships the table entry for it.

  5. Commit the evidence, the note, the profile, the golden and the list.

No mock is added for an image the lab has not run (ruling 8).

Add a platform

  1. Add a platforms row and a topology.
  2. Write a startup config with break-glass and the port49 TACACS+ server group.
  3. For TLS, add an l73Dialects row, written help first.
  4. Give it a port49 inventory platform: built in, or generic (which logs platform_unknown).
  5. Vet it as above.

Any engine gap it needs becomes an engine_gaps entry, plus Go in internal/devicemock with a test.

When a mock fails on a PR

  • A gate (an acceptance rule). The device would reject port49's reply. Fix port49, or, if the rule is wrong, re-measure it in the lab.
  • A golden difference. If port49 changed on purpose, run make qual-update and review the diff. The list shows re-vet pending until the next lab run.
  • A missing reaction. The vetter adds it from the notes.

The release-tag gate

  1. make lab-regress.
  2. make qual-vet for every profile whose golden is pending.
  3. Commit.

The lab regression run

make lab-regress reruns every closed entry in tests/lab/regress.yaml and rewrites its results notes. Each entry's measured_minutes is its wall time in a full run (the first, commit 2c680dbf, 2026-10-19/20, then the L35, L37, L66, L79, L80 and L90 entries, and XRd's L73): the manifest sums to 745.7 minutes. A clean full run is about 12 h plus NX-OS boot hangs (below), so routine use needs a subset: make lab-regress LAB_REGRESS_ARGS=-only=L35,L37. The slowest are L34 (103 min), L36 (67) and L22 (44).

Runs before the 2026-10-20 rebase name the rebased twin of the commit they ran (2c680dbf for the first full run); the tree differs by PR #29's client and spec changes only. The vJunos 23.4R2-S2.1 QUAL run names 73f7a5df, the commit its amended-away original became: the code is identical, and only two stray pcaps were dropped.

The first run passed 58 of 66 entries. Of the 8 that failed, all are resolved:

  • L35, L37: the test expected an older release's circuit-open line; port49d now goes dark on its store health checks (about 5 s). The verdict was fixed and both passed on the rerun.
  • L66 (five entries): the lookup failed for a name outside the inventory; fixed by PR #29, and L66 passed on the rerun.
  • L29-reload on vjunos: no connection open across the change was reset in the refused/rekeyed phase. Junos sends nothing on a held connection, so the phase can fail its premise; a device flake, so retry the entry.

L80 (L80-nxos106) is a regress entry: it re-measures a security fact (a command port49 denies runs over NX-OS 10.6's per-request TLS), and a run that observes nothing is INVALID. A flipped outcome shows as changed lines in the note diff, and a run that observes nothing writes no L80 note. lab-regress reports note changes and does not fail on them. It does fail an entry whose listed note the run did not rewrite (its mtime is before the entry started): notediff masks the header lines a rerun changes, so an untouched note would otherwise read "unchanged".

NX-OS boot hang. 11 of 39 NX-OS deployments never booted: the container log stops after feature netconf and port49d sees no packet within 8 min. The cause is not yet known; the device appears to hang while the launcher applies its startup config. Retry the entry. Each deploy now logs lab: <platform>: boot: and host: lines, for example lab: cisco_nxos: boot: phase=ssh_auth at=2026-11-07T01:04:10Z since_deploy=2m7s and lab: cisco_nxos: host: phase=failed mem_total_kb=32768000 mem_available_kb=900000 swap_total_kb=8388608 swap_free_kb=100000 load1=9.50 load5=7.25 load15=5.00 procs_running=6 procs_total=812. Read a hang from them: ssh_auth with no tacacs_seen is the config apply, and mem_available_kb at failed against deploy_start shows host pressure. Do not raise the timeout without them.

What a run needs beyond the base setup:

  • PORT49_LAB_ETCD_BIN names a directory holding etcd, etcdctl and etcdutl, all v3.6.15 (for example /mnt/images/etcd-v3.6.15 on the lab host). L35-restore needs etcdutl.
  • L66 needs tests/lab/l66/.collections, which is git-ignored, so a fresh worktree lacks it. Create it with <venv>/bin/ansible-galaxy collection install -p tests/lab/l66/.collections <collections>; the L66 preflight prints the exact command with the collection names.

Working around the NX-OS boot hang

make lab-regress passes LAB_REGRESS_ARGS to the runner, whose -only= flag takes a comma-separated list of entry ids and runs just those. Use it to retry an entry whose NX-OS device never booted, instead of rerunning the whole manifest:

make lab-regress LAB_REGRESS_ARGS=-only=<entry_id>,<entry_id>

-dry-run lists the entries and their last measured minutes without running anything. Each deploy logs boot: and host: lines; read the hang from them before you raise a timeout.