docs: record the stress findings and the 16-gate suite

caveats §13 documents the capacity ceiling the stress test found: the
workqueue backlog grows with the offered rate and OOM-panics a 1 GB guest
above ~1 Gbit/s. Gate count, suite status and the doc indexes move to
16/16 green.
This commit is contained in:
lgcosta committed 2026-11-12 20:11:12 -03:00
1 parent 928322edf7
commit 0a8ba69340
15 files changed
+191 -30

No files matched your search

+3 -2
View File
@@ -13,8 +13,9 @@ packet filter** to **Linux 7.3.4** — the real PF engine (`pf.c`, `pf_norm.c`,
|---|---|
| [guide.md](guide.md) | One-page architecture: components, data path, control plane, milestones |
| [install.md](install.md) | Apply/build/install recipe (kernel, module, tools) |
| [debian13.md](debian13.md) | Full from-zero implementation on Debian 13 (trixie): host prep, kernel/module/tools build, libvirt lab, stress test |
| [build.md](build.md) | Build pipeline in detail: fetch → vendorize → kernel → module → pfctl |
| [vm-lab.md](vm-lab.md) | The libvirt validation lab: topology, recreate/run, what each of the 15 gates proves |
| [vm-lab.md](vm-lab.md) | The libvirt validation lab: topology, recreate/run, what each of the 16 gates proves |
| [feature-matrix.md](feature-matrix.md) | Per-feature status: VERIFIED-in-VM / VERIFIED-offline / COMPILES-untested / NOT-ENABLED |
| [pf-guide.md](pf-guide.md) | Daily use: what works today in `/etc/pf.conf`, and the deviations to know about |
| [screencast.md](screencast.md) | The narrated videos (English and Portuguese): phases, pipeline, how to re-record |
@@ -38,4 +39,4 @@ Feature-status vocabulary used across these docs:
- **NOT-ENABLED** — the code exists but the feature is off in this build.
- **OUT-OF-SCOPE** — non-goal by design (spec §3).
The lab suite is **15/15 green** (`./scripts/vm-run-tests.sh`).
The lab suite is **16/16 green** (`./scripts/vm-run-tests.sh`).
+39
View File
@@ -186,6 +186,45 @@ The videos are a demonstration, not a test run: the lab's own gate suite
in `out/test-logs/`. The screencasts show the same commands, but a video frame
is not a verdict.
## Capacity findings (2026-11-12, from `scripts/stress/run-stress.sh`)
### 13. The workqueue backlog grows with the offered rate: above ~1 Gbit/s the guest OOM-panics
This is the sharpest consequence of §1 and it is **reproducible**, not
theoretical. PF steals every skb onto one ordered workqueue and runs the
engine in process context, so the number of packets in flight is bounded only
by the memory the backlog takes.
Measured on the lab (Alpine guest, **1 GB** RAM, 4 TCP streams through a
stateful ruleset):
| Offered rate (per stream × 4) | Result |
|---|---|
| 50 Mbit/s (200 Mbit/s total) | stable |
| 100 Mbit/s (400 Mbit/s total) | stable |
| 250 Mbit/s (**1 Gbit/s** total) | **kernel panic: "System is deadlocked on memory"** |
The panic is an OOM with no killable process; the slab at that moment shows
`kmalloc-8k` ≈ 50 MB, `skbuff_head_cache` ≈ 18 MB and `files_cache` ≈ 40 MB,
i.e. the backlog is what ate the 1 GB. The guest reboots (120 s watchdog
delay) and every connection through the firewall is lost with it.
Practical consequences:
- **Rate-limit what you point at PF**, and size the box for the backlog:
memory needed grows with the offered rate, not with the number of
concurrent flows.
- The stress test (`tests/vm/16-stress.sh`) therefore runs its traffic phases
at a **bounded** rate (default 100 Mbit/s per stream = 400 Mbit/s total) —
the regime this port survives — and reports the ceiling separately instead
of OOM-killing the firewall under test.
- The functional gates (01-15) are unaffected: they move megabytes, not gigabits
per second.
What would fix it is a real fast path (evaluate `pf_test()` inline in
softirq, as OpenBSD does) or a bounded queue with a drop policy — both are
design changes beyond this port's scope.
## What is *not* a caveat
ARP and other non-IPv4/IPv6 EtherTypes never enter the netfilter IPv4/IPv6
+1 -1
View File
@@ -14,7 +14,7 @@ As of 2026-11-12; source of per-row evidence: `out/test-logs/`,
`docs/porting-notes.md`, milestone commits, and the screencasts
([screencast.md](screencast.md)) for the visible proof.
The suite is **15/15 green** (`./scripts/vm-run-tests.sh`).
The suite is **16/16 green** (`./scripts/vm-run-tests.sh`).
## Control plane & tools
+2 -2
View File
@@ -25,7 +25,7 @@ between kernel and userland (ioctl encoding per `module/include/pfk.h`).
| `module/include/` | `pfk.h` (master compat header + OpenBSD-style `_IO*` encoding), staged `net/pfvar.h`, `pfk_mbuf.h`, `pfk_net.h`, `pfk_if.h` |
| `tools/` | Linux build of OpenBSD 8.0 `pfctl` (single binary), `pdivert` relay, ABI parity harness (`abitest` + `make abi`), offline parse tests (`make test`). Deltas vs upstream: `tools/UPSTREAM.md` |
| `scripts/` | `fetch-sources.sh`, `vendorize.sh`, `stage-port.sh` (deterministic staging), `build-kernel.sh`, `build-module.sh`, `vm-*.sh` lab automation, `screencast/` (video pipeline) |
| `tests/vm/` | VM suite: 15 gates `01-state.sh … 15-queue.sh` + `lib.sh`; `tests/confs/` rulesets deployed to the lab |
| `tests/vm/` | VM suite: 16 gates `01-state.sh … 16-stress.sh` + `lib.sh`; `tests/confs/` rulesets deployed to the lab |
| `patches/kernel/` | Core patch set — currently **empty** (`p1-needs.txt` records why no exports were needed); see [patches.md](patches.md) |
## Data path (M2, as implemented today)
@@ -100,7 +100,7 @@ rebuilt.
| M3 | NAT (`nat-to`/`rdr-to`/`binat-to`), scrub, pflog | **done** — gates 05/06/07/09 (incl. 2 MB integrity through a redirect) and pflog gate 12 |
| M4 | route-to/dup-to/reply-to, pools, divert e2e | **done** — gates 08/10/13 and divert gate 11 |
| M5 | queues: ifq back ends (hfsc/fq_codel) | **done (mechanical)** — gate 15: hfsc attaches, `pfctl -s queue -v` reports class stats, a 2 MB queued flow is counted by the class. `bandwidth` = HFSC linkshare (a guarantee under contention, not a cap) |
| M6 | docs complete, patchset tidy, full suite green, tagged releases | **done** — docs EN+PT-BR, 15/15 suite green, tagged (`v1.0.0`) |
| M6 | docs complete, patchset tidy, full suite green, tagged releases | **done** — docs EN+PT-BR, 16/16 suite green, tagged (`v1.0.0`) |
## Next reads
+1 -1
View File
@@ -6,7 +6,7 @@ This is the "what can I actually run today" guide. It follows the
[feature matrix](feature-matrix.md): the filtering core (pass/block/quick/
keep state/tables/anchors), the NAT family, scrub, forced routing, pools,
`divert`, `pflog` and the OpenBSD 8.0 queues all have **VM evidence** behind
them — 15/15 lab gates are green. What is *not* gate-verified is listed
them — 16/16 lab gates are green. What is *not* gate-verified is listed
explicitly below. Install/build is covered by [install.md](install.md) and
[build.md](build.md).
+14 -5
View File
@@ -97,7 +97,7 @@ code: the packet either arrives, is dropped, or is translated, and the proof
is read back from the firewall's own counters, state table, capture files or
peer.
The full list is in §6 (15 gates, all green). Each gate is a shell script in
The full list is in §6 (16 gates, all green). Each gate is a shell script in
`tests/vm/` that talks to the VMs through deb with nested ssh, using the
helpers in `tests/vm/lib.sh` (`pf0`, `cli`, `srv`, `gw2`, `pfput`, `pfload`,
`ok`, `bad`). `scripts/vm-run-tests.sh` runs them in order and captures
@@ -108,14 +108,22 @@ Not gated (honest gaps): the IPv6 path, OS fingerprinting, `set` options
`pf.ko` but no gate drives it. FQ-CoDel compiles and shares its stats header
with `pfctl`, but only the **hfsc** discipline is exercised.
There is **no numeric performance gate**: `bench-basic.sh` does not exist in
the tree and no throughput target is claimed.
There is **no numeric performance gate**: the design decision (spec §7) is
that *functional gates are the bar* — a perf contract would force a
non-faithful rewrite-first shim. What exists instead is a **stress test**
(`tests/vm/16-stress.sh`, driven by `scripts/stress/run-stress.sh`) that
asserts *invariants under load* — no state/table leak, no rule-load failure,
byte-identical payload under a saturating stream, no `BUG/WARNING/Oops`, the
firewall still routing — and **reports** the throughput comparison
(baseline without the module vs. with PF enabled) as information, without a
numeric pass/fail target. Report and raw log land in
`out/stress/<timestamp>/`.
## 6. Current suite status (2026-11-12)
**15/15 gates green.** A full clean run (`./scripts/vm-run-tests.sh`, ~9 min)
**16/16 gates green.** A full clean run (`./scripts/vm-run-tests.sh`, ~13 min)
recreates and reboots `pftest-pf0` before every gate, redeploys the module
and the tools, and ends with `PASS=15 FAIL=0`. The captured logs live in
and the tools, and ends with `PASS=16 FAIL=0`. The captured logs live in
`out/test-logs/<gate>.sh.log`.
| Gate | What it proves now |
@@ -135,6 +143,7 @@ and the tools, and ends with `PASS=15 FAIL=0`. The captured logs live in
| `13-replyto.sh` | `reply-to` forces the reply onto the gateway link |
| `14-soak.sh` | ruleset reload under traffic, `rmmod` cycle, leak check |
| `15-queue.sh` | hfsc queues: attach, class statistics, a 2 MB queued flow counted by the class |
| `16-stress.sh` | stress: throughput baseline vs. PF, state/rule/table churn with leak checks, 2 MB through NAT under a saturating stream, dmesg health (run by `scripts/stress/run-stress.sh`, which writes a report) |
The screencasts ([screencast.md](screencast.md)) show most of these gates
being run, with the lab's network diagram drawn over the terminal.