docs: record the stress findings and the 16-gate suite
caveats §13 documents the capacity ceiling the stress test found: the workqueue backlog grows with the offered rate and OOM-panics a 1 GB guest above ~1 Gbit/s. Gate count, suite status and the doc indexes move to 16/16 green.
This commit is contained in:
1 parent
928322edf7
commit
0a8ba69340
15 files changed
+191
-30
No files matched your search
+3
-2
@@ -13,8 +13,9 @@ packet filter** to **Linux 7.3.4** — the real PF engine (`pf.c`, `pf_norm.c`,
|
||||
|---|---|
|
||||
| [guide.md](guide.md) | One-page architecture: components, data path, control plane, milestones |
|
||||
| [install.md](install.md) | Apply/build/install recipe (kernel, module, tools) |
|
||||
| [debian13.md](debian13.md) | Full from-zero implementation on Debian 13 (trixie): host prep, kernel/module/tools build, libvirt lab, stress test |
|
||||
| [build.md](build.md) | Build pipeline in detail: fetch → vendorize → kernel → module → pfctl |
|
||||
| [vm-lab.md](vm-lab.md) | The libvirt validation lab: topology, recreate/run, what each of the 15 gates proves |
|
||||
| [vm-lab.md](vm-lab.md) | The libvirt validation lab: topology, recreate/run, what each of the 16 gates proves |
|
||||
| [feature-matrix.md](feature-matrix.md) | Per-feature status: VERIFIED-in-VM / VERIFIED-offline / COMPILES-untested / NOT-ENABLED |
|
||||
| [pf-guide.md](pf-guide.md) | Daily use: what works today in `/etc/pf.conf`, and the deviations to know about |
|
||||
| [screencast.md](screencast.md) | The narrated videos (English and Portuguese): phases, pipeline, how to re-record |
|
||||
@@ -38,4 +39,4 @@ Feature-status vocabulary used across these docs:
|
||||
- **NOT-ENABLED** — the code exists but the feature is off in this build.
|
||||
- **OUT-OF-SCOPE** — non-goal by design (spec §3).
|
||||
|
||||
The lab suite is **15/15 green** (`./scripts/vm-run-tests.sh`).
|
||||
The lab suite is **16/16 green** (`./scripts/vm-run-tests.sh`).
|
||||
@@ -186,6 +186,45 @@ The videos are a demonstration, not a test run: the lab's own gate suite
|
||||
in `out/test-logs/`. The screencasts show the same commands, but a video frame
|
||||
is not a verdict.
|
||||
|
||||
## Capacity findings (2026-11-12, from `scripts/stress/run-stress.sh`)
|
||||
|
||||
### 13. The workqueue backlog grows with the offered rate: above ~1 Gbit/s the guest OOM-panics
|
||||
|
||||
This is the sharpest consequence of §1 and it is **reproducible**, not
|
||||
theoretical. PF steals every skb onto one ordered workqueue and runs the
|
||||
engine in process context, so the number of packets in flight is bounded only
|
||||
by the memory the backlog takes.
|
||||
|
||||
Measured on the lab (Alpine guest, **1 GB** RAM, 4 TCP streams through a
|
||||
stateful ruleset):
|
||||
|
||||
| Offered rate (per stream × 4) | Result |
|
||||
|---|---|
|
||||
| 50 Mbit/s (200 Mbit/s total) | stable |
|
||||
| 100 Mbit/s (400 Mbit/s total) | stable |
|
||||
| 250 Mbit/s (**1 Gbit/s** total) | **kernel panic: "System is deadlocked on memory"** |
|
||||
|
||||
The panic is an OOM with no killable process; the slab at that moment shows
|
||||
`kmalloc-8k` ≈ 50 MB, `skbuff_head_cache` ≈ 18 MB and `files_cache` ≈ 40 MB,
|
||||
i.e. the backlog is what ate the 1 GB. The guest reboots (120 s watchdog
|
||||
delay) and every connection through the firewall is lost with it.
|
||||
|
||||
Practical consequences:
|
||||
|
||||
- **Rate-limit what you point at PF**, and size the box for the backlog:
|
||||
memory needed grows with the offered rate, not with the number of
|
||||
concurrent flows.
|
||||
- The stress test (`tests/vm/16-stress.sh`) therefore runs its traffic phases
|
||||
at a **bounded** rate (default 100 Mbit/s per stream = 400 Mbit/s total) —
|
||||
the regime this port survives — and reports the ceiling separately instead
|
||||
of OOM-killing the firewall under test.
|
||||
- The functional gates (01-15) are unaffected: they move megabytes, not gigabits
|
||||
per second.
|
||||
|
||||
What would fix it is a real fast path (evaluate `pf_test()` inline in
|
||||
softirq, as OpenBSD does) or a bounded queue with a drop policy — both are
|
||||
design changes beyond this port's scope.
|
||||
|
||||
## What is *not* a caveat
|
||||
|
||||
ARP and other non-IPv4/IPv6 EtherTypes never enter the netfilter IPv4/IPv6
|
||||
|
||||
@@ -14,7 +14,7 @@ As of 2026-11-12; source of per-row evidence: `out/test-logs/`,
|
||||
`docs/porting-notes.md`, milestone commits, and the screencasts
|
||||
([screencast.md](screencast.md)) for the visible proof.
|
||||
|
||||
The suite is **15/15 green** (`./scripts/vm-run-tests.sh`).
|
||||
The suite is **16/16 green** (`./scripts/vm-run-tests.sh`).
|
||||
|
||||
## Control plane & tools
|
||||
|
||||
|
||||
+2
-2
@@ -25,7 +25,7 @@ between kernel and userland (ioctl encoding per `module/include/pfk.h`).
|
||||
| `module/include/` | `pfk.h` (master compat header + OpenBSD-style `_IO*` encoding), staged `net/pfvar.h`, `pfk_mbuf.h`, `pfk_net.h`, `pfk_if.h` |
|
||||
| `tools/` | Linux build of OpenBSD 8.0 `pfctl` (single binary), `pdivert` relay, ABI parity harness (`abitest` + `make abi`), offline parse tests (`make test`). Deltas vs upstream: `tools/UPSTREAM.md` |
|
||||
| `scripts/` | `fetch-sources.sh`, `vendorize.sh`, `stage-port.sh` (deterministic staging), `build-kernel.sh`, `build-module.sh`, `vm-*.sh` lab automation, `screencast/` (video pipeline) |
|
||||
| `tests/vm/` | VM suite: 15 gates `01-state.sh … 15-queue.sh` + `lib.sh`; `tests/confs/` rulesets deployed to the lab |
|
||||
| `tests/vm/` | VM suite: 16 gates `01-state.sh … 16-stress.sh` + `lib.sh`; `tests/confs/` rulesets deployed to the lab |
|
||||
| `patches/kernel/` | Core patch set — currently **empty** (`p1-needs.txt` records why no exports were needed); see [patches.md](patches.md) |
|
||||
|
||||
## Data path (M2, as implemented today)
|
||||
@@ -100,7 +100,7 @@ rebuilt.
|
||||
| M3 | NAT (`nat-to`/`rdr-to`/`binat-to`), scrub, pflog | **done** — gates 05/06/07/09 (incl. 2 MB integrity through a redirect) and pflog gate 12 |
|
||||
| M4 | route-to/dup-to/reply-to, pools, divert e2e | **done** — gates 08/10/13 and divert gate 11 |
|
||||
| M5 | queues: ifq back ends (hfsc/fq_codel) | **done (mechanical)** — gate 15: hfsc attaches, `pfctl -s queue -v` reports class stats, a 2 MB queued flow is counted by the class. `bandwidth` = HFSC linkshare (a guarantee under contention, not a cap) |
|
||||
| M6 | docs complete, patchset tidy, full suite green, tagged releases | **done** — docs EN+PT-BR, 15/15 suite green, tagged (`v1.0.0`) |
|
||||
| M6 | docs complete, patchset tidy, full suite green, tagged releases | **done** — docs EN+PT-BR, 16/16 suite green, tagged (`v1.0.0`) |
|
||||
|
||||
## Next reads
|
||||
|
||||
|
||||
+1
-1
@@ -6,7 +6,7 @@ This is the "what can I actually run today" guide. It follows the
|
||||
[feature matrix](feature-matrix.md): the filtering core (pass/block/quick/
|
||||
keep state/tables/anchors), the NAT family, scrub, forced routing, pools,
|
||||
`divert`, `pflog` and the OpenBSD 8.0 queues all have **VM evidence** behind
|
||||
them — 15/15 lab gates are green. What is *not* gate-verified is listed
|
||||
them — 16/16 lab gates are green. What is *not* gate-verified is listed
|
||||
explicitly below. Install/build is covered by [install.md](install.md) and
|
||||
[build.md](build.md).
|
||||
|
||||
|
||||
+14
-5
@@ -97,7 +97,7 @@ code: the packet either arrives, is dropped, or is translated, and the proof
|
||||
is read back from the firewall's own counters, state table, capture files or
|
||||
peer.
|
||||
|
||||
The full list is in §6 (15 gates, all green). Each gate is a shell script in
|
||||
The full list is in §6 (16 gates, all green). Each gate is a shell script in
|
||||
`tests/vm/` that talks to the VMs through deb with nested ssh, using the
|
||||
helpers in `tests/vm/lib.sh` (`pf0`, `cli`, `srv`, `gw2`, `pfput`, `pfload`,
|
||||
`ok`, `bad`). `scripts/vm-run-tests.sh` runs them in order and captures
|
||||
@@ -108,14 +108,22 @@ Not gated (honest gaps): the IPv6 path, OS fingerprinting, `set` options
|
||||
`pf.ko` but no gate drives it. FQ-CoDel compiles and shares its stats header
|
||||
with `pfctl`, but only the **hfsc** discipline is exercised.
|
||||
|
||||
There is **no numeric performance gate**: `bench-basic.sh` does not exist in
|
||||
the tree and no throughput target is claimed.
|
||||
There is **no numeric performance gate**: the design decision (spec §7) is
|
||||
that *functional gates are the bar* — a perf contract would force a
|
||||
non-faithful rewrite-first shim. What exists instead is a **stress test**
|
||||
(`tests/vm/16-stress.sh`, driven by `scripts/stress/run-stress.sh`) that
|
||||
asserts *invariants under load* — no state/table leak, no rule-load failure,
|
||||
byte-identical payload under a saturating stream, no `BUG/WARNING/Oops`, the
|
||||
firewall still routing — and **reports** the throughput comparison
|
||||
(baseline without the module vs. with PF enabled) as information, without a
|
||||
numeric pass/fail target. Report and raw log land in
|
||||
`out/stress/<timestamp>/`.
|
||||
|
||||
## 6. Current suite status (2026-11-12)
|
||||
|
||||
**15/15 gates green.** A full clean run (`./scripts/vm-run-tests.sh`, ~9 min)
|
||||
**16/16 gates green.** A full clean run (`./scripts/vm-run-tests.sh`, ~13 min)
|
||||
recreates and reboots `pftest-pf0` before every gate, redeploys the module
|
||||
and the tools, and ends with `PASS=15 FAIL=0`. The captured logs live in
|
||||
and the tools, and ends with `PASS=16 FAIL=0`. The captured logs live in
|
||||
`out/test-logs/<gate>.sh.log`.
|
||||
|
||||
| Gate | What it proves now |
|
||||
@@ -135,6 +143,7 @@ and the tools, and ends with `PASS=15 FAIL=0`. The captured logs live in
|
||||
| `13-replyto.sh` | `reply-to` forces the reply onto the gateway link |
|
||||
| `14-soak.sh` | ruleset reload under traffic, `rmmod` cycle, leak check |
|
||||
| `15-queue.sh` | hfsc queues: attach, class statistics, a 2 MB queued flow counted by the class |
|
||||
| `16-stress.sh` | stress: throughput baseline vs. PF, state/rule/table churn with leak checks, 2 MB through NAT under a saturating stream, dmesg health (run by `scripts/stress/run-stress.sh`, which writes a report) |
|
||||
|
||||
The screencasts ([screencast.md](screencast.md)) show most of these gates
|
||||
being run, with the lab's network diagram drawn over the terminal.
|
||||
|
||||
Reference in new issue
Block a user