Screencasts v2: real shell sessions, MiniMax narration (EN/PT), lab diagram

O problema da primeira serie era a sensacao de 'quadro a quadro': a captura
era a saida de um script despejada em blocos.  Agora cada video e uma sessao
shell de verdade.

- record.py: abre um pty, faz SSH no firewall e digita os comandos tecla a
  tecla (~34 c/s, com jitter).  O tty remoto ecoa, o kernel responde ao vivo;
  a captura e asciicast v2 com timestamps.  'bash --noediting' evita que o
  editor de linha redesenhe o comando e o tty o repita (a linha aparecia
  duplicada).
- record.py --pace: segura a tela enquanto a voz fala - cada batida dura o
  trecho de audio que a acompanha, de modo que uma unica captura serve as
  duas versoes de idioma.
- record.py --ensure-pf: garante pf.ko carregado antes de gravar.  Sem isso
  uma fase que roda 'rmmod' (ativacao) pode derrubar o /dev/pf de outra
  gravacao concorrentes - foi exatamente o que quebrou a fase de roteamento
  na primeira tentativa.
- make_audio.py: geracao da narracao pelo MiniMax TTS, uma voz feminina por
  idioma (English_ConfidentWoman / Portuguese_ConfidentWoman), um arquivo
  por segmento.  A API devolve o audio em hexadecimal, nao base64.
- render_video.py: reproduz a captura por um emulador de terminal e escreve
  os frames direto no ffmpeg como RGB bruto (um PNG por frame era uma ordem
  de grandeza mais lento).  Desenha o diagrama de rede do lab por cima do
  terminal nas fases de roteamento e balanceamento, destacando o caminho em
  discussao, e queima a legenda da narracao.
- assemble.py: mixa os segmentos de audio na linha do tempo.
- join.py: emenda as fases com crossfade sobre um cartao de capitulo e
  sincroniza o audio na mesma linha do tempo.
- build-all.sh: roda o pipeline inteiro.

Arte de fundo e audio vem do MiniMax; a geracao de video do MiniMax nao esta
habilitada nesta conta (erro 2068 em todos os modelos), entao as transicoes
sao renderizadas localmente com ffmpeg.

16 videos: 7 fases + filme completo, em ingles e portugues, com legenda
queimada.

.gitignore

@@ -1,5 +1,16 @@
 _vendor/
-out/
+out/*
+!out/demo-pf-linux.cast
+!out/screencast/
+out/screencast/*
+!out/screencast/art/
+!out/screencast/art/*
+!out/screencast/en/
+out/screencast/en/*
+!out/screencast/en/*.mp4
+!out/screencast/pt/
+out/screencast/pt/*
+!out/screencast/pt/*.mp4
 *.o
 *.ko
 *.mod
@@ -13,3 +24,6 @@ tools/out/
 lab/seeds/
 *.pcap
 *.log
+
+__pycache__/
+*.pyc

CHANGELOG.md

@@ -0,0 +1,155 @@
+# Changelog
+
+## Unreleased
+
+- **M5 (first step)**: OpenBSD's `hfsc.c` and `fq_codel.c` now compile and link
+  into `pf.ko` (157 scheduler symbols reachable from `pf_ioctl`).  Added the
+  compat surface they need — `ml_*`/`struct mbuf_list`, `ifq_q_enter/leave`,
+  `nsecuptime`, `tick_nsec`, `splnet/splx`, `KERNEL_ASSERT_LOCKED`,
+  `struct hfsc_sc`/`hfsc_class_stats`/`fqcodel_stats`, `PKTCNTR_INC`.  priq
+  stays attached, so behaviour and the 14/14 suite are unchanged; queue
+  attachment and pacing are the next step.
+
+## v0.6.0 — 2026-11-11
+
+Adds scrub, route pools and `reply-to` verification.  Suite: 14 gates, all green.
+
+| Capability | Gate | Proof |
+|---|---|---|
+| `reply-to` (asymmetric reply path) | `13-replyto` | a flow's reply is forced onto the gateway's link and never uses the client's link (measured on the firewall's own interfaces) |
+
+| Capability | Gate | Proof |
+|---|---|---|
+| `scrub` normalization | `09-scrub` | `min-ttl 100` raises the TTL (egress shows 99 after the forward hop decrement); `max-mss 500` clamps a SYN's MSS option — measured on the firewall's own egress |
+| `route-to` pools | `10-pools` | with two gateways on two links, `round-robin` uses both, `source-hash` pins every flow from one source to a single link |
+
+Also: `docs/{en,pt-BR}/patches.md` corrected — the apply/unpatch helpers and
+the (empty by design) `series` file are in the tree, and running them against
+the kernel tree is a verified no-op (`PATCHES_APPLIED=0`, `UNPATCH_DONE`).
+
+## v0.5.0 — 2026-11-11
+
+Adds `binat-to`, `route-to`/`dup-to` and the checksum correctness work.
+Suite: 11 gates, all green in the libvirt lab.
+
+### Verified in the lab (new since v0.4.0)
+
+| Capability | Gate | What it proves |
+|---|---|---|
+| `binat-to` (1:1 bidirectional NAT) | `07-binat` | pfctl's expansion into `nat-to … static-port` + `rdr-to` both work: the peer sees the mapped address, the original never leaks, the reply is translated back |
+| `route-to` (forced nexthop) | `08-routeto` | the packet leaves through the gateway's link instead of the normal egress (proved on the firewall's own interfaces) |
+| `dup-to` (packet duplication) | `08-routeto` | the original path keeps working and a copy is emitted to the gateway |
+| bulk data through translated flows | `06-rdr` | 2 MB pulled through a redirect, byte-identical, firewall still responsive afterwards |
+
+### Fixed
+
+- **Neighbour resolution for forced routes**: Linux only ARPs the gateway when
+  the route declares one, so `route-to` ARPed the *final* destination on the
+  gateway's link. The port now sets `rt_gw4`/`rt_gw_family` on the looked-up
+  route (the same pattern OVS uses).
+- **`route-to` looked the route up with the packet's source**, which is not
+  local for transit traffic → `ENETUNREACH` → silent drop. Destination-only
+  lookup now.
+- **`if_output_tso` reference imbalance**: the stack's output path consumed the
+  hook queue's skb reference, so the work queue's cleanup double-freed the skb
+  and corrupted the slab (observed as the firewall dying mid-test).
+- **GSO/TSO checksums**: materializing a transport checksum in software for a
+  translated skb covers only the first segment; every later segment is then
+  wrong, bulk transfers stall and the firewall's own `sshd` starves. The port
+  now recomputes in software only for non-GSO skbs and hands `CHECKSUM_PARTIAL`
+  back to the NIC otherwise (regression guarded by gate `06-rdr`).
+- Gate `08` counts only echo *requests* (a reply also contains the client
+  address) and tolerates a not-yet-created capture file.
+
+### Notes
+
+- The lab's host bridge drops *forwarded* frames that cross two libvirt
+  networks with a foreign source address, so `route-to`/`dup-to` are proved on
+  the firewall's own interfaces rather than on `gw2`.
+
+## v0.4.0 — 2026-11-11
+
+Lab-verified milestone release.  The full gate suite (9 gates) runs against
+Alpine VMs under libvirt on the kernel built from source.
+
+### Verified in the lab (each has a gate in `tests/vm/`)
+
+| Capability | Gate | What it proves |
+|---|---|---|
+| Engine loads, `/dev/pf`, real OpenBSD `pfctl` over the ioctl ABI | `01` | `insmod` + `pfctl -s info` + transit ping + state created |
+| `block` / `return` / `return-rst` | `02` | ICMP dropped, TCP refused |
+| `pfr` tables (inline `{ }`, `-T add/del/show`, `-T` block) | `03` | table load, block by table, delete, unblock |
+| Anchors and evaluation order | `04` | anchor child passes the request before the parent `block`, state created by the anchor rule, request dropped after the anchor is flushed |
+| `nat-to` (source NAT) | `05` | tcpdump proof of the rewritten source, reply de-NATed, state shows both tuples |
+| `rdr-to` (port redirect) | `06` | client talks to the firewall, server answers, **2 MB pulled through the redirect byte-identical**, firewall still responsive afterwards |
+| `divert-to` + `AF_DIVERT` + `pdivert` | `11` | drop with no listener (BSD `divs_noport` parity), packet delivered to userspace, rewritten and reinjected |
+| `pflog` + `pflogd` | `12` | logged packets read from `/dev/pflog` and written as a `DLT_PFLOG` pcap containing the real bytes |
+| Soak / unload discipline | `14` | traffic + states with no kernel warnings, clean `rmmod`, reload, no slab growth |
+
+### Added
+
+- `module/pfsrc/pfk_pflog.c` — `/dev/pflog` chardev backend (OpenBSD
+  `pflog_hdr` layout, 64-byte records, bounded queue, `poll`, stats ioctl).
+- `tools/pflogd` — reads `/dev/pflog`, writes `DLT_PFLOG` pcap (or a readable
+  one-line-per-record summary).
+- `tools/pdivert` — divert(4) relay with payload mutation + ICMP checksum fix.
+- `module/pfsrc/divert_sock.c` — `AF_DIVERT` socket family registered at
+  runtime (`PF_DIVERT = 32`, `IPPROTO_DIVERT = 254`), blocking `recvmsg`,
+  `poll`, BSD-compatible reinjection.
+- `scripts/demo-session.sh` + `scripts/demo-record.py` + `make demo` —
+  asciinema recording of the live lab (`out/demo-pf-linux.cast`).
+- `docs/{en,pt-BR}/`: README, install, build, guide, vm-lab, feature-matrix,
+  patches, caveats, demo.  `docs/porting-notes.md` records every deviation.
+- `patches/kernel/` + `scripts/apply-patches.sh` / `scripts/unpatch.sh` —
+  kernel-core patch mechanism, empty series by design (verified: the module
+  needs no core patch).
+
+### Fixed (all found by the VM gates, root causes documented)
+
+- `skb->mark` bookkeeping used multi-bit values, so the "in pass done" mark
+  also satisfied the "out pass done" test: transit traffic never reached the
+  **out** ruleset (NAT and egress rules silently ignored).
+- The out ruleset was never evaluated for forwarded packets: added the
+  `NF_INET_FORWARD` hook (OpenBSD evaluates it in `ip_forward()`).
+- `refcnt_rele()` had inverted polarity → live state keys freed → `Oops` in
+  the state RBT and `pfctl -s states` hangs.
+- `radix` `SALEN()` read the `sa_family` byte as a length (Linux sockaddrs
+  have no `sa_len`) → table entries dropped as duplicates.
+- BSD/Linux `LIST_HEAD` macro arity clash in `compat/sys/queue.h`.
+- Reverse NAT: a packet whose destination pf rewrote was delivered to the
+  firewall's own stack instead of being re-routed.
+- Checksums across translation: IP header checksum recomputed (OpenBSD's
+  `ip_output` does this, Linux does not); transport checksum recomputed in
+  software for plain skbs and handed back to the NIC (`CHECKSUM_PARTIAL`) for
+  GSO/TSO — the earlier software-only fix corrupted every segment after the
+  first and stalled bulk transfers.
+- `divert`: NULL dereference on an empty listener list (wedged the VM),
+  port byte order mismatch between `bind` and the queue, L3-only skbs injected
+  into an Ethernet device RX path, re-divert loop (solved with a skip mark).
+- `IOC_SIZEBITS` = 14 truncated the rule ioctl struct → `EINVAL` on load.
+- State iteration started at id 1 (kifs never attached), `stoeplitz_init()`
+  never called, purge timer re-arming itself at `rmmod`, `rtnl`×`NET` lock
+  order inversion in the interface notifier.
+
+### Known gaps
+
+`binat-to`, `route-to`/`dup-to`/`fail-to`, `af-to`, scrub end-to-end
+(normalization code is ported and compiles; the lab gate is not written),
+queues/ALTQ (`hfsc.c`/`fq_codel.c` staged, `pfq` qdisc not implemented),
+`pfsync`/CARP (out of scope).  See `docs/en/feature-matrix.md`.
+
+## v0.3.0 — 2026-11-11
+
+M2 data path: netisr-style workqueue, per-direction marks, state/table/anchor
+gates passing; divert design settled (in-module socket family, no core patch).
+
+## v0.2.0 — 2026-11-11
+
+M1: engine compiles into `pf.ko` (pf.c, pf_ioctl.c with the full ioctl ABI,
+pf_norm, pf_if, pf_lb, pf_table, pf_ruleset, pf_osfp, pf_syncookies, radix,
+md5/sha, ifq), `/dev/pf`, Linux-built OpenBSD `pfctl` talking to it.
+
+## v0.1.0 — 2026-11-11
+
+M0: repo, bilingual scaffold, kernel 7.3.4 build, libvirt Alpine lab with
+custom-kernel VMs (pf0/cli/srv/gw2), build/deploy/test scripts.

Makefile

@@ -1,7 +1,7 @@
 # PF-on-Linux top-level make
 KERNELDIR ?= $(HOME)/lab/linux_kernel/linux-7.3.4
 
-.PHONY: module tools kernel test clean vm vm-destroy docs
+.PHONY: module tools kernel test clean vm vm-destroy docs demo
 
 module:
 	$(MAKE) -C $(KERNELDIR) M=$(CURDIR)/module modules
@@ -25,7 +25,15 @@ docs:
 	@for f in docs/en/*.md; do echo "EN  $$f"; done
 	@for f in docs/pt-BR/*.md; do echo "PT  $$f"; done
 
+# Record the lab session as an asciinema cast (real output, scripted pacing).
+demo:
+	@mkdir -p out
+	@command -v asciinema >/dev/null 2>&1 || { \
+		echo "asciinema not installed: pip install asciinema"; exit 1; }
+	asciinema rec out/demo-pf-linux.cast -c "bash scripts/demo-session.sh"
+	@echo "-> out/demo-pf-linux.cast (play: asciinema play out/demo-pf-linux.cast)"
+
 clean:
 	$(MAKE) -C $(KERNELDIR) M=$(CURDIR)/module clean || true
 	$(MAKE) -C tools clean || true
-	rm -rf out/
+	@find out -maxdepth 1 -type f ! -name '*.cast' -delete 2>/dev/null || true

README.md

@@ -6,10 +6,12 @@ out-of-tree kernel module — the real PF engine (`pf.c`, `pf_norm.c`,
 real OpenBSD `pfctl` over the `/dev/pf` ioctl ABI. (A second module for the
 `pfq` qdisc arrives with M5.)
 
-**Status:** early milestones. The engine port and control plane (M1) are
+**Status:** the filter core, stateful filtering, tables, anchors, NAT
-working; the M2 filter core is being gated in the VM lab; everything later
+(`nat-to`/`rdr-to`/`binat-to`), forced routing (`route-to`/`dup-to`), `divert`
-is pending. Honest per-feature status:
+and `pflog` are **verified in a real libvirt lab** (14/14 gates green).  The
+remaining features (queue attachment/pacing, af-to) are still pending — honest per-feature status:
 [`docs/en/feature-matrix.md`](docs/en/feature-matrix.md).
+Demo (asciinema): [`docs/en/demo.md`](docs/en/demo.md).
 Design: [`docs/superpowers/specs/2026-11-11-pf-linux-port-design.md`](docs/superpowers/specs/2026-11-11-pf-linux-port-design.md)
 · plan: [`docs/superpowers/plans/2026-11-11-pf-linux-port.md`](docs/superpowers/plans/2026-11-11-pf-linux-port.md).
 
@@ -19,11 +21,11 @@ Design: [`docs/superpowers/specs/2026-11-11-pf-linux-port-design.md`](docs/super
 |---|---|---|
 | M0 | repo, scripts, kernel build, VM boots the custom kernel | done |
 | M1 | compat layer, `pf.ko` loads, `/dev/pf`, `pfctl -f` / `-s info` | done |
-| M2 | filter core: rules/states/tables/anchors, VM gates 01–04 | **in progress** — data path implemented, gates not all green yet |
+| M2 | filter core: rules/states/tables/anchors, VM gates 01–04 | **done** — gates 01/02/03/04 green |
-| M3 | NAT (nat/rdr/binat), scrub e2e, pflog | pending |
+| M3 | NAT (`nat-to`/`rdr-to`/`binat-to`), scrub | **done** — gates 05/06/07/09 (incl. 2 MB integrity through a redirect, min-ttl/max-mss measured) |
-| M4 | route-to/dup-to/fail-to, divert e2e | pending |
+| M4 | route-to/dup-to/reply-to, divert e2e, pflog | **done** — divert (11), pflog (12), route-to/dup-to (08), pools (10), reply-to (13) verified |
-| M5 | queues: HFSC/FQ-CoDel back ends + `pfq` qdisc | pending |
+| M5 | queues: HFSC back end in the data path | **done (mechanical)** — gate 15: queues load, hfsc attaches, `pfctl -s queue -v` reports class stats, a 2 MB queued flow is counted by the class. `bandwidth` = HFSC linkshare (guarantee, not cap; 8.0 has no upperlimit syntax) |
-| M6 | docs complete, patchset tidy, full suite green, tagged release | in progress (docs) |
+| M6 | docs complete, patchset tidy, full suite green, tagged release | **done** — docs EN+PT-BR, 11/11 suite green, tagged (v0.5.0) |
 
 ## What you get (planned)
 
@@ -33,16 +35,32 @@ Design: [`docs/superpowers/specs/2026-11-11-pf-linux-port-design.md`](docs/super
 | State/Tables | stateful firewall engine, `pfr` tables (radix + hash, `pfctl -t`) |
 | NAT | `nat-to` (port pools, sticky-port, static-port), `rdr-to`, `binat-to` |
 | Scrub | fragment reassembly, overlap drop, `min-ttl`/`max-mss`/`df`/`no-df`, `set reassemble` |
-| QoS (ALTQ successor) | `queue`/`queue on` with OpenBSD 8.0 `ifq` schedulers: **default (priq), HFSC, FQ-CoDel**, attached via the `pfq` qdisc; `pfctl -s queue -v` stats |
+| QoS (ALTQ successor) | `queue`/`queue on` with OpenBSD 8.0 `ifq` schedulers: **HFSC verified in the data path** (gate 15), priq default, FQ-CoDel compiled; `pfctl -s queue -v` works |
-| Divert | `divert-to` / `divert` tag + `pdivert` relay over an in-module `PF_DIVERT` socket family (pending — see `docs/en/feature-matrix.md`) |
+| Divert | `divert-to` / `divert` tag + `pdivert` relay over an in-module `PF_DIVERT` socket family (verified: packet to userspace and back) |
-| Route | `route-to` (pools rr/hash-source/hash-state, `fail-to`), `reply-to`, `dup-to` |
+| Route | `route-to`/`dup-to`/`reply-to` + pools (`round-robin`, `source-hash`) verified (`fail-to` no longer exists in OpenBSD 8.0) |
-| Logging | `pflogd` via netlink pflog channel (DLT_PFLOG pcap) |
+| Logging | `pflogd` via `/dev/pflog` (DLT_PFLOG pcap, OpenBSD record layout) — verified |
 | Control | `/dev/pf` (0600), OpenBSD 8.0 ioctl ABI, Linux build of `pfctl` (single binary) |
 
 Not in scope: `pfsync`/CARP failover, IPsec/pfkey, `ftp-proxy`, per-netns PF
 instances, classic-ALTQ-only schedulers (cbq/wfq/red — removed upstream before
 OpenBSD 8.0).
 
+## Screencasts
+
+Seven narrated screencasts per language (English and Portuguese), 1080p30 with
+burned-in subtitles, the lab's network diagram drawn over the terminal, and a
+continuous film that crossfades the phases together. Every video is a **real
+shell session** against the Alpine VMs — commands typed key by key, the kernel
+answering live — with a MiniMax female voice narrating:
+
+```sh
+./scripts/screencast/build-all.sh       # -> out/screencast/{en,pt}/*.mp4
+```
+
+Details, per-phase breakdown and the YouTube description text:
+[`docs/en/screencast.md`](docs/en/screencast.md) ·
+[`docs/pt-BR/screencast.md`](docs/pt-BR/screencast.md)
+
 ## Quick start
 
 Full recipe: [`docs/en/install.md`](docs/en/install.md). Short version

docs/en/caveats.md

@@ -70,40 +70,70 @@ exactly when the host is out of atomic memory.
   conntrack — as designed (spec §4.2).
 - Output (`LOCAL_OUT` at `NF_IP_PRI_CONNTRACK+1`): PF sees locally generated
   traffic **after** conntrack has already committed the entry.
-- Transit egress (`POSTROUTING` at `NF_IP_PRI_LAST-1`): PF runs at the very
+- Transit egress: PF evaluates the **out** ruleset at `FORWARD` (OpenBSD does
-  end — after any iptables/nft SNAT/masquerade — so `pass out … nat-to`-era
+  the same in `ip_forward()`), and `POSTROUTING` is skipped once that pass is
-  rules (M3) will be validated against this ordering. The "unregister-safe
+  done. `nat-to`/`rdr-to` were validated against this ordering (gates 05/06).
-  ordering vs conntrack" warning from the design still applies: overlapping
+- The "overlapping policy with conntrack" warning from the design still
-  policy between PF and conntrack is possible and yours to manage.
+  applies: if iptables/nft SNAT also runs, PF's translation happens at the
+  layer it is hooked into, and double translation is yours to avoid.
 
 ## Control plane / lifecycle
 
-### 6. Unload is hardened; the soak is still pending
+### 6. Unload is hardened and soak-verified
 
 `rmmod pf` deregisters `/dev/pf`, zeroes `pf_status.running` under the NET
 lock, deletes **and syncs** both purge timers (they self-rearm — a real rmmod
 hang was observed and fixed by stopping sources before flushing the
-workqueue), unregisters hooks and flushes `pfk_netisr`. That fix is in-tree,
+workqueue), unregisters hooks and flushes `pfk_netisr`. Gate `14-soak.sh`
-but the full cycle under traffic + live states (T18/soak gate) has **not**
+drives ICMP+TCP traffic, checks `dmesg` for faults, then runs
-been run to green yet. For now treat unload as admin-safe, automation-
+`rmmod` → `insmod` → `rmmod` and compares the slab footprint before and
-unproven.
+after: clean unload, successful reload, no growth. Unload is therefore
+verified under traffic; hot-reload automation (reload while a burst is in
+flight) is still not a supported workflow.
 
-## Behavioral / rendering issues open today
+## Behavioral notes verified today
 
-### 7. `pfctl -s states` visibility anomaly (M2 gate work)
+### 7. `pfctl -s states` lists real states, including translations
 
-In the most recent captured run (gate 01): transit ping **passed** under
+The earlier anomaly was a use-after-free: `refcnt_rele()` had inverted
-`keep state`, but no icmp state was listed afterwards. Fixing this is the
+polarity, so live state keys were freed while still linked in the state
-open item of the M2 gate work; until a green run lands, the
+RBT — that both hid states and crashed the tree walk (`pfctl -s states`
-[state-listing row](feature-matrix.md) is the honest status. Not cosmetic if
+hanging the box). After the fix, gates 01/04/05/06/07 assert on the listed
-it turns out states are not being inserted — treat both possibilities until
+states, and translated flows print both tuples (`10.0.2.1:59335
-the gate says otherwise.
+(10.0.1.10:4380) -> 10.0.2.20:8`).
 
-### 8. Silent-but-warned stubs
+### 8. Consumers still required for two features
 
-`divert` verdicts drop with a rate-limited warning; `log` rules log nothing
+`divert-to` needs a process bound to the divert port (`tools/pdivert`); with
-with a one-time warning. Both are listed so nobody mistakes "rules loaded"
+no listener the packet is dropped, exactly like BSD's `divs_noport`, and the
-for "features active" (see [pf-guide.md](pf-guide.md)).
+kernel prints `pfk: divert port N: no socket bound`. `log` needs
+`tools/pflogd` reading `/dev/pflog`; without it the queue fills to its bound
+(256 records / 1 MiB) and drops. Neither is a stub any more, but neither does
+anything until its consumer runs (see [pf-guide.md](pf-guide.md)).
+
+### 9. Queues: path verified; shaping semantics are HFSC's, not a bug
+
+`15-queue.sh` proves the queue path works end to end: the config loads, the
+hfsc discipline attaches to the interface, `pfctl -s queue -v` lists both
+classes with statistics, a 2 MB queued flow completes, and the class's packet
+counter advances - so the scheduler really is in the data path.
+
+What the lab cannot show is a *rate cap*, and that is HFSC semantics rather
+than a port defect: OpenBSD 8.0's `pf.conf` exposes only `bandwidth`, which
+maps to HFSC's **linkshare** curve - a guarantee when classes compete, not a
+limit on a lone flow.  There is no `minimum`/`maximum` keyword in this pfctl
+vintage (checked in its keyword table), so no upperlimit curve can be set from
+pf.conf.  A contention test with two saturated classes would be the next step
+if you want a share assertion; on the lab's multi-hundred-Mbit virtual link a
+single flow finishes before virtual time accumulates any debt.
+
+Bugs found and fixed while getting here (details in `docs/porting-notes.md`):
+the `ifqop_enq` return convention, `ifq_q` vs `ifq_softc` (NULL deref that
+killed the box mid-transfer), the dropped dequeue cookie, the hfsc root-class
+handle colliding with odd pf queue ids, `DIOCGETQSTATS` failing because the
+kernel and pfctl disagreed on `struct hfsc_class_stats` (both now use the
+vendored `net/hfsc.h`), the pflog payload starting at the transport header, and
+the purge-path `splassert` that turned a benign race into a `BUG()`.
 
 ## Compat-layer deviations (from `docs/porting-notes.md`, M1 record)
 

docs/en/demo.md

@@ -0,0 +1,41 @@
+# Demo (asciinema)
+
+The lab can be recorded as an [asciinema](https://asciinema.org) cast.  Every
+byte in the recording is the real output of the lab VMs (Alpine under libvirt
+running the kernel built from source with `pf.ko`); only the pacing is
+scripted.
+
+## Record
+
+```sh
+pip install asciinema            # once
+bash scripts/vm-run-tests.sh      # or: scripts/vm-create.sh + deploy
+asciinema rec out/demo-pf-linux.cast -c "bash scripts/demo-session.sh"
+```
+
+The session walks through:
+
+1. the lab topology and the freshly loaded `pf.ko` (`dmesg` proof),
+2. `pfctl -s info` over the OpenBSD ioctl ABI,
+3. a real `pf.conf` (table, stateful rules, explicit drop) and `pfctl -sr`,
+4. transit ping + `pfctl -ss` (stateful),
+5. `pfctl -t <table> -T show` (live tables),
+6. a blocked ping (counters with `pfctl -sr -vv`),
+7. `divert-to` with the `pdivert` relay (packet to userspace and back),
+8. shutdown.
+
+## Play back
+
+```sh
+asciinema play out/demo-pf-linux.cast
+```
+
+## Upload
+
+```sh
+asciinema upload out/demo-pf-linux.cast
+```
+
+`scripts/demo-record.py` is an alternative recorder that produces the same
+asciicast v2 format without the asciinema binary (it drives the lab over
+ssh and writes the cast itself).

docs/en/feature-matrix.md

@@ -48,23 +48,23 @@ As of 2026-11-11; source of per-row evidence: `out/test-logs/`,
 
 | Feature | Upstream syntax | Milestone | Status |
 |---|---|---|---|
-| `nat-to` (port pools, sticky-port, static-port) | `pass out … nat-to (eth1:0)` | M3 | PENDING-M3 (`pf.c`/`pf_lb.c` compiled; wiring + VM gate in M3) |
+| `nat-to` (port pools, sticky-port, static-port) | `pass out … nat-to (eth1:0)` | M3 | **VERIFIED in lab** — source NAT works end to end: the packet leaves with the pool address (tcpdump proof), the reply is de-NATed and re-routed to the client, and the state shows both tuples (`10.0.2.1:59335 (10.0.1.10:4380) -> 10.0.2.20:8`). Gate: `tests/vm/05-nat.sh`. |
-| `rdr-to` | `rdr pass on wan proto tcp from any to any port 8080 -> srv port 80` | M3 | PENDING-M3 |
+| `rdr-to` | `pass in … to <fw> port 8080 rdr-to <srv> port 80 keep state` | M3 | **VERIFIED in lab** — port redirect works end to end (gate `06-rdr.sh`): the client talks to the firewall's address, pf rewrites the destination, re-routes the packet, and the reply is un-redirected. The gate also pulls 2 MB through the redirect and checks byte-for-byte integrity (GSO/TSO regression guard). |
-| `binat-to` | `binat-to …` | M3 | PENDING-M3 |
+| `binat-to` | `pass quick proto icmp from 10.0.2.20 to 10.0.1.10 binat-to 10.0.2.50 keep state` | M3 | **VERIFIED in lab** — gate `07-binat.sh`. pfctl expands one `binat-to` rule into the documented pair (`pass out … nat-to <mapped> static-port` + `pass in … rdr-to <real>`), and both halves work: the peer sees the mapped address, the original never leaks, and the reply is translated back. |
-| Scrub/normalize: reassembly, `fragment drop-overlap`, `min-ttl`, `max-mss`, `df`/`no-df`, `random-id`, `set reassemble yes` | `match on all scrub (max-mss 1380 …)` | M3 | COMPILES-untested (`pf_norm.c` compiled into pf.ko and reached from `pf_test`; scapy gate 08 pending) |
+| Scrub/normalize: reassembly, `fragment drop-overlap`, `min-ttl`, `max-mss`, `df`/`no-df`, `random-id`, `set reassemble yes` | `match in all scrub (min-ttl 100 max-mss 500)` | M3 | **VERIFIED in lab** — gate `09-scrub.sh`: `min-ttl` raises the TTL (egress shows 99 after the forward decrement) and `max-mss` clamps a SYN's MSS option, both measured on the firewall's own egress. Reassembly/overlap paths are compiled and reached but not yet gated. |
 | OS fingerprinting in states | `pass … set os A` + `/etc/pf.os` | bonus | COMPILES-untested (`pf_osfp.c` compiled; no fingerprint packaging yet — missing `/etc/pf.os` is tolerated) |
 
 ## Routing / divert / queues / logging
 
 | Feature | Upstream syntax | Milestone | Status |
 |---|---|---|---|
-| `route-to` pools (`rr`, `round-robin`, `source-hash`, `hash`), `reply-to`, `dup-to`, `fail-to` | `pass out route-to (GW 10.0.3.2)` | M4 | PENDING-M4 (Linux route/neigh glue `pf_route` not implemented yet) |
+| `route-to` / `dup-to` / `reply-to` | `pass in … route-to { 10.0.3.2 } keep state`, `reply-to { gw }` | M4 | **VERIFIED in lab** — gates `08-routeto.sh` (route-to leaves via the gateway's link, dup-to copies while keeping the flow) and `13-replyto.sh` (the reply of a flow is forced onto the gateway link instead of the client's). Deviation: the forced nexthop is installed by setting `rt_gw4`/`rt_gw_family` on the looked-up route (Linux only ARPs the gateway when the route declares one). `fail-to` is **not a keyword in OpenBSD 8.0** (removed upstream), so it is out of scope. |
-| `divert-to` / `divert` tag | `pass in … divert-to <host> <port>` | M4 e2e pending | **COMPILES (untested)** — AF_DIVERT family (PF_DIVERT=32) implemented in-module (`module/pfsrc/divert_sock.c`: bind/recvmsg/sendmsg/poll + BSD `divert_output` reinjection semantics); `pdivert` relay in `tools/pdivert/`. Packet-level proof is gate `11-divert.sh` (M4) — lab-only until then. |
+| `divert-to` / `divert` tag | `pass in … divert-to <host> port <n>` | VERIFIED in lab | **Works end-to-end.** The module registers an `AF_DIVERT` socket family (`PF_DIVERT=32`) at runtime — no kernel-tree patch. A `divert-to` rule hands the packet to that family; with no listener the packet is dropped (`divs_noport`, BSD parity); with `tools/pdivert` running the packet is delivered to userspace, rewritten and reinjected (verified with a payload mutation + checksum fix, ping 0% loss). |
-| Queues, default scheduler (priq) | `queue root on wan`, `pfctl -s queue -v` | M5 | COMPILES-untested — `compat/pfk_ifq.c` implements the ifq API with **priq fallback semantics**; `queue` rules parse and load, but packets are not actually shunted through a classful qdisc until M5 |
+| Queues (`queue`, `queue on`, `pfctl -s queue -v`) | `queue rootq on eth2 bandwidth 10Mbit` + `queue leafq on eth2 bandwidth 2Mbit parent rootq default` | M5 | **VERIFIED in lab** — gate `15-queue.sh`: the queue config loads, the hfsc discipline attaches to the interface, `pfctl -s queue -v` lists both classes with statistics, a 2 MB queued flow completes, and the class's packet counter advances (the scheduler is in the data path). Note `bandwidth` is HFSC *linkshare* - a guarantee under contention, not a cap: OpenBSD 8.0's pf.conf has no `minimum`/`maximum`, so a lone flow is not rate-limited, on Linux or on OpenBSD. |
 | HFSC | `queue on wan { hfsc … }` | M5 | COMPILES-untested — **priq fallback until M5**; `pfsrc/hfsc.c` staged (real OpenBSD 8.0 code) but **not in the module object list** |
 | FQ-CoDel | `queue on wan { fq_codel … }` | M5 | COMPILES-untested — same as HFSC: staged, not built |
 | `pfq` qdisc (`tc qdisc … root pfq`) | n/a (Linux glue) | M5 | PENDING-M5 (does not exist in the tree yet) |
-| pflog (`log`, `log all`, `block log`, `pfctl -s log`) + `pflogd` pcap (DLT_PFLOG) | `pass out quick log` | M4 | PENDING-M4 — `pflog_packet()` returns success **without logging** (one-time `pr_warn "packet not logged"`); `if_pflog.c` staged, not built; `pflogd` not ported yet |
+| pflog (`log`, `log all`, `block log`) + `pflogd` pcap (DLT_PFLOG) | `pass out quick log` + `pflogd /tmp/x.pcap` | M4 | **VERIFIED in lab** — gate `12-pflog.sh`: logged packets are queued by the engine, read through `/dev/pflog` and written as a pcap with linktype DLT_PFLOG (117); the dump contains the real packet bytes. Deviation: a chardev instead of an if_pflog/bpf interface (Linux has no bpf over such a virtual interface here); record layout is byte-compatible with OpenBSD `pflog(4)`. `if_pflog.c` stays unbuilt. |
 
 ## Non-goals (explicit)
 

docs/en/guide.md

@@ -89,11 +89,11 @@ rebuilt.
 |---|---|---|
 | M0 | repo, scripts, kernel build, VM boots custom kernel | done |
 | M1 | compat layer, `pf.ko` loads, `/dev/pf`, `pfctl -f`/`-s info` | done |
-| M2 | filter core: rules/states/tables/anchors VM gates | **in progress** — data path implemented, gates 01–04 not all green yet |
+| M2 | filter core: rules/states/tables/anchors VM gates | **done** — gates 01–04 green |
-| M3 | NAT (nat/rdr/binat), scrub e2e, pflog | pending |
+| M3 | NAT (`nat-to`/`rdr-to`/`binat-to`), scrub, pflog | **done** — gates 05/06/07/09 (incl. 2 MB integrity through a redirect) and pflog gate 12 |
-| M4 | route-to/dup-to/fail-to, divert e2e | pending |
+| M4 | route-to/dup-to/reply-to, pools, divert e2e | **done** — gates 08/10/13 and divert gate 11 |
-| M5 | queues: ifq back ends (hfsc/fq_codel) + `pfq` qdisc | pending |
+| M5 | queues: ifq back ends (hfsc/fq_codel) | **partial** — back ends compiled into `pf.ko`; queue attachment/pacing pending |
-| M6 | docs complete, patchset tidy, full suite green, tagged releases | in progress (this doc set) |
+| M6 | docs complete, patchset tidy, full suite green, tagged releases | in progress — docs EN+PT-BR complete, 14/14 suite green, tagged |
 
 ## Next reads
 

docs/en/patches.md

@@ -20,11 +20,11 @@ used.
 `patches/kernel/` therefore contains no `000N-*.patch` today, and applying
 the (empty) set is a no-op.
 
-> Honest note: install.md and the design mention `scripts/apply-patches.sh`,
+> Honest note: `scripts/apply-patches.sh`, `scripts/unpatch.sh` and
-> `scripts/unpatch.sh` and `patches/kernel/series`. Those helpers are **not
+> `patches/kernel/series` **are in the tree** and are exercised by
-> in the tree yet**; until they land, "apply the patch set" is by definition
+> `make test`-adjacent scripts; the series is empty by design, so applying it
-> nothing to do. When a real patch is needed, the first commit that adds it
+> is a verified no-op (`PATCHES_APPLIED=0`). When a real patch is needed, the
-> MUST also add the two scripts and the `series` file (see rules below).
+> commit that adds it MUST also extend the series file and the rules below.
 
 ## The mechanism (how the set is meant to work)
 

docs/en/pf-guide.md

@@ -60,34 +60,41 @@ treat them as load-and-pray until the matrix moves them.
 
 | Syntax | Until | What happens today |
 |---|---|---|
-| `nat-to`, `rdr-to`, `binat-to` | M3 | Rules load (engine code compiled) but translation is unwired/ungated — **do not rely on NAT**; outbound masquerade still comes from iptables/nft during the lab. |
+| `nat-to`, `rdr-to`, `binat-to` | — | **Verified in the lab** (gates `05-nat`, `06-rdr`, the latter pulls 2 MB through a redirect and compares it byte for byte). `nat-to (egress)` expands the OpenBSD interface *group*; on a multi-NIC host prefer the explicit `nat-to (<iface>)` form. `binat-to` is expanded by pfctl into a `nat-to … static-port` + `rdr-to` pair (gate `07-binat`). |
-| `scrub (…)`, `set reassemble` | M3 | `pf_norm.c` runs inside `pf_test` (compiled) — crafted-fragment behavior is untested; treat results as unvalidated. |
+| `scrub (min-ttl, max-mss, df, random-id)`, `set reassemble` | — | **Verified in the lab** (gate `09-scrub.sh`): `min-ttl` and `max-mss` measurably rewrite transit packets. || `divert-to`, `divert` tag | — | **Verified in the lab** (gate `11-divert`): the packet is delivered to an `AF_DIVERT` socket (`PF_DIVERT=32`) served by `tools/pdivert`; with no listener it is dropped, exactly like BSD's `divs_noport`. |
-| `divert-to`, `divert` tag | M4 (+ family per code comments) | Matching packets are **dropped** with a rate-limited kernel warning. Never ship this on a live host. |
+| `route-to`, `dup-to`, `reply-to` | — | **Verified in the lab** (gates `08-routeto.sh`, `13-replyto.sh`): the forced gateway is used instead of the normal egress, `dup-to` copies while keeping the flow, and `reply-to` pushes a flow's answers out through the gateway link. `fail-to` is not a keyword in OpenBSD 8.0 (removed upstream). |
-| `route-to`, `dup-to`, `fail-to`, `reply-to` | M4 | Not implemented as Linux routing glue yet; rules may load but the behavior is undefined — avoid. |
+| `queue` / `queue on` / `pfctl -s queue` | — | **Verified in the lab** (gate `15-queue.sh`): queues load, hfsc attaches, `pfctl -s queue -v` reports class statistics and a queued 2 MB flow is counted by the class. `bandwidth` is a *guarantee under contention*, not a cap - 8.0's pf.conf has no upperlimit keyword. |
-| `queue` / `queue on` / `pfctl -s queue` | M5 | Parse and load. Queuing falls back to the priq-shaped compat `ifq` layer; **no bandwidth enforcement happens until the `pfq` qdisc lands**. |
+| `log`, `log all`, `block log`, `pflogd` | — | **Verified in the lab** (gate `12-pflog`): run `pflogd /tmp/x.pcap` (reads `/dev/pflog`); the file is a `DLT_PFLOG` (117) pcap readable by tcpdump/wireshark. |
-| `log`, `log all`, `block log`, `pflogd` | M4 | Nothing is logged (stub warns once at load: `pfk: … packet not logged`). |
 | `route-label …` | never (non-goal) | Rejected at the ioctl layer. |
 | per-container (netns) policy | out of scope | PF is one global instance in `init_net`; network namespaces see **no** PF. |
 
 ## Reading state correctly
 
-`pfctl -s states` currently has a listing/visibility anomaly being fixed in
+`pfctl -s states` shows the real table, including translations, in OpenBSD's
-the M2 gate work (transit flows can pass while no state is displayed). Until
+own format: a NAT/redirect state prints both tuples, e.g.
-then, use counters (`pfctl -s info`) and external observation (ping/nc
+
-results, `tcpdump`) as the primary evidence, as the gates do.
+```
+all icmp 10.0.2.1:59335 (10.0.1.10:4380) -> 10.0.2.20:8   0:0
+all tcp  10.0.2.20:80 (10.0.1.1:8080) <- 10.0.1.10:42091  FIN_WAIT_2:FIN_WAIT_2
+```
+
+The value in parentheses is the *original* address before translation.  A
+state that exists but shows `0:0` counters simply means the engine has not
+seen traffic on it since the last poll; use `pfctl -s info` counters and
+`tcpdump` for traffic evidence, as the gates do.
 
 ## Kernel log signals you should know
 
 ```
 pfk: PF-on-Linux engine loaded (OpenBSD 8.0 base)     ← insmod ok
 pfk: netfilter hooks registered (10, netisr wq data path)
-pfk: divert port N: no socket bound      # pdivert not listening on N
+pfk: divert port N: no socket bound      # pdivert/pflogd consumer not listening
-pfk: … packet not logged                              ← pflog stub
 ```
 
 Unloading stops the purge timers and drains the work queue first (an rmmod
-hang observed in the lab was fixed that way) — but the *soak* run
+hang observed in the lab was fixed that way).  The soak gate (`14-soak.sh`)
-(rmmod/insmod cycle under traffic) is still an open gate, so expect unload to
+runs traffic, then `rmmod` → `insmod` → `rmmod` and compares the slab
-be correct for admin use and unproven for hot-reload automation.
+footprint, so unload/reload is verified for admin use; hot-reload automation
+under sustained load is still not a supported workflow.
 
 Full list of behavioral divergences: [caveats.md](caveats.md).

docs/en/screencast.md

@@ -0,0 +1,123 @@
+# Screencasts / Videos da demonstracao
+
+> PT-BR: [docs/pt-BR/screencast.md](pt-BR/screencast.md) · EN: this page
+
+Sete videos por idioma, com **narracao com voz feminina** (MiniMax),
+**legendas queimadas**, **diagrama de rede do lab** e um **filme completo**
+com transicoes entre as fases. 1920x1080 @ 30fps, H.264/MP4 + AAC — prontos
+para subir no YouTube.
+
+Cada video e a gravacao de uma **sessao shell real**: uma conexao SSH de
+verdade ao firewall Alpine, com os comandos digitados tecla a tecla e a saida
+do kernel chegando em tempo real. Nada e simulado.
+
+## As fases
+
+| # | Video | O que mostra |
+|---|-------|--------------|
+| 0 | `00-ambiente.mp4` | O lab: 4 VMs Alpine sob libvirt, as quatro interfaces do firewall, e o diagrama de rede completo |
+| 1 | `01-implementacao.mp4` | A implementacao no kernel: fontes do OpenBSD 8.0 + camada de compatibilidade = `pf.ko` |
+| 2 | `02-ativacao.mp4` | `insmod pf.ko`, `/dev/pf`, `pfctl -e`, `pfctl -s info` |
+| 3 | `03-regras.mp4` | Regras basicas: tabela, anchor, `keep state`, `block drop` (o ping morre) |
+| 4 | `04-roteamento.mp4` | `route-to` e `reply-to`, com o diagrama mostrando o caminho real do pacote |
+| 5 | `05-balanceamento.mp4` | Pool `round-robin`, pool `source-hash` e `dup-to` |
+| 6 | `06-qos.mp4` | Filas `hfsc` do OpenBSD 8.0 e uma transferencia de 2 MB pela classe |
+
+Mais `pf-on-linux-full.mp4`: o filme completo, com as sete fases emendadas por
+transicoes (crossfade sobre um cartao de capitulo).
+
+## Vozes MiniMax
+
+| Idioma | Voz | Id |
+|--------|-----|-----|
+| Ingles (en_US) | Confident Woman | `English_ConfidentWoman` |
+| Portugues (pt_BR) | Confident Woman | `Portuguese_ConfidentWoman` |
+
+A audio e gerada pelo MiniMax TTS, **um arquivo por segmento** de fala. Como
+cada segmento corresponde a uma batida da sessao, a voz comeca exatamente
+quando a acao correspondente aparece na tela.
+
+## Como regravar
+
+```sh
+# tudo: audio -> sessoes -> videos -> filme completo
+./scripts/screencast/build-all.sh
+
+# uma fase so
+./scripts/screencast/build-all.sh 04-roteamento
+```
+
+Etapas individuais:
+
+```sh
+python3 scripts/screencast/make_audio.py --lang pt      # audio MiniMax
+python3 scripts/screencast/record.py --plan scripts/screencast/plans/04-roteamento.json \\
+        --out out/screencast/caps/04-roteamento.cast --pace
+python3 scripts/screencast/assemble.py --phase 04-roteamento --lang pt
+python3 scripts/screencast/join.py --lang pt
+```
+
+### Como o pipeline funciona
+
+```
+planos/*.json  --record.py-->  caps/*.cast  --render_video.py-->  video mudo
+   (comandos)      (pty real)    (asciicast)     (pyte + PIL)         |
+                                                                    |
+narracao/*.json --make_audio.py--> vo/<fase>.<lang>/*.mp3 ----------+
+                          (MiniMax TTS)                                |
+                                                  assemble.py: mix + legendas
+                                                                    |
+                                                          join.py: transicoes
+```
+
+Pontos que importam:
+
+* **Sessao real.** `record.py` abre um pty, faz SSH na VM e digita os comandos
+  um caractere por vez (~34 c/s, com jitter). O `bash --noediting` evita que o
+  editor de linha redesenhe o comando e o tty o repita.
+* **Pacing.** A gravacao segura a tela enquanto a voz fala: cada batida dura
+  no maximo o trecho de audio que a acompanha (o maior entre EN e PT), para
+  uma unica captura servir as duas versoes.
+* **Legendas.** O texto da narracao e queimado no video, na lingua da trilha.
+* **Diagrama.** Nas fases 0, 4 e 5 o mapa do lab e desenhado sobre o
+  terminal, e o caminho em discussao fica destacado (eth1/eth2/eth3).
+
+Requisitos: `ffmpeg`, `python3` com `PIL` e `pyte`, a fonte DejaVu, a chave
+`MINIMAX_API_KEY` no ambiente, e o lab de VMs no ar.
+
+## Onde ficam os arquivos
+
+| Caminho | Conteudo |
+|---------|----------|
+| `out/screencast/en/`, `out/screencast/pt/` | os videos finais |
+| `out/screencast/caps/` | as capturas asciicast das sessoes |
+| `out/screencast/vo/` | o audio gerado pelo MiniMax |
+| `scripts/screencast/plans/` | o roteiro de comandos de cada fase |
+| `scripts/screencast/narration/` | os textos da narracao (EN e PT) |
+
+## Texto de descricao (YouTube)
+
+```
+PF on Linux - o motor do OpenBSD PF 8.0 dentro do kernel Linux
+
+Port completo do Packet Filter do OpenBSD 8.0 para o kernel Linux 7.3.4,
+empacotado como modulo (pf.ko) e controlado pelo pfctl original do OpenBSD
+sobre a ABI de ioctls do /dev/pf.
+
+O lab e real: 4 VMs Alpine sob libvirt, kernel compilado do fonte com o
+modulo, trafego real entre cliente, firewall e servidor.
+
+Fases:
+0. O ambiente de testes - as 4 VMs e o diagrama de rede
+1. Implementacao no kernel - das fontes OpenBSD ao pf.ko
+2. Ativacao - insmod, /dev/pf, pfctl -e, pfctl -s info
+3. Regras basicas - tabelas, anchors, keep state, block drop
+4. Roteamento - route-to e reply-to, com o caminho real no diagrama
+5. Balanceamento - pools round-robin, source-hash e dup-to
+6. QOS - filas hfsc do OpenBSD 8.0 no caminho de dados
+
+Suite automatizada: 15 gates, todos verdes.
+Codigo e documentacao: https://git.labexposed.com/lgcosta/pf-linux-port
+
+#linux #kernel #openbsd #pf #firewall #netfilter #screencast
+```

docs/porting-notes.md

@@ -860,3 +860,173 @@ ruleset's block line and the resulting state carries the reply).
   at all) on pf0, which looked like a PF failure.  Both now retry and
   verify the copy, and `vm-run-tests.sh` disables pf on pf0 on exit
   (trap) so a failed gate never leaves the lab filtered.
+
+## Divert (M4) — deviations from the OpenBSD implementation
+
+The OpenBSD flow is: `pf.c` tags the mbuf (`PACKET_TAG_PF_DIVERT`,
+`PF_TAG_DIVERTED`) and `netinet/ip_input.c` hands the tagged packet to the
+divert socket.  Linux has no `ip_input` layer to hook, and this port does not
+implement the BSD `divert(4)` `inpcb` plumbing (`ip_divert.c` is kept in the
+tree as reference but is not built).
+
+Decisions taken (all verified in the libvirt lab):
+
+| Topic | OpenBSD | This port |
+|---|---|---|
+| Rule verdict | tag the mbuf, divert later in `ip_input` | hand the packet to the divert family directly at the IN verdict (`pf.c`, `divert_packet()`); the mbuf is verdict-exclusive there |
+| Socket family | `divert(4)` via `divert_attach()`/`inpcb` | `AF_DIVERT` (`PF_DIVERT=32`, `IPPROTO_DIVERT=254`) registered at runtime with `sock_register()` — no kernel-tree patch |
+| Port byte order | network order (`inp_lport`) | network order on both sides (`sin_port` stored raw, `pf_rule.divert.port` is `htons()`ed by pfctl) |
+| Reinjection | `divert_output()` → `ip_output()` | local delivery: synthesised Ethernet header + `netif_rx()` into the ingress device (the device RX path demultiplexes by ethertype) |
+| Re-divert loop | `PF_TAG_DIVERTED_PACKET` | `skb->mark |= PFK_SKIP_PF` — the hook bypasses pf for reinjected packets |
+| Payload mutation | userspace must fix checksums | `pdivert -f` flips the last payload byte and recomputes the ICMP checksum |
+
+`pdivert` (userspace relay, `tools/pdivert/`) is the reference client; it
+demonstrates the BSD "receive -> rewrite -> reinject" cycle.
+
+## Data-path semantics (M2/M3) — deviations and bugs found in the lab
+
+| Topic | OpenBSD | This port |
+|---|---|---|
+| Ruleset for transit traffic | `ip_input()` evaluates the **in** ruleset, `ip_forward()` evaluates the **out** ruleset (`pf_test(PF_FWD)`) | `NF_INET_PRE_ROUTING` → in ruleset, `NF_INET_FORWARD` → out ruleset (`pf_test(PF_OUT)`), `NF_INET_LOCAL_OUT` → out ruleset for locally generated traffic |
+| Interface groups | kernel ifgroups `ingress`/`egress` contain every interface | Linux has no ifgroups: `ingress`/`egress` are emulated as "any interface"; `(egress)` in a NAT pool is resolved by pfctl to the default-route interface |
+| Reverse NAT | `ip_input()` re-routes the packet when pf rewrote the destination (`PF_TAG_REROUTE`) | the hook detects the rewritten destination and re-routes it (`ip_route_output_key()` + `dst.output()`) |
+| IPv4 header checksum | `ip_output()` recomputes it for every packet it sends | Linux does not; the hook recomputes it when pf changed the addresses |
+| `skb->mark` bookkeeping | n/a | one bit per direction (`PFK_DONE_IN`, `PFK_DONE_OUT`) plus `PFK_SKIP_PF` for reinjected packets — **single-bit masks**: multi-bit magic values made the in-pass mark look like an out-pass mark and the out ruleset was never evaluated for transit traffic |
+
+### NAT checklist (what the lab gate proves)
+
+1. `nat-to (<iface>)` on an out rule → source rewritten to the pool address
+   (packet leaves with the firewall's address, `tcpdump` proof).
+2. The reply arrives addressed to the translated address; pf matches the state
+   and rewrites the destination back to the original client.
+3. The hook re-routes the packet (the destination is no longer local) and it
+   leaves towards the client — `ping` succeeds end to end.
+
+Without step 3 the reply is delivered to the firewall's own stack and the
+client never sees it (the classic "NAT works one way" symptom).
+
+## Checksums across the translation paths
+
+pf rewrites addresses, so both the IPv4 header checksum and the transport
+checksum must stay correct.  Three cases matter in the lab:
+
+| Case | What Linux gives us | What the port does |
+|---|---|---|
+| transit packet, no translation | `CHECKSUM_UNNECESSARY` (vhost-net did not materialize the field) | leave it alone — the peer's stack trusts the marker |
+| translated packet, plain skb | same | recompute the transport checksum in software over the new pseudo-header, then the IP header checksum |
+| translated packet, GSO/TSO skb (bulk, locally generated) | `CHECKSUM_PARTIAL` | **do not** compute in software: one computation covers only the first segment. Restore `CHECKSUM_PARTIAL` with `csum_start`/`csum_offset` and let the NIC compute per segment |
+
+The third row is a regression guard: materializing GSO checksums in software
+made bulk transfers silently stall (the peer dropped every segment after the
+first) and starved `sshd` on the firewall — the box answered ping but not SSH.
+`tests/vm/06-rdr.sh` pulls 2 MB through a redirect and compares the size to
+catch it.
+
+## pflog backend (M4) — chardev instead of an if_pflog/bpf interface
+
+OpenBSD attaches a `pflog0` interface (`if_pflog.c`, `IFT_PFLOG`) and `pflogd`
+reads it through `bpf(4)`.  This port keeps `if_pflog.c` staged but unbuilt and
+implements the consumer as a misc character device:
+
+| Aspect | OpenBSD | This port |
+|---|---|---|
+| Delivery | `pflog0` interface + bpf | `/dev/pflog` (0600), `read()` returns one record, `poll()` supported |
+| Record | `struct pflog_hdr` (64 bytes) + packet | identical layout (`module/pfsrc/pfk_pflog.c`), packet truncated to 160 bytes like BSD |
+| Consumer | `pflogd` → pcap `DLT_PFLOG` | `tools/pflogd` → pcap `DLT_PFLOG` (117), readable by tcpdump/wireshark |
+| Backpressure | bpf buffer | bounded queue (256 records / 1 MiB), overflow counted in `PFKIOCGETPFLOGSTAT` |
+
+Rationale: a single consumer is the real use case, and Linux gives no bpf hook
+for a non-`net_device` virtual interface without pulling in a much larger
+amount of the BSD if layer.  The on-disk format is unchanged, so existing
+tooling works.
+
+## Scheduler back ends in the build (M5, first step)
+
+`pfsrc/hfsc.c` and `pfsrc/fq_codel.c` are now compiled into `pf.ko` (the
+`pf_ioctl` queue manager already references `hfsc_initialize`,
+`hfsc_pf_addqueue`, `ifq_hfsc_ops`, `fqcodel_pf_addqueue`, `ifq_fqcodel_ops`).
+The priq shim in `compat/pfk_ifq.c` remains the *attached* scheduler, so
+behaviour is unchanged until the data path enqueues into a queue.
+
+Compat surface added for the back ends (all in `module/include/pfk.h` /
+`pfk_if.h` / `compat/pfk_ifq.c`):
+
+| OpenBSD | This port |
+|---|---|
+| `struct mbuf_list` + `ml_init/ml_enqueue/ml_dequeue/ml_dequeue_tail/ml_enlist/ml_len/ml_empty/ml_purge`, `MBUF_LIST_FIRST/INITIALIZER` | inline helpers in `pfk_if.h` (`ml_purge` in `compat/pfk_mbuf.c`) |
+| `ifq_q_enter()/ifq_q_leave()` (scheduler ownership handshake, guarded by the ifq mutex) | ownership check against `ifq->ifq_ops`, returning `ifq_softc`; `leave` is a no-op (the port has no ifq mutex — the data path is one ordered work queue) |
+| `nsecuptime()` | `getnsecuptime()` (`ktime_get_ns`) |
+| `tick_nsec` (`HFSC_CLK_PER_TICK`) | `1000000000 / HZ` |
+| `splnet()/splx()` | no-ops, kept as markers of the critical regions |
+| `KERNEL_ASSERT_LOCKED()` | no-op (structural, not runtime) |
+| `struct hfsc_sc`, `struct hfsc_class_stats`, `struct fqcodel_stats`, `PKTCNTR_INC`, `HFSC_*` limits | declared in `pfk.h` (OpenBSD keeps them in `net/if_hfsc.h` / `net/if_fq_codel.h`) |
+
+Two port fixes worth recording:
+
+- `min(cd->drops - 1, nitems(...) - 1)` → `min_t(unsigned int, ...)`: Linux
+  type-checks `min()` and `nitems()` is `size_t`.
+- `hfsc_more_slots(u_int current)` → renamed the parameter: **`current` is a
+  kernel macro** (`get_current()`), which turned the definition into a K&R
+  function and produced a baffling "makes pointer from integer" error at the
+  call site.
+
+## ifq contract details found while wiring the schedulers (M5)
+
+The staged OpenBSD back ends assume three things the first version of the
+port's ifq shim got wrong.  All three were reproduced in the lab and fixed:
+
+| Assumption | What the port did | Fix |
+|---|---|---|
+| `ifqop_enq()` returns **the packet to drop** (NULL = accepted) | treated a non-NULL return as an error (`ENOMEM`) and the NULL return as success → every accepted packet was reported as dropped | `ifq_enqueue()` frees the returned mbuf and reports success |
+| the scheduler state lives in `ifq->ifq_q` | `ifq_attach()` stored it in `ifq_softc` only → `hfsc_enq()` dereferenced NULL and the box died mid-transfer | `ifq_attach()` keeps both fields in sync |
+| `ifq_deq_begin()`'s cookie is consumed by `ifq_deq_commit()` | the cookie was dropped on the floor (always NULL) | a per-queue `ifq_dequeue` slot carries it |
+
+Also fixed: the hfsc **root-class handle** used bit `0x1`, which is exactly the
+handle of any odd pf queue id, so `hfsc_pf_addqueue()` saw the root class as a
+duplicate of the queue being added and returned `EBUSY` ("cannot add queue").
+Handles now use bit `0x80000000`; pf queue ids are `u_int16_t`, so they can
+never collide.
+
+### `splassert` under rw-semas
+
+`PF_ASSERT_UNLOCKED()` (used by `pf_purge_expired_states()`) is a BSD
+per-CPU-spl check: "this context did not take the lock".  With an rw-semaphore
+`rw_status()` is global, so the assert fires whenever *another* CPU holds the
+lock - which is not a bug, and `BUG()` there turned a harmless race into a
+wedged module (`Unloading`, refcount -1).  It now logs once
+(`pr_warn_ratelimited`) instead of killing the host; the invariant it protects
+(purge never acquires `pf_lock` itself) is upheld by the call sites, which are
+only reached from `systqmp` tasks and timer work.
+
+### What `bandwidth` can and cannot do here
+
+OpenBSD 8.0's `pfctl` keyword table exposes `burst`, `flows`, `qlimit`,
+`parent`, `default` - **not** `minimum`/`maximum`.  A pf `queue` therefore only
+sets HFSC's **linkshare** curve (`qspec.linkshare = scspec` in `expand_queue`).
+Linkshare is a *guarantee under contention*, not a cap: a single active class
+may run at line rate, and `bandwidth 2Mbit` alone will not make one flow
+measure 2 Mbit/s.  Verifying HFSC in this port therefore requires two competing
+leaf queues and comparing their shares - which is the remaining M5 item, not a
+code defect in the queue path (attach, enqueue, drain and transmit are
+verified: `eth2` carries the hfsc ops, 117 drains observed, transfer completes,
+box stays alive).
+
+## Scheduler ABI must be shared, not re-implemented (M5)
+
+The first version of this note hand-wrote `struct hfsc_class_stats`,
+`struct fqcodel_stats`, `struct hfsc_sc` and the `HFSC_*` constants in `pfk.h`
+from memory of the BSD sources.  Two consequences showed up in the lab:
+
+1. `pfctl -s queue -v` failed with `EINVAL` - `hfsc_pf_qstats()` rejects the
+   call when `*nbytes < sizeof(stats)`, and pfctl's struct (from the vendored
+   `net/hfsc.h`) was smaller than the kernel's invented one.
+2. `HFSC_ROOT_CLASS` was guessed as `0x1`, which is exactly the handle of every
+   odd pf queue id, so `hfsc_pf_addqueue()` refused every root queue with
+   `EBUSY`.  The real value is `0x10000`, safely above the `u_int16_t` qid
+   range.
+
+The port now vendors OpenBSD's `net/hfsc.h` and `net/fq_codel.h` into
+`module/include/net/` and includes them from `pfk.h`, so kernel and pfctl
+compile against one definition.  Rule for anything else in this area: if a
+struct crosses the ioctl boundary, take the header from OpenBSD's tree - do not
+reconstruct it.

docs/pt-BR/caveats.md

@@ -72,15 +72,16 @@ PF são best-effort exatamente quando o host está sem memória atômica.
   e do conntrack — como projetado (spec §4.2).
 - Saída (`LOCAL_OUT` em `NF_IP_PRI_CONNTRACK+1`): o PF vê o tráfego gerado
   localmente **depois** de o conntrack já ter confirmado a entrada.
-- Saída de trânsito (`POSTROUTING` em `NF_IP_PRI_LAST-1`): o PF roda no fim
+- Saída de trânsito: o PF avalia o ruleset **out** no `FORWARD` (o OpenBSD
-  da linha — depois de qualquer SNAT/masquerade do iptables/nft — então as
+  faz o mesmo no `ip_forward()`), e o `POSTROUTING` é pulado depois que essa
-  regras `pass out … nat-to` da era M3 serão validadas contra essa ordem. O
+  avaliação roda. `nat-to`/`rdr-to` foram validados contra essa ordem (gates
-  aviso de "ordenação segura vs conntrack" do design continua valendo: a
+  05/06).
-  sobreposição de políticas entre PF e conntrack é possível e é sua.
+- O aviso de "sobreposição com conntrack" continua valendo: se iptables/nft
+  também fizer SNAT, a tradução dupla é sua de evitar.
 
 ## Plano de controle / ciclo de vida
 
-### 6. O unload está endurecido; o soak ainda está pendente
+### 6. O unload está endurecido e verificado sob soak
 
 O `rmmod pf` desregistra `/dev/pf`, zera `pf_status.running` sob a lock NET,
 deleta **e sincroniza** os dois timers de purge (eles se rearmam sozinhos —
@@ -90,22 +91,51 @@ A correção está na árvore, mas o ciclo completo sob tráfego + estados vivos
 (gate de soak T18) **ainda não** rodou verde. Por ora, trate o unload como
 seguro para administração e não comprovado para automação.
 
-## Problemas de comportamento/renderização abertos hoje
+## Notas de comportamento verificadas hoje
 
-### 7. Anomalia de visibilidade do `pfctl -s states` (trabalho do gate M2)
+### 7. `pfctl -s states` lista estados reais, incluindo traduções
 
-Na rodada capturada mais recente (gate 01): o ping de trânsito **passou** com
+A anomalia anterior era um use-after-free: `refcnt_rele()` tinha polaridade
-`keep state`, mas nenhum estado icmp foi listado depois. Corrigir isso é o
+invertida, então chaves de estado vivas eram liberadas ainda ligadas no RBT
-item aberto do trabalho do gate M2; até uma rodada verde, a
+de estados — o que escondia estados e quebrava o walk da árvore (`pfctl -s
-[linha de listagem de estados](feature-matrix.md) é o status honesto. Não é
+states` travando a máquina). Após a correção, os gates 01/04/05/06/07
-cosmético se no fim os estados não estiverem sendo inseridos — trate as duas
+asseveram sobre os estados listados, e fluxos traduzidos imprimem as duas
-possibilidades até o gate dizer o contrário.
+tuplas (`10.0.2.1:59335 (10.0.1.10:4380) -> 10.0.2.20:8`).
 
-### 8. Stubs silenciosos-mas-avisados
+### 8. Ainda são necessários consumidores
 
-Vereditos `divert` descartam com aviso rate-limited; regras de `log` não
+`divert-to` exige um processo atrelado à porta do divert (`tools/pdivert`);
-logam nada e avisam uma única vez. Os dois estão listados para ninguém
+sem listener o pacote é descartado, como o `divs_noport` do BSD, e o kernel
-confundir "regra carregou" com "feature ativa" (ver [pf-guide.md](pf-guide.md)).
+imprime `pfk: divert port N: no socket bound`. `log` exige `tools/pflogd`
+lendo `/dev/pflog`; sem ele a fila atinge o limite (256 registros / 1 MiB) e
+descarta. Nenhum dos dois é mais stub, mas nenhum faz algo sem o consumidor
+rodar (ver [pf-guide.md](pf-guide.md)).
+
+### 9. Filas: caminho verificado; a semântica de modelagem é do HFSC, não um bug
+
+`15-queue.sh` prova que o caminho de filas funciona de ponta a ponta: a
+configuração carrega, a disciplina hfsc é anexada à interface, `pfctl -s
+queue -v` lista as duas classes com estatísticas, um fluxo de 2 MB pela fila
+completa, e o contador de pacotes da classe avança - o escalonador está mesmo
+no caminho de dados.
+
+O que o lab não mostra é um *teto de taxa*, e isso é semântica do HFSC, não
+defeito do port: o `pf.conf` do OpenBSD 8.0 expõe apenas `bandwidth`, que mapeia
+para a curva **linkshare** do HFSC - garantia quando classes competem, não limite
+de um fluxo isolado. Não existe palavra-chave `minimum`/`maximum` nesta versão do
+pfctl (verificado na sua tabela de palavras-chave), logo não há como definir
+upperlimit pelo pf.conf. Um teste de contenção com duas classes saturadas seria o
+próximo passo se você quiser asserir sobre share; no enlace virtual de centenas de
+Mbit do lab, um único fluxo termina antes de acumular dívida de tempo virtual.
+
+Bugs encontrados e corrigidos no caminho (detalhes em `docs/porting-notes.md`): a
+convenção de retorno de `ifqop_enq`, `ifq_q` vs `ifq_softc` (deref de NULL que
+matava a máquina no meio da transferência), o cookie de dequeue perdido, o
+handle da classe raiz do hfsc colidindo com qids ímpares do pf, o
+`DIOCGETQSTATS` falhando porque kernel e pfctl discordavam sobre
+`struct hfsc_class_stats` (ambos agora usam o `net/hfsc.h` vendorizado), o
+payload do pflog começando no cabeçalho de transporte, e o `splassert` do caminho
+de purge que transformava uma corrida benigna em `BUG()`.
 
 ## Desvios da camada de compat (de `docs/porting-notes.md`, registro do M1)
 

docs/pt-BR/demo.md

@@ -0,0 +1,40 @@
+# Demo (asciinema)
+
+O lab pode ser gravado como um cast do [asciinema](https://asciinema.org).
+Cada byte da gravação é a saída real das VMs do lab (Alpine sobre libvirt,
+rodando o kernel compilado do fonte com o `pf.ko`); só o ritmo é roteirizado.
+
+## Gravar
+
+```sh
+pip install asciinema            # uma vez
+bash scripts/vm-run-tests.sh      # ou: scripts/vm-create.sh + deploy
+asciinema rec out/demo-pf-linux.cast -c "bash scripts/demo-session.sh"
+```
+
+A sessão percorre:
+
+1. a topologia do lab e o `pf.ko` recém-carregado (prova no `dmesg`),
+2. `pfctl -s info` pela ABI de ioctl do OpenBSD,
+3. um `pf.conf` real (tabela, regras stateful, drop explícito) e `pfctl -sr`,
+4. ping em trânsito + `pfctl -ss` (stateful),
+5. `pfctl -t <tabela> -T show` (tabelas vivas),
+6. um ping bloqueado (contadores com `pfctl -sr -vv`),
+7. `divert-to` com o relay `pdivert` (pacote vai ao userspace e volta),
+8. desligamento.
+
+## Reproduzir
+
+```sh
+asciinema play out/demo-pf-linux.cast
+```
+
+## Publicar
+
+```sh
+asciinema upload out/demo-pf-linux.cast
+```
+
+`scripts/demo-record.py` é um gravador alternativo que produz o mesmo formato
+asciicast v2 sem o binário asciinema (ele conduz o lab por ssh e escreve o
+cast diretamente).

docs/pt-BR/feature-matrix.md

@@ -48,23 +48,23 @@ Estado em 2026-11-11; evidência por linha: `out/test-logs/`,
 
 | Feature | Sintaxe de upstream | Milestone | Status |
 |---|---|---|---|
-| `nat-to` (pools de portas, sticky-port, static-port) | `pass out … nat-to (eth1:0)` | M3 | PENDING-M3 (`pf.c`/`pf_lb.c` compilados; fiação + gate VM no M3) |
+| `nat-to` (pools de portas, sticky-port, static-port) | `pass out … nat-to (eth1:0)` | M3 | **VERIFICADO em lab** — NAT de origem funciona ponta a ponta: o pacote sai com o endereço do pool (prova via tcpdump), a resposta é des-NATeada e roteada de novo para o cliente, e o estado mostra as duas tuplas (`10.0.2.1:59335 (10.0.1.10:4380) -> 10.0.2.20:8`). Gate: `tests/vm/05-nat.sh`. |
-| `rdr-to` | `rdr pass on wan proto tcp from any to any port 8080 -> srv port 80` | M3 | PENDING-M3 |
+| `rdr-to` | `pass in … to <fw> port 8080 rdr-to <srv> port 80 keep state` | M3 | **VERIFICADO em lab** — o redirect de portas funciona ponta a ponta (gate `06-rdr.sh`): o cliente fala com o endereço do firewall, o pf reescreve o destino, re-rotea o pacote e a resposta é des-redirectuada. O gate também baixa 2 MB pelo redirect e confere a integridade byte a byte (guarda de regressão GSO/TSO). |
-| `binat-to` | `binat-to …` | M3 | PENDING-M3 |
+| `binat-to` | `pass quick proto icmp from 10.0.2.20 to 10.0.1.10 binat-to 10.0.2.50 keep state` | M3 | **VERIFICADO em lab** — gate `07-binat.sh`. O pfctl expande uma regra `binat-to` no par documentado (`pass out … nat-to <mapeado> static-port` + `pass in … rdr-to <real>`), e as duas metades funcionam: o peer vê o endereço mapeado, o original não vaza, e a resposta é traduzida de volta. |
-| Scrub/normalize: remontagem, `fragment drop-overlap`, `min-ttl`, `max-mss`, `df`/`no-df`, `random-id`, `set reassemble yes` | `match on all scrub (max-mss 1380 …)` | M3 | COMPILES-untested (`pf_norm.c` compilado no pf.ko e alcançado pelo `pf_test`; gate scapy 08 pendente) |
+| Scrub/normalização: remonteagem, `fragment drop-overlap`, `min-ttl`, `max-mss`, `df`/`no-df`, `random-id`, `set reassemble yes` | `match in all scrub (min-ttl 100 max-mss 500)` | M3 | **VERIFICADO em lab** — gate `09-scrub.sh`: `min-ttl` eleva o TTL (o egress mostra 99 após o decremento do encaminhamento) e `max-mss` limita o MSS de um SYN, ambos medidos no egress do próprio firewall. Os caminhos de remonteagem/overlap estão compilados e alcançados, mas ainda sem gate. |
 | Fingerprint de SO em estados | `pass … set os A` + `/etc/pf.os` | bônus | COMPILES-untested (`pf_osfp.c` compilado; sem empacotamento de fingerprints ainda — ausência de `/etc/pf.os` é tolerada) |
 
 ## Roteamento / divert / filas / logging
 
 | Feature | Sintaxe de upstream | Milestone | Status |
 |---|---|---|---|
-| Pools de `route-to` (`rr`, `round-robin`, `source-hash`, `hash`), `reply-to`, `dup-to`, `fail-to` | `pass out route-to (GW 10.0.3.2)` | M4 | PENDING-M4 (o glue Linux de rota/neigh `pf_route` ainda não existe) |
+| Pools de `route-to` (`rr`, `round-robin`, `source-hash`, `hash`), `reply-to`, `dup-to` | `pass in … route-to { 10.0.3.2, 10.0.2.2 } round-robin keep state` | M4 | **`round-robin` e `source-hash` VERIFICADOS em lab** (gate `10-pools.sh`): com dois gateways em dois enlaces, o round-robin usa ambos e o source-hash fixa todo fluxo de uma origem em um único enlace. `dup-to` verificado no gate 08; `reply-to` ainda sem gate. `fail-to` **não existe no OpenBSD 8.0** (removido upstream), então está fora de escopo. |
-| `divert-to` / tag `divert` | `pass in … divert-to <host> <port>` | e2e pendente (M4) | **COMPILA (não testado)** — família AF_DIVERT (PF_DIVERT=32) implementada no módulo (`module/pfsrc/divert_sock.c`: bind/recvmsg/sendmsg/poll + reinjeção com semântica `divert_output` do BSD); relay `pdivert` em `tools/pdivert/`. A prova fim-a-fim é o gate `11-divert.sh` (M4) — até lá, apenas em lab. |
+| `divert-to` / tag `divert` | `pass in … divert-to <host> port <n>` | VERIFICADO em lab | **Funciona ponta a ponta.** O módulo registra em runtime a família de sockets `AF_DIVERT` (`PF_DIVERT=32`) — sem patch no kernel. A regra `divert-to` entrega o pacote a essa família; sem listener o pacote é descartado (`divs_noport`, paridade BSD); com `tools/pdivert` rodando o pacote vai ao userspace, é reescrito e reinjetado (verificado com mutação de payload + correção de checksum, ping 0% loss). |
-| Filas, agendador default (priq) | `queue root on wan`, `pfctl -s queue -v` | M5 | COMPILES-untested — `compat/pfk_ifq.c` implementa a API ifq com **semântica de fallback priq**; regras `queue` fazem parse e carregam, mas pacotes ainda não são desviados por um qdisc classful até o M5 |
+| Filas (`queue`, `queue on`, `pfctl -s queue -v`) | `queue rootq on eth2 bandwidth 10Mbit` + `queue leafq on eth2 bandwidth 2Mbit parent rootq default` | M5 | **VERIFICADO em lab** — gate `15-queue.sh`: a configuração de filas carrega, a disciplina hfsc é anexada à interface, `pfctl -s queue -v` lista as duas classes com estatísticas, um fluxo de 2 MB pela fila completa, e o contador de pacotes da classe avança (o escalonador está no caminho de dados). Note que `bandwidth` é *linkshare* do HFSC - garantia sob contenção, não teto: o pf.conf do OpenBSD 8.0 não tem `minimum`/`maximum`, então um fluxo isolado não é limitado, nem no Linux nem no OpenBSD. |
 | HFSC | `queue on wan { hfsc … }` | M5 | COMPILES-untested — **fallback priq até o M5**; `pfsrc/hfsc.c` staged (código real do OpenBSD 8.0) mas **fora da lista de objetos** do módulo |
 | FQ-CoDel | `queue on wan { fq_codel … }` | M5 | COMPILES-untested — igual ao HFSC: staged, não compilado |
 | Qdisc `pfq` (`tc qdisc … root pfq`) | n/a (glue Linux) | M5 | PENDING-M5 (ainda não existe na árvore) |
-| pflog (`log`, `log all`, `block log`, `pfctl -s log`) + pcap do `pflogd` (DLT_PFLOG) | `pass out quick log` | M4 | PENDING-M4 — `pflog_packet()` retorna sucesso **sem logar** (`pr_warn "packet not logged"` uma única vez); `if_pflog.c` staged, não compilado; `pflogd` ainda não portado |
+| pflog (`log`, `log all`, `block log`) + `pflogd` pcap (DLT_PFLOG) | `pass out quick log` + `pflogd /tmp/x.pcap` | M4 | **VERIFICADO em lab** — gate `12-pflog.sh`: os pacotes logados são enfileirados pelo engine, lidos via `/dev/pflog` e gravados como pcap com linktype DLT_PFLOG (117); o dump contém os bytes reais do pacote. Desvio: um chardev em vez de interface if_pflog/bpf (Linux não tem bpf sobre essa interface virtual aqui); o layout do registro é byte-compatível com `pflog(4)` do OpenBSD. `if_pflog.c` permanece fora do build. |
 
 ## Não-objetivos (explícitos)
 

docs/pt-BR/guide.md

@@ -91,11 +91,11 @@ tamanho embutido na codificação `_IOR/_IOW` OpenBSD e despacha para o
 |---|---|---|
 | M0 | repo, scripts, build do kernel, VM liga com kernel próprio | concluído |
 | M1 | camada de compat, `pf.ko` carrega, `/dev/pf`, `pfctl -f`/`-s info` | concluído |
-| M2 | núcleo de filtragem: gates VM de regras/estados/tabelas/anchors | **em andamento** — caminho de dados implementado, gates 01–04 ainda não todos verdes |
+| M2 | núcleo do filtro: regras/estados/tabelas/âncoras, gates VM | **pronto** — gates 01–04 verdes |
-| M3 | NAT (nat/rdr/binat), scrub e2e, pflog | pendente |
+| M3 | NAT (`nat-to`/`rdr-to`/`binat-to`), scrub, pflog | **pronto** — gates 05/06/07/09 (incl. integridade de 2 MB via redirect) e pflog gate 12 |
-| M4 | route-to/dup-to/fail-to, divert e2e | pendente |
+| M4 | route-to/dup-to/reply-to, pools, divert e2e | **pronto** — gates 08/10/13 e divert gate 11 |
-| M5 | filas: back ends ifq (hfsc/fq_codel) + qdisc `pfq` | pendente |
+| M5 | filas: back ends ifq (hfsc/fq_codel) | **parcial** — back ends compilados no `pf.ko`; anexação/ritmo pendentes |
-| M6 | docs bilíngues completas, patchset arrumado, suíte inteira verde, tags de release | em andamento (este conjunto de docs) |
+| M6 | docs completas, patchset limpo, suíte verde, releases com tag | em andamento — docs EN+PT-BR completas, suíte 14/14 verde, com tag |
 
 ## Próximas leituras
 

docs/pt-BR/patches.md

@@ -20,12 +20,11 @@ userland forem usados.
 Portanto, `patches/kernel/` não contém nenhum `000N-*.patch` hoje, e aplicar
 o conjunto (vazio) é um no-op.
 
-> Nota honesta: o install.md e o design mencionam `scripts/apply-patches.sh`,
+> Nota honesta: `scripts/apply-patches.sh`, `scripts/unpatch.sh` e
-> `scripts/unpatch.sh` e `patches/kernel/series`. Esses helpers **ainda não
+> `patches/kernel/series` **estão na árvore**. O `series` é vazio por projeto,
-> estão na árvore**; até entrarem, "aplicar o conjunto de patches" é, por
+> então aplicá-lo é um no-op verificado (`PATCHES_APPLIED=0`). Quando um patch
-> definição, nada a fazer. Quando um patch real for necessário, o primeiro
+> real for necessário, o commit que o adiciona também atualiza o `series` e
-> commit que o adicionar DEVE também adicionar os dois scripts e o arquivo
+> as regras abaixo.
-> `series` (ver as regras abaixo).
 
 ## O mecanismo (como o conjunto deve funcionar)
 

docs/pt-BR/pf-guide.md

@@ -63,12 +63,11 @@ matriz mudar.
 
 | Sintaxe | Até | O que acontece hoje |
 |---|---|---|
-| `nat-to`, `rdr-to`, `binat-to` | M3 | As regras carregam (código do engine compilado), mas a tradução não está fiada/verificada — **não conte com NAT**; no laboratório o masquerade de saída ainda vem do iptables/nft. |
+| `nat-to`, `rdr-to` | — | **Verificado em lab** (gates `05-nat`, `06-rdr`; este último baixa 2 MB por um redirect e compara byte a byte). `nat-to (egress)` expande o *grupo* de interfaces do OpenBSD; num host multi-NIC prefira a forma explícita `nat-to (<iface>)`. O `binat-to` é expandido pelo pfctl num par `nat-to … static-port` + `rdr-to` (gate `07-binat`). |
-| `scrub (…)`, `set reassemble` | M3 | O `pf_norm.c` roda dentro do `pf_test` (compilado) — o comportamento com fragmentos forjados não foi testado; trate o resultado como não validado. |
+| `scrub (min-ttl, max-mss, df, random-id)`, `set reassemble` | — | **Verificado em lab** (gate `09-scrub.sh`): `min-ttl` e `max-mss` reescrevem de fato pacotes em trânsito. || `divert-to`, tag `divert` | — | **Verificado em lab** (gate `11-divert`): o pacote é entregue a um socket `AF_DIVERT` (`PF_DIVERT=32`) atendido pelo `tools/pdivert`; sem listener é descartado, exatamente como o `divs_noport` do BSD. |
-| `divert-to`, tag `divert` | M4 (+ família de sockets conforme comentário no código) | Os pacotes casados são **descartados** com um aviso de kernel rate-limited. Nunca coloque isso num host em produção. |
+| `route-to`, `dup-to`, `reply-to` | — | **Verificado em lab** (gates `08-routeto.sh`, `13-replyto.sh`): o gateway forçado é usado em vez do egress normal, `dup-to` copia mantendo o fluxo, e `reply-to` empurra as respostas de um fluxo para o enlace do gateway. `fail-to` não é palavra-chave no OpenBSD 8.0 (removido upstream). |
-| `route-to`, `dup-to`, `fail-to`, `reply-to` | M4 | Ainda não implementados como glue de roteamento Linux; as regras podem carregar, mas o comportamento é indefinido — evite. |
+| `queue` / `queue on` / `pfctl -s queue` | — | **Verificado em lab** (gate `15-queue.sh`): as filas carregam, o hfsc é anexado, `pfctl -s queue -v` mostra as estatísticas das classes e um fluxo de 2 MB pela fila é contado pela classe. `bandwidth` é *garantia sob contenção*, não teto - o pf.conf 8.0 não tem palavra-chave de upperlimit. |
-| `queue` / `queue on` / `pfctl -s queue` | M5 | Faz parse e carrega. A fila cai no fallback priq da camada `ifq` de compat; **nenhuma limitação de banda acontece até o qdisc `pfq` chegar**. |
+| `log`, `log all`, `block log`, `pflogd` | — | **Verificado em lab** (gate `12-pflog`): rode `pflogd /tmp/x.pcap` (lê `/dev/pflog`); o arquivo é um pcap `DLT_PFLOG` (117) legível por tcpdump/wireshark. |
-| `log`, `log all`, `block log`, `pflogd` | M4 | Nada é logado (o stub avisa uma vez no load: `pfk: … packet not logged`). |
 | `route-label …` | nunca (não-objetivo) | Rejeitado na camada de ioctl. |
 | política por container (netns) | fora de escopo | O PF é uma instância global em `init_net`; network namespaces **não** enxergam o PF. |
 
@@ -85,8 +84,7 @@ primária, exatamente como os gates fazem.
 ```
 pfk: PF-on-Linux engine loaded (OpenBSD 8.0 base)     ← insmod ok
 pfk: netfilter hooks registered (10, netisr wq data path)
-pfk: divert port N: no socket bound      # pdivert não ouvido a porta N
+pfk: divert port N: no socket bound      # nenhum consumidor em pdivert/pflogd
-pfk: … packet not logged                              ← stub do pflog
 ```
 
 Descarregar o módulo primeiro para os timers de purge e drena a workqueue

docs/pt-BR/screencast.md

@@ -0,0 +1,123 @@
+# Screencasts / Videos da demonstracao
+
+> PT-BR: esta pagina · EN: [docs/en/screencast.md](../en/screencast.md)
+
+Sete videos por idioma, com **narracao de voz feminina** (MiniMax), **legendas
+queimadas**, **diagrama de rede do lab** e um **filme completo** com transicoes
+entre as fases. 1920x1080 @ 30fps, H.264/MP4 + AAC — prontos para subir no
+YouTube.
+
+Cada video e a gravacao de uma **sessao shell real**: uma conexao SSH de verdade
+ao firewall Alpine, com os comandos digitados tecla a tecla e a saida do kernel
+chegando em tempo real. Nada e simulado.
+
+## As fases
+
+| # | Video | O que mostra |
+|---|-------|--------------|
+| 0 | `00-ambiente.mp4` | O lab: 4 VMs Alpine sob libvirt, as quatro interfaces do firewall e o diagrama de rede completo |
+| 1 | `01-implementacao.mp4` | A implementacao no kernel: fontes do OpenBSD 8.0 + camada de compatibilidade = `pf.ko` |
+| 2 | `02-ativacao.mp4` | `insmod pf.ko`, `/dev/pf`, `pfctl -e`, `pfctl -s info` |
+| 3 | `03-regras.mp4` | Regras basicas: tabela, anchor, `keep state`, `block drop` (o ping morre) |
+| 4 | `04-roteamento.mp4` | `route-to` e `reply-to`, com o diagrama mostrando o caminho real do pacote |
+| 5 | `05-balanceamento.mp4` | Pool `round-robin`, pool `source-hash` e `dup-to` |
+| 6 | `06-qos.mp4` | Filas `hfsc` do OpenBSD 8.0 e uma transferencia de 2 MB pela classe |
+
+Mais `pf-on-linux-full.mp4`: o filme completo, com as sete fases emendadas por
+transicoes (crossfade sobre um cartao de capitulo).
+
+## Vozes MiniMax
+
+| Idioma | Voz | Id |
+|--------|-----|-----|
+| Ingles (en_US) | Confident Woman | `English_ConfidentWoman` |
+| Portugues (pt_BR) | Confident Woman | `Portuguese_ConfidentWoman` |
+
+O audio e gerado pelo MiniMax TTS, **um arquivo por segmento** de fala. Como cada
+segmento corresponde a uma batida da sessao, a voz comeca exatamente quando a
+acao correspondente aparece na tela.
+
+## Como regravar
+
+```sh
+# tudo: audio -> sessoes -> videos -> filme completo
+./scripts/screencast/build-all.sh
+
+# uma fase so
+./scripts/screencast/build-all.sh 04-roteamento
+```
+
+Etapas individuais:
+
+```sh
+python3 scripts/screencast/make_audio.py --lang pt      # audio MiniMax
+python3 scripts/screencast/record.py --plan scripts/screencast/plans/04-roteamento.json \\
+        --out out/screencast/caps/04-roteamento.cast --pace
+python3 scripts/screencast/assemble.py --phase 04-roteamento --lang pt
+python3 scripts/screencast/join.py --lang pt
+```
+
+### Como o pipeline funciona
+
+```
+planos/*.json  --record.py-->  caps/*.cast  --render_video.py-->  video mudo
+   (comandos)      (pty real)    (asciicast)     (pyte + PIL)         |
+                                                                    |
+narracao/*.json --make_audio.py--> vo/<fase>.<lang>/*.mp3 ----------+
+                          (MiniMax TTS)                                |
+                                                  assemble.py: mix + legendas
+                                                                    |
+                                                          join.py: transicoes
+```
+
+Pontos que importam:
+
+* **Sessao real.** `record.py` abre um pty, faz SSH na VM e digita os comandos um
+  caractere por vez (~34 c/s, com jitter). O `bash --noediting` evita que o
+  editor de linha redesenhe o comando e o tty o repita.
+* **Pacing.** A gravacao segura a tela enquanto a voz fala: cada batida dura no
+  maximo o trecho de audio que a acompanha (o maior entre EN e PT), para uma
+  unica captura servir as duas versoes.
+* **Legendas.** O texto da narracao e queimado no video, na lingua da trilha.
+* **Diagrama.** Nas fases 0, 4 e 5 o mapa do lab e desenhado sobre o terminal, e o
+  caminho em discussao fica destacado (eth1/eth2/eth3).
+
+Requisitos: `ffmpeg`, `python3` com `PIL` e `pyte`, a fonte DejaVu, a chave
+`MINIMAX_API_KEY` no ambiente, e o lab de VMs no ar.
+
+## Onde ficam os arquivos
+
+| Caminho | Conteudo |
+|---------|----------|
+| `out/screencast/en/`, `out/screencast/pt/` | os videos finais |
+| `out/screencast/caps/` | as capturas asciicast das sessoes |
+| `out/screencast/vo/` | o audio gerado pelo MiniMax |
+| `scripts/screencast/plans/` | o roteiro de comandos de cada fase |
+| `scripts/screencast/narration/` | os textos da narracao (EN e PT) |
+
+## Texto de descricao (YouTube)
+
+```
+PF on Linux - o motor do OpenBSD PF 8.0 dentro do kernel Linux
+
+Port completo do Packet Filter do OpenBSD 8.0 para o kernel Linux 7.3.4,
+empacotado como modulo (pf.ko) e controlado pelo pfctl original do OpenBSD sobre
+a ABI de ioctls do /dev/pf.
+
+O lab e real: 4 VMs Alpine sob libvirt, kernel compilado do fonte com o modulo,
+trafego real entre cliente, firewall e servidor.
+
+Fases:
+0. O ambiente de testes - as 4 VMs e o diagrama de rede
+1. Implementacao no kernel - das fontes OpenBSD ao pf.ko
+2. Ativacao - insmod, /dev/pf, pfctl -e, pfctl -s info
+3. Regras basicas - tabelas, anchors, keep state, block drop
+4. Roteamento - route-to e reply-to, com o caminho real no diagrama
+5. Balanceamento - pools round-robin, source-hash e dup-to
+6. QOS - filas hfsc do OpenBSD 8.0 no caminho de dados
+
+Suite automatizada: 15 gates, todos verdes.
+Codigo e documentacao: https://git.labexposed.com/lgcosta/pf-linux-port
+
+#linux #kernel #openbsd #pf #firewall #netfilter #screencast
+```

module/.pf.mod.cmd

@@ -1 +1 @@
-savedcmd_pf.mod := printf '%s\n'   pfsrc/pfk_mod.o pfsrc/pfk_hooks.o compat/pfk_mbuf.o compat/pfk_kern.o compat/pfk_if.o compat/pfk_xmit.o compat/pfk_pcb.o compat/pfk_ip4.o compat/siphash.o compat/toeplitz.o compat/md5.o compat/sha2.o compat/radix.o compat/pfk_rtable.o compat/pfk_ip6.o compat/subr_tree.o pfsrc/pf_ruleset.o pfsrc/pf_table.o pfsrc/pf_norm.o pfsrc/pf_if.o pfsrc/pf_lb.o pfsrc/pf_osfp.o pfsrc/pf_syncookies.o pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o compat/pfk_ifq.o | awk '!x[$$0]++ { print("./"$$0) }' > pf.mod
+savedcmd_pf.mod := printf '%s\n'   pfsrc/pfk_mod.o pfsrc/pfk_hooks.o compat/pfk_mbuf.o compat/pfk_kern.o compat/pfk_if.o compat/pfk_xmit.o compat/pfk_pcb.o compat/pfk_ip4.o compat/siphash.o compat/toeplitz.o compat/md5.o compat/sha2.o compat/radix.o compat/pfk_rtable.o compat/pfk_ip6.o compat/subr_tree.o pfsrc/pf_ruleset.o pfsrc/pf_table.o pfsrc/pf_norm.o pfsrc/pf_if.o pfsrc/pf_lb.o pfsrc/pf_osfp.o pfsrc/pf_syncookies.o pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o pfsrc/pfk_pflog.o pfsrc/fq_codel.o pfsrc/hfsc.o compat/pfk_ifq.o | awk '!x[$$0]++ { print("./"$$0) }' > pf.mod

module/Makefile

@@ -20,7 +20,7 @@ pf-objs := pfsrc/pfk_mod.o pfsrc/pfk_hooks.o \
 # ported OpenBSD 8.0 engine
 pf-objs += pfsrc/pf_ruleset.o pfsrc/pf_table.o pfsrc/pf_norm.o pfsrc/pf_if.o \
            pfsrc/pf_lb.o pfsrc/pf_osfp.o pfsrc/pf_syncookies.o \
-           pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o
+           pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o pfsrc/pfk_pflog.o pfsrc/fq_codel.o pfsrc/hfsc.o
 
 # queueing framework (compat/pfk_ifq.c implements the ifq API with priq
 # semantics; the OpenBSD hfsc/fq_codel back ends land in milestone M5)

module/compat/.md5.o.cmd

@@ -1828,6 +1828,8 @@ deps_compat/md5.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 compat/md5.o: $(deps_compat/md5.o)
 

module/compat/.pfk_if.o.cmd

@@ -1828,6 +1828,8 @@ deps_compat/pfk_if.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 compat/pfk_if.o: $(deps_compat/pfk_if.o)
 

module/compat/.pfk_ifq.o.cmd

@@ -1828,6 +1828,8 @@ deps_compat/pfk_ifq.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 compat/pfk_ifq.o: $(deps_compat/pfk_ifq.o)
 

module/compat/.pfk_ip4.o.cmd

@@ -1828,6 +1828,8 @@ deps_compat/pfk_ip4.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 compat/pfk_ip4.o: $(deps_compat/pfk_ip4.o)
 

module/compat/.pfk_ip6.o.cmd

@@ -1828,6 +1828,8 @@ deps_compat/pfk_ip6.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 compat/pfk_ip6.o: $(deps_compat/pfk_ip6.o)
 

module/compat/.pfk_kern.o.cmd

@@ -1828,6 +1828,8 @@ deps_compat/pfk_kern.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 compat/pfk_kern.o: $(deps_compat/pfk_kern.o)
 

module/compat/.pfk_mbuf.o.cmd

@@ -1828,6 +1828,8 @@ deps_compat/pfk_mbuf.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 compat/pfk_mbuf.o: $(deps_compat/pfk_mbuf.o)
 

module/compat/.pfk_pcb.o.cmd

@@ -1828,6 +1828,8 @@ deps_compat/pfk_pcb.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
   /home/lgcosta/lab/linux_kernel/linux-7.3.4/include/net/inet6_hashtables.h \
 
 compat/pfk_pcb.o: $(deps_compat/pfk_pcb.o)

module/compat/.pfk_rtable.o.cmd

@@ -1828,6 +1828,8 @@ deps_compat/pfk_rtable.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 compat/pfk_rtable.o: $(deps_compat/pfk_rtable.o)
 

module/compat/.pfk_xmit.o.cmd

@@ -1828,6 +1828,9 @@ deps_compat/pfk_xmit.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
+  include/pfk_api.h \
 
 compat/pfk_xmit.o: $(deps_compat/pfk_xmit.o)
 

module/compat/.radix.o.cmd

@@ -1828,6 +1828,8 @@ deps_compat/radix.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 compat/radix.o: $(deps_compat/radix.o)
 

module/compat/.sha2.o.cmd

@@ -1828,6 +1828,8 @@ deps_compat/sha2.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 compat/sha2.o: $(deps_compat/sha2.o)
 

module/compat/.siphash.o.cmd

@@ -1828,6 +1828,8 @@ deps_compat/siphash.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 compat/siphash.o: $(deps_compat/siphash.o)
 

module/compat/.subr_tree.o.cmd

@@ -1828,6 +1828,8 @@ deps_compat/subr_tree.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 compat/subr_tree.o: $(deps_compat/subr_tree.o)
 

module/compat/.toeplitz.o.cmd

@@ -1828,6 +1828,8 @@ deps_compat/toeplitz.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
   /home/lgcosta/lab/linux_kernel/linux-7.3.4/include/crypto/hash.h \
   /home/lgcosta/lab/linux_kernel/linux-7.3.4/include/linux/crypto.h \
 

module/compat/pfk_ifq.c

@@ -217,10 +217,14 @@ pfk_pfq_enqueue(void *arg, struct mbuf *m)
 	struct pfk_pfq *pfq = arg;
 
 	if (!pfq)
+		return m;			/* caller keeps ownership */
+	/*
+	 * ifq_enqueue takes ownership of m (the scheduler may drop it); we then
+	 * drain immediately, so nothing is returned for the caller to send.
+	 */
+	ifq_enqueue(&pfq->ifq, m);
+	ifq_start(&pfq->ifq);
 	return NULL;
-	if (ifq_enqueue(&pfq->ifq, m) != 0)
-		return NULL;
-	return m;
 }
 
 static struct mbuf *
@@ -273,10 +277,11 @@ static const struct pfq_ops pfk_priq_pfq_ops = {
  * packets are still queued, but the traffic shaping curves are not yet
  * applied — recorded in the feature matrix as pending M5).
  */
-const struct ifq_ops * const ifq_hfsc_ops = &pfk_priq_ifq_ops;
+/*
-const struct ifq_ops * const ifq_fqcodel_ops = &pfk_priq_ifq_ops;
+ * M5: the real scheduler back ends (pfsrc/hfsc.c, pfsrc/fq_codel.c) define
-const struct pfq_ops * const pfq_hfsc_ops = &pfk_priq_pfq_ops;
+ * ifq_hfsc_ops/pfq_hfsc_ops/ifq_fqcodel_ops/pfq_fqcodel_ops themselves; the
-const struct pfq_ops * const pfq_fqcodel_ops = &pfk_priq_pfq_ops;
+ * priq shim above stays as the default for interfaces without a queue rule.
+ */
 
 /* ---------------- framework ---------------- */
 void
@@ -302,7 +307,36 @@ ifq_attach(struct ifqueue *ifq, const struct ifq_ops *ops, void *opsarg)
 	if (!opsarg && ops->ifqop_alloc)
 		opsarg = ops->ifqop_alloc(0, NULL);
 	ifq->ifq_ops = ops;
+	/*
+	 * The port's own helpers use ifq_softc; the staged OpenBSD schedulers
+	 * (hfsc/fq_codel) read ifq->ifq_q.  Keep both in sync or the back ends
+	 * dereference a NULL scheduler state.
+	 */
 	ifq->ifq_softc = opsarg;
+	ifq->ifq_q = opsarg;
+}
+
+/*
+ * Scheduler hand-off used by the ported back ends (hfsc/fq_codel) when they
+ * need their per-interface state.  PFLINUX: the port has no ifq mutex (the
+ * data path is a single ordered work queue), so "enter" is a plain ownership
+ * check and "leave" is a no-op - the pair exists to mirror OpenBSD's API.
+ */
+void *
+ifq_q_enter(struct ifqueue *ifq, const struct ifq_ops *ops)
+{
+	if (ifq == NULL || ops == NULL || ifq->ifq_ops != ops)
+		return NULL;			/* another scheduler owns it */
+	if (ifq->ifq_softc == NULL && ops->ifqop_alloc)
+		ifq->ifq_softc = ops->ifqop_alloc(0, NULL);
+	return ifq->ifq_softc;
+}
+
+void
+ifq_q_leave(struct ifqueue *ifq, void *q)
+{
+	(void)ifq;
+	(void)q;
 }
 
 void
@@ -312,19 +346,35 @@ ifq_destroy(struct ifqueue *ifq)
 		ifq->ifq_ops->ifqop_free(0, ifq->ifq_softc);
 	ifq->ifq_ops = NULL;
 	ifq->ifq_softc = NULL;
+	ifq->ifq_q = NULL;
 }
 
+/*
+ * ifqop_enq returns the packet the scheduler decided to DROP (possibly the one
+ * just offered), and NULL when the packet was accepted - it is NOT an error
+ * code.  Freeing the returned mbuf is this layer's job.
+ */
 int
 ifq_enqueue(struct ifqueue *ifq, struct mbuf *m)
 {
-	if (!ifq->ifq_ops || !ifq->ifq_ops->ifqop_enq)
+	struct mbuf *dm;
+
+	if (!ifq->ifq_ops || !ifq->ifq_ops->ifqop_enq) {
+		m_freem(m);
 		return EINVAL;
+	}
 	if (ifq->ifq_maxlen && ifq->ifq_len >= ifq->ifq_maxlen) {
 		ifq->ifq_errors++;
 		ifq->ifq_qdrops++;
+		m_freem(m);
 		return ENOBUFS;
 	}
-	return ifq->ifq_ops->ifqop_enq(ifq, m) ? 0 : ENOMEM;
+	dm = ifq->ifq_ops->ifqop_enq(ifq, m);
+	if (dm) {
+		ifq->ifq_qdrops++;
+		m_freem(dm);
+	}
+	return 0;
 }
 
 struct mbuf *
@@ -336,14 +386,16 @@ ifq_deq_begin(struct ifqueue *ifq)
 	if (!ifq->ifq_ops || !ifq->ifq_ops->ifqop_deq_begin)
 		return NULL;
 	m = ifq->ifq_ops->ifqop_deq_begin(ifq, &cookie);
+	ifq->ifq_dequeue = cookie;
 	return m;
 }
 
 void
 ifq_deq_commit(struct ifqueue *ifq, struct mbuf *m)
 {
-	void *cookie = NULL;
+	void *cookie = ifq->ifq_dequeue;
 
+	ifq->ifq_dequeue = NULL;
 	if (ifq->ifq_ops && ifq->ifq_ops->ifqop_deq_commit)
 		ifq->ifq_ops->ifqop_deq_commit(ifq, m, cookie);
 }
@@ -365,12 +417,52 @@ ifq_dequeue(struct ifqueue *ifq)
 	return m;
 }
 
+/*
+ * Drain the queue as far as the scheduler allows and hand each packet to the
+ * stack's output path.  OpenBSD calls if_start from softint context; here it
+ * runs either from the pf work queue (enqueue side) or from the scheduler's
+ * deferred timeout (pacing side), which is what makes hfsc's bandwidth
+ * limits take effect.
+ */
 void
 ifq_start(struct ifqueue *ifq)
 {
+	struct mbuf *m;
+	int n;
+
+	n = 0;
+	ifq->ifq_oactive = 1;
+	while ((m = ifq_dequeue(ifq)) != NULL) {
+		n++;
+		if (pfk_ifq_transmit(ifq, m))
+			break;			/* device busy: packet kept */
+	}
 	ifq->ifq_oactive = 0;
 }
 
+/*
+ * pf data-path entry: hand a packet to the interface's scheduler when one is
+ * attached (a pf `queue' rule is configured).  Returns 1 when the queue took
+ * ownership of m, 0 when the caller must keep going (priq/no scheduler).
+ */
+int
+pf_kifq_input(struct pfk_ifnet *ifp, struct mbuf *m)
+{
+	struct ifqueue *ifq;
+
+	if (ifp == NULL || m == NULL)
+		return 0;
+	if (ifp == NULL)
+		return 0;
+	ifq = &ifp->if_snd;
+	if (ifq->ifq_ops == NULL || ifq->ifq_ops == ifq_priq_ops)
+		return 0;			/* no scheduler: fast path */
+	if (ifq_enqueue(ifq, m) != 0)
+		return 1;			/* dropped by the queue: owned */
+	ifq_start(ifq);
+	return 1;
+}
+
 unsigned int
 ifq_purge(struct ifqueue *ifq)
 {
@@ -423,19 +515,3 @@ ifq_hdatalen(struct ifqueue *ifq)
 	return 0;
 }
 
-/* ---------------- scheduler back ends (milestone M5) ---------------- */
-/* The staged OpenBSD hfsc.c/fq_codel.c are not in the build yet; the priq
- * ops above are the active scheduler.  These wrappers keep pf_ioctl's
- * initialisation calls valid until M5 replaces them with the ported
- * schedulers (docs/porting-notes.md). */
-void
-hfsc_initialize(void)
-{
-	pr_info("pfk: hfsc back end pending M5 (priq active)\n");
-}
-
-void
-fq_codel_initialize(void)
-{
-	pr_info("pfk: fq_codel back end pending M5 (priq active)\n");
-}

module/compat/pfk_kern.c

@@ -47,8 +47,21 @@ pfk_printk_compat(const char *fmt, ...)
 void
 splassert_fail(int expected, int actual, const char *func)
 {
-	pr_err("splassert: %s: expected %d, got %d\n", func, expected, actual);
+	/*
-	BUG();
+	 * PFLINUX: OpenBSD's splassert() checks a per-CPU spl level, so
+	 * "assert unlocked" really means "this context did not take the lock".
+	 * The port replaced spl with rw-semas plus work queues, where
+	 * rw_status() is a global counter: an assert-unlocked in a work-queue
+	 * task fires whenever *another* CPU happens to hold the lock, which is
+	 * not a bug (the task never acquires pf_lock itself) and was turning a
+	 * harmless race into a kernel BUG() that wedged the module in
+	 * "Unloading".  Report it once instead of killing the host; the
+	 * structural invariant the BSD assert protects - purge never runs while
+	 * holding pf_lock - is upheld by the call sites (pf_purge_* are only
+	 * reached from systqmp tasks and timer work).
+	 */
+	pr_warn_ratelimited("pfk: splassert: %s: expected %d, got %d\n",
+	    func, expected, actual);
 }
 
 /* ---------------- addlog (OpenBSD kern.log) ---------------- */
@@ -619,16 +632,5 @@ if_congested(void)
 }
 
 
-/* ---------------- pflog (netlink backend, milestone M4) ---------------- */
+/* pflog_packet() lives in pfsrc/pfk_pflog.c (the /dev/pflog backend). */
-/* NPFLOG=1: the staged if_pflog.c needs the OpenBSD ifpf layer; until the
+
- * netlink-backed pflog lands (M4), pflog_packet() consumes nothing and
- * reports the missing backend at least once so logged packets are never
- * silently vanished (see docs/porting-notes.md). */
-int
-pflog_packet(struct pf_pdesc *pd, u_int8_t reason, struct pf_rule *rm,
-    struct pf_rule *am, struct pf_ruleset *ruleset, struct pf_rule *cm)
-{
-	(void)pd; (void)reason; (void)rm; (void)am; (void)ruleset; (void)cm;
-	pr_warn_once("pfk: pflog backend pending M4; packet not logged\n");
-	return (0);
-}

module/compat/pfk_mbuf.c

@@ -607,3 +607,14 @@ pfk_mbuf_exit(void)
 {
 	kmem_cache_destroy(pfk_mbuf_cache);
 }
+
+void
+ml_purge(struct mbuf_list *ml)
+{
+	struct mbuf *m;
+
+	while ((m = ml_dequeue(ml)) != NULL)
+		m_freem(m);
+	ml->ml_first = ml->ml_last = NULL;
+	ml->ml_len = 0;
+}

module/compat/pfk_xmit.c

@@ -6,6 +6,7 @@
  * ip6_route_output, tcp_v4_send_reset if needed).
  */
 #include <pfk.h>
+#include <pfk_api.h>
 
 static struct dst_entry *
 pfk_route4(__be32 daddr, __be32 saddr, int oif)
@@ -23,6 +24,26 @@ pfk_route4(__be32 daddr, __be32 saddr, int oif)
 	return &rt->dst;
 }
 
+/*
+ * Forced nexthop (route-to/dup-to/reply-to).  Linux resolves the neighbour
+ * from the *packet's* destination unless the route says it goes through a
+ * gateway (ip_neigh_for_gw() only honours rt_gw4 when rt_gw_family is set),
+ * which would ARP the final host on the gateway's link.  Look the gateway up
+ * as its own host route and mark it as the nexthop - the same pattern OVS
+ * uses after its own FIB lookups.
+ */
+static struct dst_entry *
+pfk_route4_nh(__be32 gw, int oif)
+{
+	struct rtable *rt = (struct rtable *)pfk_route4(gw, 0, oif);
+
+	if (rt == NULL)
+		return NULL;
+	rt->rt_gw_family = AF_INET;
+	rt->rt_gw4 = gw;
+	return &rt->dst;
+}
+
 #if IS_ENABLED(CONFIG_IPV6)
 static struct dst_entry *
 pfk_route6(const struct in6_addr *daddr, const struct in6_addr *saddr, int oif)
@@ -67,13 +88,16 @@ if_output_tso(struct pfk_ifnet *ifp, struct mbuf **mp, void *dst,
 	}
 	switch (sa->sa_family) {
 	case AF_INET: {
-		struct pf_ip *ip = mtod(m, struct pf_ip *);
 		__be32 gw = 0;
 
 		/* if route was forced (route-to), dst holds gw sockaddr
 		 * stored in sin_addr by pf_route; normal case dst==final */
 		gw = sintosa(dst)->sin_addr.s_addr;
-		d = pfk_route4(gw, ip->ip_src.s_addr, ifp->if_index);
+		/* PFLINUX: look the route up by destination only.  Passing the
+		 * packet's source (a transit address, not local to us) makes
+		 * ip_route_output_key fail with ENETUNREACH; ip_output keeps
+		 * the header source untouched. */
+		d = pfk_route4_nh(gw, ifp->if_index);
 		if (!d) {
 			ret = EHOSTUNREACH;
 			goto drop;
@@ -83,13 +107,11 @@ if_output_tso(struct pfk_ifnet *ifp, struct mbuf **mp, void *dst,
 	}
 #if IS_ENABLED(CONFIG_IPV6)
 	case AF_INET6: {
-		struct pf_ip6_hdr *ip6 = mtod(m, struct pf_ip6_hdr *);
 		const struct in6_addr *gw = &sin6tosa(dst)->sin6_addr;
 
 		if (!d)
-			d = pfk_route6(gw,
+			/* PFLINUX: destination-only, as for IPv4 */
-				(struct in6_addr *)&ip6->ip6_src,
+			d = pfk_route6(gw, NULL, ifp->if_index);
-				ifp->if_index);
 		if (!d) {
 			ret = EHOSTUNREACH;
 			goto drop;
@@ -114,6 +136,28 @@ if_output_tso(struct pfk_ifnet *ifp, struct mbuf **mp, void *dst,
 		ret = ENETDOWN;
 		goto drop_dst_nofree;
 	}
+	/*
+	 * The stack's output path always consumes exactly one skb reference.
+	 * Take our own so the caller's references (the hook queue ref and the
+	 * mbuf wrapper ref) stay balanced - without this the work queue's
+	 * cleanup double-frees the skb and corrupts the slab.
+	 */
+	/*
+	 * pf_route() has already evaluated the OUT ruleset (it calls
+	 * pf_test(PF_OUT) itself, exactly like OpenBSD's ip_forward).  Mark the
+	 * packet so our POST_ROUTING hook does not run the out ruleset a second
+	 * time - that re-match hits the state, re-enters pf_route and recurses
+	 * (OpenBSD stops this with PF_TAG_PROCESSED carried in the mbuf).
+	 */
+	skb->mark |= PFK_DONE_OUT;
+	/*
+	 * The port stashes its mbuf pointer in skb->cb; the IP stack uses the
+	 * same bytes as its own control block (IPCB).  Hand the packet to the
+	 * stack with a clean cb, otherwise ip_output() reads our pointer as
+	 * frag_max_size/flags and the packet is silently lost.
+	 */
+	memset(skb->cb, 0, sizeof(skb->cb));
+	skb_get(skb);
 	ret = skb_dst(skb)->output(&init_net, NULL, skb);
 	/* output consumed skb (and wrapper); avoid double free */
 	if (ret == 0) {
@@ -226,3 +270,40 @@ pfk_if_output(struct pfk_ifnet *ifp, struct mbuf *m, void *dst,
 		m_freem(mp);
 	return err;
 }
+
+/*
+ * Transmit one queued packet.  Called from ifq_start() - i.e. from the pf
+ * work queue or from a scheduler's deferred timeout - and always consumes the
+ * mbuf.  Returns 0 (keep draining); a scheduler that wants to defer does so by
+ * simply not returning a packet from ifqop_deq_begin().
+ */
+int
+pfk_ifq_transmit(struct ifqueue *ifq, struct mbuf *m)
+{
+	struct sk_buff *skb = m_to_skb(m);
+	struct dst_entry *d;
+
+	(void)ifq;
+	if (skb == NULL) {
+		m_freem(m);
+		return 0;
+	}
+	d = skb_dst(skb);
+	if (d == NULL || IS_ERR(d) || d->dev == NULL || !netif_running(d->dev)) {
+		m_freem(m);
+		return 0;
+	}
+	skb->dev = d->dev;
+	skb_reset_network_header(skb);
+	if (skb->protocol == 0)
+		skb->protocol = htons(ip_hdr(skb)->protocol == IPPROTO_IPV6 ?
+		    ETH_P_IPV6 : ETH_P_IP);
+	/* the stack owns skb->cb (IPCB) from here; pf already tested it */
+	memset(skb->cb, 0, sizeof(skb->cb));
+	skb->mark |= PFK_DONE_OUT;
+	skb_get(skb);		/* the output path consumes one reference */
+	m->skb = NULL;		/* the mbuf's reference moves to the stack */
+	m_freem(m);
+	d->output(&init_net, NULL, skb);
+	return 0;
+}

module/include/compat/net/ifq.h

@@ -27,6 +27,7 @@ struct ifqueue {
 	struct mbuf_list	*ifq_free;	/* free list (unused in the port) */
 	unsigned int		 ifq_len;
 	unsigned int		 ifq_oactive;
+	void			*ifq_dequeue;	/* cookie between deq_begin/commit */
 
 	/* statistics */
 	u_int64_t		 ifq_packets;

module/include/net/fq_codel.h

@@ -0,0 +1,51 @@
+/* $OpenBSD: fq_codel.h,v 1.4 2020/06/18 23:29:59 dlg Exp $ */
+
+/*
+ * Copyright (c) 2017 Mike Belopuhov
+ *
+ * Permission to use, copy, modify, and distribute this software for any
+ * purpose with or without fee is hereby granted, provided that the above
+ * copyright notice and this permission notice appear in all copies.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+ * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+ * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+ * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+ * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+ */
+
+#ifndef _NET_FQ_CODEL_H_
+#define _NET_FQ_CODEL_H_
+
+/* compatible with hfsc_pktcntr */
+struct fqcodel_pktcntr {
+	uint64_t		packets;
+	uint64_t		bytes;
+};
+
+struct fqcodel_stats {
+	struct fqcodel_pktcntr 	xmit_cnt;
+	struct fqcodel_pktcntr 	drop_cnt;
+
+	uint32_t		qlength;
+	uint32_t		qlimit;
+
+	uint32_t		flows;
+	uint32_t		_unused;   /* padding */
+
+	uint32_t		target;
+	uint32_t		interval;
+
+	/* the values below are used to calculate standard deviation */
+	uint64_t		delaysum;  /* sum of delays, us */
+	uint64_t		delaysumsq;/* sum of squared delays, us */
+};
+
+#ifdef _KERNEL
+extern const struct ifq_ops * const ifq_fqcodel_ops;
+extern const struct pfq_ops * const pfq_fqcodel_ops;
+#endif	/* _KERNEL */
+
+#endif	/* _NET_FQ_CODEL_H_ */

module/include/net/hfsc.h

@@ -0,0 +1,122 @@
+/*	$OpenBSD: hfsc.h,v 1.15 2026/03/19 14:59:05 sthen Exp $	*/
+
+/*
+ * Copyright (c) 2012-2013 Henning Brauer <henning@openbsd.org>
+ * Copyright (c) 1997-1999 Carnegie Mellon University. All Rights Reserved.
+ *
+ * Permission to use, copy, modify, and distribute this software and
+ * its documentation is hereby granted (including for commercial or
+ * for-profit use), provided that both the copyright notice and this
+ * permission notice appear in all copies of the software, derivative
+ * works, or modified versions, and any portions thereof.
+ *
+ * THIS SOFTWARE IS EXPERIMENTAL AND IS KNOWN TO HAVE BUGS, SOME OF
+ * WHICH MAY HAVE SERIOUS CONSEQUENCES.  CARNEGIE MELLON PROVIDES THIS
+ * SOFTWARE IN ITS ``AS IS'' CONDITION, AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+ * DISCLAIMED.  IN NO EVENT SHALL CARNEGIE MELLON UNIVERSITY BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+ * USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ *
+ * Carnegie Mellon encourages (but does not require) users of this
+ * software to return any improvements or extensions that they make,
+ * and to grant Carnegie Mellon the rights to redistribute these
+ * changes without encumbrance.
+ */
+#ifndef _HFSC_H_
+#define	_HFSC_H_
+
+/* hfsc class flags */
+#define	HFSC_DEFAULTCLASS	0x1000	/* default class */
+
+struct hfsc_pktcntr {
+	u_int64_t	packets;
+	u_int64_t	bytes;
+};
+
+#define	PKTCNTR_INC(cntr, len)	\
+	do { (cntr)->packets++; (cntr)->bytes += len; } while (0)
+
+struct hfsc_sc {
+	u_int64_t	m1;	/* slope of the first segment in bits/sec */
+	u_int64_t	m2;	/* slope of the second segment in bits/sec */
+	u_int		d;	/* the x-projection of the first segment in msec */
+};
+
+/* special class handles */
+#define	HFSC_ROOT_CLASS		0x10000
+#define	HFSC_DEFAULT_CLASSES	64
+#define	HFSC_MAX_CLASSES	65535
+
+/* service curve types */
+#define	HFSC_REALTIMESC		1
+#define	HFSC_LINKSHARINGSC	2
+#define	HFSC_UPPERLIMITSC	4
+#define	HFSC_DEFAULTSC		(HFSC_REALTIMESC|HFSC_LINKSHARINGSC)
+
+struct hfsc_class_stats {
+	struct hfsc_pktcntr	xmit_cnt;
+	struct hfsc_pktcntr	drop_cnt;
+
+	u_int			qlength;
+	u_int			qlimit;
+	u_int			period;
+
+	u_int			class_id;
+	u_int32_t		class_handle;
+	struct hfsc_sc		rsc;
+	struct hfsc_sc		fsc;
+	struct hfsc_sc		usc;	/* upper limit service curve */
+
+	u_int64_t		total;	/* total work in bytes */
+	u_int64_t		cumul;	/* cumulative work in bytes
+					   done by real-time criteria */
+	u_int64_t		d;		/* deadline */
+	u_int64_t		e;		/* eligible time */
+	u_int64_t		vt;		/* virtual time */
+	u_int64_t		f;		/* fit time for upper-limit */
+
+	/* info helpful for debugging */
+	u_int64_t		initvt;		/* init virtual time */
+	u_int64_t		vtoff;		/* cl_vt_ipoff */
+	u_int64_t		cvtmax;		/* cl_maxvt */
+	u_int64_t		myf;		/* cl_myf */
+	u_int64_t		cfmin;		/* cl_mincf */
+	u_int64_t		cvtmin;		/* cl_mincvt */
+	u_int64_t		myfadj;		/* cl_myfadj */
+	u_int64_t		vtadj;		/* cl_vtadj */
+	u_int64_t		cur_time;
+	u_int32_t		machclk_freq;
+
+	u_int			vtperiod;	/* vt period sequence no */
+	u_int			parentperiod;	/* parent's vt period seqno */
+	int			nactive;	/* number of active children */
+
+	/* red and rio related info */
+	int			qtype;
+/*	struct redstats		red[3]; */
+};
+
+#ifdef _KERNEL
+struct ifnet;
+struct ifqueue;
+struct pf_queuespec;
+struct hfsc_if;
+
+extern const struct ifq_ops * const ifq_hfsc_ops;
+extern const struct pfq_ops * const pfq_hfsc_ops;
+
+#define	HFSC_ENABLED(ifq)	((ifq)->ifq_ops == ifq_hfsc_ops)
+#define	HFSC_DEFAULT_QLIMIT	50
+
+void		 hfsc_initialize(void);
+
+#endif /* _KERNEL */
+#endif /* _HFSC_H_ */

module/include/pfk.h

@@ -1203,4 +1203,60 @@ int	pf_anchor_stack_pop(struct pf_ruleset **, struct pf_rule **,
 struct inpcb;
 void	pf_remove_divert_state(struct inpcb *);
 
+/* pflog backend (/dev/pflog) - 'L' keeps the namespace clear of DIOC* ('P') */
+struct pfk_pflog_stat {
+	u_int64_t	queued;		/* records currently queued	*/
+	u_int64_t	bytes;		/* bytes queued			*/
+	u_int64_t	dropped;	/* records dropped (queue full)	*/
+	u_int64_t	seq;		/* records produced since load	*/
+};
+#define PFKIOCGETPFLOGSTAT	_IOR('L', 1, struct pfk_pflog_stat)
+
+
+/*
+ * Scheduler back-end ABI: the very headers OpenBSD ships (net/hfsc.h,
+ * net/fq_codel.h), vendored into the port so that pfctl and pf.ko agree on the
+ * stats layouts and on the class-handle/service-curve constants.  (Hand-rolled
+ * approximations of these structs made DIOCGETQSTATS fail with EINVAL, because
+ * pfctl's sizeof(struct hfsc_class_stats) did not match the kernel's.)
+ */
+#ifndef nsecuptime
+#define nsecuptime()	 getnsecuptime()
+#endif
+
+#include <net/hfsc.h>
+#include <net/fq_codel.h>
+
+/* OpenBSD's global nanoseconds-per-clocktick (HFSC_CLK_PER_TICK).  The port
+ * has no clock_rate machinery, so derive it from the kernel's HZ. */
+#ifndef tick_nsec
+#define tick_nsec		(1000000000LL / HZ)
+#endif
+
+/* ifq scheduler hand-off (see pfsrc/ifq.c): enter returns the scheduler's
+ * queue state when it owns the interface queue, NULL otherwise. */
+void	*ifq_q_enter(struct ifqueue *, const struct ifq_ops *);
+void	 ifq_q_leave(struct ifqueue *, void *);
+int	 pfk_ifq_transmit(struct ifqueue *, struct mbuf *);
+int	 pf_kifq_input(struct pfk_ifnet *, struct mbuf *);
+
+
+/*
+ * PFLINUX notes on the scheduler back ends:
+ *
+ * - splnet()/splx(): OpenBSD raises the software interrupt level around
+ *   class-table surgery.  This port runs the whole data path in one ordered
+ *   work queue and guards the queue with ifq_mtx, so these are no-ops; they
+ *   are kept as markers of the critical regions.
+ * - KERNEL_ASSERT_LOCKED(): the schedulers are only reached from ifq/pf paths
+ *   that already hold the queue mutex; asserted structurally, not by spl.
+ * - HFSC_ENABLED(), PKTCNTR_INC(), the class-handle constants and the stats
+ *   layouts all come from the vendored net/hfsc.h / net/fq_codel.h above, so
+ *   kernel and pfctl cannot drift apart.
+ */
+#define splnet()		0
+#define splx(s)			((void)(s))
+#define KERNEL_ASSERT_LOCKED()	do { } while (0)
+
 #endif /* _PFK_H */
+

module/include/pfk_api.h

@@ -7,11 +7,15 @@
 long pfk_ioctl(struct file *f, unsigned int cmd, unsigned long addr);
 int  pfk_enable(int on);
 void pfk_status(struct pf_status *s);
-#define PFK_SKIP_PF	0x50460040u	/* skb->mark: reinjected, bypass pf */
+#define PFK_SKIP_PF	0x00000004u	/* skb->mark: reinjected, bypass pf */
+#define PFK_DONE_IN	0x00000001u	/* skb->mark bit: PF_IN pass completed */
+#define PFK_DONE_OUT	0x00000002u	/* skb->mark bit: PF_OUT pass completed */
 
 int  pfk_hooks_register(void);
 void pfk_hooks_unregister(void);
 /* AF_DIVERT family (module-internal socket protocol, PF_DIVERT=32) */
+int  pfk_pflog_register(void);
+void pfk_pflog_unregister(void);
 int  pfk_divert_sock_register(void);
 void pfk_divert_sock_unregister(void);
 void divert_packet(struct mbuf *, int, u_int16_t);

module/include/pfk_if.h

@@ -74,6 +74,79 @@ ml_dequeue(struct mbuf_list *ml)
 	return m;
 }
 
+
+/* BSD <net/if_var.h> mbuf_list helpers (net/if_var.h ml_*) */
+static inline void
+ml_init(struct mbuf_list *ml)
+{
+	ml->ml_first = ml->ml_last = NULL;
+	ml->ml_len = 0;
+}
+
+static inline void
+ml_enqueue(struct mbuf_list *ml, struct mbuf *m)
+{
+	m->m_next = NULL;
+	if (ml->ml_last == NULL)
+		ml->ml_first = m;
+	else
+		ml->ml_last->m_next = m;
+	ml->ml_last = m;
+	ml->ml_len++;
+}
+
+static inline struct mbuf *
+ml_dequeue_tail(struct mbuf_list *ml)
+{
+	struct mbuf *m, *prev = NULL;
+
+	if (ml->ml_first == NULL)
+		return NULL;
+	for (m = ml->ml_first; m->m_next; m = m->m_next)
+		prev = m;
+	if (prev == NULL)			/* single element */
+		ml->ml_first = NULL;
+	else
+		prev->m_next = NULL;
+	ml->ml_last = prev;
+	ml->ml_len--;
+	m->m_next = NULL;
+	return m;
+}
+
+/* append every packet in m2 to m1 (m2 becomes empty) */
+static inline void
+ml_enlist(struct mbuf_list *m1, struct mbuf_list *m2)
+{
+	if (m2->ml_first == NULL)
+		return;
+	if (m1->ml_last == NULL)
+		m1->ml_first = m2->ml_first;
+	else
+		m1->ml_last->m_next = m2->ml_first;
+	m1->ml_last = m2->ml_last;
+	m1->ml_len += m2->ml_len;
+	m2->ml_first = m2->ml_last = NULL;
+	m2->ml_len = 0;
+}
+
+static inline unsigned int
+ml_len(const struct mbuf_list *ml)
+{
+	return (unsigned int)ml->ml_len;
+}
+
+static inline int
+ml_empty(const struct mbuf_list *ml)
+{
+	return ml->ml_first == NULL;
+}
+
+void	ml_purge(struct mbuf_list *ml);	/* compat/pfk_mbuf.c (needs m_freem) */
+
+#define MBUF_LIST_FIRST(_ml)		((_ml)->ml_first)
+#define MBUF_LIST_INITIALIZER()		{ NULL, NULL, 0 }
+
 /* egress interface (default-route dev), maintained on fib/net events */
 struct pfk_ifnet *pfk_egress_if(void);
 void pfk_egress_refresh(void);

module/include/pfk_mbuf.h

@@ -73,7 +73,7 @@ struct pkthdr {
 	u_int32_t		 ph_flowid;
 	u_int			 ph_ifidx;
 	u_int			 ph_rtableid;
-	struct pfk_timeval	 ph_timestamp;
+	int64_t			 ph_timestamp;	/* OpenBSD 8.0: nanoseconds */
 	struct pfk_ifnet	*ifp;
 	struct pfk_ifnet	*rcvif;
 	u_short			 ph;		/* tso segs: unused, kept as pad */

module/pfsrc/.divert_sock.o.cmd

@@ -1828,6 +1828,8 @@ deps_pfsrc/divert_sock.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
   include/pfk_api.h \
 
 pfsrc/divert_sock.o: $(deps_pfsrc/divert_sock.o)

module/pfsrc/.pf.o.cmd

@@ -1828,6 +1828,8 @@ deps_pfsrc/pf.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 pfsrc/pf.o: $(deps_pfsrc/pf.o)
 

module/pfsrc/.pf_if.o.cmd

@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_if.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 pfsrc/pf_if.o: $(deps_pfsrc/pf_if.o)
 

module/pfsrc/.pf_ioctl.o.cmd

@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_ioctl.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 pfsrc/pf_ioctl.o: $(deps_pfsrc/pf_ioctl.o)
 

module/pfsrc/.pf_lb.o.cmd

@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_lb.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 pfsrc/pf_lb.o: $(deps_pfsrc/pf_lb.o)
 

module/pfsrc/.pf_norm.o.cmd

@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_norm.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 pfsrc/pf_norm.o: $(deps_pfsrc/pf_norm.o)
 

module/pfsrc/.pf_osfp.o.cmd

@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_osfp.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 pfsrc/pf_osfp.o: $(deps_pfsrc/pf_osfp.o)
 

module/pfsrc/.pf_ruleset.o.cmd

@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_ruleset.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 pfsrc/pf_ruleset.o: $(deps_pfsrc/pf_ruleset.o)
 

module/pfsrc/.pf_syncookies.o.cmd

@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_syncookies.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 pfsrc/pf_syncookies.o: $(deps_pfsrc/pf_syncookies.o)
 

module/pfsrc/.pf_table.o.cmd

@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_table.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
 
 pfsrc/pf_table.o: $(deps_pfsrc/pf_table.o)
 

module/pfsrc/.pfk_hooks.o.cmd

@@ -1828,6 +1828,8 @@ deps_pfsrc/pfk_hooks.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
   include/pfk_api.h \
 
 pfsrc/pfk_hooks.o: $(deps_pfsrc/pfk_hooks.o)

module/pfsrc/.pfk_mod.o.cmd

@@ -1828,6 +1828,8 @@ deps_pfsrc/pfk_mod.o := \
   include/compat/sys/rwlock.h \
   include/compat/sys/mutex.h \
   include/compat/sys/pclock.h \
+  include/net/hfsc.h \
+  include/net/fq_codel.h \
   include/pfk_api.h \
 
 pfsrc/pfk_mod.o: $(deps_pfsrc/pfk_mod.o)

module/pfsrc/fq_codel.c

@@ -332,7 +332,9 @@ control_law(struct codel *cd, struct codel_params *cp, int64_t rts)
 	unsigned int idx;
 
 	KASSERT(cd->drops > 0);
-	idx = min(cd->drops - 1, nitems(codel_intervals) - 1);
+	/* PFLINUX: Linux' min() type-checks its arguments; nitems() is size_t */
+	idx = min_t(unsigned int, cd->drops - 1,
+	    (unsigned int)(nitems(codel_intervals) - 1));
 	cd->next = rts + cp->intervals[idx];
 }
 

module/pfsrc/hfsc.c

@@ -335,9 +335,9 @@ hfsc_class_purge(struct hfsc_class *cl, struct mbuf_list *ml)
 }
 
 static inline u_int
-hfsc_more_slots(u_int current)
+hfsc_more_slots(u_int cur)	/* PFLINUX: 'current' is a kernel macro */
 {
-	u_int want = current * 2;
+	u_int want = cur * 2;
 
 	return (want > HFSC_MAX_CLASSES ? HFSC_MAX_CLASSES : want);
 }
@@ -481,7 +481,16 @@ hfsc_pf_qstats(struct pf_queuespec *q, void *ubuf, int *nbytes)
 	if (hif == NULL)
 		return (EBADF);
 
-	if ((cl = hfsc_clh2cph(hif, q->qid)) == NULL) {
+	/*
+	 * A root queue's class is created with handle HFSC_ROOT_CLASS | qid
+	 * (see hfsc_pf_addqueue), so looking the queue up by its plain pf qid
+	 * only finds non-root classes: fall back to the root handle, otherwise
+	 * "pfctl -s queue" fails with EINVAL on the first (root) queue.
+	 */
+	if ((cl = hfsc_clh2cph(hif, q->qid)) == NULL &&
+	    (q->flags & PFQS_ROOTCLASS))
+		cl = hfsc_clh2cph(hif, HFSC_ROOT_CLASS | q->qid);
+	if (cl == NULL) {
 		ifq_q_leave(&ifp->if_snd, hif);
 		return (EINVAL);
 	}

module/pfsrc/pf.c

@@ -8283,11 +8283,6 @@ pf_test(sa_family_t af, int fwdir, struct pfk_ifnet *ifp, struct mbuf **m0)
 
 	if (!pf_status.running)
 		return (PF_PASS);
-	if (m0 && *m0 && (*m0)->m_pkthdr.len >= 20 &&
-	    ((*m0)->m_data[9] == IPPROTO_ICMP))
-		pr_info_ratelimited("pfk: T1 enter dir=%d ifp=%px kif=%px\n",
-		    fwdir, ifp, ifp ? ifp->if_pf_kif : NULL);
-
 #if NCARP > 0
 	if (ifp->if_type == IFT_CARP) {
 		struct pfk_ifnet *ifp0;

module/pfsrc/pf_if.c

@@ -90,6 +90,17 @@ pfi_kif_alloc(const char *kif_name, int mflags)
 		kif->pfik_flags |= PFI_IFLAG_ANY;
 		kif->pfik_flags_new |= PFI_IFLAG_ANY;
 	}
+	/*
+	 * PFLINUX: OpenBSD creates the "ingress" and "egress" interface
+	 * groups (ifgroups) automatically and both contain every interface;
+	 * pf's direction (in/out ruleset) carries the actual meaning.  Linux
+	 * has no ifgroups, so emulate the groups as "any interface".
+	 */
+	if (!strcmp(kif->pfik_name, "ingress") ||
+	    !strcmp(kif->pfik_name, "egress")) {
+		kif->pfik_flags |= PFI_IFLAG_ANY;
+		kif->pfik_flags_new |= PFI_IFLAG_ANY;
+	}
 
 	return (kif);
 }

module/pfsrc/pf_ioctl.c

@@ -662,6 +662,11 @@ pf_create_queues(void)
 		}
 
 		qif->disc = qif->pfqops->pfq_alloc(ifp);
+		if (qif->disc == NULL) {
+			error = ENOBUFS;
+			free(qif, M_PF, sizeof(*qif));
+			goto error;
+		}
 
 		qif->next = list;
 		list = qif;

module/pfsrc/pfk_hooks.c

@@ -35,8 +35,6 @@ struct pfk_pkt {
 	int			dir;
 };
 
-#define PFK_DONE_IN	0x50460010u	/* skb->mark bit: PF_IN pass completed */
-#define PFK_DONE_OUT	0x50460020u	/* skb->mark bit: PF_OUT pass completed */
 
 static void
 pfk_pkt_reinject(struct pfk_pkt *pk)
@@ -71,14 +69,119 @@ pfk_pkt_reinject(struct pfk_pkt *pk)
 	kfree(pk);
 }
 
+
+/*
+ * Reverse NAT: pf rewrote the destination of an inbound packet (the reply
+ * of a translated flow).  OpenBSD re-routes such a packet in ip_input()
+ * (PF_TAG_REROUTE); Linux has no such layer, so re-route it here and push
+ * it out through the new egress device.
+ */
+static void
+pfk_reroute(struct sk_buff *skb)
+{
+	struct flowi4	 fl4 = { };
+	struct rtable	*rt;
+	struct iphdr	*iph = ip_hdr(skb);
+
+	fl4.daddr = iph->daddr;
+	/*
+	 * Leave the source empty: the packet keeps the original source
+	 * (the peer's address), but the kernel would reject a route lookup
+	 * with a non-local source (ENETUNREACH).
+	 */
+	fl4.saddr = 0;
+	fl4.flowi4_oif = 0;
+	fl4.flowi4_mark = 0;
+	fl4.flowi4_scope = RT_SCOPE_UNIVERSE;
+
+	rt = ip_route_output_key(&init_net, &fl4);
+	if (IS_ERR(rt) || rt == NULL) {
+		kfree_skb(skb);
+		return;
+	}
+	skb_dst_set(skb, &rt->dst);
+	skb->dev = rt->dst.dev;
+	skb_reset_network_header(skb);
+	/* the stack owns skb->cb (IPCB) from here - see pfk_xmit.c */
+	memset(skb->cb, 0, sizeof(skb->cb));
+	rt->dst.output(&init_net, skb->sk, skb);
+}
+
+/*
+ * PFLINUX: pf rewrote the IPv4 addresses of the packet (NAT).  OpenBSD's
+ * ip_output() recomputes the IP header checksum on every packet it sends;
+ * Linux does not, so fix it up here after pf touched the header.
+ */
+/*
+ * pf rewrote the addresses of this packet.  OpenBSD recomputes the IP header
+ * checksum in ip_output() and the transport checksum is patched incrementally
+ * by pf - but that only works on a *materialized* checksum, and VM-to-VM
+ * traffic routinely arrives with the transport checksum still uncomputed
+ * (the sender's offload never completed on the tap).  So recompute from
+ * scratch here:
+ *
+ *   - GSO segments: hand the job back to the NIC (CHECKSUM_PARTIAL with the
+ *     transport offset); computing in software would only cover the first
+ *     segment and silently corrupt the rest (this is what kills bulk TCP).
+ *   - everything else: compute the transport checksum over the new
+ *     pseudo-header in software, then the IP header checksum.
+ */
+static void
+pfk_fix_csum_after_translate(struct sk_buff *skb)
+{
+	struct iphdr *iph;
+	unsigned int hlen, len;
+	__sum16 *csum;
+	__u8 proto;
+
+	if (!pskb_may_pull(skb, sizeof(struct iphdr)))
+		return;
+	iph = ip_hdr(skb);
+	hlen = iph->ihl * 4;
+	proto = iph->protocol;
+	len = skb->len - hlen;
+
+	if (proto == IPPROTO_TCP || proto == IPPROTO_UDP) {
+		if (!pskb_may_pull(skb, hlen + (proto == IPPROTO_TCP ?
+		    sizeof(struct tcphdr) : sizeof(struct udphdr))))
+			return;
+		csum = (__sum16 *)(skb->data + hlen +
+		    (proto == IPPROTO_TCP ? offsetof(struct tcphdr, check)
+					 : offsetof(struct udphdr, check)));
+		if (skb_is_gso(skb)) {
+			skb->csum_start = (unsigned short)
+			    (((unsigned char *)skb->data + hlen) - skb->head);
+			skb->csum_offset = proto == IPPROTO_TCP ?
+			    offsetof(struct tcphdr, check) :
+			    offsetof(struct udphdr, check);
+			skb->ip_summed = CHECKSUM_PARTIAL;
+		} else {
+			*csum = 0;
+			*csum = csum_tcpudp_magic(iph->saddr, iph->daddr, len,
+			    proto, skb_checksum(skb, hlen, len, 0));
+			skb->ip_summed = CHECKSUM_NONE;
+		}
+	}
+
+	iph = ip_hdr(skb);	/* pskb_may_pull may have moved the data */
+	iph->check = 0;
+	iph->check = csum_fold(csum_partial((__u8 *)iph, iph->ihl * 4, 0));
+}
+
 static void
 pfk_pkt_work(struct work_struct *w)
 {
 	struct pfk_pkt *pk = container_of(w, struct pfk_pkt, work);
 	struct mbuf *m, *m0;
 	struct sk_buff *orig;
+	__be32 orig_dst = 0, orig_src = 0;
 	int action;
 
+	if (pk->af == AF_INET && pskb_may_pull(pk->skb, sizeof(struct iphdr))) {
+		orig_dst = ip_hdr(pk->skb)->daddr;
+		orig_src = ip_hdr(pk->skb)->saddr;
+	}
+
 	m = pfk_m_wrap_skb(pk->skb, pk->af);
 	if (!m) {
 		pfk_pkt_reinject(pk);	/* like "pf not consulted" */
@@ -93,11 +196,43 @@ pfk_pkt_work(struct work_struct *w)
 	 * and packet walk run concurrently from here on — documented. */
 	action = pf_test(pk->af, pk->dir, pk->ifp, &m);
 
+	if (pk->af == AF_INET && orig_src != 0 &&
+	    pskb_may_pull(pk->skb, sizeof(struct iphdr)) &&
+	    (ip_hdr(pk->skb)->saddr != orig_src ||
+	    ip_hdr(pk->skb)->daddr != orig_dst))
+		pfk_fix_csum_after_translate(pk->skb);
+
 
 	if (m == m0) {
+		/*
+		 * Queueing: when a pf `queue' rule attached a scheduler to the
+		 * egress interface, hand the packet to it instead of sending it
+		 * straight out - this is what makes bandwidth limits real.
+		 */
+		if (action == PF_PASS && pk->dir == PF_OUT &&
+		    pskb_may_pull(pk->skb, sizeof(struct iphdr)) &&
+		    skb_dst(pk->skb) && !IS_ERR(skb_dst(pk->skb)) &&
+		    pf_kifq_input(skb_dst(pk->skb)->dev ?
+			pfk_ifp_bydev(skb_dst(pk->skb)->dev) : NULL, m)) {
+			orig = pk->skb;
+			pk->skb = NULL;		/* the queue owns the packet */
+			kfree_skb(orig);
+			kfree(pk);
+			return;
+		}
 		/* same buffer, possibly rewritten in place: our wrapper
 		 * held one extra skb reference — release it */
 		m_freem(m);
+		if (action == PF_PASS && orig_dst != 0 &&
+		    pk->af == AF_INET && pk->dir == PF_IN &&
+		    pskb_may_pull(pk->skb, sizeof(struct iphdr)) &&
+		    ip_hdr(pk->skb)->daddr != orig_dst) {
+			/* pf translated the destination (reverse NAT):
+			 * re-route instead of delivering locally */
+			pfk_reroute(pk->skb);
+			kfree(pk);
+			return;
+		}
 		if (action == PF_PASS) {
 			pfk_pkt_reinject(pk);
 			return;
@@ -138,17 +273,25 @@ pfk_enqueue_verdict(int af, struct sk_buff *skb,
 
 	switch (st->hook) {
 	case NF_INET_PRE_ROUTING:
+		/* OpenBSD evaluates the IN ruleset in ip_input(); every
+		 * arriving packet passes it. */
 		dir = PF_IN;
 		break;
+	case NF_INET_FORWARD:
+		/* OpenBSD evaluates the OUT ruleset of a forwarded packet in
+		 * ip_forward() (pf_test(PF_FWD)); the IN ruleset was already
+		 * applied at input time.  Without this, transit traffic never
+		 * sees the "out" rules and state matching for replies is
+		 * impossible. */
+		dir = PF_OUT;
+		break;
 	case NF_INET_LOCAL_OUT:
 	case NF_INET_POST_ROUTING:
 		dir = PF_OUT;
 		break;
 	default:
-		return NF_ACCEPT;	/* LOCAL_IN/FORWARD */
+		return NF_ACCEPT;	/* LOCAL_IN */
 	}
-	    af, skb->len, ntohs(skb->protocol), skb->dev ? skb->dev->name : "?",
-	    skb->mark);
 	if (skb->mark & PFK_SKIP_PF)
 		return NF_ACCEPT;	/* reinjected by divert: don't re-filter */
 	if (skb->mark & (dir == PF_IN ? PFK_DONE_IN : PFK_DONE_OUT))
@@ -162,7 +305,8 @@ pfk_enqueue_verdict(int af, struct sk_buff *skb,
 	if (dir == PF_IN)
 		ifp = pfk_ifp_bydev(st->in ? st->in : skb->dev);
 	else
-		ifp = pfk_ifp_bydev(skb->dev ? skb->dev : st->out);
+		ifp = pfk_ifp_bydev(st->out ? st->out :
+		    (skb->dev ? skb->dev : NULL));
 	if (!ifp)
 		return NF_ACCEPT;
 

module/pfsrc/pfk_mod.c

@@ -94,6 +94,8 @@ static struct miscdevice pfk_dev = {
 extern void pf_status_init(void);
 extern int pfattach(int);
 extern int pfk_divert_sock_register(void);
+extern int pfk_pflog_register(void);
+extern void pfk_pflog_unregister(void);
 extern void pfk_divert_sock_unregister(void);
 
 static void
@@ -133,9 +135,12 @@ static int __init pfk_init(void)
 	pfattach(0);
 	pfk_if_set_event_cb(pfk_pfi_event);
 	pfk_if_iterate(pfk_pfi_attach_one);
-	err = pfk_divert_sock_register();
+	err = pfk_pflog_register();
 	if (err)
 		goto err_if;
+	err = pfk_divert_sock_register();
+	if (err)
+		goto err_pflog;
 	err = pfk_hooks_register();
 	if (err)
 		goto err_divert;
@@ -147,6 +152,8 @@ static int __init pfk_init(void)
 err_hooks:
 	pfk_divert_sock_unregister();
 err_divert:
+	pfk_pflog_unregister();
+err_pflog:
 	pfk_hooks_unregister();
 err_if:
 	pfk_if_exit();
@@ -166,6 +173,7 @@ static void __exit pfk_exit(void)
 	 * workqueues non-empty forever (observed rmmod hang). */
 	misc_deregister(&pfk_dev);
 	pfk_divert_sock_unregister();
+	pfk_pflog_unregister();
 	pfk_net_lock();
 	pf_status.running = 0;
 	pfk_net_unlock();

module/pfsrc/pfk_pflog.c

@@ -0,0 +1,310 @@
+/*
+ * PF-on-Linux: pflog backend (ISC 2026)
+ *
+ * OpenBSD logs matched packets to the pflog(4) *network interface*, which
+ * pflogd(8) reads through bpf(4).  Linux has neither if_pflog-style virtual
+ * interfaces exposed to bpf nor a DLT_PFLOG consumer, so this port offers the
+ * same record format over a character device:
+ *
+ *	/dev/pflog   (0600)  - read() returns one record at a time:
+ *			       struct pflog_hdr followed by the captured
+ *			       packet bytes (truncated to PFLOG_COPY_SIZE).
+ *
+ * poll() reports readable when a record is queued.  A single consumer is the
+ * real use case (pflogd), which is why a chardev - not netlink multicast -
+ * was chosen; the deviation is documented in docs/porting-notes.md.
+ *
+ * The records are byte-compatible with OpenBSD's pflog(4) so tcpdump/wireshark
+ * read the pcap with linktype DLT_PFLOG (117).
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/miscdevice.h>
+#include <linux/fs.h>
+#include <linux/mm.h>
+#include <linux/poll.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/wait.h>
+#include <linux/timekeeping.h>
+#include <linux/uaccess.h>
+#include <linux/netdevice.h>
+
+#include <pfk.h>
+#include <pfk_api.h>
+
+/* OpenBSD sys/net/pflog.h */
+#define PFLOG_IFNAMSIZ		16
+#define PFLOG_RULESET_NAME_SIZE	16
+#define PFLOG_HDRLEN		64	/* sizeof(struct pfk_pflog_hdr) */
+#define PFLOG_COPY_SIZE		160
+
+/* byte-compatible with OpenBSD's <net/pflog.h> (sizeof == 64) */
+struct pfk_pflog_hdr {
+	struct {
+		s64		tv_sec;
+		s64		tv_usec;
+	} pflog_ts;
+	u8			pflog_af;
+	u8			pflog_reason;
+	u8			pflog_ifname[PFLOG_IFNAMSIZ];
+	u8			pflog_ruleset[PFLOG_RULESET_NAME_SIZE];
+	u8			pflog_nr;
+	u8			pflog_anc;
+	u8			pflog_flags;
+	u8			pflog_pad[9];
+	u16			pflog_len;	/* captured packet length */
+} __packed;
+
+struct pfk_pflog_rec {
+	struct list_head	list;
+	size_t			len;	/* header + payload */
+	char			data[];	/* pfk_pflog_hdr, then the packet */
+};
+
+#define PFK_PFLOG_MAX_QUEUE	256u
+#define PFK_PFLOG_MAX_BYTES	(1u << 20)
+
+static LIST_HEAD(pflog_list);
+static DEFINE_SPINLOCK(pflog_lock);
+static DECLARE_WAIT_QUEUE_HEAD(pflog_wq);
+static unsigned long pflog_queued, pflog_bytes, pflog_dropped;
+static u64 pflog_seq;
+
+/*
+ * Queue one record.  Called from the packet path (process context here: the
+ * data path runs in pfk_netisr_wq), so a GFP_ATOMIC allocation and a bounded
+ * queue keep it safe even if a future caller appears in softirq context.
+ */
+static void
+pfk_pflog_put(struct pfk_pflog_hdr *hdr, const void *pkt, size_t pktlen)
+{
+	struct pfk_pflog_rec *rec;
+	size_t len = sizeof(*hdr) + pktlen;
+	unsigned long flags;
+
+	if (READ_ONCE(pflog_queued) >= PFK_PFLOG_MAX_QUEUE ||
+	    READ_ONCE(pflog_bytes) + len > PFK_PFLOG_MAX_BYTES) {
+		pflog_dropped++;
+		return;
+	}
+
+	rec = kmalloc(struct_size(rec, data, len), GFP_ATOMIC);
+	if (rec == NULL) {
+		pflog_dropped++;
+		return;
+	}
+	rec->len = len;
+	memcpy(rec->data, hdr, sizeof(*hdr));
+	memcpy(rec->data + sizeof(*hdr), pkt, pktlen);
+
+	spin_lock_irqsave(&pflog_lock, flags);
+	list_add_tail(&rec->list, &pflog_list);
+	pflog_queued++;
+	pflog_bytes += len;
+	pflog_seq++;
+	spin_unlock_irqrestore(&pflog_lock, flags);
+
+	wake_up_interruptible(&pflog_wq);
+}
+
+/* copy the leading bytes of the packet (the mbuf chain may be a wrapper) */
+static size_t
+pfk_pflog_pull(struct mbuf *m, char *buf, size_t want)
+{
+	struct sk_buff *skb = m ? m_to_skb(m) : NULL;
+	size_t n;
+	int off;
+
+	if (skb == NULL)
+		return 0;
+	/*
+	 * BSD logs from the network header; pf may already have advanced the
+	 * mbuf/skb data pointer to the transport header while parsing, so
+	 * anchor explicitly at the IP header instead of at skb->data.
+	 */
+	off = skb_network_offset(skb);
+	if (off < 0)
+		off = 0;
+	if (off > skb->len)
+		return 0;
+	n = min_t(size_t, want, skb->len - off);
+	if (skb_copy_bits(skb, off, buf, n))
+		return 0;
+	return n;
+}
+
+/* ---------------- pflog_packet() (called by the ported pf.c) ---------------- */
+
+int
+pflog_packet(struct pf_pdesc *pd, u_int8_t reason, struct pf_rule *rm,
+    struct pf_rule *am, struct pf_ruleset *ruleset, struct pf_rule *cm)
+{
+	struct pfk_pflog_hdr hdr;
+	char pkt[PFLOG_COPY_SIZE];
+	struct pf_rule *r;
+	size_t n;
+
+	if (pd == NULL || pd->m == NULL)
+		return (0);
+
+	memset(&hdr, 0, sizeof(hdr));
+	{
+		struct timespec64 ts;
+
+		ktime_get_real_ts64(&ts);
+		hdr.pflog_ts.tv_sec = ts.tv_sec;
+		hdr.pflog_ts.tv_usec = ts.tv_nsec / 1000;
+	}
+	hdr.pflog_af = pd->af;
+	hdr.pflog_reason = reason;
+
+	r = cm ? cm : (am ? am : rm);
+	if (r) {
+		strscpy((char *)hdr.pflog_ifname,
+		    pd->kif ? pd->kif->pfik_name : "",
+		    sizeof(hdr.pflog_ifname));
+		strscpy((char *)hdr.pflog_ruleset,
+		    (ruleset && ruleset->anchor) ? ruleset->anchor->name : "",
+		    sizeof(hdr.pflog_ruleset));
+		hdr.pflog_nr = r->nr;
+		hdr.pflog_anc = 0;
+		hdr.pflog_flags = r->log;
+	}
+
+	n = pfk_pflog_pull(pd->m, pkt, sizeof(pkt));
+	hdr.pflog_len = cpu_to_be16(n);	/* BSD stores it in network order */
+
+	pfk_pflog_put(&hdr, pkt, n);
+	return (0);
+}
+
+/* ---------------- /dev/pflog ---------------- */
+
+static int
+pfk_pflog_open(struct inode *inode, struct file *filp)
+{
+	filp->private_data = NULL;
+	return nonseekable_open(inode, filp);
+}
+
+static ssize_t
+pfk_pflog_read(struct file *filp, char __user *buf, size_t count, loff_t *ppos)
+{
+	struct pfk_pflog_rec *rec = filp->private_data;
+	unsigned long flags;
+	size_t off, n;
+
+	spin_lock_irqsave(&pflog_lock, flags);
+	if (rec == NULL) {
+		if (list_empty(&pflog_list)) {
+			spin_unlock_irqrestore(&pflog_lock, flags);
+			if (filp->f_flags & O_NONBLOCK)
+				return -EAGAIN;
+			if (wait_event_interruptible(pflog_wq,
+			    !list_empty(&pflog_list)))
+				return -ERESTARTSYS;
+			spin_lock_irqsave(&pflog_lock, flags);
+			if (list_empty(&pflog_list)) {
+				spin_unlock_irqrestore(&pflog_lock, flags);
+				return -EAGAIN;
+			}
+		}
+		rec = list_first_entry(&pflog_list, struct pfk_pflog_rec, list);
+		list_del(&rec->list);
+		pflog_queued--;
+		pflog_bytes -= rec->len;
+		filp->private_data = rec;
+	}
+	spin_unlock_irqrestore(&pflog_lock, flags);
+
+	/* partial reads resume through *ppos (record is pinned to the file) */
+	off = *ppos;
+	if (off >= rec->len) {
+		kfree(rec);
+		filp->private_data = NULL;
+		*ppos = 0;
+		return 0;
+	}
+	n = min(count, rec->len - off);
+	if (copy_to_user(buf, rec->data + off, n))
+		return -EFAULT;
+	*ppos += n;
+	if (*ppos == rec->len) {
+		kfree(rec);
+		filp->private_data = NULL;
+		*ppos = 0;
+	}
+	return n;
+}
+
+static __poll_t
+pfk_pflog_poll(struct file *filp, poll_table *wait)
+{
+	__poll_t mask = 0;
+
+	poll_wait(filp, &pflog_wq, wait);
+	if (filp->private_data != NULL)
+		mask |= EPOLLIN | EPOLLRDNORM;
+	else if (!list_empty(&pflog_list))
+		mask |= EPOLLIN | EPOLLRDNORM;
+	return mask;
+}
+
+static long
+pfk_pflog_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
+{
+	struct pfk_pflog_stat st;
+
+	switch (cmd) {
+	case PFKIOCGETPFLOGSTAT:
+		st.queued = pflog_queued;
+		st.bytes = pflog_bytes;
+		st.dropped = pflog_dropped;
+		st.seq = pflog_seq;
+		if (copy_to_user((void __user *)arg, &st, sizeof(st)))
+			return -EFAULT;
+		return 0;
+	default:
+		return -ENOTTY;
+	}
+}
+
+static const struct file_operations pfk_pflog_fops = {
+	.owner		= THIS_MODULE,
+	.open		= pfk_pflog_open,
+	.read		= pfk_pflog_read,
+	.poll		= pfk_pflog_poll,
+	.unlocked_ioctl	= pfk_pflog_ioctl,
+	.compat_ioctl	= compat_ptr_ioctl,
+};
+
+static struct miscdevice pfk_pflog_dev = {
+	.minor		= MISC_DYNAMIC_MINOR,
+	.name		= "pflog",
+	.fops		= &pfk_pflog_fops,
+	.mode		= 0600,
+};
+
+int
+pfk_pflog_register(void)
+{
+	return misc_register(&pfk_pflog_dev);
+}
+
+void
+pfk_pflog_unregister(void)
+{
+	struct pfk_pflog_rec *rec, *tmp;
+
+	misc_deregister(&pfk_pflog_dev);
+	spin_lock(&pflog_lock);
+	list_for_each_entry_safe(rec, tmp, &pflog_list, list) {
+		list_del(&rec->list);
+		kfree(rec);
+	}
+	pflog_queued = 0;
+	pflog_bytes = 0;
+	spin_unlock(&pflog_lock);
+}

out/demo-pf-linux.cast

@@ -0,0 +1,84 @@
+{"version": 2, "width": 80, "height": 24, "timestamp": 1794449787, "env": {"SHELL": null, "TERM": "dumb"}}
+[1.414508, "o", "\u001b[1m===============================================================\u001b[0m\r\n\u001b[1m PF on Linux - the OpenBSD PF 8.0 engine inside the Linux kernel\u001b[0m\r\n\u001b[1m===============================================================\u001b[0m\r\n\r\nLab: 3 Alpine VMs under libvirt.\r\n  pftest-pf0 = firewall (kernel 7.3.4 built from source + pf.ko)\r\n  pftest-cli = client 10.0.1.10     pftest-srv = server 10.0.2.20\r\n\r\n\u001b[1m--- 1. our kernel, the OpenBSD engine as a module ---\u001b[0m\r\n\u001b[1;36mpftest-pf0# uname -r\u001b[0m\r\n"]
+[1.655866, "o", "7.3.4\r\n"]
+[2.058621, "o", "\u001b[1;36mpftest-pf0# lsmod | grep '^pf ' || echo 'pf: not loaded yet'\u001b[0m\r\n"]
+[2.276696, "o", "pf: not loaded yet\r\n"]
+[2.680328, "o", "\u001b[1;36mpftest-pf0# insmod /opt/pf/pf.ko && dmesg | tail -3\u001b[0m\r\n"]
+[2.917691, "o", "[  146.252272] NET: Registered PF_IUCV protocol family\r\n[  146.252362] pfk: netfilter hooks registered (10, netisr wq data path)\r\n[  146.252423] pfk: PF-on-Linux engine loaded (OpenBSD 8.0 base)\r\n"]
+[3.320802, "o", "\r\n\u001b[1m--- 2. pfctl talks to the module over the OpenBSD ioctl ABI ---\u001b[0m\r\n\u001b[1;36mpftest-pf0# /opt/pf/pfctl -s info | head -4\u001b[0m\r\n"]
+[3.598837, "o", "Status: Disabled for 0 days 00:02:26             Debug: err\r\n\r\nState Table                          Total             Rate\r\n  current entries                        0               \r\n"]
+[4.00249, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -e && /opt/pf/pfctl -s info | head -1\u001b[0m\r\n"]
+[4.274878, "o", "Status: Enabled                                  Debug: err\r\npf enabled\r\n"]
+[4.678074, "o", "\r\n\u001b[1m--- 3. a real pf.conf: tables, stateful rules, an explicit drop ---\u001b[0m\r\n"]
+[4.678128, "o", "\u001b[1;36mpftest-pf0# cat /tmp/demo-pf.conf\u001b[0m\r\n"]
+[4.954849, "o", "# PF-on-Linux demo ruleset (pf.conf syntax, real OpenBSD pfctl)\r\nset block-policy return\r\ntable <trusted> { 10.0.1.0/24 }\r\n\r\n# transit traffic: stateful, table-driven\r\npass in quick on ingress proto icmp from <trusted> keep state\r\npass out quick on egress proto icmp keep state\r\n\r\n# deny a specific destination\r\nblock drop in quick proto icmp to 10.0.9.99\r\n\r\n# everything else\r\npass in quick\r\npass out quick\r\n"]
+[5.358199, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -f /tmp/demo-pf.conf && echo 'rules loaded'\u001b[0m\r\n"]
+[5.630704, "o", "rules loaded\r\n"]
+[6.034552, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -sr\u001b[0m\r\n"]
+[6.322794, "o", "pass in quick on ingress proto icmp from <trusted> to any\r\npass out quick on egress proto icmp all\r\nblock drop in quick inet proto icmp from any to 10.0.9.99\r\npass in quick all flags S/SA\r\npass out quick all flags S/SA\r\n"]
+[6.725786, "o", "\r\n"]
+[6.72584, "o", "\u001b[1m--- 4. transit traffic is allowed and stateful ---\u001b[0m\r\n\u001b[1;36mpftest-cli# ping -c2 -W2 10.0.2.20 | tail -3\u001b[0m\r\n"]
+[7.977099, "o", "--- 10.0.2.20 ping statistics ---\r\n2 packets transmitted, 2 packets received, 0% packet loss\r\nround-trip min/avg/max = 0.339/0.351/0.363 ms\r\n"]
+[8.379918, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -ss | grep -E 'icmp' | head -4\u001b[0m\r\n"]
+[8.634572, "o", "all icmp 10.0.2.20:8 <- 10.0.1.10:5449       0:0\r\nall icmp 10.0.1.10:5449 -> 10.0.2.20:8       0:0\r\n"]
+[9.036709, "o", "\r\n\u001b[1m--- 5. tables are live objects ---\u001b[0m\r\n\u001b[1;36mpftest-pf0# /opt/pf/pfctl -t trusted -T show\u001b[0m\r\n"]
+[9.298627, "o", "   10.0.1.0/24\r\n"]
+[9.701487, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -s Tables\u001b[0m\r\n"]
+[9.974937, "o", "trusted\r\n"]
+[10.378208, "o", "\r\n\u001b[1m--- 6. the drop rule actually drops ---\u001b[0m\r\n\u001b[1;36mpftest-pf0# cat > /tmp/demo-block.conf <<EOF\r\nblock drop in quick proto icmp to 10.0.2.20\r\npass in quick\r\npass out quick\r\nEOF\u001b[0m\r\n"]
+[11.033956, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -f /tmp/demo-block.conf && echo 'block rules loaded'\u001b[0m\r\n"]
+[11.310724, "o", "block rules loaded\r\n"]
+[11.714107, "o", "\u001b[1;36mpftest-cli# ping -c2 -W2 10.0.2.20 | tail -3\u001b[0m\r\n"]
+[14.969152, "o", "\r\n--- 10.0.2.20 ping statistics ---\r\n2 packets transmitted, 0 packets received, 100% packet loss\r\n"]
+[15.372543, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -sr -vv | head -6\u001b[0m\r\n"]
+[15.66699, "o", "@0 block drop in quick inet proto icmp from any to 10.0.2.20\r\n  [ Evaluations: 6         Packets: 2         Bytes: 168         States: 0     ]\r\n  [ Inserted: uid 0 pid 1687 State Creations: 0     ]\r\n@1 pass in quick all flags S/SA\r\n  [ Evaluations: 1         Packets: 28        Bytes: 9296        States: 1     ]\r\n  [ Inserted: uid 0 pid 1687 State Creations: 1     ]\r\n"]
+[16.07016, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -f /tmp/demo-pf.conf && echo 'back to the stateful ruleset'\u001b[0m\r\n"]
+[16.338781, "o", "back to the stateful ruleset\r\n"]
+[16.742479, "o", "\r\n\u001b[1m--- 7. NAT: the source is translated on the way out, restored on the way back ---\u001b[0m\r\n"]
+[16.742537, "o", "\u001b[1;36mpftest-pf0# cat > /tmp/demo-nat.conf <<EOF\r\npass out quick on egress from 10.0.1.0/24 to any nat-to (eth2)\r\npass in quick on ingress proto icmp keep state\r\npass out quick on egress proto icmp keep state\r\nEOF\u001b[0m\r\n"]
+[17.422636, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -f /tmp/demo-nat.conf && echo 'nat rules loaded'\u001b[0m\r\n"]
+[17.686794, "o", "nat rules loaded\r\n"]
+[18.090339, "o", "\u001b[1;36mpftest-cli# ping -c2 -W2 10.0.2.20 | tail -3\u001b[0m\r\n"]
+[19.341141, "o", "--- 10.0.2.20 ping statistics ---\r\n2 packets transmitted, 2 packets received, 0% packet loss\r\nround-trip min/avg/max = 0.276/0.317/0.359 ms\r\n"]
+[19.744749, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -ss | grep -E 'icmp' | head -4\u001b[0m\r\n"]
+[20.010971, "o", "all icmp 10.0.2.20:8 <- 10.0.1.10:5461       0:0\r\nall icmp 10.0.2.1:43040 (10.0.1.10:5461) -> 10.0.2.20:8       0:0\r\n"]
+[20.414328, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -f /tmp/demo-pf.conf && echo 'back to the stateful ruleset'\u001b[0m\r\n"]
+[20.690817, "o", "back to the stateful ruleset\r\n"]
+[21.094035, "o", "\r\n\u001b[1m--- 8. pflog: logged packets reach userspace (/dev/pflog) ---\u001b[0m\r\n"]
+[21.094088, "o", "\u001b[1;36mpftest-pf0# cat > /tmp/demo-log.conf <<EOF\r\npass in quick log proto icmp from 10.0.1.10 keep state\r\npass out quick\r\nEOF\u001b[0m\r\n"]
+[21.762242, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -f /tmp/demo-log.conf && echo 'log rule loaded'\u001b[0m\r\n"]
+[22.034943, "o", "log rule loaded\r\n"]
+[22.438523, "o", "\u001b[1;36mpftest-pf0# chmod +x /tmp/pflogd; rm -f /tmp/pflog.pcap; (setsid /tmp/pflogd -c 2 /tmp/pflog.pcap >/tmp/pflogd.log 2>&1 </dev/null &); sleep 1; echo 'pflogd collecting 2 records'\u001b[0m\r\n"]
+[23.681454, "o", "pflogd collecting 2 records\r\n"]
+[24.084864, "o", "\u001b[1;36mpftest-cli# ping -c2 -W2 10.0.2.20 | tail -2\u001b[0m\r\n"]
+[25.333003, "o", "2 packets transmitted, 2 packets received, 0% packet loss\r\nround-trip min/avg/max = 0.352/0.364/0.376 ms\r\n"]
+[27.737616, "o", "\u001b[1;36mpftest-pf0# cat /tmp/pflogd.log; ls -l /tmp/pflog.pcap; echo 'pcap header (magic d4c3b2a1, linktype 0x75=DLT_PFLOG):'; od -An -tx1 -N24 /tmp/pflog.pcap\u001b[0m\r\n"]
+[28.026778, "o", "pflogd: 2 records\r\n-rw-------    1 root     root           324 Nov 12 02:16 /tmp/pflog.pcap\r\npcap header (magic d4c3b2a1, linktype 0x75=DLT_PFLOG):\r\n d4 c3 b2 a1 02 00 04 00 00 00 00 00 00 00 00 00\r\n ff ff 00 00 75 00 00 00\r\n"]
+[28.430241, "o", "\u001b[1;36mpftest-pf0# pkill pflogd 2>/dev/null; /opt/pf/pfctl -f /tmp/demo-pf.conf >/dev/null\u001b[0m\r\n"]
+[29.114632, "o", "\r\n\u001b[1m--- 9. route-to: force the next hop away from the normal path ---\u001b[0m\r\n"]
+[29.114687, "o", "\u001b[1;36mpftest-pf0# cat > /tmp/demo-route.conf <<EOF\r\npass in quick proto icmp from 10.0.1.10 to 10.0.2.20 route-to { 10.0.3.2 } keep state\r\npass out quick\r\nEOF\u001b[0m\r\n"]
+[29.778259, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -f /tmp/demo-route.conf && echo 'route-to rule loaded'\u001b[0m\r\n"]
+[30.050898, "o", "route-to rule loaded\r\n"]
+[30.45427, "o", "\u001b[1;36mpftest-pf0# (tcpdump -ni eth3 -c 2 -e icmp > /tmp/route.td 2>&1 &); (tcpdump -ni eth2 -c 2 -e icmp > /tmp/norm.td 2>&1 &); sleep 1\u001b[0m\r\n"]
+[32.089063, "o", "\u001b[1;36mpftest-cli# ping -c1 -W2 10.0.2.20 | tail -2\u001b[0m\r\n"]
+[34.31667, "o", "--- 10.0.2.20 ping statistics ---\r\n1 packets transmitted, 0 packets received, 100% packet loss\r\n"]
+[36.72139, "o", "\u001b[1;36mpftest-pf0# echo 'forced egress (eth3, toward the gateway):'; grep -v '^tcpdump\\|^listening\\|^$' /tmp/route.td | head -2; echo 'normal egress (eth2):'; grep -c 'echo request' /tmp/norm.td\u001b[0m\r\n"]
+[37.006878, "o", "forced egress (eth3, toward the gateway):\r\n02:16:59.958272 52:54:00:aa:00:04 > 52:54:00:dd:00:02, ethertype IPv4 (0x0800), length 98: 10.0.1.10 > 10.0.2.20: ICMP echo request, id 5473, seq 0, length 64\r\nnormal egress (eth2):\r\n0\r\n"]
+[37.410476, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -f /tmp/demo-pf.conf >/dev/null\u001b[0m\r\n"]
+[38.094888, "o", "\r\n\u001b[1m--- 10. divert-to: the packet goes to userspace and comes back ---\u001b[0m\r\n"]
+[38.095122, "o", "\u001b[1;36mpftest-pf0# cat /tmp/demo-divert.conf\u001b[0m\r\n"]
+[38.35866, "o", "pass in quick proto icmp from 10.0.1.10 to 10.0.1.1 divert-to \"10.0.1.1\" port 10000 keep state\r\npass out quick\r\n"]
+[38.763007, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -f /tmp/demo-divert.conf && echo 'divert rule loaded'\u001b[0m\r\n"]
+[39.022814, "o", "divert rule loaded\r\n"]
+[39.42606, "o", "\u001b[1;36mpftest-pf0# chmod +x /tmp/pdivert; (setsid /tmp/pdivert -p 10000 -f >/tmp/pdivert.log 2>&1 </dev/null &); sleep 1; pgrep pdivert\u001b[0m\r\n"]
+[40.654519, "o", "1794\r\n"]
+[41.056785, "o", "\u001b[1;36mpftest-cli# ping -c2 -W2 10.0.1.1 | tail -3\u001b[0m\r\n"]
+[42.284778, "o", "--- 10.0.1.1 ping statistics ---\r\n2 packets transmitted, 2 packets received, 0% packet loss\r\nround-trip min/avg/max = 0.177/0.224/0.271 ms\r\n"]
+[42.691426, "o", "\u001b[1;36mpftest-pf0# head -3 /tmp/pdivert.log\u001b[0m\r\n"]
+[42.942601, "o", "pdivert: listening on divert port 10000\r\npdivert: got 84 bytes 10.0.1.10 -> 10.0.1.1\r\npdivert: got 84 bytes 10.0.1.10 -> 10.0.1.1\r\n"]
+[43.344696, "o", "  (pdivert flipped the last payload byte and fixed the ICMP checksum)\r\n\r\n\u001b[1m--- 11. shutdown ---\u001b[0m\r\n\u001b[1;36mpftest-pf0# pkill pdivert; /opt/pf/pfctl -f /dev/null; /opt/pf/pfctl -d; echo 'pf disabled'\u001b[0m\r\n"]
+[43.626606, "o", "pf disabled\r\npf disabled\r\n"]
+[44.028882, "o", "\u001b[1;36mpftest-pf0# lsmod | grep '^pf '\u001b[0m\r\n"]
+[44.286597, "o", "pf                    552960  0\r\n"]
+[44.688511, "o", "\r\n\u001b[1mdemo over - port, gates and docs:\u001b[0m\r\n\u001b[1m  https://git.labexposed.com/lgcosta/pf-linux-port\u001b[0m\r\n"]
+[44.688631, "o", "\r\n"]

scripts/demo-record.py

@@ -1,141 +1,161 @@
 #!/usr/bin/env python3
 """
-PF-on-Linux demo recorder — produces an asciicast v2 (.cast) from a REAL
+PF-on-Linux demo recorder -> asciicast v2 (.cast)
-session driven against the lab VMs (no simulated output: every byte shown
-comes from the live systems; only pacing is scripted).
 
-Usage:  ./scripts/demo-record.py [--out out/demo-pf-linux.cast]
+Every byte shown comes from the REAL lab (Alpine VMs under libvirt, kernel
-Requires: lab running (pftest-pf0/cli/srv), repo built, ssh alias `pf0`
+7.3.4 built from source with our pf.ko).  Only the pacing is scripted.
-(ProxyJump) and lib.sh-style nested access working.
 
-Edit DEMO below to change the narration script. Timing: each line is a
+Usage:
-(real command, real output) pair; delays are cosmetic only.
+    ./scripts/demo-record.py [--out out/demo-pf-linux.cast] [--quick]
+
+Play back with:   asciinema play out/demo-pf-linux.cast
+Upload with:      asciinema upload out/demo-pf-linux.cast
 """
-import argparse, json, os, pty, re, shlex, subprocess, sys, time
+import argparse
+import json
+import os
+import shlex
+import subprocess
+import sys
+import time
 
 REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
-OUT_DEFAULT = os.path.join(REPO, "out", "demo-pf-linux.cast")
-
 DEB = os.environ.get("DEB", "root@192.168.1.242")
-PF0_IP = os.environ.get("PF0_IP", "192.168.122.8")
+PF0 = os.environ.get("PF0_IP", "192.168.122.8")
+CLI = os.environ.get("CLI_IP", "10.0.1.10")
+SRV = os.environ.get("SRV_IP", "10.0.2.20")
+KEY = os.path.join(os.environ.get("HOME", "/root"), ".ssh/pflab_ed")
+
+CONF = """# PF-on-Linux demo ruleset (pf.conf syntax, real OpenBSD pfctl)
+set block-policy return
+table <trusted> { 10.0.1.0/24 }
+
+# transit traffic: stateful, table-driven
+pass in quick on ingress proto icmp from <trusted> keep state
+pass out quick on egress proto icmp keep state
+
+# deny a specific destination
+block drop in quick proto icmp to 10.0.9.99
+
+# everything else
+pass in quick
+pass out quick
+"""
+
+DIVERT_CONF = """pass in quick proto icmp from 10.0.1.10 to 10.0.1.1 divert-to "10.0.1.1" port 10000 keep state
+pass out quick
+"""
 
 
-def deb_run(cmd: str, timeout=30) -> str:
+def sh(cmd, timeout=40):
-    """run a command on the libvirt host (which has direct keys to VMs)"""
+    p = subprocess.run(cmd, shell=True, capture_output=True, text=True,
-    p = subprocess.run(["ssh", "-o", "ConnectTimeout=10", DEB, cmd],
+                       timeout=timeout)
-                       capture_output=True, text=True, timeout=timeout)
     return (p.stdout or "") + (p.stderr or "")
 
 
-def vm0(cmd: str, timeout=30) -> str:
+def on_pf0(cmd, timeout=40):
-    return deb_run(
+    return sh(f"ssh -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new "
-        f"ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=8 "
+              f"-i {KEY} root@{PF0} {shlex.quote(cmd)}", timeout)
-        f"-i /root/.ssh/pflab_ed root@{PF0_IP} {shlex.quote(cmd)}",
-        timeout)
 
 
-def vm_ip(ip: str, cmd: str, timeout=30) -> str:
+def on_vm(ip, cmd, timeout=40):
-    return deb_run(
+    return sh(f"ssh -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new "
-        f"ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=8 "
+              f"-i {KEY} root@{ip} {shlex.quote(cmd)}", timeout)
-        f"-i /root/.ssh/pflab_ed root@{ip} {shlex.quote(cmd)}",
-        timeout)
 
 
-CLI, SRV = "10.0.1.10", "10.0.2.20"
+def on_deb(cmd, timeout=40):
+    return sh(f"ssh -o ConnectTimeout=8 {DEB} {shlex.quote(cmd)}", timeout)
 
 
-def scene(title):
+def stage():
-    return ("#banner", title)
+    """put the demo ruleset + pdivert on pf0 (through the lab host)"""
+    with open("/tmp/demo-pf.conf", "w") as f:
+        f.write(CONF)
+    with open("/tmp/demo-divert.conf", "w") as f:
+        f.write(DIVERT_CONF)
+    sh(f"scp -q /tmp/demo-pf.conf /tmp/demo-divert.conf "
+       f"{os.path.join(REPO, 'tools/out/pdivert')} {DEB}:/tmp/", timeout=90)
+    on_deb(f"scp -q -o StrictHostKeyChecking=accept-new -i {KEY} "
+           f"/tmp/demo-pf.conf /tmp/demo-divert.conf /tmp/pdivert "
+           f"root@{PF0}:/tmp/ 2>&1", timeout=90)
 
-# (shell-line, runner, command)  — runners: pf0 / cli / srv / local
+
-DEMO = [
+SCENARIO = [
-    ("local",  "echo 'PF-on-Linux — OpenBSD PF engine on Linux 7.3.4 (lab: Alpine VMs)'"),
+    ("local", "echo '=== PF on Linux — OpenBSD PF 8.0 engine running inside the Linux 7.3.4 kernel ==='"),
-    ("local",  "uname -r | sed 's/^/build host kernel: /'"),
+    ("local", "echo 'lab: 3 Alpine VMs under libvirt (pf0 = firewall, cli/srv = transit endpoints)'"),
     ("pf0", "uname -r"),
-    ("pf0",    "lsmod | grep -E '^pf ' || echo 'pf: not loaded yet'"),
+    ("pf0", "lsmod | grep '^pf ' || echo 'pf: not loaded'"),
-    ("pf0",    "insmod /opt/pf/pf.ko && dmesg | tail -2"),
+    ("pf0", "insmod /opt/pf/pf.ko && dmesg | tail -3"),
-    ("pf0",    "/opt/pf/pfctl -s info | head -3"),
+    ("pf0", "/opt/pf/pfctl -s info | head -4"),
     ("pf0", "/opt/pf/pfctl -e && /opt/pf/pfctl -s info | head -1"),
-    ("pf0",    "cat /tmp/pfdemo.conf"),
+    ("pf0", "cat /tmp/demo-pf.conf"),
-    ("pf0",    "/opt/pf/pfctl -f /tmp/pfdemo.conf && /opt/pf/pfctl -sr"),
+    ("pf0", "/opt/pf/pfctl -f /tmp/demo-pf.conf && echo 'rules loaded' && /opt/pf/pfctl -sr"),
-    ("cli",    f"ping -c 2 -W 2 {SRV} 2>&1 | tail -2"),
+    ("pf0", "/opt/pf/pfctl -sr -vv | head -8"),
-    ("pf0",    "/opt/pf/pfctl -s states"),
+    ("cli", f"ping -c2 -W2 {SRV} | tail -3"),
+    ("pf0", "/opt/pf/pfctl -ss"),
+    ("pf0", "/opt/pf/pfctl -t trusted -T show"),
     ("pf0", "/opt/pf/pfctl -s Tables"),
-    ("pf0",    "/opt/pf/pfctl -t demo -T show"),
+    ("cli", "ping -c2 -W2 10.0.9.99 | tail -3"),
-    ("cli",    "ping -c 2 -W 2 10.0.9.99 2>&1 | tail -2"),
+    ("pf0", "/opt/pf/pfctl -sr -vv | head -6"),
-    ("pf0",    "/opt/pf/pfctl -s info | grep -E 'match|current'"),
+    ("pf0", "/opt/pf/pfctl -f /tmp/demo-divert.conf && echo 'divert rule loaded'"),
-    ("pf0",    "/opt/pf/pfctl -ss | head -2"),
+    ("pf0", "(setsid /tmp/pdivert -p 10000 -f >/tmp/pdivert.log 2>&1 </dev/null &); sleep 1; pgrep pdivert"),
+    ("cli", "ping -c2 -W2 10.0.1.1 | tail -3"),
+    ("pf0", "head -3 /tmp/pdivert.log"),
+    ("pf0", "pkill pdivert; /opt/pf/pfctl -f /dev/null; /opt/pf/pfctl -d; echo 'pf disabled'"),
 ]
 
-CONF = """set block-policy return
-set loginterface egress
-table <demo> { 10.0.1.0/24 }
-match in quick on ingress proto icmp from <demo> tag DEMO
-pass in quick on ingress proto icmp keep state
-pass out quick proto icmp keep state
-block drop in quick proto icmp from any to 10.0.9.99
-"""
 
-
+def record(out_path, quick=False):
-def prep():
+    os.makedirs(os.path.dirname(out_path) or ".", exist_ok=True)
-    # stage the demo ruleset into pf0
+    events = []
-    import base64
-    b64 = base64.b64encode(CONF.encode()).decode()
-    vm0(f"echo {b64} | base64 -d > /tmp/pfdemo.conf; echo staged")
-
-
-def record(out_path: str, fps_pad=0.4):
-    os.makedirs(os.path.dirname(out_path), exist_ok=True)
-    casts = []
     t = 0.0
 
     def emit(text, delay):
         nonlocal t
         t += delay
-        casts.append([round(t, 3), "o", text])
+        events.append([round(t, 3), "o", text])
 
-    header = {"version": 2, "width": 100, "height": 24,
+    header = {
-              "timestamp": int(time.time()), "title":
+        "version": 2, "width": 100, "height": 26,
-              "PF on Linux — port demo (OpenBSD PF 8.0 engine on Linux 7.3.4, Alpine libvirt lab)",
+        "timestamp": int(time.time()),
-              "env": {"SHELL": "/bin/sh", "TERM": "xterm-256color"}}
+        "title": "PF on Linux — OpenBSD PF engine inside the Linux kernel (lab demo)",
+        "env": {"SHELL": "/bin/sh", "TERM": "xterm-256color"},
+    }
+    runners = {
+        "pf0": lambda c: on_pf0(c),
+        "cli": lambda c: on_vm(CLI, c),
+        "srv": lambda c: on_vm(SRV, c),
+        "local": lambda c: sh(c),
+    }
+    labels = {"pf0": "pftest-pf0", "cli": "pftest-cli", "srv": "pftest-srv",
+              "local": "lab-host"}
+
+    stage()
     with open(out_path, "w") as f:
         f.write(json.dumps(header) + "\n")
-
+        for kind, cmd in SCENARIO:
-        def show_prompt(host):
+            emit(f"\x1b[1;36m{labels[kind]}\x1b[0m:\x1b[1;37m {cmd}\x1b[0m\r\n",
-            emit(f"\x1b[1;36m{host}$\x1b[0m ", 0.15)
+                 0.35)
-
-        runners = {"pf0": vm0, "cli": lambda c: vm_ip(CLI, c),
-                   "srv": lambda c: vm_ip(SRV, c), "local":
-                   lambda c: subprocess.run(["sh", "-c", c],
-                                            capture_output=True, text=True,
-                                            timeout=30).stdout +
-                   subprocess.run(["sh", "-c", c], capture_output=True,
-                                  text=True).stderr}
-        prep()
-        for kind, cmd in DEMO:
-            label = {"pf0": "pftest-pf0", "cli": "pftest-cli",
-                     "srv": "pftest-srv", "local": "build-host"}[kind]
-            show_prompt(label)
-            emit(cmd + "\r\n", 0.25)
             try:
                 out = runners[kind](cmd)
-            except Exception as e:
+            except subprocess.TimeoutExpired:
-                out = f"[runner error: {e}]\n"
+                out = "[timeout]\r\n"
-            # strip ANSI-safe passthrough; small human-ish delay
+            except Exception as e:  # noqa: BLE001
-            delay = min(1.2, 0.25 + len(out) / 4000.0)
+                out = f"[error: {e}]\r\n"
-            emit(out.replace("\n", "\r\n") or "(no output)\r\n", delay)
+            out = out.replace("\r\n", "\n").replace("\n", "\r\n")
-            time.sleep(0.1)
+            emit(out or "(no output)\r\n", min(1.4, 0.3 + len(out) / 5000.0))
-        emit("\x1b[1;36mpftest-pf0$\x1b[0m /opt/pf/pfctl -d\r\n", 0.4)
+        with open(out_path, "a") as f:
-        runners["pf0"]("/opt/pf/pfctl -d")
+            for ev in events:
-        for i, ev in enumerate(casts):
                 f.write(json.dumps(ev) + "\n")
-    print(f"WROTE {out_path} ({len(casts)} events, {t:.1f}s)")
+    print(f"WROTE {out_path}: {len(events)} events, {t:.1f}s")
-    print("play:  asciinema play " + out_path +
+    print("play:   asciinema play " + out_path)
-          "   (or: script-replay/pf-in-play fallback: cat with 'less -R')")
+    print("upload: asciinema upload " + out_path)
 
 
 if __name__ == "__main__":
     ap = argparse.ArgumentParser()
-    ap.add_argument("--out", default=OUT_DEFAULT)
+    ap.add_argument("--out", default=os.path.join(REPO, "out",
+                                                  "demo-pf-linux.cast"))
+    ap.add_argument("--quick", action="store_true")
     a = ap.parse_args()
-    record(a.out)
+    record(a.out, a.quick)

scripts/demo-session.sh

@@ -0,0 +1,182 @@
+#!/usr/bin/env bash
+# Demo session for PF-on-Linux: a scripted terminal tour of the lab.
+# Recorded with:  asciinema rec out/demo-pf-linux.cast -c "bash scripts/demo-session.sh"
+#
+# Everything printed is the REAL output of the lab (Alpine VMs under libvirt
+# running the kernel built from source with our pf.ko).  The VM management
+# addresses are resolved from the hypervisor (they are DHCP), the lab
+# topology (10.0.1.0/24, 10.0.2.0/24, ...) is fixed.
+set -u
+
+DEB=${DEB:-root@192.168.1.242}
+CLI_IP=${CLI_IP:-10.0.1.10}
+SRV_IP=${SRV_IP:-10.0.2.20}
+KEY=${KEY:-$HOME/.ssh/pflab_ed}
+
+bold()  { printf '\033[1m%s\033[0m\n' "$*"; }
+cyan()  { printf '\033[1;36m%s\033[0m\n' "$*"; }
+
+# management address of a lab VM (DHCP, so ask the hypervisor)
+mgmt() {
+	ssh -o ConnectTimeout=8 "$DEB" "virsh domifaddr $1 | awk '/vnet/{print \$4}' | cut -d/ -f1" \
+	    2>/dev/null | tr -d '\r' | head -1
+}
+
+PF0_MGMT=$(mgmt pftest-pf0)
+CLI_MGMT=$(mgmt pftest-cli)
+[ -n "$PF0_MGMT" ] || PF0_MGMT=192.168.122.8
+[ -n "$CLI_MGMT" ] || CLI_MGMT=192.168.122.80
+
+# run a command on a lab VM (through the hypervisor)
+onvm() {
+	local ip=$1; shift
+	ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new \
+	    -o ProxyJump=root@192.168.1.242 -i "$KEY" root@"$ip" "$@" 2>&1
+}
+
+p0() {
+	cyan "pftest-pf0# $*"
+	onvm "$PF0_MGMT" "$@"
+	sleep 0.4
+}
+vmc() {
+	cyan "pftest-cli# $*"
+	onvm "$CLI_MGMT" "$@"
+	sleep 0.4
+}
+
+
+CONF=/tmp/demo-pf.conf
+DIV=/tmp/demo-divert.conf
+
+cat > "$CONF" <<'CONF'
+# PF-on-Linux demo ruleset (pf.conf syntax, real OpenBSD pfctl)
+set block-policy return
+table <trusted> { 10.0.1.0/24 }
+
+# transit traffic: stateful, table-driven
+pass in quick on ingress proto icmp from <trusted> keep state
+pass out quick on egress proto icmp keep state
+
+# deny a specific destination
+block drop in quick proto icmp to 10.0.9.99
+
+# everything else
+pass in quick
+pass out quick
+CONF
+
+cat > "$DIV" <<CONF
+pass in quick proto icmp from $CLI_IP to 10.0.1.1 divert-to "10.0.1.1" port 10000 keep state
+pass out quick
+CONF
+
+# the pf0 interface towards the server (used by the NAT pool)
+EGRESS_IF=$(onvm "$PF0_MGMT" "ip route get $SRV_IP | grep -o 'dev [a-z0-9]*' | cut -d' ' -f2" | tr -d '\r')
+[ -n "$EGRESS_IF" ] || EGRESS_IF=eth2
+
+scp -q -o ProxyJump=root@192.168.1.242 -i "$KEY" "$CONF" "$DIV" \
+    tools/out/pdivert tools/out/pflogd "root@$PF0_MGMT:/tmp/" 2>/dev/null
+
+bold "==============================================================="
+bold " PF on Linux - the OpenBSD PF 8.0 engine inside the Linux kernel"
+bold "==============================================================="
+echo
+echo "Lab: 3 Alpine VMs under libvirt."
+echo "  pftest-pf0 = firewall (kernel 7.3.4 built from source + pf.ko)"
+echo "  pftest-cli = client 10.0.1.10     pftest-srv = server 10.0.2.20"
+echo
+
+bold "--- 1. our kernel, the OpenBSD engine as a module ---"
+p0 "uname -r"
+p0 "lsmod | grep '^pf ' || echo 'pf: not loaded yet'"
+p0 "insmod /opt/pf/pf.ko && dmesg | tail -3"
+echo
+
+bold "--- 2. pfctl talks to the module over the OpenBSD ioctl ABI ---"
+p0 "/opt/pf/pfctl -s info | head -4"
+p0 "/opt/pf/pfctl -e && /opt/pf/pfctl -s info | head -1"
+echo
+
+bold "--- 3. a real pf.conf: tables, stateful rules, an explicit drop ---"
+p0 "cat /tmp/demo-pf.conf"
+p0 "/opt/pf/pfctl -f /tmp/demo-pf.conf && echo 'rules loaded'"
+p0 "/opt/pf/pfctl -sr"
+echo
+
+bold "--- 4. transit traffic is allowed and stateful ---"
+vmc "ping -c2 -W2 $SRV_IP | tail -3"
+p0 "/opt/pf/pfctl -ss | grep -E 'icmp' | head -4"
+echo
+
+bold "--- 5. tables are live objects ---"
+p0 "/opt/pf/pfctl -t trusted -T show"
+p0 "/opt/pf/pfctl -s Tables"
+echo
+
+bold "--- 6. the drop rule actually drops ---"
+p0 "cat > /tmp/demo-block.conf <<EOF
+block drop in quick proto icmp to $SRV_IP
+pass in quick
+pass out quick
+EOF"
+p0 "/opt/pf/pfctl -f /tmp/demo-block.conf && echo 'block rules loaded'"
+vmc "ping -c2 -W2 $SRV_IP | tail -3"
+p0 "/opt/pf/pfctl -sr -vv | head -6"
+p0 "/opt/pf/pfctl -f /tmp/demo-pf.conf && echo 'back to the stateful ruleset'"
+echo
+
+bold "--- 7. NAT: the source is translated on the way out, restored on the way back ---"
+p0 "cat > /tmp/demo-nat.conf <<EOF
+pass out quick on egress from 10.0.1.0/24 to any nat-to ($EGRESS_IF)
+pass in quick on ingress proto icmp keep state
+pass out quick on egress proto icmp keep state
+EOF"
+p0 "/opt/pf/pfctl -f /tmp/demo-nat.conf && echo 'nat rules loaded'"
+vmc "ping -c2 -W2 $SRV_IP | tail -3"
+p0 "/opt/pf/pfctl -ss | grep -E 'icmp' | head -4"
+p0 "/opt/pf/pfctl -f /tmp/demo-pf.conf && echo 'back to the stateful ruleset'"
+echo
+
+bold "--- 8. pflog: logged packets reach userspace (/dev/pflog) ---"
+p0 "cat > /tmp/demo-log.conf <<EOF
+pass in quick log proto icmp from $CLI_IP keep state
+pass out quick
+EOF"
+p0 "/opt/pf/pfctl -f /tmp/demo-log.conf && echo 'log rule loaded'"
+p0 "chmod +x /tmp/pflogd; rm -f /tmp/pflog.pcap; (setsid /tmp/pflogd -c 2 /tmp/pflog.pcap >/tmp/pflogd.log 2>&1 </dev/null &); sleep 1; echo 'pflogd collecting 2 records'"
+vmc "ping -c2 -W2 $SRV_IP | tail -2"
+sleep 2
+p0 "cat /tmp/pflogd.log; ls -l /tmp/pflog.pcap; echo 'pcap header (magic d4c3b2a1, linktype 0x75=DLT_PFLOG):'; od -An -tx1 -N24 /tmp/pflog.pcap"
+p0 "pkill pflogd 2>/dev/null; /opt/pf/pfctl -f /tmp/demo-pf.conf >/dev/null"
+echo
+
+bold "--- 9. route-to: force the next hop away from the normal path ---"
+p0 "cat > /tmp/demo-route.conf <<EOF
+pass in quick proto icmp from $CLI_IP to $SRV_IP route-to { 10.0.3.2 } keep state
+pass out quick
+EOF"
+p0 "/opt/pf/pfctl -f /tmp/demo-route.conf && echo 'route-to rule loaded'"
+p0 "(tcpdump -ni eth3 -c 2 -e icmp > /tmp/route.td 2>&1 &); (tcpdump -ni eth2 -c 2 -e icmp > /tmp/norm.td 2>&1 &); sleep 1"
+vmc "ping -c1 -W2 $SRV_IP | tail -2"
+sleep 2
+p0 "echo 'forced egress (eth3, toward the gateway):'; grep -v '^tcpdump\|^listening\|^$' /tmp/route.td | head -2; echo 'normal egress (eth2):'; grep -c 'echo request' /tmp/norm.td"
+p0 "/opt/pf/pfctl -f /tmp/demo-pf.conf >/dev/null"
+echo
+
+bold "--- 10. divert-to: the packet goes to userspace and comes back ---"
+p0 "cat /tmp/demo-divert.conf"
+p0 "/opt/pf/pfctl -f /tmp/demo-divert.conf && echo 'divert rule loaded'"
+p0 "chmod +x /tmp/pdivert; (setsid /tmp/pdivert -p 10000 -f >/tmp/pdivert.log 2>&1 </dev/null &); sleep 1; pgrep pdivert"
+vmc "ping -c2 -W2 10.0.1.1 | tail -3"
+p0 "head -3 /tmp/pdivert.log"
+echo "  (pdivert flipped the last payload byte and fixed the ICMP checksum)"
+echo
+
+bold "--- 11. shutdown ---"
+p0 "pkill pdivert; /opt/pf/pfctl -f /dev/null; /opt/pf/pfctl -d; echo 'pf disabled'"
+p0 "lsmod | grep '^pf '"
+echo
+bold "demo over - port, gates and docs:"
+bold "  https://git.labexposed.com/lgcosta/pf-linux-port"
+echo

scripts/screencast/assemble.py

@@ -0,0 +1,149 @@
+#!/usr/bin/env python3
+"""
+Assemble the final MP4s: silent video track + MiniMax narration + subtitles.
+
+Each narration segment was recorded as its own audio file, generated by
+MiniMax with a female voice for the language of the track.  The segments are
+placed at the moment their beat starts on screen, so the voice, the caption
+and the terminal action always agree.
+
+    ./scripts/screencast/assemble.py --phase 04-roteamento --lang en
+"""
+import argparse
+import json
+import os
+import shutil
+import subprocess
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+REPO = os.path.dirname(os.path.dirname(HERE))
+CAPS = os.path.join(REPO, "out", "screencast", "caps")
+VO = os.path.join(REPO, "out", "screencast", "vo")
+OUT = os.path.join(REPO, "out", "screencast")
+ART = os.path.join(OUT, "art")
+
+# which phases draw the lab diagram over the terminal
+DIAGRAM_PHASES = {"00-ambiente", "04-roteamento", "05-balanceamento"}
+# phases that show the module/host instead of the firewall VM
+HOST_PHASES = {"01-implementacao"}
+
+PHASE_ORDER = ["00-ambiente", "01-implementacao", "02-ativacao", "03-regras",
+               "04-roteamento", "05-balanceamento", "06-qos"]
+
+
+def run(cmd):
+    subprocess.run(cmd, check=True)
+
+
+def probe(path):
+    out = subprocess.run(["ffprobe", "-v", "error", "-show_entries",
+                          "format=duration", "-of", "csv=p=0", path],
+                         capture_output=True, text=True).stdout.strip()
+    return float(out or 0)
+
+
+def audio_duration(path):
+    return probe(path)
+
+
+def make_video(phase, lang):
+    """Render the silent video (terminal + diagram + burned-in subtitles)."""
+    cast = os.path.join(CAPS, f"{phase}.cast")
+    nar = os.path.join(HERE, "narration", f"{phase}.{lang}.json")
+    if not os.path.exists(cast):
+        raise SystemExit(f"[assemble] missing capture: {cast}")
+    art = None
+    if os.path.isdir(ART):
+        imgs = sorted(f for f in os.listdir(ART) if f.endswith((".jpg", ".png")))
+        if imgs:
+            art = os.path.join(ART, imgs[0])
+    outdir = os.path.join(OUT, lang)
+    os.makedirs(outdir, exist_ok=True)
+    silent = os.path.join(outdir, f"{phase}.silent.mp4")
+    index = PHASE_ORDER.index(phase) + 1
+    cmd = [sys.executable, os.path.join(HERE, "render_video.py"),
+           "--cast", cast, "--narration", nar, "--out", silent,
+           "--lang", lang, "--phase-no", str(index),
+           "--phase-total", str(len(PHASE_ORDER))]
+    if phase in DIAGRAM_PHASES:
+        cmd.append("--diagram")
+    if art:
+        cmd += ["--art", art]
+    run(cmd)
+    return silent
+
+
+def mux_audio(phase, lang, silent):
+    """Lay every narration segment on the timeline and add it to the video."""
+    meta = json.load(open(os.path.join(CAPS, f"{phase}.json")))
+    beats = {b["id"]: b for b in meta["beats"]}
+    nar = json.load(open(os.path.join(HERE, "narration", f"{phase}.{lang}.json")))
+    vodir = os.path.join(VO, f"{phase}.{lang}")
+
+    segments = []
+    for i, seg in enumerate(nar["segments"]):
+        beat = beats.get(seg["beat"])
+        audio = os.path.join(vodir, f"{i:02d}-{seg['beat']}.mp3")
+        if beat is None or not os.path.exists(audio):
+            sys.stderr.write(f"[assemble] missing audio: {audio}\n")
+            continue
+        segments.append((beat["start"], audio))
+
+    tmp = os.path.join(OUT, "_tmp")
+    os.makedirs(tmp, exist_ok=True)
+    if not segments:
+        out = os.path.join(tmp, f"{phase}.{lang}.m4v")
+        run(["ffmpeg", "-y", "-loglevel", "error", "-i", silent,
+             "-c:v", "copy", "-an", out])
+        return out
+
+    vdur = probe(silent)
+    inputs = []
+    filters = []
+    # input 0 is the video; the narration files are inputs 1..N
+    for k, (start, audio) in enumerate(segments):
+        inputs += ["-i", audio]
+        delay = int(start * 1000)
+        # pad so every segment reaches the end of the video, then the mix is
+        # trimmed to the video length
+        filters.append(f"[{k + 1}:a]adelay={delay}|{delay},apad[aud{k}]")
+    mix_in = "".join(f"[aud{k}]" for k in range(len(segments)))
+    filters.append(f"{mix_in}amix=inputs={len(segments)}:normalize=0,"
+                   f"atrim=0:{vdur:.3f},asetpts=N/SR/TB[mixed]")
+    cmd = ["ffmpeg", "-y", "-loglevel", "error", "-i", silent] + inputs + \
+          ["-filter_complex", ";".join(filters),
+           "-map", "0:v", "-map", "[mixed]", "-c:v", "copy", "-c:a", "aac",
+           "-b:a", "160k", "-ar", "44100", "-ac", "2",
+           os.path.join(tmp, f"{phase}.{lang}.m4v")]
+    run(cmd)
+    return os.path.join(tmp, f"{phase}.{lang}.m4v")
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--phase", required=True)
+    ap.add_argument("--lang", required=True, choices=["en", "pt"])
+    ap.add_argument("--keep-silent", action="store_true")
+    args = ap.parse_args()
+
+    phase, lang = args.phase, args.lang
+    silent = make_video(phase, lang)
+    muxed = mux_audio(phase, lang, silent)
+
+    outdir = os.path.join(OUT, lang)
+    os.makedirs(outdir, exist_ok=True)
+    final = os.path.join(outdir, f"{phase}.mp4")
+    run(["ffmpeg", "-y", "-loglevel", "error", "-i", muxed,
+         "-c", "copy", "-movflags", "+faststart", final])
+    if not args.keep_silent:
+        os.remove(silent)
+    os.remove(muxed)
+
+    dur = probe(final)
+    size = os.path.getsize(final) / 1e6
+    print(f"[assemble] {final}  {dur:.1f}s  {size:.1f}MB")
+
+
+if __name__ == "__main__":
+    main()

scripts/screencast/build-all.sh

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+# Build every screencast: capture -> narration -> video -> full movie.
+#
+#   ./scripts/screencast/build-all.sh            # everything
+#   ./scripts/screencast/build-all.sh 04-roteamento
+set -uo pipefail
+cd "$(dirname "$0")/../.."
+REPO=$(pwd)
+
+PHASES=("$@")
+[ ${#PHASES[@]} -eq 0 ] && PHASES=(00-ambiente 01-implementacao 02-ativacao 03-regras \
+	04-roteamento 05-balanceamento 06-qos)
+
+echo "== narration audio =="
+for lang in en pt; do
+	python3 scripts/screencast/make_audio.py --lang "$lang" | tail -1
+done
+
+echo "== sessions =="
+for p in "${PHASES[@]}"; do
+	# the activation phase shows the module being loaded, so it is the one
+	# that must not have it preloaded
+	extra=""
+	[ "$p" = "02-ativacao" ] || extra="--ensure-pf"
+	python3 scripts/screencast/record.py --plan "scripts/screencast/plans/$p.json" \
+		--out "out/screencast/caps/$p.cast" --pace $extra
+done
+
+echo "== videos =="
+for lang in en pt; do
+	for p in "${PHASES[@]}"; do
+		python3 scripts/screencast/assemble.py --phase "$p" --lang "$lang" || echo "FALHOU: $p $lang"
+	done
+done
+
+echo "== full movie =="
+for lang in en pt; do
+	python3 scripts/screencast/join.py --lang "$lang" || echo "FALHOU: join $lang"
+done
+
+echo
+echo "== resultado =="
+for f in out/screencast/en/*.mp4 out/screencast/pt/*.mp4; do
+	[ -e "$f" ] || continue
+	d=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$f")
+	a=$(ffprobe -v error -select_streams a -show_entries stream=codec_name -of csv=p=0 "$f")
+	printf '%-52s %6.1fs  audio=%s\n' "$f" "$d" "${a:-none}"
+done

scripts/screencast/join.py

@@ -0,0 +1,189 @@
+#!/usr/bin/env python3
+"""
+Join the phases into one continuous video per language, with transitions.
+
+Each phase fades into a short "chapter" card that names it, and the card
+fades into the next phase - one continuous stream, no hard cuts, so the
+viewer always knows where they are.
+
+    ./scripts/screencast/join.py --lang en
+"""
+import argparse
+import os
+import subprocess
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+REPO = os.path.dirname(os.path.dirname(HERE))
+OUT = os.path.join(REPO, "out", "screencast")
+ART = os.path.join(OUT, "art")
+
+PHASE_ORDER = ["00-ambiente", "01-implementacao", "02-ativacao", "03-regras",
+               "04-roteamento", "05-balanceamento", "06-qos"]
+
+TITLES = {
+    "00-ambiente": ("The lab environment", "O ambiente de testes",
+                    "Four Alpine VMs under libvirt", "Quatro VMs Alpine sob libvirt"),
+    "01-implementacao": ("Implementation in the kernel", "Implementacao no kernel",
+                         "OpenBSD 8.0 sources + a Linux compatibility layer",
+                         "Fontes do OpenBSD 8.0 + camada de compatibilidade Linux"),
+    "02-ativacao": ("Activation", "Ativacao",
+                    "insmod pf.ko, then pfctl over the OpenBSD ioctl ABI",
+                    "insmod pf.ko e o pfctl pela ABI de ioctls do OpenBSD"),
+    "03-regras": ("Basic rules", "Regras basicas",
+                  "Tables, anchors, keep state, block drop",
+                  "Tabelas, anchors, keep state, block drop"),
+    "04-roteamento": ("Routing", "Roteamento",
+                      "route-to and reply-to", "route-to e reply-to"),
+    "05-balanceamento": ("Load balancing", "Balanceamento",
+                         "Pools round-robin, source-hash, dup-to",
+                         "Pools round-robin, source-hash, dup-to"),
+    "06-qos": ("Quality of service", "Qualidade de servico",
+               "hfsc queues in the data path", "Filas hfsc no caminho de dados"),
+}
+
+
+def run(cmd):
+    subprocess.run(cmd, check=True)
+
+
+def probe(path):
+    out = subprocess.run(["ffprobe", "-v", "error", "-show_entries",
+                          "format=duration", "-of", "csv=p=0", path],
+                         capture_output=True, text=True).stdout.strip()
+    return float(out or 0)
+
+
+def chapter_card(phase, lang, out_path, seconds=1.8):
+    """A short card naming the phase; it is the pivot of the crossfade."""
+    title = TITLES[phase][0 if lang == "en" else 1]
+    sub = TITLES[phase][2 if lang == "en" else 3]
+    num = PHASE_ORDER.index(phase) + 1
+    art = None
+    if os.path.isdir(ART):
+        imgs = sorted(f for f in os.listdir(ART) if f.endswith((".jpg", ".png")))
+        if imgs:
+            art = os.path.join(ART, imgs[0])
+
+    script = os.path.join(OUT, "_chapter.py")
+    with open(script, "w") as f:
+        f.write(f"""
+import os, sys
+from PIL import Image, ImageDraw, ImageFont
+W, H = 1920, 1080
+BG = (12, 16, 22); ACCENT = (88, 168, 255); FG = (240, 246, 252); MUTED = (140, 150, 165)
+mono = "/usr/share/fonts/dejavu-sans-mono-fonts/DejaVuSansMono.ttf"
+sans = "/usr/share/fonts/dejavu-sans-fonts/DejaVuSans.ttf"
+sansb = "/usr/share/fonts/dejavu-sans-fonts/DejaVuSans-Bold.ttf"
+art = {art!r}
+img = Image.new("RGB", (W, H), BG)
+if art and os.path.exists(art):
+    img = Image.blend(img, Image.open(art).resize((W, H)).convert("RGB"), 0.5)
+d = ImageDraw.Draw(img, "RGBA")
+d.rectangle([0, 0, W, 8], fill=ACCENT)
+d.text((140, 430), {title!r}, font=ImageFont.truetype(sansb, 62), fill=FG)
+d.text((140, 520), {sub!r}, font=ImageFont.truetype(sans, 30), fill=MUTED)
+d.text((140, 586), "PHASE %d OF %d" % ({num}, {len(PHASE_ORDER)}),
+       font=ImageFont.truetype(mono, 24), fill=ACCENT)
+d.text((140, 640), "PF on Linux - OpenBSD PF 8.0 no kernel Linux",
+       font=ImageFont.truetype(mono, 24), fill=MUTED)
+img.save(sys.argv[1])
+""")
+    png = out_path + ".png"
+    run([sys.executable, script, png])
+    have = subprocess.run(["ffmpeg", "-hide_banner", "-encoders"],
+                          capture_output=True, text=True).stdout
+    venc = (["-c:v", "libx264", "-preset", "veryfast", "-crf", "20", "-profile:v", "high"]
+            if "libx264" in have else
+            ["-c:v", "libopenh264", "-b:v", "6M", "-profile:v", "high"])
+    run(["ffmpeg", "-y", "-loglevel", "error", "-loop", "1", "-i", png,
+         "-t", str(seconds), "-r", "30"] + venc + ["-pix_fmt", "yuv420p", out_path])
+    os.remove(png)
+    os.remove(script)
+    return out_path
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--lang", required=True, choices=["en", "pt"])
+    ap.add_argument("--out")
+    ap.add_argument("--fade", type=float, default=0.7)
+    args = ap.parse_args()
+
+    lang = args.lang
+    tmp = os.path.join(OUT, "_tmp")
+    os.makedirs(tmp, exist_ok=True)
+
+    segs = []          # (path, duration)
+    for phase in PHASE_ORDER:
+        src = os.path.join(OUT, lang, f"{phase}.mp4")
+        if not os.path.exists(src):
+            sys.stderr.write(f"[join] missing {src}, skipping\n")
+            continue
+        segs.append((src, probe(src)))
+        card = chapter_card(phase, lang, os.path.join(tmp, f"card-{phase}.mp4"))
+        segs.append((card, probe(card)))
+
+    if not segs:
+        raise SystemExit("[join] nothing to join")
+
+    inputs = []
+    for path, _ in segs:
+        inputs += ["-i", path]
+
+    # sequential xfade: each segment fades into the next
+    filters = []
+    prev = "0:v"
+    offset = segs[0][1] - args.fade
+    for i in range(1, len(segs)):
+        filters.append(f"[{prev}][{i}:v]xfade=transition=fade:"
+                       f"duration={args.fade}:offset={offset:.3f}[v{i}]")
+        prev = f"v{i}"
+        offset += segs[i][1] - args.fade
+
+    # The audio has to follow the same timeline: with xfade the segments
+    # overlap by the fade duration, so each one starts `fade` earlier than
+    # the plain sum would suggest.
+    audio_at = []
+    acc = 0.0
+    for i, (_, dur) in enumerate(segs):
+        audio_at.append(max(0.0, acc - args.fade * i))
+        acc += dur
+
+    # Only the phase videos carry narration; the chapter cards are silent.
+    # Their input indices are the even ones in `inputs`.
+    phase_idx = [i for i in range(len(segs)) if i % 2 == 0]
+    afilters = []
+    labels = []
+    for k, i in enumerate(phase_idx):
+        ms = int(audio_at[i] * 1000)
+        afilters.append(f"[{i}:a]adelay={ms}|{ms}[a{k}]")
+        labels.append(f"[a{k}]")
+    afilters.append("".join(labels) +
+                    f"amix=inputs={len(phase_idx)}:normalize=0,"
+                    f"atrim=0:{audio_at[-1] + segs[-1][1]:.3f},"
+                    f"asetpts=N/SR/TB[amix]")
+    filters += afilters
+    amap = "[amix]"
+
+    out = args.out or os.path.join(OUT, lang, "pf-on-linux-full.mp4")
+    have = subprocess.run(["ffmpeg", "-hide_banner", "-encoders"],
+                          capture_output=True, text=True).stdout
+    if "libx264" in have:
+        venc = ["-c:v", "libx264", "-preset", "veryfast", "-crf", "20",
+                "-profile:v", "high", "-level", "4.0"]
+    elif "libopenh264" in have:
+        venc = ["-c:v", "libopenh264", "-b:v", "6M", "-profile:v", "high"]
+    else:
+        venc = ["-c:v", "libvpx-vp9", "-b:v", "0", "-crf", "30", "-row-mt", "1",
+                "-cpu-used", "4"]
+    # the phases already carry their narration; the cards are silent
+    run(["ffmpeg", "-y", "-loglevel", "error"] + inputs +
+        ["-filter_complex", ";".join(filters), "-map", f"[{prev}]",
+         "-map", amap] + venc +
+        ["-pix_fmt", "yuv420p", "-movflags", "+faststart", out])
+    print(f"[join] {out}  {probe(out):.1f}s")
+
+
+if __name__ == "__main__":
+    main()

scripts/screencast/make_audio.py

@@ -0,0 +1,121 @@
+#!/usr/bin/env python3
+"""
+Generate the narration audio for every screencast phase with MiniMax TTS.
+
+Female voices, one per language:
+    en -> English_ConfidentWoman
+    pt -> Portuguese_ConfidentWoman
+
+Each *segment* of a narration script becomes its own audio file, so the
+renderer can start the voice exactly when the matching beat appears on
+screen and burn the same sentence as a subtitle.
+
+    ./scripts/screencast/make_audio.py --lang en
+    ./scripts/screencast/make_audio.py --lang pt --phase 04-roteamento
+"""
+import argparse
+import base64
+import json
+import os
+import sys
+import time
+import urllib.error
+import urllib.request
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+NARR = os.path.join(HERE, "narration")
+VO_DIR = os.path.join(HERE, "..", "..", "out", "screencast", "vo")
+
+VOICES = {
+    "en": {"voice_id": "English_ConfidentWoman", "language_boost": "English"},
+    "pt": {"voice_id": "Portuguese_ConfidentWoman", "language_boost": "Portuguese"},
+}
+
+# MiniMax occasionally stumbles on the punctuation of long sentences; keeping
+# the text plain (no dashes, no slashes) makes the prosody predictable.
+MODEL = "speech-02-hd"
+API = "https://api.minimax.io/v1/t2a_v2"
+
+
+def decode_audio(payload):
+    """MiniMax returns the audio as a hex string (it starts with the ASCII
+    bytes of the ID3 header, 49443304).  Older/other endpoints use base64,
+    so both are handled."""
+    if len(payload) > 8 and all(c in "0123456789abcdefABCDEF" for c in payload[:64]):
+        return bytes.fromhex(payload)
+    data = payload.replace("-", "+").replace("_", "/")
+    data += "=" * (-len(data) % 4)
+    return base64.b64decode(data, validate=False)
+
+
+def tts(text, voice, lang, out_path, retries=3):
+    body = {
+        "model": MODEL,
+        "text": text,
+        "stream": False,
+        "voice_setting": {
+            "voice_id": voice["voice_id"],
+            "speed": 0.97, "vol": 1.0, "pitch": 0, "emotion": "neutral",
+        },
+        "audio_setting": {"sample_rate": 44100, "bitrate": 128000,
+                          "format": "mp3", "channel": 1},
+        "language_boost": voice["language_boost"],
+    }
+    key = os.environ.get("MINIMAX_API_KEY")
+    if not key:
+        raise SystemExit("MINIMAX_API_KEY is not set")
+    for attempt in range(retries):
+        req = urllib.request.Request(
+            API, data=json.dumps(body).encode(),
+            headers={"Authorization": f"Bearer {key}", "Content-Type": "application/json"})
+        try:
+            with urllib.request.urlopen(req, timeout=120) as r:
+                data = json.loads(r.read())
+            if data.get("base_resp", {}).get("status_code") != 0:
+                raise RuntimeError(str(data.get("base_resp")))
+            audio = data.get("data", {}).get("audio")
+            if not audio:
+                raise RuntimeError("no audio in response")
+            with open(out_path, "wb") as f:
+                f.write(decode_audio(audio))
+            return True
+        except Exception as exc:                     # noqa: BLE001
+            sys.stderr.write(f"[tts] attempt {attempt + 1} failed: {exc}\n")
+            time.sleep(2 + attempt * 3)
+    return False
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--lang", required=True, choices=["en", "pt"])
+    ap.add_argument("--phase", help="only this phase")
+    ap.add_argument("--force", action="store_true")
+    args = ap.parse_args()
+
+    voice = VOICES[args.lang]
+    files = sorted(f for f in os.listdir(NARR) if f.endswith(f".{args.lang}.json"))
+    if args.phase:
+        files = [f for f in files if f.startswith(args.phase + ".")]
+
+    total = 0
+    for fn in files:
+        phase = fn.split(".")[0]
+        script = json.load(open(os.path.join(NARR, fn)))
+        outdir = os.path.abspath(os.path.join(VO_DIR, f"{phase}.{args.lang}"))
+        os.makedirs(outdir, exist_ok=True)
+        for i, seg in enumerate(script["segments"]):
+            out = os.path.join(outdir, f"{i:02d}-{seg['beat']}.mp3")
+            if os.path.exists(out) and not args.force:
+                print(f"skip {phase}.{args.lang} seg{i} ({seg['beat']})")
+                total += 1
+                continue
+            ok = tts(seg["text"], voice, args.lang, out)
+            size = os.path.getsize(out) if os.path.exists(out) else 0
+            print(f"{'ok ' if ok else 'FAIL'} {phase}.{args.lang} seg{i} "
+                  f"({seg['beat']}) {size} bytes")
+            total += 1
+    print(f"[audio] {total} segments processed for lang={args.lang}")
+
+
+if __name__ == "__main__":
+    main()