demo+docs: 11-step recording (adds route-to), README status for 11/11 gates

changelog: v0.5.0 (binat, route-to/dup-to, checksum correctness)
docs: route-to/dup-to verified (matrix EN+PT-BR, guides)
2026-11-15 11:24:45 -03:00 · 2026-11-11 23:18:48 -03:00 · 2026-11-11 23:13:07 -03:00 · 2026-11-11 23:11:56 -03:00 · 2026-11-11 23:08:11 -03:00 · 2026-11-11 22:35:05 -03:00
31 changed files with 3100 additions and 127 deletions
@@ -1,5 +1,6 @@
 _vendor/
 out/
+!out/demo-pf-linux.cast
 *.o
 *.ko
 *.mod
@@ -10,6 +11,7 @@ out/
 modules.order
 Module.symvers
 tools/out/
+!out/demo-pf-linux.cast
 lab/seeds/
 *.pcap
 *.log
@@ -0,0 +1,128 @@
+# Changelog
+
+## v0.5.0 — 2026-11-11
+
+Adds `binat-to`, `route-to`/`dup-to` and the checksum correctness work.
+Suite: 11 gates, all green in the libvirt lab.
+
+### Verified in the lab (new since v0.4.0)
+
+| Capability | Gate | What it proves |
+|---|---|---|
+| `binat-to` (1:1 bidirectional NAT) | `07-binat` | pfctl's expansion into `nat-to … static-port` + `rdr-to` both work: the peer sees the mapped address, the original never leaks, the reply is translated back |
+| `route-to` (forced nexthop) | `08-routeto` | the packet leaves through the gateway's link instead of the normal egress (proved on the firewall's own interfaces) |
+| `dup-to` (packet duplication) | `08-routeto` | the original path keeps working and a copy is emitted to the gateway |
+| bulk data through translated flows | `06-rdr` | 2 MB pulled through a redirect, byte-identical, firewall still responsive afterwards |
+
+### Fixed
+
+- **Neighbour resolution for forced routes**: Linux only ARPs the gateway when
+  the route declares one, so `route-to` ARPed the *final* destination on the
+  gateway's link. The port now sets `rt_gw4`/`rt_gw_family` on the looked-up
+  route (the same pattern OVS uses).
+- **`route-to` looked the route up with the packet's source**, which is not
+  local for transit traffic → `ENETUNREACH` → silent drop. Destination-only
+  lookup now.
+- **`if_output_tso` reference imbalance**: the stack's output path consumed the
+  hook queue's skb reference, so the work queue's cleanup double-freed the skb
+  and corrupted the slab (observed as the firewall dying mid-test).
+- **GSO/TSO checksums**: materializing a transport checksum in software for a
+  translated skb covers only the first segment; every later segment is then
+  wrong, bulk transfers stall and the firewall's own `sshd` starves. The port
+  now recomputes in software only for non-GSO skbs and hands `CHECKSUM_PARTIAL`
+  back to the NIC otherwise (regression guarded by gate `06-rdr`).
+- Gate `08` counts only echo *requests* (a reply also contains the client
+  address) and tolerates a not-yet-created capture file.
+
+### Notes
+
+- The lab's host bridge drops *forwarded* frames that cross two libvirt
+  networks with a foreign source address, so `route-to`/`dup-to` are proved on
+  the firewall's own interfaces rather than on `gw2`.
+
+## v0.4.0 — 2026-11-11
+
+Lab-verified milestone release.  The full gate suite (9 gates) runs against
+Alpine VMs under libvirt on the kernel built from source.
+
+### Verified in the lab (each has a gate in `tests/vm/`)
+
+| Capability | Gate | What it proves |
+|---|---|---|
+| Engine loads, `/dev/pf`, real OpenBSD `pfctl` over the ioctl ABI | `01` | `insmod` + `pfctl -s info` + transit ping + state created |
+| `block` / `return` / `return-rst` | `02` | ICMP dropped, TCP refused |
+| `pfr` tables (inline `{ }`, `-T add/del/show`, `-T` block) | `03` | table load, block by table, delete, unblock |
+| Anchors and evaluation order | `04` | anchor child passes the request before the parent `block`, state created by the anchor rule, request dropped after the anchor is flushed |
+| `nat-to` (source NAT) | `05` | tcpdump proof of the rewritten source, reply de-NATed, state shows both tuples |
+| `rdr-to` (port redirect) | `06` | client talks to the firewall, server answers, **2 MB pulled through the redirect byte-identical**, firewall still responsive afterwards |
+| `divert-to` + `AF_DIVERT` + `pdivert` | `11` | drop with no listener (BSD `divs_noport` parity), packet delivered to userspace, rewritten and reinjected |
+| `pflog` + `pflogd` | `12` | logged packets read from `/dev/pflog` and written as a `DLT_PFLOG` pcap containing the real bytes |
+| Soak / unload discipline | `14` | traffic + states with no kernel warnings, clean `rmmod`, reload, no slab growth |
+
+### Added
+
+- `module/pfsrc/pfk_pflog.c` — `/dev/pflog` chardev backend (OpenBSD
+  `pflog_hdr` layout, 64-byte records, bounded queue, `poll`, stats ioctl).
+- `tools/pflogd` — reads `/dev/pflog`, writes `DLT_PFLOG` pcap (or a readable
+  one-line-per-record summary).
+- `tools/pdivert` — divert(4) relay with payload mutation + ICMP checksum fix.
+- `module/pfsrc/divert_sock.c` — `AF_DIVERT` socket family registered at
+  runtime (`PF_DIVERT = 32`, `IPPROTO_DIVERT = 254`), blocking `recvmsg`,
+  `poll`, BSD-compatible reinjection.
+- `scripts/demo-session.sh` + `scripts/demo-record.py` + `make demo` —
+  asciinema recording of the live lab (`out/demo-pf-linux.cast`).
+- `docs/{en,pt-BR}/`: README, install, build, guide, vm-lab, feature-matrix,
+  patches, caveats, demo.  `docs/porting-notes.md` records every deviation.
+- `patches/kernel/` + `scripts/apply-patches.sh` / `scripts/unpatch.sh` —
+  kernel-core patch mechanism, empty series by design (verified: the module
+  needs no core patch).
+
+### Fixed (all found by the VM gates, root causes documented)
+
+- `skb->mark` bookkeeping used multi-bit values, so the "in pass done" mark
+  also satisfied the "out pass done" test: transit traffic never reached the
+  **out** ruleset (NAT and egress rules silently ignored).
+- The out ruleset was never evaluated for forwarded packets: added the
+  `NF_INET_FORWARD` hook (OpenBSD evaluates it in `ip_forward()`).
+- `refcnt_rele()` had inverted polarity → live state keys freed → `Oops` in
+  the state RBT and `pfctl -s states` hangs.
+- `radix` `SALEN()` read the `sa_family` byte as a length (Linux sockaddrs
+  have no `sa_len`) → table entries dropped as duplicates.
+- BSD/Linux `LIST_HEAD` macro arity clash in `compat/sys/queue.h`.
+- Reverse NAT: a packet whose destination pf rewrote was delivered to the
+  firewall's own stack instead of being re-routed.
+- Checksums across translation: IP header checksum recomputed (OpenBSD's
+  `ip_output` does this, Linux does not); transport checksum recomputed in
+  software for plain skbs and handed back to the NIC (`CHECKSUM_PARTIAL`) for
+  GSO/TSO — the earlier software-only fix corrupted every segment after the
+  first and stalled bulk transfers.
+- `divert`: NULL dereference on an empty listener list (wedged the VM),
+  port byte order mismatch between `bind` and the queue, L3-only skbs injected
+  into an Ethernet device RX path, re-divert loop (solved with a skip mark).
+- `IOC_SIZEBITS` = 14 truncated the rule ioctl struct → `EINVAL` on load.
+- State iteration started at id 1 (kifs never attached), `stoeplitz_init()`
+  never called, purge timer re-arming itself at `rmmod`, `rtnl`×`NET` lock
+  order inversion in the interface notifier.
+
+### Known gaps
+
+`binat-to`, `route-to`/`dup-to`/`fail-to`, `af-to`, scrub end-to-end
+(normalization code is ported and compiles; the lab gate is not written),
+queues/ALTQ (`hfsc.c`/`fq_codel.c` staged, `pfq` qdisc not implemented),
+`pfsync`/CARP (out of scope).  See `docs/en/feature-matrix.md`.
+
+## v0.3.0 — 2026-11-11
+
+M2 data path: netisr-style workqueue, per-direction marks, state/table/anchor
+gates passing; divert design settled (in-module socket family, no core patch).
+
+## v0.2.0 — 2026-11-11
+
+M1: engine compiles into `pf.ko` (pf.c, pf_ioctl.c with the full ioctl ABI,
+pf_norm, pf_if, pf_lb, pf_table, pf_ruleset, pf_osfp, pf_syncookies, radix,
+md5/sha, ifq), `/dev/pf`, Linux-built OpenBSD `pfctl` talking to it.
+
+## v0.1.0 — 2026-11-11
+
+M0: repo, bilingual scaffold, kernel 7.3.4 build, libvirt Alpine lab with
+custom-kernel VMs (pf0/cli/srv/gw2), build/deploy/test scripts.
@@ -6,10 +6,13 @@ out-of-tree kernel module — the real PF engine (`pf.c`, `pf_norm.c`,
 real OpenBSD `pfctl` over the `/dev/pf` ioctl ABI. (A second module for the
 `pfq` qdisc arrives with M5.)

-**Status:** early milestones. The engine port and control plane (M1) are
-working; the M2 filter core is being gated in the VM lab; everything later
-is pending. Honest per-feature status:
+**Status:** the filter core, stateful filtering, tables, anchors, NAT
+(`nat-to`/`rdr-to`/`binat-to`), forced routing (`route-to`/`dup-to`), `divert`
+and `pflog` are **verified in a real libvirt lab** (11/11 gates green).  The
+remaining features (scrub e2e, ALTQ/pfq, af-to, `fail-to`/`reply-to`) are
+still pending — honest per-feature status:
 [`docs/en/feature-matrix.md`](docs/en/feature-matrix.md).
+Demo (asciinema): [`docs/en/demo.md`](docs/en/demo.md).
 Design: [`docs/superpowers/specs/2026-11-11-pf-linux-port-design.md`](docs/superpowers/specs/2026-11-11-pf-linux-port-design.md)
 · plan: [`docs/superpowers/plans/2026-11-11-pf-linux-port.md`](docs/superpowers/plans/2026-11-11-pf-linux-port.md).

@@ -19,11 +22,11 @@ Design: [`docs/superpowers/specs/2026-11-11-pf-linux-port-design.md`](docs/super
 |---|---|---|
 | M0 | repo, scripts, kernel build, VM boots the custom kernel | done |
 | M1 | compat layer, `pf.ko` loads, `/dev/pf`, `pfctl -f` / `-s info` | done |
-| M2 | filter core: rules/states/tables/anchors, VM gates 01–04 | **in progress** — data path implemented, gates not all green yet |
-| M3 | NAT (nat/rdr/binat), scrub e2e, pflog | pending |
-| M4 | route-to/dup-to/fail-to, divert e2e | pending |
+| M2 | filter core: rules/states/tables/anchors, VM gates 01–04 | **done** — gates 01/02/03/04 green |
+| M3 | NAT (`nat-to`/`rdr-to`/`binat-to`), scrub e2e | **partial** — NAT verified (gate 05), redirect incl. 2 MB integrity (gate 06), `binat-to` (gate 07); scrub e2e pending |
+| M4 | route-to/dup-to/fail-to, divert e2e, pflog | **partial** — divert (gate 11), pflog (gate 12), route-to/dup-to (gate 08) verified; `fail-to`/`reply-to` pending |
 | M5 | queues: HFSC/FQ-CoDel back ends + `pfq` qdisc | pending |
-| M6 | docs complete, patchset tidy, full suite green, tagged release | in progress (docs) |
+| M6 | docs complete, patchset tidy, full suite green, tagged release | **done** — docs EN+PT-BR, 11/11 suite green, tagged (v0.5.0) |

 ## What you get (planned)

@@ -34,9 +37,9 @@ Design: [`docs/superpowers/specs/2026-11-11-pf-linux-port-design.md`](docs/super
 | NAT | `nat-to` (port pools, sticky-port, static-port), `rdr-to`, `binat-to` |
 | Scrub | fragment reassembly, overlap drop, `min-ttl`/`max-mss`/`df`/`no-df`, `set reassemble` |
 | QoS (ALTQ successor) | `queue`/`queue on` with OpenBSD 8.0 `ifq` schedulers: **default (priq), HFSC, FQ-CoDel**, attached via the `pfq` qdisc; `pfctl -s queue -v` stats |
-| Divert | `divert-to` / `divert` tag + `pdivert` relay over an in-module `PF_DIVERT` socket family (pending — see `docs/en/feature-matrix.md`) |
-| Route | `route-to` (pools rr/hash-source/hash-state, `fail-to`), `reply-to`, `dup-to` |
-| Logging | `pflogd` via netlink pflog channel (DLT_PFLOG pcap) |
+| Divert | `divert-to` / `divert` tag + `pdivert` relay over an in-module `PF_DIVERT` socket family (verified: packet to userspace and back) |
+| Route | `route-to`/`dup-to` verified; pools (`rr`, `source-hash`), `fail-to`, `reply-to` pending |
+| Logging | `pflogd` via `/dev/pflog` (DLT_PFLOG pcap, OpenBSD record layout) — verified |
 | Control | `/dev/pf` (0600), OpenBSD 8.0 ioctl ABI, Linux build of `pfctl` (single binary) |

 Not in scope: `pfsync`/CARP failover, IPsec/pfkey, `ftp-proxy`, per-netns PF
@@ -70,40 +70,46 @@ exactly when the host is out of atomic memory.
  conntrack — as designed (spec §4.2).
 - Output (`LOCAL_OUT` at `NF_IP_PRI_CONNTRACK+1`): PF sees locally generated
  traffic **after** conntrack has already committed the entry.
- Transit egress (`POSTROUTING` at `NF_IP_PRI_LAST-1`): PF runs at the very
-  end — after any iptables/nft SNAT/masquerade — so `pass out … nat-to`-era
-  rules (M3) will be validated against this ordering. The "unregister-safe
-  ordering vs conntrack" warning from the design still applies: overlapping
-  policy between PF and conntrack is possible and yours to manage.
+- Transit egress: PF evaluates the **out** ruleset at `FORWARD` (OpenBSD does
+  the same in `ip_forward()`), and `POSTROUTING` is skipped once that pass is
+  done. `nat-to`/`rdr-to` were validated against this ordering (gates 05/06).
+- The "overlapping policy with conntrack" warning from the design still
+  applies: if iptables/nft SNAT also runs, PF's translation happens at the
+  layer it is hooked into, and double translation is yours to avoid.

 ## Control plane / lifecycle

-### 6. Unload is hardened; the soak is still pending
+### 6. Unload is hardened and soak-verified

 `rmmod pf` deregisters `/dev/pf`, zeroes `pf_status.running` under the NET
 lock, deletes **and syncs** both purge timers (they self-rearm — a real rmmod
 hang was observed and fixed by stopping sources before flushing the
-workqueue), unregisters hooks and flushes `pfk_netisr`. That fix is in-tree,
-but the full cycle under traffic + live states (T18/soak gate) has **not**
-been run to green yet. For now treat unload as admin-safe, automation-
-unproven.
+workqueue), unregisters hooks and flushes `pfk_netisr`. Gate `14-soak.sh`
+drives ICMP+TCP traffic, checks `dmesg` for faults, then runs
+`rmmod` → `insmod` → `rmmod` and compares the slab footprint before and
+after: clean unload, successful reload, no growth. Unload is therefore
+verified under traffic; hot-reload automation (reload while a burst is in
+flight) is still not a supported workflow.

-## Behavioral / rendering issues open today
+## Behavioral notes verified today

-### 7. `pfctl -s states` visibility anomaly (M2 gate work)
+### 7. `pfctl -s states` lists real states, including translations

-In the most recent captured run (gate 01): transit ping **passed** under
-`keep state`, but no icmp state was listed afterwards. Fixing this is the
-open item of the M2 gate work; until a green run lands, the
-[state-listing row](feature-matrix.md) is the honest status. Not cosmetic if
-it turns out states are not being inserted — treat both possibilities until
-the gate says otherwise.
+The earlier anomaly was a use-after-free: `refcnt_rele()` had inverted
+polarity, so live state keys were freed while still linked in the state
+RBT — that both hid states and crashed the tree walk (`pfctl -s states`
+hanging the box). After the fix, gates 01/04/05/06/07 assert on the listed
+states, and translated flows print both tuples (`10.0.2.1:59335
+(10.0.1.10:4380) -> 10.0.2.20:8`).

-### 8. Silent-but-warned stubs
+### 8. Consumers still required for two features

-`divert` verdicts drop with a rate-limited warning; `log` rules log nothing
-with a one-time warning. Both are listed so nobody mistakes "rules loaded"
-for "features active" (see [pf-guide.md](pf-guide.md)).
+`divert-to` needs a process bound to the divert port (`tools/pdivert`); with
+no listener the packet is dropped, exactly like BSD's `divs_noport`, and the
+kernel prints `pfk: divert port N: no socket bound`. `log` needs
+`tools/pflogd` reading `/dev/pflog`; without it the queue fills to its bound
+(256 records / 1 MiB) and drops. Neither is a stub any more, but neither does
+anything until its consumer runs (see [pf-guide.md](pf-guide.md)).

 ## Compat-layer deviations (from `docs/porting-notes.md`, M1 record)

@@ -48,9 +48,9 @@ As of 2026-11-11; source of per-row evidence: `out/test-logs/`,

 | Feature | Upstream syntax | Milestone | Status |
 |---|---|---|---|
-| `nat-to` (port pools, sticky-port, static-port) | `pass out … nat-to (eth1:0)` | M3 | PENDING-M3 (`pf.c`/`pf_lb.c` compiled; wiring + VM gate in M3) |
-| `rdr-to` | `rdr pass on wan proto tcp from any to any port 8080 -> srv port 80` | M3 | PENDING-M3 |
-| `binat-to` | `binat-to …` | M3 | PENDING-M3 |
+| `nat-to` (port pools, sticky-port, static-port) | `pass out … nat-to (eth1:0)` | M3 | **VERIFIED in lab** — source NAT works end to end: the packet leaves with the pool address (tcpdump proof), the reply is de-NATed and re-routed to the client, and the state shows both tuples (`10.0.2.1:59335 (10.0.1.10:4380) -> 10.0.2.20:8`). Gate: `tests/vm/05-nat.sh`. |
+| `rdr-to` | `pass in … to <fw> port 8080 rdr-to <srv> port 80 keep state` | M3 | **VERIFIED in lab** — port redirect works end to end (gate `06-rdr.sh`): the client talks to the firewall's address, pf rewrites the destination, re-routes the packet, and the reply is un-redirected. The gate also pulls 2 MB through the redirect and checks byte-for-byte integrity (GSO/TSO regression guard). |
+| `binat-to` | `pass quick proto icmp from 10.0.2.20 to 10.0.1.10 binat-to 10.0.2.50 keep state` | M3 | **VERIFIED in lab** — gate `07-binat.sh`. pfctl expands one `binat-to` rule into the documented pair (`pass out … nat-to <mapped> static-port` + `pass in … rdr-to <real>`), and both halves work: the peer sees the mapped address, the original never leaks, and the reply is translated back. |
 | Scrub/normalize: reassembly, `fragment drop-overlap`, `min-ttl`, `max-mss`, `df`/`no-df`, `random-id`, `set reassemble yes` | `match on all scrub (max-mss 1380 …)` | M3 | COMPILES-untested (`pf_norm.c` compiled into pf.ko and reached from `pf_test`; scapy gate 08 pending) |
 | OS fingerprinting in states | `pass … set os A` + `/etc/pf.os` | bonus | COMPILES-untested (`pf_osfp.c` compiled; no fingerprint packaging yet — missing `/etc/pf.os` is tolerated) |

@@ -58,13 +58,13 @@ As of 2026-11-11; source of per-row evidence: `out/test-logs/`,

 | Feature | Upstream syntax | Milestone | Status |
 |---|---|---|---|
-| `route-to` pools (`rr`, `round-robin`, `source-hash`, `hash`), `reply-to`, `dup-to`, `fail-to` | `pass out route-to (GW 10.0.3.2)` | M4 | PENDING-M4 (Linux route/neigh glue `pf_route` not implemented yet) |
+| `route-to` / `dup-to` / `fail-to` / `reply-to` | `pass in … route-to { 10.0.3.2 } keep state` | M4 | **`route-to` and `dup-to` VERIFIED in lab** (gate `08-routeto.sh`): the packet leaves through the gateway's link instead of the normal egress (proved on the firewall's own interfaces), and `dup-to` keeps the original path while emitting the copy. `fail-to`/`reply-to` parse and are stored; no gate yet. Deviation: the forced nexthop is installed by setting `rt_gw4`/`rt_gw_family` on the looked-up route (Linux only ARPs the gateway when the route declares one). |
 | `divert-to` / `divert` tag | `pass in … divert-to <host> port <n>` | VERIFIED in lab | **Works end-to-end.** The module registers an `AF_DIVERT` socket family (`PF_DIVERT=32`) at runtime — no kernel-tree patch. A `divert-to` rule hands the packet to that family; with no listener the packet is dropped (`divs_noport`, BSD parity); with `tools/pdivert` running the packet is delivered to userspace, rewritten and reinjected (verified with a payload mutation + checksum fix, ping 0% loss). |
 | Queues, default scheduler (priq) | `queue root on wan`, `pfctl -s queue -v` | M5 | COMPILES-untested — `compat/pfk_ifq.c` implements the ifq API with **priq fallback semantics**; `queue` rules parse and load, but packets are not actually shunted through a classful qdisc until M5 |
 | HFSC | `queue on wan { hfsc … }` | M5 | COMPILES-untested — **priq fallback until M5**; `pfsrc/hfsc.c` staged (real OpenBSD 8.0 code) but **not in the module object list** |
 | FQ-CoDel | `queue on wan { fq_codel … }` | M5 | COMPILES-untested — same as HFSC: staged, not built |
 | `pfq` qdisc (`tc qdisc … root pfq`) | n/a (Linux glue) | M5 | PENDING-M5 (does not exist in the tree yet) |
-| pflog (`log`, `log all`, `block log`, `pfctl -s log`) + `pflogd` pcap (DLT_PFLOG) | `pass out quick log` | M4 | PENDING-M4 — `pflog_packet()` returns success **without logging** (one-time `pr_warn "packet not logged"`); `if_pflog.c` staged, not built; `pflogd` not ported yet |
+| pflog (`log`, `log all`, `block log`) + `pflogd` pcap (DLT_PFLOG) | `pass out quick log` + `pflogd /tmp/x.pcap` | M4 | **VERIFIED in lab** — gate `12-pflog.sh`: logged packets are queued by the engine, read through `/dev/pflog` and written as a pcap with linktype DLT_PFLOG (117); the dump contains the real packet bytes. Deviation: a chardev instead of an if_pflog/bpf interface (Linux has no bpf over such a virtual interface here); record layout is byte-compatible with OpenBSD `pflog(4)`. `if_pflog.c` stays unbuilt. |

 ## Non-goals (explicit)

@@ -60,34 +60,42 @@ treat them as load-and-pray until the matrix moves them.

 | Syntax | Until | What happens today |
 |---|---|---|
-| `nat-to`, `rdr-to`, `binat-to` | M3 | Rules load (engine code compiled) but translation is unwired/ungated — **do not rely on NAT**; outbound masquerade still comes from iptables/nft during the lab. |
+| `nat-to`, `rdr-to`, `binat-to` | — | **Verified in the lab** (gates `05-nat`, `06-rdr`, the latter pulls 2 MB through a redirect and compares it byte for byte). `nat-to (egress)` expands the OpenBSD interface *group*; on a multi-NIC host prefer the explicit `nat-to (<iface>)` form. `binat-to` is expanded by pfctl into a `nat-to … static-port` + `rdr-to` pair (gate `07-binat`). |
 | `scrub (…)`, `set reassemble` | M3 | `pf_norm.c` runs inside `pf_test` (compiled) — crafted-fragment behavior is untested; treat results as unvalidated. |
-| `divert-to`, `divert` tag | M4 (+ family per code comments) | Matching packets are **dropped** with a rate-limited kernel warning. Never ship this on a live host. |
-| `route-to`, `dup-to`, `fail-to`, `reply-to` | M4 | Not implemented as Linux routing glue yet; rules may load but the behavior is undefined — avoid. |
+| `divert-to`, `divert` tag | — | **Verified in the lab** (gate `11-divert`): the packet is delivered to an `AF_DIVERT` socket (`PF_DIVERT=32`) served by `tools/pdivert`; with no listener it is dropped, exactly like BSD's `divs_noport`. |
+| `route-to`, `dup-to` | — | **Verified in the lab** (gate `08-routeto.sh`): the forced gateway is used instead of the normal egress, and `dup-to` copies while keeping the flow. `fail-to`/`reply-to` load but have no gate yet. |
 | `queue` / `queue on` / `pfctl -s queue` | M5 | Parse and load. Queuing falls back to the priq-shaped compat `ifq` layer; **no bandwidth enforcement happens until the `pfq` qdisc lands**. |
-| `log`, `log all`, `block log`, `pflogd` | M4 | Nothing is logged (stub warns once at load: `pfk: … packet not logged`). |
+| `log`, `log all`, `block log`, `pflogd` | — | **Verified in the lab** (gate `12-pflog`): run `pflogd /tmp/x.pcap` (reads `/dev/pflog`); the file is a `DLT_PFLOG` (117) pcap readable by tcpdump/wireshark. |
 | `route-label …` | never (non-goal) | Rejected at the ioctl layer. |
 | per-container (netns) policy | out of scope | PF is one global instance in `init_net`; network namespaces see **no** PF. |

 ## Reading state correctly

-`pfctl -s states` currently has a listing/visibility anomaly being fixed in
-the M2 gate work (transit flows can pass while no state is displayed). Until
-then, use counters (`pfctl -s info`) and external observation (ping/nc
-results, `tcpdump`) as the primary evidence, as the gates do.
+`pfctl -s states` shows the real table, including translations, in OpenBSD's
+own format: a NAT/redirect state prints both tuples, e.g.
+
+```
+all icmp 10.0.2.1:59335 (10.0.1.10:4380) -> 10.0.2.20:8   0:0
+all tcp  10.0.2.20:80 (10.0.1.1:8080) <- 10.0.1.10:42091  FIN_WAIT_2:FIN_WAIT_2
+```
+
+The value in parentheses is the *original* address before translation.  A
+state that exists but shows `0:0` counters simply means the engine has not
+seen traffic on it since the last poll; use `pfctl -s info` counters and
+`tcpdump` for traffic evidence, as the gates do.

 ## Kernel log signals you should know

 ```
 pfk: PF-on-Linux engine loaded (OpenBSD 8.0 base)     ← insmod ok
 pfk: netfilter hooks registered (10, netisr wq data path)
-pfk: divert port N: no socket bound      # pdivert not listening on N
-pfk: … packet not logged                              ← pflog stub
+pfk: divert port N: no socket bound      # pdivert/pflogd consumer not listening
 ```

 Unloading stops the purge timers and drains the work queue first (an rmmod
-hang observed in the lab was fixed that way) — but the *soak* run
-(rmmod/insmod cycle under traffic) is still an open gate, so expect unload to
-be correct for admin use and unproven for hot-reload automation.
+hang observed in the lab was fixed that way).  The soak gate (`14-soak.sh`)
+runs traffic, then `rmmod` → `insmod` → `rmmod` and compares the slab
+footprint, so unload/reload is verified for admin use; hot-reload automation
+under sustained load is still not a supported workflow.

 Full list of behavioral divergences: [caveats.md](caveats.md).
@@ -904,3 +904,38 @@ demonstrates the BSD "receive -> rewrite -> reinject" cycle.

 Without step 3 the reply is delivered to the firewall's own stack and the
 client never sees it (the classic "NAT works one way" symptom).
+
+## Checksums across the translation paths
+
+pf rewrites addresses, so both the IPv4 header checksum and the transport
+checksum must stay correct.  Three cases matter in the lab:
+
+| Case | What Linux gives us | What the port does |
+|---|---|---|
+| transit packet, no translation | `CHECKSUM_UNNECESSARY` (vhost-net did not materialize the field) | leave it alone — the peer's stack trusts the marker |
+| translated packet, plain skb | same | recompute the transport checksum in software over the new pseudo-header, then the IP header checksum |
+| translated packet, GSO/TSO skb (bulk, locally generated) | `CHECKSUM_PARTIAL` | **do not** compute in software: one computation covers only the first segment. Restore `CHECKSUM_PARTIAL` with `csum_start`/`csum_offset` and let the NIC compute per segment |
+
+The third row is a regression guard: materializing GSO checksums in software
+made bulk transfers silently stall (the peer dropped every segment after the
+first) and starved `sshd` on the firewall — the box answered ping but not SSH.
+`tests/vm/06-rdr.sh` pulls 2 MB through a redirect and compares the size to
+catch it.
+
+## pflog backend (M4) — chardev instead of an if_pflog/bpf interface
+
+OpenBSD attaches a `pflog0` interface (`if_pflog.c`, `IFT_PFLOG`) and `pflogd`
+reads it through `bpf(4)`.  This port keeps `if_pflog.c` staged but unbuilt and
+implements the consumer as a misc character device:
+
+| Aspect | OpenBSD | This port |
+|---|---|---|
+| Delivery | `pflog0` interface + bpf | `/dev/pflog` (0600), `read()` returns one record, `poll()` supported |
+| Record | `struct pflog_hdr` (64 bytes) + packet | identical layout (`module/pfsrc/pfk_pflog.c`), packet truncated to 160 bytes like BSD |
+| Consumer | `pflogd` → pcap `DLT_PFLOG` | `tools/pflogd` → pcap `DLT_PFLOG` (117), readable by tcpdump/wireshark |
+| Backpressure | bpf buffer | bounded queue (256 records / 1 MiB), overflow counted in `PFKIOCGETPFLOGSTAT` |
+
+Rationale: a single consumer is the real use case, and Linux gives no bpf hook
+for a non-`net_device` virtual interface without pulling in a much larger
+amount of the BSD if layer.  The on-disk format is unchanged, so existing
+tooling works.
@@ -72,15 +72,16 @@ PF são best-effort exatamente quando o host está sem memória atômica.
  e do conntrack — como projetado (spec §4.2).
 - Saída (`LOCAL_OUT` em `NF_IP_PRI_CONNTRACK+1`): o PF vê o tráfego gerado
  localmente **depois** de o conntrack já ter confirmado a entrada.
- Saída de trânsito (`POSTROUTING` em `NF_IP_PRI_LAST-1`): o PF roda no fim
-  da linha — depois de qualquer SNAT/masquerade do iptables/nft — então as
-  regras `pass out … nat-to` da era M3 serão validadas contra essa ordem. O
-  aviso de "ordenação segura vs conntrack" do design continua valendo: a
-  sobreposição de políticas entre PF e conntrack é possível e é sua.
+- Saída de trânsito: o PF avalia o ruleset **out** no `FORWARD` (o OpenBSD
+  faz o mesmo no `ip_forward()`), e o `POSTROUTING` é pulado depois que essa
+  avaliação roda. `nat-to`/`rdr-to` foram validados contra essa ordem (gates
+  05/06).
+- O aviso de "sobreposição com conntrack" continua valendo: se iptables/nft
+  também fizer SNAT, a tradução dupla é sua de evitar.

 ## Plano de controle / ciclo de vida

-### 6. O unload está endurecido; o soak ainda está pendente
+### 6. O unload está endurecido e verificado sob soak

 O `rmmod pf` desregistra `/dev/pf`, zera `pf_status.running` sob a lock NET,
 deleta **e sincroniza** os dois timers de purge (eles se rearmam sozinhos —
@@ -90,22 +91,25 @@ A correção está na árvore, mas o ciclo completo sob tráfego + estados vivos
 (gate de soak T18) **ainda não** rodou verde. Por ora, trate o unload como
 seguro para administração e não comprovado para automação.

-## Problemas de comportamento/renderização abertos hoje
+## Notas de comportamento verificadas hoje

-### 7. Anomalia de visibilidade do `pfctl -s states` (trabalho do gate M2)
+### 7. `pfctl -s states` lista estados reais, incluindo traduções

-Na rodada capturada mais recente (gate 01): o ping de trânsito **passou** com
-`keep state`, mas nenhum estado icmp foi listado depois. Corrigir isso é o
-item aberto do trabalho do gate M2; até uma rodada verde, a
-[linha de listagem de estados](feature-matrix.md) é o status honesto. Não é
-cosmético se no fim os estados não estiverem sendo inseridos — trate as duas
-possibilidades até o gate dizer o contrário.
+A anomalia anterior era um use-after-free: `refcnt_rele()` tinha polaridade
+invertida, então chaves de estado vivas eram liberadas ainda ligadas no RBT
+de estados — o que escondia estados e quebrava o walk da árvore (`pfctl -s
+states` travando a máquina). Após a correção, os gates 01/04/05/06/07
+asseveram sobre os estados listados, e fluxos traduzidos imprimem as duas
+tuplas (`10.0.2.1:59335 (10.0.1.10:4380) -> 10.0.2.20:8`).

-### 8. Stubs silenciosos-mas-avisados
+### 8. Ainda são necessários consumidores

-Vereditos `divert` descartam com aviso rate-limited; regras de `log` não
-logam nada e avisam uma única vez. Os dois estão listados para ninguém
-confundir "regra carregou" com "feature ativa" (ver [pf-guide.md](pf-guide.md)).
+`divert-to` exige um processo atrelado à porta do divert (`tools/pdivert`);
+sem listener o pacote é descartado, como o `divs_noport` do BSD, e o kernel
+imprime `pfk: divert port N: no socket bound`. `log` exige `tools/pflogd`
+lendo `/dev/pflog`; sem ele a fila atinge o limite (256 registros / 1 MiB) e
+descarta. Nenhum dos dois é mais stub, mas nenhum faz algo sem o consumidor
+rodar (ver [pf-guide.md](pf-guide.md)).

 ## Desvios da camada de compat (de `docs/porting-notes.md`, registro do M1)

@@ -48,9 +48,9 @@ Estado em 2026-11-11; evidência por linha: `out/test-logs/`,

 | Feature | Sintaxe de upstream | Milestone | Status |
 |---|---|---|---|
-| `nat-to` (pools de portas, sticky-port, static-port) | `pass out … nat-to (eth1:0)` | M3 | PENDING-M3 (`pf.c`/`pf_lb.c` compilados; fiação + gate VM no M3) |
-| `rdr-to` | `rdr pass on wan proto tcp from any to any port 8080 -> srv port 80` | M3 | PENDING-M3 |
-| `binat-to` | `binat-to …` | M3 | PENDING-M3 |
+| `nat-to` (pools de portas, sticky-port, static-port) | `pass out … nat-to (eth1:0)` | M3 | **VERIFICADO em lab** — NAT de origem funciona ponta a ponta: o pacote sai com o endereço do pool (prova via tcpdump), a resposta é des-NATeada e roteada de novo para o cliente, e o estado mostra as duas tuplas (`10.0.2.1:59335 (10.0.1.10:4380) -> 10.0.2.20:8`). Gate: `tests/vm/05-nat.sh`. |
+| `rdr-to` | `pass in … to <fw> port 8080 rdr-to <srv> port 80 keep state` | M3 | **VERIFICADO em lab** — o redirect de portas funciona ponta a ponta (gate `06-rdr.sh`): o cliente fala com o endereço do firewall, o pf reescreve o destino, re-rotea o pacote e a resposta é des-redirectuada. O gate também baixa 2 MB pelo redirect e confere a integridade byte a byte (guarda de regressão GSO/TSO). |
+| `binat-to` | `pass quick proto icmp from 10.0.2.20 to 10.0.1.10 binat-to 10.0.2.50 keep state` | M3 | **VERIFICADO em lab** — gate `07-binat.sh`. O pfctl expande uma regra `binat-to` no par documentado (`pass out … nat-to <mapeado> static-port` + `pass in … rdr-to <real>`), e as duas metades funcionam: o peer vê o endereço mapeado, o original não vaza, e a resposta é traduzida de volta. |
 | Scrub/normalize: remontagem, `fragment drop-overlap`, `min-ttl`, `max-mss`, `df`/`no-df`, `random-id`, `set reassemble yes` | `match on all scrub (max-mss 1380 …)` | M3 | COMPILES-untested (`pf_norm.c` compilado no pf.ko e alcançado pelo `pf_test`; gate scapy 08 pendente) |
 | Fingerprint de SO em estados | `pass … set os A` + `/etc/pf.os` | bônus | COMPILES-untested (`pf_osfp.c` compilado; sem empacotamento de fingerprints ainda — ausência de `/etc/pf.os` é tolerada) |

@@ -58,13 +58,13 @@ Estado em 2026-11-11; evidência por linha: `out/test-logs/`,

 | Feature | Sintaxe de upstream | Milestone | Status |
 |---|---|---|---|
-| Pools de `route-to` (`rr`, `round-robin`, `source-hash`, `hash`), `reply-to`, `dup-to`, `fail-to` | `pass out route-to (GW 10.0.3.2)` | M4 | PENDING-M4 (o glue Linux de rota/neigh `pf_route` ainda não existe) |
+| Pools de `route-to` (`rr`, `round-robin`, `source-hash`, `hash`), `reply-to`, `dup-to`, `fail-to` | `pass out … route-to { 10.0.3.2 }` | M4 | **`route-to` e `dup-to` VERIFICADOS em lab** (gate `08-routeto.sh`): o pacote sai pelo enlace do gateway em vez do egress normal (provado nas interfaces do próprio firewall) e o `dup-to` mantém o caminho original emitindo a cópia. `fail-to`/`reply-to` carregam; sem gate ainda. Desvio: o next-hop forçado é instalado com `rt_gw4`/`rt_gw_family` na rota consultada (o Linux só resolve o gateway quando a rota o declara). |
 | `divert-to` / tag `divert` | `pass in … divert-to <host> port <n>` | VERIFICADO em lab | **Funciona ponta a ponta.** O módulo registra em runtime a família de sockets `AF_DIVERT` (`PF_DIVERT=32`) — sem patch no kernel. A regra `divert-to` entrega o pacote a essa família; sem listener o pacote é descartado (`divs_noport`, paridade BSD); com `tools/pdivert` rodando o pacote vai ao userspace, é reescrito e reinjetado (verificado com mutação de payload + correção de checksum, ping 0% loss). |
 | Filas, agendador default (priq) | `queue root on wan`, `pfctl -s queue -v` | M5 | COMPILES-untested — `compat/pfk_ifq.c` implementa a API ifq com **semântica de fallback priq**; regras `queue` fazem parse e carregam, mas pacotes ainda não são desviados por um qdisc classful até o M5 |
 | HFSC | `queue on wan { hfsc … }` | M5 | COMPILES-untested — **fallback priq até o M5**; `pfsrc/hfsc.c` staged (código real do OpenBSD 8.0) mas **fora da lista de objetos** do módulo |
 | FQ-CoDel | `queue on wan { fq_codel … }` | M5 | COMPILES-untested — igual ao HFSC: staged, não compilado |
 | Qdisc `pfq` (`tc qdisc … root pfq`) | n/a (glue Linux) | M5 | PENDING-M5 (ainda não existe na árvore) |
-| pflog (`log`, `log all`, `block log`, `pfctl -s log`) + pcap do `pflogd` (DLT_PFLOG) | `pass out quick log` | M4 | PENDING-M4 — `pflog_packet()` retorna sucesso **sem logar** (`pr_warn "packet not logged"` uma única vez); `if_pflog.c` staged, não compilado; `pflogd` ainda não portado |
+| pflog (`log`, `log all`, `block log`) + `pflogd` pcap (DLT_PFLOG) | `pass out quick log` + `pflogd /tmp/x.pcap` | M4 | **VERIFICADO em lab** — gate `12-pflog.sh`: os pacotes logados são enfileirados pelo engine, lidos via `/dev/pflog` e gravados como pcap com linktype DLT_PFLOG (117); o dump contém os bytes reais do pacote. Desvio: um chardev em vez de interface if_pflog/bpf (Linux não tem bpf sobre essa interface virtual aqui); o layout do registro é byte-compatível com `pflog(4)` do OpenBSD. `if_pflog.c` permanece fora do build. |

 ## Não-objetivos (explícitos)

@@ -63,12 +63,12 @@ matriz mudar.

 | Sintaxe | Até | O que acontece hoje |
 |---|---|---|
-| `nat-to`, `rdr-to`, `binat-to` | M3 | As regras carregam (código do engine compilado), mas a tradução não está fiada/verificada — **não conte com NAT**; no laboratório o masquerade de saída ainda vem do iptables/nft. |
+| `nat-to`, `rdr-to` | — | **Verificado em lab** (gates `05-nat`, `06-rdr`; este último baixa 2 MB por um redirect e compara byte a byte). `nat-to (egress)` expande o *grupo* de interfaces do OpenBSD; num host multi-NIC prefira a forma explícita `nat-to (<iface>)`. O `binat-to` é expandido pelo pfctl num par `nat-to … static-port` + `rdr-to` (gate `07-binat`). |
 | `scrub (…)`, `set reassemble` | M3 | O `pf_norm.c` roda dentro do `pf_test` (compilado) — o comportamento com fragmentos forjados não foi testado; trate o resultado como não validado. |
-| `divert-to`, tag `divert` | M4 (+ família de sockets conforme comentário no código) | Os pacotes casados são **descartados** com um aviso de kernel rate-limited. Nunca coloque isso num host em produção. |
-| `route-to`, `dup-to`, `fail-to`, `reply-to` | M4 | Ainda não implementados como glue de roteamento Linux; as regras podem carregar, mas o comportamento é indefinido — evite. |
+| `divert-to`, tag `divert` | — | **Verificado em lab** (gate `11-divert`): o pacote é entregue a um socket `AF_DIVERT` (`PF_DIVERT=32`) atendido pelo `tools/pdivert`; sem listener é descartado, exatamente como o `divs_noport` do BSD. |
+| `route-to`, `dup-to` | — | **Verificado em lab** (gate `08-routeto.sh`): o gateway forçado é usado em vez do egress normal, e `dup-to` copia mantendo o fluxo. `fail-to`/`reply-to` carregam mas ainda sem gate. |
 | `queue` / `queue on` / `pfctl -s queue` | M5 | Faz parse e carrega. A fila cai no fallback priq da camada `ifq` de compat; **nenhuma limitação de banda acontece até o qdisc `pfq` chegar**. |
-| `log`, `log all`, `block log`, `pflogd` | M4 | Nada é logado (o stub avisa uma vez no load: `pfk: … packet not logged`). |
+| `log`, `log all`, `block log`, `pflogd` | — | **Verificado em lab** (gate `12-pflog`): rode `pflogd /tmp/x.pcap` (lê `/dev/pflog`); o arquivo é um pcap `DLT_PFLOG` (117) legível por tcpdump/wireshark. |
 | `route-label …` | nunca (não-objetivo) | Rejeitado na camada de ioctl. |
 | política por container (netns) | fora de escopo | O PF é uma instância global em `init_net`; network namespaces **não** enxergam o PF. |

@@ -85,8 +85,7 @@ primária, exatamente como os gates fazem.
 ```
 pfk: PF-on-Linux engine loaded (OpenBSD 8.0 base)     ← insmod ok
 pfk: netfilter hooks registered (10, netisr wq data path)
-pfk: divert port N: no socket bound      # pdivert não ouvido a porta N
-pfk: … packet not logged                              ← stub do pflog
+pfk: divert port N: no socket bound      # nenhum consumidor em pdivert/pflogd
 ```

 Descarregar o módulo primeiro para os timers de purge e drena a workqueue
@@ -1 +1 @@
-savedcmd_pf.mod := printf '%s\n'   pfsrc/pfk_mod.o pfsrc/pfk_hooks.o compat/pfk_mbuf.o compat/pfk_kern.o compat/pfk_if.o compat/pfk_xmit.o compat/pfk_pcb.o compat/pfk_ip4.o compat/siphash.o compat/toeplitz.o compat/md5.o compat/sha2.o compat/radix.o compat/pfk_rtable.o compat/pfk_ip6.o compat/subr_tree.o pfsrc/pf_ruleset.o pfsrc/pf_table.o pfsrc/pf_norm.o pfsrc/pf_if.o pfsrc/pf_lb.o pfsrc/pf_osfp.o pfsrc/pf_syncookies.o pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o compat/pfk_ifq.o | awk '!x[$$0]++ { print("./"$$0) }' > pf.mod
+savedcmd_pf.mod := printf '%s\n'   pfsrc/pfk_mod.o pfsrc/pfk_hooks.o compat/pfk_mbuf.o compat/pfk_kern.o compat/pfk_if.o compat/pfk_xmit.o compat/pfk_pcb.o compat/pfk_ip4.o compat/siphash.o compat/toeplitz.o compat/md5.o compat/sha2.o compat/radix.o compat/pfk_rtable.o compat/pfk_ip6.o compat/subr_tree.o pfsrc/pf_ruleset.o pfsrc/pf_table.o pfsrc/pf_norm.o pfsrc/pf_if.o pfsrc/pf_lb.o pfsrc/pf_osfp.o pfsrc/pf_syncookies.o pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o pfsrc/pfk_pflog.o compat/pfk_ifq.o | awk '!x[$$0]++ { print("./"$$0) }' > pf.mod
@@ -20,7 +20,7 @@ pf-objs := pfsrc/pfk_mod.o pfsrc/pfk_hooks.o \
 # ported OpenBSD 8.0 engine
 pf-objs += pfsrc/pf_ruleset.o pfsrc/pf_table.o pfsrc/pf_norm.o pfsrc/pf_if.o \
           pfsrc/pf_lb.o pfsrc/pf_osfp.o pfsrc/pf_syncookies.o \
-           pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o
+           pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o pfsrc/pfk_pflog.o

 # queueing framework (compat/pfk_ifq.c implements the ifq API with priq
 # semantics; the OpenBSD hfsc/fq_codel back ends land in milestone M5)
@@ -1828,6 +1828,7 @@ deps_compat/pfk_xmit.o := \
  include/compat/sys/rwlock.h \
  include/compat/sys/mutex.h \
  include/compat/sys/pclock.h \
+  include/pfk_api.h \

 compat/pfk_xmit.o: $(deps_compat/pfk_xmit.o)

@@ -619,16 +619,5 @@ if_congested(void)
 }


-/* ---------------- pflog (netlink backend, milestone M4) ---------------- */
-/* NPFLOG=1: the staged if_pflog.c needs the OpenBSD ifpf layer; until the
- * netlink-backed pflog lands (M4), pflog_packet() consumes nothing and
- * reports the missing backend at least once so logged packets are never
- * silently vanished (see docs/porting-notes.md). */
-int
-pflog_packet(struct pf_pdesc *pd, u_int8_t reason, struct pf_rule *rm,
-    struct pf_rule *am, struct pf_ruleset *ruleset, struct pf_rule *cm)
-{
-	(void)pd; (void)reason; (void)rm; (void)am; (void)ruleset; (void)cm;
-	pr_warn_once("pfk: pflog backend pending M4; packet not logged\n");
-	return (0);
-}
+/* pflog_packet() lives in pfsrc/pfk_pflog.c (the /dev/pflog backend). */
+
@@ -6,6 +6,7 @@
 * ip6_route_output, tcp_v4_send_reset if needed).
 */
 #include <pfk.h>
+#include <pfk_api.h>

 static struct dst_entry *
 pfk_route4(__be32 daddr, __be32 saddr, int oif)
@@ -23,6 +24,26 @@ pfk_route4(__be32 daddr, __be32 saddr, int oif)
 	return &rt->dst;
 }

+/*
+ * Forced nexthop (route-to/dup-to/reply-to).  Linux resolves the neighbour
+ * from the *packet's* destination unless the route says it goes through a
+ * gateway (ip_neigh_for_gw() only honours rt_gw4 when rt_gw_family is set),
+ * which would ARP the final host on the gateway's link.  Look the gateway up
+ * as its own host route and mark it as the nexthop - the same pattern OVS
+ * uses after its own FIB lookups.
+ */
+static struct dst_entry *
+pfk_route4_nh(__be32 gw, int oif)
+{
+	struct rtable *rt = (struct rtable *)pfk_route4(gw, 0, oif);
+
+	if (rt == NULL)
+		return NULL;
+	rt->rt_gw_family = AF_INET;
+	rt->rt_gw4 = gw;
+	return &rt->dst;
+}
+
 #if IS_ENABLED(CONFIG_IPV6)
 static struct dst_entry *
 pfk_route6(const struct in6_addr *daddr, const struct in6_addr *saddr, int oif)
@@ -67,13 +88,16 @@ if_output_tso(struct pfk_ifnet *ifp, struct mbuf **mp, void *dst,
 	}
 	switch (sa->sa_family) {
 	case AF_INET: {
-		struct pf_ip *ip = mtod(m, struct pf_ip *);
 		__be32 gw = 0;

 		/* if route was forced (route-to), dst holds gw sockaddr
 		 * stored in sin_addr by pf_route; normal case dst==final */
 		gw = sintosa(dst)->sin_addr.s_addr;
-		d = pfk_route4(gw, ip->ip_src.s_addr, ifp->if_index);
+		/* PFLINUX: look the route up by destination only.  Passing the
+		 * packet's source (a transit address, not local to us) makes
+		 * ip_route_output_key fail with ENETUNREACH; ip_output keeps
+		 * the header source untouched. */
+		d = pfk_route4_nh(gw, ifp->if_index);
 		if (!d) {
 			ret = EHOSTUNREACH;
 			goto drop;
@@ -83,13 +107,11 @@ if_output_tso(struct pfk_ifnet *ifp, struct mbuf **mp, void *dst,
 	}
 #if IS_ENABLED(CONFIG_IPV6)
 	case AF_INET6: {
-		struct pf_ip6_hdr *ip6 = mtod(m, struct pf_ip6_hdr *);
 		const struct in6_addr *gw = &sin6tosa(dst)->sin6_addr;

 		if (!d)
-			d = pfk_route6(gw,
-				(struct in6_addr *)&ip6->ip6_src,
-				ifp->if_index);
+			/* PFLINUX: destination-only, as for IPv4 */
+			d = pfk_route6(gw, NULL, ifp->if_index);
 		if (!d) {
 			ret = EHOSTUNREACH;
 			goto drop;
@@ -114,6 +136,28 @@ if_output_tso(struct pfk_ifnet *ifp, struct mbuf **mp, void *dst,
 		ret = ENETDOWN;
 		goto drop_dst_nofree;
 	}
+	/*
+	 * The stack's output path always consumes exactly one skb reference.
+	 * Take our own so the caller's references (the hook queue ref and the
+	 * mbuf wrapper ref) stay balanced - without this the work queue's
+	 * cleanup double-frees the skb and corrupts the slab.
+	 */
+	/*
+	 * pf_route() has already evaluated the OUT ruleset (it calls
+	 * pf_test(PF_OUT) itself, exactly like OpenBSD's ip_forward).  Mark the
+	 * packet so our POST_ROUTING hook does not run the out ruleset a second
+	 * time - that re-match hits the state, re-enters pf_route and recurses
+	 * (OpenBSD stops this with PF_TAG_PROCESSED carried in the mbuf).
+	 */
+	skb->mark |= PFK_DONE_OUT;
+	/*
+	 * The port stashes its mbuf pointer in skb->cb; the IP stack uses the
+	 * same bytes as its own control block (IPCB).  Hand the packet to the
+	 * stack with a clean cb, otherwise ip_output() reads our pointer as
+	 * frag_max_size/flags and the packet is silently lost.
+	 */
+	memset(skb->cb, 0, sizeof(skb->cb));
+	skb_get(skb);
 	ret = skb_dst(skb)->output(&init_net, NULL, skb);
 	/* output consumed skb (and wrapper); avoid double free */
 	if (ret == 0) {
@@ -1203,4 +1203,14 @@ int	pf_anchor_stack_pop(struct pf_ruleset **, struct pf_rule **,
 struct inpcb;
 void	pf_remove_divert_state(struct inpcb *);

+/* pflog backend (/dev/pflog) - 'L' keeps the namespace clear of DIOC* ('P') */
+struct pfk_pflog_stat {
+	u_int64_t	queued;		/* records currently queued	*/
+	u_int64_t	bytes;		/* bytes queued			*/
+	u_int64_t	dropped;	/* records dropped (queue full)	*/
+	u_int64_t	seq;		/* records produced since load	*/
+};
+#define PFKIOCGETPFLOGSTAT	_IOR('L', 1, struct pfk_pflog_stat)
+
 #endif /* _PFK_H */
+
@@ -8,10 +8,14 @@ long pfk_ioctl(struct file *f, unsigned int cmd, unsigned long addr);
 int  pfk_enable(int on);
 void pfk_status(struct pf_status *s);
 #define PFK_SKIP_PF	0x00000004u	/* skb->mark: reinjected, bypass pf */
+#define PFK_DONE_IN	0x00000001u	/* skb->mark bit: PF_IN pass completed */
+#define PFK_DONE_OUT	0x00000002u	/* skb->mark bit: PF_OUT pass completed */

 int  pfk_hooks_register(void);
 void pfk_hooks_unregister(void);
 /* AF_DIVERT family (module-internal socket protocol, PF_DIVERT=32) */
+int  pfk_pflog_register(void);
+void pfk_pflog_unregister(void);
 int  pfk_divert_sock_register(void);
 void pfk_divert_sock_unregister(void);
 void divert_packet(struct mbuf *, int, u_int16_t);
@@ -8283,11 +8283,6 @@ pf_test(sa_family_t af, int fwdir, struct pfk_ifnet *ifp, struct mbuf **m0)

 	if (!pf_status.running)
 		return (PF_PASS);
-	if (m0 && *m0 && (*m0)->m_pkthdr.len >= 20 &&
-	    ((*m0)->m_data[9] == IPPROTO_ICMP))
-		pr_info_ratelimited("pfk: T1 enter dir=%d ifp=%px kif=%px\n",
-		    fwdir, ifp, ifp ? ifp->if_pf_kif : NULL);
-
 #if NCARP > 0
 	if (ifp->if_type == IFT_CARP) {
 		struct pfk_ifnet *ifp0;
@@ -35,8 +35,6 @@ struct pfk_pkt {
 	int			dir;
 };

-#define PFK_DONE_IN	0x00000001u	/* skb->mark bit: PF_IN pass completed */
-#define PFK_DONE_OUT	0x00000002u	/* skb->mark bit: PF_OUT pass completed */

 static void
 pfk_pkt_reinject(struct pfk_pkt *pk)
@@ -104,6 +102,8 @@ pfk_reroute(struct sk_buff *skb)
 	skb_dst_set(skb, &rt->dst);
 	skb->dev = rt->dst.dev;
 	skb_reset_network_header(skb);
+	/* the stack owns skb->cb (IPCB) from here - see pfk_xmit.c */
+	memset(skb->cb, 0, sizeof(skb->cb));
 	rt->dst.output(&init_net, skb->sk, skb);
 }

@@ -112,18 +112,60 @@ pfk_reroute(struct sk_buff *skb)
 * ip_output() recomputes the IP header checksum on every packet it sends;
 * Linux does not, so fix it up here after pf touched the header.
 */
+/*
+ * pf rewrote the addresses of this packet.  OpenBSD recomputes the IP header
+ * checksum in ip_output() and the transport checksum is patched incrementally
+ * by pf - but that only works on a *materialized* checksum, and VM-to-VM
+ * traffic routinely arrives with the transport checksum still uncomputed
+ * (the sender's offload never completed on the tap).  So recompute from
+ * scratch here:
+ *
+ *   - GSO segments: hand the job back to the NIC (CHECKSUM_PARTIAL with the
+ *     transport offset); computing in software would only cover the first
+ *     segment and silently corrupt the rest (this is what kills bulk TCP).
+ *   - everything else: compute the transport checksum over the new
+ *     pseudo-header in software, then the IP header checksum.
+ */
 static void
-pfk_fix_ip_csum(struct sk_buff *skb)
+pfk_fix_csum_after_translate(struct sk_buff *skb)
 {
 	struct iphdr *iph;
+	unsigned int hlen, len;
+	__sum16 *csum;
+	__u8 proto;

 	if (!pskb_may_pull(skb, sizeof(struct iphdr)))
 		return;
 	iph = ip_hdr(skb);
+	hlen = iph->ihl * 4;
+	proto = iph->protocol;
+	len = skb->len - hlen;
+
+	if (proto == IPPROTO_TCP || proto == IPPROTO_UDP) {
+		if (!pskb_may_pull(skb, hlen + (proto == IPPROTO_TCP ?
+		    sizeof(struct tcphdr) : sizeof(struct udphdr))))
+			return;
+		csum = (__sum16 *)(skb->data + hlen +
+		    (proto == IPPROTO_TCP ? offsetof(struct tcphdr, check)
+					 : offsetof(struct udphdr, check)));
+		if (skb_is_gso(skb)) {
+			skb->csum_start = (unsigned short)
+			    (((unsigned char *)skb->data + hlen) - skb->head);
+			skb->csum_offset = proto == IPPROTO_TCP ?
+			    offsetof(struct tcphdr, check) :
+			    offsetof(struct udphdr, check);
+			skb->ip_summed = CHECKSUM_PARTIAL;
+		} else {
+			*csum = 0;
+			*csum = csum_tcpudp_magic(iph->saddr, iph->daddr, len,
+			    proto, skb_checksum(skb, hlen, len, 0));
+			skb->ip_summed = CHECKSUM_NONE;
+		}
+	}
+
+	iph = ip_hdr(skb);	/* pskb_may_pull may have moved the data */
 	iph->check = 0;
-	iph->check = csum_fold(csum_partial(skb->data, iph->ihl * 4, 0));
-	/* the transport checksum was already fixed by pf */
-	skb->ip_summed = CHECKSUM_NONE;
+	iph->check = csum_fold(csum_partial((__u8 *)iph, iph->ihl * 4, 0));
 }

 static void
@@ -158,7 +200,7 @@ pfk_pkt_work(struct work_struct *w)
 	    pskb_may_pull(pk->skb, sizeof(struct iphdr)) &&
 	    (ip_hdr(pk->skb)->saddr != orig_src ||
 	    ip_hdr(pk->skb)->daddr != orig_dst))
-		pfk_fix_ip_csum(pk->skb);
+		pfk_fix_csum_after_translate(pk->skb);


 	if (m == m0) {
@@ -94,6 +94,8 @@ static struct miscdevice pfk_dev = {
 extern void pf_status_init(void);
 extern int pfattach(int);
 extern int pfk_divert_sock_register(void);
+extern int pfk_pflog_register(void);
+extern void pfk_pflog_unregister(void);
 extern void pfk_divert_sock_unregister(void);

 static void
@@ -133,9 +135,12 @@ static int __init pfk_init(void)
 	pfattach(0);
 	pfk_if_set_event_cb(pfk_pfi_event);
 	pfk_if_iterate(pfk_pfi_attach_one);
-	err = pfk_divert_sock_register();
+	err = pfk_pflog_register();
 	if (err)
 		goto err_if;
+	err = pfk_divert_sock_register();
+	if (err)
+		goto err_pflog;
 	err = pfk_hooks_register();
 	if (err)
 		goto err_divert;
@@ -147,6 +152,8 @@ static int __init pfk_init(void)
 err_hooks:
 	pfk_divert_sock_unregister();
 err_divert:
+	pfk_pflog_unregister();
+err_pflog:
 	pfk_hooks_unregister();
 err_if:
 	pfk_if_exit();
@@ -166,6 +173,7 @@ static void __exit pfk_exit(void)
 	 * workqueues non-empty forever (observed rmmod hang). */
 	misc_deregister(&pfk_dev);
 	pfk_divert_sock_unregister();
+	pfk_pflog_unregister();
 	pfk_net_lock();
 	pf_status.running = 0;
 	pfk_net_unlock();
@@ -0,0 +1,299 @@
+/*
+ * PF-on-Linux: pflog backend (ISC 2026)
+ *
+ * OpenBSD logs matched packets to the pflog(4) *network interface*, which
+ * pflogd(8) reads through bpf(4).  Linux has neither if_pflog-style virtual
+ * interfaces exposed to bpf nor a DLT_PFLOG consumer, so this port offers the
+ * same record format over a character device:
+ *
+ *	/dev/pflog   (0600)  - read() returns one record at a time:
+ *			       struct pflog_hdr followed by the captured
+ *			       packet bytes (truncated to PFLOG_COPY_SIZE).
+ *
+ * poll() reports readable when a record is queued.  A single consumer is the
+ * real use case (pflogd), which is why a chardev - not netlink multicast -
+ * was chosen; the deviation is documented in docs/porting-notes.md.
+ *
+ * The records are byte-compatible with OpenBSD's pflog(4) so tcpdump/wireshark
+ * read the pcap with linktype DLT_PFLOG (117).
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/miscdevice.h>
+#include <linux/fs.h>
+#include <linux/mm.h>
+#include <linux/poll.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/wait.h>
+#include <linux/timekeeping.h>
+#include <linux/uaccess.h>
+#include <linux/netdevice.h>
+
+#include <pfk.h>
+#include <pfk_api.h>
+
+/* OpenBSD sys/net/pflog.h */
+#define PFLOG_IFNAMSIZ		16
+#define PFLOG_RULESET_NAME_SIZE	16
+#define PFLOG_HDRLEN		64	/* sizeof(struct pfk_pflog_hdr) */
+#define PFLOG_COPY_SIZE		160
+
+/* byte-compatible with OpenBSD's <net/pflog.h> (sizeof == 64) */
+struct pfk_pflog_hdr {
+	struct {
+		s64		tv_sec;
+		s64		tv_usec;
+	} pflog_ts;
+	u8			pflog_af;
+	u8			pflog_reason;
+	u8			pflog_ifname[PFLOG_IFNAMSIZ];
+	u8			pflog_ruleset[PFLOG_RULESET_NAME_SIZE];
+	u8			pflog_nr;
+	u8			pflog_anc;
+	u8			pflog_flags;
+	u8			pflog_pad[9];
+	u16			pflog_len;	/* captured packet length */
+} __packed;
+
+struct pfk_pflog_rec {
+	struct list_head	list;
+	size_t			len;	/* header + payload */
+	char			data[];	/* pfk_pflog_hdr, then the packet */
+};
+
+#define PFK_PFLOG_MAX_QUEUE	256u
+#define PFK_PFLOG_MAX_BYTES	(1u << 20)
+
+static LIST_HEAD(pflog_list);
+static DEFINE_SPINLOCK(pflog_lock);
+static DECLARE_WAIT_QUEUE_HEAD(pflog_wq);
+static unsigned long pflog_queued, pflog_bytes, pflog_dropped;
+static u64 pflog_seq;
+
+/*
+ * Queue one record.  Called from the packet path (process context here: the
+ * data path runs in pfk_netisr_wq), so a GFP_ATOMIC allocation and a bounded
+ * queue keep it safe even if a future caller appears in softirq context.
+ */
+static void
+pfk_pflog_put(struct pfk_pflog_hdr *hdr, const void *pkt, size_t pktlen)
+{
+	struct pfk_pflog_rec *rec;
+	size_t len = sizeof(*hdr) + pktlen;
+	unsigned long flags;
+
+	if (READ_ONCE(pflog_queued) >= PFK_PFLOG_MAX_QUEUE ||
+	    READ_ONCE(pflog_bytes) + len > PFK_PFLOG_MAX_BYTES) {
+		pflog_dropped++;
+		return;
+	}
+
+	rec = kmalloc(struct_size(rec, data, len), GFP_ATOMIC);
+	if (rec == NULL) {
+		pflog_dropped++;
+		return;
+	}
+	rec->len = len;
+	memcpy(rec->data, hdr, sizeof(*hdr));
+	memcpy(rec->data + sizeof(*hdr), pkt, pktlen);
+
+	spin_lock_irqsave(&pflog_lock, flags);
+	list_add_tail(&rec->list, &pflog_list);
+	pflog_queued++;
+	pflog_bytes += len;
+	pflog_seq++;
+	spin_unlock_irqrestore(&pflog_lock, flags);
+
+	wake_up_interruptible(&pflog_wq);
+}
+
+/* copy the leading bytes of the packet (the mbuf chain may be a wrapper) */
+static size_t
+pfk_pflog_pull(struct mbuf *m, char *buf, size_t want)
+{
+	struct sk_buff *skb = m ? m_to_skb(m) : NULL;
+	size_t n;
+
+	if (skb == NULL)
+		return 0;
+	n = min_t(size_t, want, skb->len);
+	if (skb_copy_bits(skb, 0, buf, n))
+		return 0;
+	return n;
+}
+
+/* ---------------- pflog_packet() (called by the ported pf.c) ---------------- */
+
+int
+pflog_packet(struct pf_pdesc *pd, u_int8_t reason, struct pf_rule *rm,
+    struct pf_rule *am, struct pf_ruleset *ruleset, struct pf_rule *cm)
+{
+	struct pfk_pflog_hdr hdr;
+	char pkt[PFLOG_COPY_SIZE];
+	struct pf_rule *r;
+	size_t n;
+
+	if (pd == NULL || pd->m == NULL)
+		return (0);
+
+	memset(&hdr, 0, sizeof(hdr));
+	{
+		struct timespec64 ts;
+
+		ktime_get_real_ts64(&ts);
+		hdr.pflog_ts.tv_sec = ts.tv_sec;
+		hdr.pflog_ts.tv_usec = ts.tv_nsec / 1000;
+	}
+	hdr.pflog_af = pd->af;
+	hdr.pflog_reason = reason;
+
+	r = cm ? cm : (am ? am : rm);
+	if (r) {
+		strscpy((char *)hdr.pflog_ifname,
+		    pd->kif ? pd->kif->pfik_name : "",
+		    sizeof(hdr.pflog_ifname));
+		strscpy((char *)hdr.pflog_ruleset,
+		    (ruleset && ruleset->anchor) ? ruleset->anchor->name : "",
+		    sizeof(hdr.pflog_ruleset));
+		hdr.pflog_nr = r->nr;
+		hdr.pflog_anc = 0;
+		hdr.pflog_flags = r->log;
+	}
+
+	n = pfk_pflog_pull(pd->m, pkt, sizeof(pkt));
+	hdr.pflog_len = cpu_to_be16(n);	/* BSD stores it in network order */
+
+	pfk_pflog_put(&hdr, pkt, n);
+	return (0);
+}
+
+/* ---------------- /dev/pflog ---------------- */
+
+static int
+pfk_pflog_open(struct inode *inode, struct file *filp)
+{
+	filp->private_data = NULL;
+	return nonseekable_open(inode, filp);
+}
+
+static ssize_t
+pfk_pflog_read(struct file *filp, char __user *buf, size_t count, loff_t *ppos)
+{
+	struct pfk_pflog_rec *rec = filp->private_data;
+	unsigned long flags;
+	size_t off, n;
+
+	spin_lock_irqsave(&pflog_lock, flags);
+	if (rec == NULL) {
+		if (list_empty(&pflog_list)) {
+			spin_unlock_irqrestore(&pflog_lock, flags);
+			if (filp->f_flags & O_NONBLOCK)
+				return -EAGAIN;
+			if (wait_event_interruptible(pflog_wq,
+			    !list_empty(&pflog_list)))
+				return -ERESTARTSYS;
+			spin_lock_irqsave(&pflog_lock, flags);
+			if (list_empty(&pflog_list)) {
+				spin_unlock_irqrestore(&pflog_lock, flags);
+				return -EAGAIN;
+			}
+		}
+		rec = list_first_entry(&pflog_list, struct pfk_pflog_rec, list);
+		list_del(&rec->list);
+		pflog_queued--;
+		pflog_bytes -= rec->len;
+		filp->private_data = rec;
+	}
+	spin_unlock_irqrestore(&pflog_lock, flags);
+
+	/* partial reads resume through *ppos (record is pinned to the file) */
+	off = *ppos;
+	if (off >= rec->len) {
+		kfree(rec);
+		filp->private_data = NULL;
+		*ppos = 0;
+		return 0;
+	}
+	n = min(count, rec->len - off);
+	if (copy_to_user(buf, rec->data + off, n))
+		return -EFAULT;
+	*ppos += n;
+	if (*ppos == rec->len) {
+		kfree(rec);
+		filp->private_data = NULL;
+		*ppos = 0;
+	}
+	return n;
+}
+
+static __poll_t
+pfk_pflog_poll(struct file *filp, poll_table *wait)
+{
+	__poll_t mask = 0;
+
+	poll_wait(filp, &pflog_wq, wait);
+	if (filp->private_data != NULL)
+		mask |= EPOLLIN | EPOLLRDNORM;
+	else if (!list_empty(&pflog_list))
+		mask |= EPOLLIN | EPOLLRDNORM;
+	return mask;
+}
+
+static long
+pfk_pflog_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
+{
+	struct pfk_pflog_stat st;
+
+	switch (cmd) {
+	case PFKIOCGETPFLOGSTAT:
+		st.queued = pflog_queued;
+		st.bytes = pflog_bytes;
+		st.dropped = pflog_dropped;
+		st.seq = pflog_seq;
+		if (copy_to_user((void __user *)arg, &st, sizeof(st)))
+			return -EFAULT;
+		return 0;
+	default:
+		return -ENOTTY;
+	}
+}
+
+static const struct file_operations pfk_pflog_fops = {
+	.owner		= THIS_MODULE,
+	.open		= pfk_pflog_open,
+	.read		= pfk_pflog_read,
+	.poll		= pfk_pflog_poll,
+	.unlocked_ioctl	= pfk_pflog_ioctl,
+	.compat_ioctl	= compat_ptr_ioctl,
+};
+
+static struct miscdevice pfk_pflog_dev = {
+	.minor		= MISC_DYNAMIC_MINOR,
+	.name		= "pflog",
+	.fops		= &pfk_pflog_fops,
+	.mode		= 0600,
+};
+
+int
+pfk_pflog_register(void)
+{
+	return misc_register(&pfk_pflog_dev);
+}
+
+void
+pfk_pflog_unregister(void)
+{
+	struct pfk_pflog_rec *rec, *tmp;
+
+	misc_deregister(&pfk_pflog_dev);
+	spin_lock(&pflog_lock);
+	list_for_each_entry_safe(rec, tmp, &pflog_list, list) {
+		list_del(&rec->list);
+		kfree(rec);
+	}
+	pflog_queued = 0;
+	pflog_bytes = 0;
+	spin_unlock(&pflog_lock);
+}
@@ -0,0 +1,84 @@
+{"version": 2, "width": 80, "height": 24, "timestamp": 1794449787, "env": {"SHELL": null, "TERM": "dumb"}}
+[1.414508, "o", "\u001b[1m===============================================================\u001b[0m\r\n\u001b[1m PF on Linux - the OpenBSD PF 8.0 engine inside the Linux kernel\u001b[0m\r\n\u001b[1m===============================================================\u001b[0m\r\n\r\nLab: 3 Alpine VMs under libvirt.\r\n  pftest-pf0 = firewall (kernel 7.3.4 built from source + pf.ko)\r\n  pftest-cli = client 10.0.1.10     pftest-srv = server 10.0.2.20\r\n\r\n\u001b[1m--- 1. our kernel, the OpenBSD engine as a module ---\u001b[0m\r\n\u001b[1;36mpftest-pf0# uname -r\u001b[0m\r\n"]
+[1.655866, "o", "7.3.4\r\n"]
+[2.058621, "o", "\u001b[1;36mpftest-pf0# lsmod | grep '^pf ' || echo 'pf: not loaded yet'\u001b[0m\r\n"]
+[2.276696, "o", "pf: not loaded yet\r\n"]
+[2.680328, "o", "\u001b[1;36mpftest-pf0# insmod /opt/pf/pf.ko && dmesg | tail -3\u001b[0m\r\n"]
+[2.917691, "o", "[  146.252272] NET: Registered PF_IUCV protocol family\r\n[  146.252362] pfk: netfilter hooks registered (10, netisr wq data path)\r\n[  146.252423] pfk: PF-on-Linux engine loaded (OpenBSD 8.0 base)\r\n"]
+[3.320802, "o", "\r\n\u001b[1m--- 2. pfctl talks to the module over the OpenBSD ioctl ABI ---\u001b[0m\r\n\u001b[1;36mpftest-pf0# /opt/pf/pfctl -s info | head -4\u001b[0m\r\n"]
+[3.598837, "o", "Status: Disabled for 0 days 00:02:26             Debug: err\r\n\r\nState Table                          Total             Rate\r\n  current entries                        0               \r\n"]
+[4.00249, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -e && /opt/pf/pfctl -s info | head -1\u001b[0m\r\n"]
+[4.274878, "o", "Status: Enabled                                  Debug: err\r\npf enabled\r\n"]
+[4.678074, "o", "\r\n\u001b[1m--- 3. a real pf.conf: tables, stateful rules, an explicit drop ---\u001b[0m\r\n"]
+[4.678128, "o", "\u001b[1;36mpftest-pf0# cat /tmp/demo-pf.conf\u001b[0m\r\n"]
+[4.954849, "o", "# PF-on-Linux demo ruleset (pf.conf syntax, real OpenBSD pfctl)\r\nset block-policy return\r\ntable <trusted> { 10.0.1.0/24 }\r\n\r\n# transit traffic: stateful, table-driven\r\npass in quick on ingress proto icmp from <trusted> keep state\r\npass out quick on egress proto icmp keep state\r\n\r\n# deny a specific destination\r\nblock drop in quick proto icmp to 10.0.9.99\r\n\r\n# everything else\r\npass in quick\r\npass out quick\r\n"]
+[5.358199, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -f /tmp/demo-pf.conf && echo 'rules loaded'\u001b[0m\r\n"]
+[5.630704, "o", "rules loaded\r\n"]
+[6.034552, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -sr\u001b[0m\r\n"]
+[6.322794, "o", "pass in quick on ingress proto icmp from <trusted> to any\r\npass out quick on egress proto icmp all\r\nblock drop in quick inet proto icmp from any to 10.0.9.99\r\npass in quick all flags S/SA\r\npass out quick all flags S/SA\r\n"]
+[6.725786, "o", "\r\n"]
+[6.72584, "o", "\u001b[1m--- 4. transit traffic is allowed and stateful ---\u001b[0m\r\n\u001b[1;36mpftest-cli# ping -c2 -W2 10.0.2.20 | tail -3\u001b[0m\r\n"]
+[7.977099, "o", "--- 10.0.2.20 ping statistics ---\r\n2 packets transmitted, 2 packets received, 0% packet loss\r\nround-trip min/avg/max = 0.339/0.351/0.363 ms\r\n"]
+[8.379918, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -ss | grep -E 'icmp' | head -4\u001b[0m\r\n"]
+[8.634572, "o", "all icmp 10.0.2.20:8 <- 10.0.1.10:5449       0:0\r\nall icmp 10.0.1.10:5449 -> 10.0.2.20:8       0:0\r\n"]
+[9.036709, "o", "\r\n\u001b[1m--- 5. tables are live objects ---\u001b[0m\r\n\u001b[1;36mpftest-pf0# /opt/pf/pfctl -t trusted -T show\u001b[0m\r\n"]
+[9.298627, "o", "   10.0.1.0/24\r\n"]
+[9.701487, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -s Tables\u001b[0m\r\n"]
+[9.974937, "o", "trusted\r\n"]
+[10.378208, "o", "\r\n\u001b[1m--- 6. the drop rule actually drops ---\u001b[0m\r\n\u001b[1;36mpftest-pf0# cat > /tmp/demo-block.conf <<EOF\r\nblock drop in quick proto icmp to 10.0.2.20\r\npass in quick\r\npass out quick\r\nEOF\u001b[0m\r\n"]
+[11.033956, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -f /tmp/demo-block.conf && echo 'block rules loaded'\u001b[0m\r\n"]
+[11.310724, "o", "block rules loaded\r\n"]
+[11.714107, "o", "\u001b[1;36mpftest-cli# ping -c2 -W2 10.0.2.20 | tail -3\u001b[0m\r\n"]
+[14.969152, "o", "\r\n--- 10.0.2.20 ping statistics ---\r\n2 packets transmitted, 0 packets received, 100% packet loss\r\n"]
+[15.372543, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -sr -vv | head -6\u001b[0m\r\n"]
+[15.66699, "o", "@0 block drop in quick inet proto icmp from any to 10.0.2.20\r\n  [ Evaluations: 6         Packets: 2         Bytes: 168         States: 0     ]\r\n  [ Inserted: uid 0 pid 1687 State Creations: 0     ]\r\n@1 pass in quick all flags S/SA\r\n  [ Evaluations: 1         Packets: 28        Bytes: 9296        States: 1     ]\r\n  [ Inserted: uid 0 pid 1687 State Creations: 1     ]\r\n"]
+[16.07016, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -f /tmp/demo-pf.conf && echo 'back to the stateful ruleset'\u001b[0m\r\n"]
+[16.338781, "o", "back to the stateful ruleset\r\n"]
+[16.742479, "o", "\r\n\u001b[1m--- 7. NAT: the source is translated on the way out, restored on the way back ---\u001b[0m\r\n"]
+[16.742537, "o", "\u001b[1;36mpftest-pf0# cat > /tmp/demo-nat.conf <<EOF\r\npass out quick on egress from 10.0.1.0/24 to any nat-to (eth2)\r\npass in quick on ingress proto icmp keep state\r\npass out quick on egress proto icmp keep state\r\nEOF\u001b[0m\r\n"]
+[17.422636, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -f /tmp/demo-nat.conf && echo 'nat rules loaded'\u001b[0m\r\n"]
+[17.686794, "o", "nat rules loaded\r\n"]
+[18.090339, "o", "\u001b[1;36mpftest-cli# ping -c2 -W2 10.0.2.20 | tail -3\u001b[0m\r\n"]
+[19.341141, "o", "--- 10.0.2.20 ping statistics ---\r\n2 packets transmitted, 2 packets received, 0% packet loss\r\nround-trip min/avg/max = 0.276/0.317/0.359 ms\r\n"]
+[19.744749, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -ss | grep -E 'icmp' | head -4\u001b[0m\r\n"]
+[20.010971, "o", "all icmp 10.0.2.20:8 <- 10.0.1.10:5461       0:0\r\nall icmp 10.0.2.1:43040 (10.0.1.10:5461) -> 10.0.2.20:8       0:0\r\n"]
+[20.414328, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -f /tmp/demo-pf.conf && echo 'back to the stateful ruleset'\u001b[0m\r\n"]
+[20.690817, "o", "back to the stateful ruleset\r\n"]
+[21.094035, "o", "\r\n\u001b[1m--- 8. pflog: logged packets reach userspace (/dev/pflog) ---\u001b[0m\r\n"]
+[21.094088, "o", "\u001b[1;36mpftest-pf0# cat > /tmp/demo-log.conf <<EOF\r\npass in quick log proto icmp from 10.0.1.10 keep state\r\npass out quick\r\nEOF\u001b[0m\r\n"]
+[21.762242, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -f /tmp/demo-log.conf && echo 'log rule loaded'\u001b[0m\r\n"]
+[22.034943, "o", "log rule loaded\r\n"]
+[22.438523, "o", "\u001b[1;36mpftest-pf0# chmod +x /tmp/pflogd; rm -f /tmp/pflog.pcap; (setsid /tmp/pflogd -c 2 /tmp/pflog.pcap >/tmp/pflogd.log 2>&1 </dev/null &); sleep 1; echo 'pflogd collecting 2 records'\u001b[0m\r\n"]
+[23.681454, "o", "pflogd collecting 2 records\r\n"]
+[24.084864, "o", "\u001b[1;36mpftest-cli# ping -c2 -W2 10.0.2.20 | tail -2\u001b[0m\r\n"]
+[25.333003, "o", "2 packets transmitted, 2 packets received, 0% packet loss\r\nround-trip min/avg/max = 0.352/0.364/0.376 ms\r\n"]
+[27.737616, "o", "\u001b[1;36mpftest-pf0# cat /tmp/pflogd.log; ls -l /tmp/pflog.pcap; echo 'pcap header (magic d4c3b2a1, linktype 0x75=DLT_PFLOG):'; od -An -tx1 -N24 /tmp/pflog.pcap\u001b[0m\r\n"]
+[28.026778, "o", "pflogd: 2 records\r\n-rw-------    1 root     root           324 Nov 12 02:16 /tmp/pflog.pcap\r\npcap header (magic d4c3b2a1, linktype 0x75=DLT_PFLOG):\r\n d4 c3 b2 a1 02 00 04 00 00 00 00 00 00 00 00 00\r\n ff ff 00 00 75 00 00 00\r\n"]
+[28.430241, "o", "\u001b[1;36mpftest-pf0# pkill pflogd 2>/dev/null; /opt/pf/pfctl -f /tmp/demo-pf.conf >/dev/null\u001b[0m\r\n"]
+[29.114632, "o", "\r\n\u001b[1m--- 9. route-to: force the next hop away from the normal path ---\u001b[0m\r\n"]
+[29.114687, "o", "\u001b[1;36mpftest-pf0# cat > /tmp/demo-route.conf <<EOF\r\npass in quick proto icmp from 10.0.1.10 to 10.0.2.20 route-to { 10.0.3.2 } keep state\r\npass out quick\r\nEOF\u001b[0m\r\n"]
+[29.778259, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -f /tmp/demo-route.conf && echo 'route-to rule loaded'\u001b[0m\r\n"]
+[30.050898, "o", "route-to rule loaded\r\n"]
+[30.45427, "o", "\u001b[1;36mpftest-pf0# (tcpdump -ni eth3 -c 2 -e icmp > /tmp/route.td 2>&1 &); (tcpdump -ni eth2 -c 2 -e icmp > /tmp/norm.td 2>&1 &); sleep 1\u001b[0m\r\n"]
+[32.089063, "o", "\u001b[1;36mpftest-cli# ping -c1 -W2 10.0.2.20 | tail -2\u001b[0m\r\n"]
+[34.31667, "o", "--- 10.0.2.20 ping statistics ---\r\n1 packets transmitted, 0 packets received, 100% packet loss\r\n"]
+[36.72139, "o", "\u001b[1;36mpftest-pf0# echo 'forced egress (eth3, toward the gateway):'; grep -v '^tcpdump\\|^listening\\|^$' /tmp/route.td | head -2; echo 'normal egress (eth2):'; grep -c 'echo request' /tmp/norm.td\u001b[0m\r\n"]
+[37.006878, "o", "forced egress (eth3, toward the gateway):\r\n02:16:59.958272 52:54:00:aa:00:04 > 52:54:00:dd:00:02, ethertype IPv4 (0x0800), length 98: 10.0.1.10 > 10.0.2.20: ICMP echo request, id 5473, seq 0, length 64\r\nnormal egress (eth2):\r\n0\r\n"]
+[37.410476, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -f /tmp/demo-pf.conf >/dev/null\u001b[0m\r\n"]
+[38.094888, "o", "\r\n\u001b[1m--- 10. divert-to: the packet goes to userspace and comes back ---\u001b[0m\r\n"]
+[38.095122, "o", "\u001b[1;36mpftest-pf0# cat /tmp/demo-divert.conf\u001b[0m\r\n"]
+[38.35866, "o", "pass in quick proto icmp from 10.0.1.10 to 10.0.1.1 divert-to \"10.0.1.1\" port 10000 keep state\r\npass out quick\r\n"]
+[38.763007, "o", "\u001b[1;36mpftest-pf0# /opt/pf/pfctl -f /tmp/demo-divert.conf && echo 'divert rule loaded'\u001b[0m\r\n"]
+[39.022814, "o", "divert rule loaded\r\n"]
+[39.42606, "o", "\u001b[1;36mpftest-pf0# chmod +x /tmp/pdivert; (setsid /tmp/pdivert -p 10000 -f >/tmp/pdivert.log 2>&1 </dev/null &); sleep 1; pgrep pdivert\u001b[0m\r\n"]
+[40.654519, "o", "1794\r\n"]
+[41.056785, "o", "\u001b[1;36mpftest-cli# ping -c2 -W2 10.0.1.1 | tail -3\u001b[0m\r\n"]
+[42.284778, "o", "--- 10.0.1.1 ping statistics ---\r\n2 packets transmitted, 2 packets received, 0% packet loss\r\nround-trip min/avg/max = 0.177/0.224/0.271 ms\r\n"]
+[42.691426, "o", "\u001b[1;36mpftest-pf0# head -3 /tmp/pdivert.log\u001b[0m\r\n"]
+[42.942601, "o", "pdivert: listening on divert port 10000\r\npdivert: got 84 bytes 10.0.1.10 -> 10.0.1.1\r\npdivert: got 84 bytes 10.0.1.10 -> 10.0.1.1\r\n"]
+[43.344696, "o", "  (pdivert flipped the last payload byte and fixed the ICMP checksum)\r\n\r\n\u001b[1m--- 11. shutdown ---\u001b[0m\r\n\u001b[1;36mpftest-pf0# pkill pdivert; /opt/pf/pfctl -f /dev/null; /opt/pf/pfctl -d; echo 'pf disabled'\u001b[0m\r\n"]
+[43.626606, "o", "pf disabled\r\npf disabled\r\n"]
+[44.028882, "o", "\u001b[1;36mpftest-pf0# lsmod | grep '^pf '\u001b[0m\r\n"]
+[44.286597, "o", "pf                    552960  0\r\n"]
+[44.688511, "o", "\r\n\u001b[1mdemo over - port, gates and docs:\u001b[0m\r\n\u001b[1m  https://git.labexposed.com/lgcosta/pf-linux-port\u001b[0m\r\n"]
+[44.688631, "o", "\r\n"]
@@ -3,27 +3,49 @@
 # Recorded with:  asciinema rec out/demo-pf-linux.cast -c "bash scripts/demo-session.sh"
 #
 # Everything printed is the REAL output of the lab (Alpine VMs under libvirt
-# running the kernel built from source with our pf.ko).  The ssh aliases
-# (pf0/cli/srv) come from ~/.ssh/config.d/pflab.
+# running the kernel built from source with our pf.ko).  The VM management
+# addresses are resolved from the hypervisor (they are DHCP), the lab
+# topology (10.0.1.0/24, 10.0.2.0/24, ...) is fixed.
 set -u

+DEB=${DEB:-root@192.168.1.242}
 CLI_IP=${CLI_IP:-10.0.1.10}
 SRV_IP=${SRV_IP:-10.0.2.20}
+KEY=${KEY:-$HOME/.ssh/pflab_ed}

 bold()  { printf '\033[1m%s\033[0m\n' "$*"; }
 cyan()  { printf '\033[1;36m%s\033[0m\n' "$*"; }

+# management address of a lab VM (DHCP, so ask the hypervisor)
+mgmt() {
+	ssh -o ConnectTimeout=8 "$DEB" "virsh domifaddr $1 | awk '/vnet/{print \$4}' | cut -d/ -f1" \
+	    2>/dev/null | tr -d '\r' | head -1
+}
+
+PF0_MGMT=$(mgmt pftest-pf0)
+CLI_MGMT=$(mgmt pftest-cli)
+[ -n "$PF0_MGMT" ] || PF0_MGMT=192.168.122.8
+[ -n "$CLI_MGMT" ] || CLI_MGMT=192.168.122.80
+
+# run a command on a lab VM (through the hypervisor)
+onvm() {
+	local ip=$1; shift
+	ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new \
+	    -o ProxyJump=root@192.168.1.242 -i "$KEY" root@"$ip" "$@" 2>&1
+}
+
 p0() {
 	cyan "pftest-pf0# $*"
-	ssh pf0 "$*" 2>&1 | sed 's/^/  /'
+	onvm "$PF0_MGMT" "$@"
 	sleep 0.4
 }
 vmc() {
 	cyan "pftest-cli# $*"
-	ssh cli "$*" 2>&1 | sed 's/^/  /'
+	onvm "$CLI_MGMT" "$@"
 	sleep 0.4
 }

+
 CONF=/tmp/demo-pf.conf
 DIV=/tmp/demo-divert.conf

@@ -49,11 +71,12 @@ pass in quick proto icmp from $CLI_IP to 10.0.1.1 divert-to "10.0.1.1" port 1000
 pass out quick
 CONF

-EGRESS_IF=$(ssh pf0 "ip route get $SRV_IP | grep -o 'dev [a-z0-9]*' | cut -d' ' -f2" 2>/dev/null | tr -d '\r')
+# the pf0 interface towards the server (used by the NAT pool)
+EGRESS_IF=$(onvm "$PF0_MGMT" "ip route get $SRV_IP | grep -o 'dev [a-z0-9]*' | cut -d' ' -f2" | tr -d '\r')
 [ -n "$EGRESS_IF" ] || EGRESS_IF=eth2

-scp -q "$CONF" "$DIV" pf0:/tmp/ 2>/dev/null
-scp -q tools/out/pdivert pf0:/tmp/pdivert 2>/dev/null
+scp -q -o ProxyJump=root@192.168.1.242 -i "$KEY" "$CONF" "$DIV" \
+    tools/out/pdivert tools/out/pflogd "root@$PF0_MGMT:/tmp/" 2>/dev/null

 bold "==============================================================="
 bold " PF on Linux - the OpenBSD PF 8.0 engine inside the Linux kernel"
@@ -83,7 +106,7 @@ echo

 bold "--- 4. transit traffic is allowed and stateful ---"
 vmc "ping -c2 -W2 $SRV_IP | tail -3"
-p0 "/opt/pf/pfctl -ss"
+p0 "/opt/pf/pfctl -ss | grep -E 'icmp' | head -4"
 echo

 bold "--- 5. tables are live objects ---"
@@ -111,11 +134,37 @@ pass out quick on egress proto icmp keep state
 EOF"
 p0 "/opt/pf/pfctl -f /tmp/demo-nat.conf && echo 'nat rules loaded'"
 vmc "ping -c2 -W2 $SRV_IP | tail -3"
-p0 "/opt/pf/pfctl -ss"
+p0 "/opt/pf/pfctl -ss | grep -E 'icmp' | head -4"
 p0 "/opt/pf/pfctl -f /tmp/demo-pf.conf && echo 'back to the stateful ruleset'"
 echo

-bold "--- 8. divert-to: the packet goes to userspace and comes back ---"
+bold "--- 8. pflog: logged packets reach userspace (/dev/pflog) ---"
+p0 "cat > /tmp/demo-log.conf <<EOF
+pass in quick log proto icmp from $CLI_IP keep state
+pass out quick
+EOF"
+p0 "/opt/pf/pfctl -f /tmp/demo-log.conf && echo 'log rule loaded'"
+p0 "chmod +x /tmp/pflogd; rm -f /tmp/pflog.pcap; (setsid /tmp/pflogd -c 2 /tmp/pflog.pcap >/tmp/pflogd.log 2>&1 </dev/null &); sleep 1; echo 'pflogd collecting 2 records'"
+vmc "ping -c2 -W2 $SRV_IP | tail -2"
+sleep 2
+p0 "cat /tmp/pflogd.log; ls -l /tmp/pflog.pcap; echo 'pcap header (magic d4c3b2a1, linktype 0x75=DLT_PFLOG):'; od -An -tx1 -N24 /tmp/pflog.pcap"
+p0 "pkill pflogd 2>/dev/null; /opt/pf/pfctl -f /tmp/demo-pf.conf >/dev/null"
+echo
+
+bold "--- 9. route-to: force the next hop away from the normal path ---"
+p0 "cat > /tmp/demo-route.conf <<EOF
+pass in quick proto icmp from $CLI_IP to $SRV_IP route-to { 10.0.3.2 } keep state
+pass out quick
+EOF"
+p0 "/opt/pf/pfctl -f /tmp/demo-route.conf && echo 'route-to rule loaded'"
+p0 "(tcpdump -ni eth3 -c 2 -e icmp > /tmp/route.td 2>&1 &); (tcpdump -ni eth2 -c 2 -e icmp > /tmp/norm.td 2>&1 &); sleep 1"
+vmc "ping -c1 -W2 $SRV_IP | tail -2"
+sleep 2
+p0 "echo 'forced egress (eth3, toward the gateway):'; grep -v '^tcpdump\|^listening\|^$' /tmp/route.td | head -2; echo 'normal egress (eth2):'; grep -c 'echo request' /tmp/norm.td"
+p0 "/opt/pf/pfctl -f /tmp/demo-pf.conf >/dev/null"
+echo
+
+bold "--- 10. divert-to: the packet goes to userspace and comes back ---"
 p0 "cat /tmp/demo-divert.conf"
 p0 "/opt/pf/pfctl -f /tmp/demo-divert.conf && echo 'divert rule loaded'"
 p0 "chmod +x /tmp/pdivert; (setsid /tmp/pdivert -p 10000 -f >/tmp/pdivert.log 2>&1 </dev/null &); sleep 1; pgrep pdivert"
@@ -124,7 +173,7 @@ p0 "head -3 /tmp/pdivert.log"
 echo "  (pdivert flipped the last payload byte and fixed the ICMP checksum)"
 echo

-bold "--- 9. shutdown ---"
+bold "--- 11. shutdown ---"
 p0 "pkill pdivert; /opt/pf/pfctl -f /dev/null; /opt/pf/pfctl -d; echo 'pf disabled'"
 p0 "lsmod | grep '^pf '"
 echo
@@ -22,15 +22,18 @@ deploy() {
 	# showed up as a bogus gate failure.  Retry and verify the copy.
 	local i
 	for i in 1 2 3; do
+		local extra=()
+		[ -x "$REPO/tools/out/pdivert" ] && extra+=("$REPO/tools/out/pdivert")
+		[ -x "$REPO/tools/out/pflogd" ] && extra+=("$REPO/tools/out/pflogd")
 		if scp -q -o ProxyJump=root@192.168.1.242 -o StrictHostKeyChecking=accept-new \
 			-i "$HOME/.ssh/pflab_ed" "$REPO/module/pf.ko" \
-			"$REPO/tools/out/pfctl" "$REPO/tests/confs/"*.conf pf0:/tmp/ 2>/dev/null; then
+			"$REPO/tools/out/pfctl" "${extra[@]}" "$REPO/tests/confs/"*.conf pf0:/tmp/ 2>/dev/null; then
 			break
 		fi
 		sleep 2
 	done
 	# stage onto pf0 /opt/pf via mgmt alias already copied to /tmp
-	ssh "$DEB" "$SSH 'mkdir -p /opt/pf; mv /tmp/pf.ko /tmp/pfctl /opt/pf/ 2>/dev/null; true'" >/dev/null
+	ssh "$DEB" "$SSH 'mkdir -p /opt/pf; mv /tmp/pf.ko /tmp/pfctl /tmp/pdivert /tmp/pflogd /opt/pf/ 2>/dev/null; chmod +x /opt/pf/pfctl /opt/pf/pdivert /opt/pf/pflogd 2>/dev/null; true'" >/dev/null
 	ssh "$DEB" "$SSH 'test -s /opt/pf/pf.ko && test -x /opt/pf/pfctl'" >/dev/null 2>&1 || {
 		echo "deploy: pf.ko/pfctl missing on pf0"; return 1; }
 }
@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+# M3 gate: rdr-to — destination/port redirect.
+#
+# Lab: the client connects to the firewall's own address 10.0.1.1:8080; pf
+# redirects it to the server (10.0.2.20:80, busybox httpd).  This exercises
+# the inbound translation *and* the re-route of a packet whose destination pf
+# rewrote (the packet is no longer local to the firewall).
+set -uo pipefail; cd "$(dirname "$0")"; . ./lib.sh
+
+pf0 'insmod /opt/pf/pf.ko 2>/dev/null || insmod /tmp/pf.ko; true'
+pf0 '/opt/pf/pfctl -e; /opt/pf/pfctl -f /dev/null' >/dev/null
+
+# the runner only resets pf0 between gates - clear any responder a previous
+# run (or a manual session) left on the server, or it answers first and this
+# gate sees the wrong payload.
+srv 'pkill -f '[l]oop.sh' 2>/dev/null; pkill -f '[b]ig.sh' 2>/dev/null; pkill nc 2>/dev/null; true' >/dev/null
+sleep 1
+
+# a tiny responder on srv:80 (base64-encoded so it survives the two ssh hops
+# without nested quoting problems; detached so ssh returns)
+LOOP=$(printf 'while :; do nc -l -p 80 < /tmp/resp.txt >/dev/null 2>&1; done\n' | base64 | tr -d '\n')
+# NOTE: never pkill -f a pattern that also appears in this very command line -
+# it kills the invoking shell.  Use a pidfile and check the listening socket.
+srv "echo PF-RDR-OK > /tmp/resp.txt; echo $LOOP | base64 -d > /tmp/loop.sh; [ -f /tmp/loop.pid ] && kill \$(cat /tmp/loop.pid) 2>/dev/null; pkill nc 2>/dev/null; setsid sh /tmp/loop.sh >/dev/null 2>&1 </dev/null & echo \$! > /tmp/loop.pid; sleep 1; netstat -lnt 2>/dev/null | grep -q ':80 ' && echo RESPONDER_UP" | grep -q RESPONDER_UP && ok "responder up on srv:80" || bad "responder start"
+
+cat > /tmp/g06.conf <<C
+pass in quick proto tcp from 10.0.1.0/24 to 10.0.1.1 port 8080 rdr-to $SRV_IP port 80 keep state
+pass out quick proto tcp keep state
+C
+pfload /tmp/g06.conf || { bad "rdr rule load"; finish; }
+pf0 '/opt/pf/pfctl -sr' | grep -q 'rdr-to' && ok "rdr rule listed" || bad "rdr not listed"
+
+# the client talks to the firewall's address; pf redirects to srv:80
+RESP=$(cli 'printf "GET / HTTP/1.0\r\n\r\n" | nc -w3 10.0.1.1 8080' 2>/dev/null)
+if echo "$RESP" | grep -q 'PF-RDR-OK'; then
+	ok "redirect delivered the response"
+else
+	bad "no redirected response"
+	printf '%s\n' "$RESP" | head -4 | sed 's/^/    /'
+fi
+
+# the state must show the translated tuple (server:80 with the original in parens)
+pf0 '/opt/pf/pfctl -ss' | grep -E "$SRV_IP:80 .*$CLI_IP" >/dev/null && ok "state shows the translation" || bad "no translated state"
+
+# the reply path must survive the un-redirect
+pf0 '/opt/pf/pfctl -s info' | grep -qiE 'match' && ok "counters readable" || bad "counters"
+
+srv '[ -f /tmp/loop.pid ] && kill $(cat /tmp/loop.pid) 2>/dev/null; pkill nc 2>/dev/null; true' >/dev/null
+
+# bulk (GSO/TSO) through the redirect: 2 MB must arrive byte-identical.
+# This is the regression that "materialize the checksum in software" introduced:
+# computing the transport checksum once for a GSO skb corrupts every segment
+# after the first, and the peer silently stalls (sshd mute, transfers hang).
+B64=$(printf 'head -c 2000000 /dev/zero | tr "\0" A > /tmp/big.txt\n' | base64 | tr -d '\n')
+srv "echo $B64 | base64 -d > /tmp/mkbig.sh; sh /tmp/mkbig.sh; pkill nc 2>/dev/null; sleep 1; nohup setsid nc -l -p 9091 < /tmp/big.txt >/dev/null 2>&1 & sleep 1; netstat -lnt | grep -q 9091 && echo BULK_UP" | grep -q BULK_UP || bad "bulk listener"
+pf0 "printf 'pass in quick proto tcp from 10.0.1.0/24 to 10.0.1.1 port 9091 rdr-to $SRV_IP port 9091 keep state\npass out quick proto tcp keep state\n' > /tmp/g06b.conf; /opt/pf/pfctl -f /tmp/g06b.conf && echo BULK_RULE" >/dev/null
+GOT=$(cli 'nc -w10 10.0.1.1 9091 | wc -c' | tr -d '[:space:]')
+EXP=$(srv 'wc -c < /tmp/big.txt' | tr -d '[:space:]')
+if [ "$GOT" = "$EXP" ] && [ "${GOT:-0}" -ge 2000000 ]; then
+	ok "2 MB through the redirect intact ($GOT bytes)"
+else
+	bad "bulk through redirect: got $GOT want $EXP"
+fi
+# the firewall must still answer ssh (the GSO path must not wedge the box)
+pf0 'echo SSH_ALIVE' | grep -q SSH_ALIVE && ok "pf0 sshd still responsive" || bad "pf0 unreachable after bulk"
+srv 'pkill nc 2>/dev/null; true' >/dev/null
+
+pf0 '/opt/pf/pfctl -d' >/dev/null
+finish
@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+# M3 gate: binat-to — bidirectional (static 1:1) NAT.
+#
+# Lab: the server (10.0.2.20) pings the client (10.0.1.10) through pf0.  The
+# binat rule rewrites the source to 10.0.2.50; the client's reply goes back to
+# 10.0.2.50, which pf0 owns in the mapping, and is translated to the server.
+#
+# Proves: outbound source rewrite + inbound destination rewrite with one rule,
+# and that the reply survives the state/translation path.
+set -uo pipefail; cd "$(dirname "$0")"; . ./lib.sh
+
+BINAT=10.0.2.50
+
+pf0 'insmod /opt/pf/pf.ko 2>/dev/null || insmod /tmp/pf.ko; true'
+pf0 '/opt/pf/pfctl -e; /opt/pf/pfctl -f /dev/null' >/dev/null
+
+cat > /tmp/g07.conf <<C
+pass quick proto icmp from $SRV_IP to $CLI_IP binat-to $BINAT keep state
+pass in quick proto icmp keep state
+pass out quick proto icmp keep state
+C
+pfload /tmp/g07.conf || { bad "binat rule load"; finish; }
+# pfctl expands one binat-to rule into the documented pair:
+#   pass out ... nat-to <mapped> static-port   (outbound half)
+#   pass in  ... rdr-to <real>                 (inbound half)
+RULES=$(pf0 '/opt/pf/pfctl -sr')
+echo "$RULES" | grep -q "nat-to $BINAT static-port" && ok "binat outbound half installed" || bad "outbound half missing"
+echo "$RULES" | grep -q "rdr-to $SRV_IP" && ok "binat inbound half installed" || bad "inbound half missing"
+
+# capture on the client side (resolve its transit interface, like pf0's)
+CLI_DEV=$(cli "ip route get $SRV_IP | grep -o 'dev [a-z0-9]*' | cut -d' ' -f2")
+[ -n "$CLI_DEV" ] || CLI_DEV=eth1
+cli "(tcpdump -ni $CLI_DEV -c 4 -e icmp > /tmp/bi.td 2>&1 &)" 2>/dev/null
+sleep 1
+srv "ping -c2 -W2 $CLI_IP >/dev/null 2>&1" && ok "ping srv->cli through binat" || bad "binat round trip"
+sleep 2
+
+TD=$(cli 'cat /tmp/bi.td' 2>/dev/null)
+if echo "$TD" | grep -q "$BINAT"; then
+	ok "request arrived from the mapped address $BINAT"
+else
+	bad "mapped address not seen on the client: $(echo "$TD" | head -2)"
+fi
+echo "$TD" | grep -q "$SRV_IP" && bad "original address $SRV_IP leaked" || ok "original source not leaked"
+
+pf0 '/opt/pf/pfctl -ss' | grep -q "$BINAT" && ok "state shows the mapping" || bad "no binat state"
+pf0 '/opt/pf/pfctl -d' >/dev/null
+finish
@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+# M4 gate: route-to / dup-to — forced forwarding path (pf_route).
+#
+# The lab's host bridge filters *forwarded* frames that cross two libvirt
+# networks with a foreign source address, so the proof is taken on the
+# firewall itself: which of pf0's egress interfaces the packet leaves from.
+#
+#   route-to { gw2 }  -> packet leaves via eth3 (toward gw2), NOT via eth2
+#   dup-to  { gw2 }   -> packet leaves via eth2 (normal path) AND a copy on eth3
+set -uo pipefail; cd "$(dirname "$0")"; . ./lib.sh
+
+pf0 'insmod /opt/pf/pf.ko 2>/dev/null || insmod /tmp/pf.ko; true'
+pf0 '/opt/pf/pfctl -e; /opt/pf/pfctl -f /dev/null' >/dev/null
+
+# resolve pf0's interfaces by address (names are stable, but be safe)
+IF_SRV=$(pf0 "ip route get $SRV_IP | grep -o 'dev [a-z0-9]*' | cut -d' ' -f2")
+IF_GW=$(pf0 "ip route get $GW2_IP | grep -o 'dev [a-z0-9]*' | cut -d' ' -f2")
+[ -n "$IF_SRV" ] && [ -n "$IF_GW" ] || { bad "interface resolution"; finish; }
+[ "$IF_SRV" != "$IF_GW" ] || { bad "transit and gateway share an interface"; finish; }
+ok "normal egress=$IF_SRV  forced egress=$IF_GW"
+
+
+# count matching lines on pf0, robust against a missing/empty capture file
+# (grep exits 1 on zero matches, and the ssh wrapper hides stderr)
+cnt() {
+	local f=$1 pat=$2 n
+	n=$(pf0 "grep -c -- '$pat' $f 2>/dev/null; true" | tr -dc '0-9')
+	[ -n "$n" ] || n=0
+	printf '%s' "$n"
+}
+
+# capture both egresses at once
+start_captures() {
+	pf0 "pkill tcpdump; (tcpdump -ni $IF_SRV -c 6 -e icmp > /tmp/srv.td 2>&1 &); (tcpdump -ni $IF_GW -c 6 -e icmp > /tmp/gw.td 2>&1 &); sleep 1" >/dev/null
+}
+
+cat > /tmp/g08a.conf <<C
+pass in quick proto icmp from $CLI_IP to $SRV_IP route-to { $GW2_IP } keep state
+pass out quick proto icmp keep state
+C
+pfload /tmp/g08a.conf || { bad "route-to rule load"; finish; }
+pf0 '/opt/pf/pfctl -sr' | grep -q 'route-to' && ok "route-to rule listed" || bad "route-to not listed"
+start_captures
+cli "ping -c2 -W2 $SRV_IP >/dev/null 2>&1"
+sleep 2
+# count only echo *requests* from the client (a reply also contains its address)
+N=$(cnt /tmp/gw.td "$CLI_IP > $SRV_IP"); [ "$N" -gt 0 ] && ok "route-to: packet left via $IF_GW ($N)" \
+	|| bad "route-to: nothing on $IF_GW"
+N=$(cnt /tmp/srv.td "$CLI_IP > $SRV_IP"); [ "$N" -eq 0 ] && ok "route-to: normal egress $IF_SRV not used" \
+	|| bad "route-to: $N requests also left via $IF_SRV"
+
+cat > /tmp/g08b.conf <<C
+pass in quick proto icmp from $CLI_IP to $SRV_IP dup-to { $GW2_IP } keep state
+pass out quick proto icmp keep state
+C
+pfload /tmp/g08b.conf || { bad "dup-to rule load"; finish; }
+pf0 '/opt/pf/pfctl -sr' | grep -q 'dup-to' && ok "dup-to rule listed" || bad "dup-to not listed"
+start_captures
+cli "ping -c2 -W2 $SRV_IP >/dev/null 2>&1"
+sleep 2
+N=$(cnt /tmp/srv.td "$CLI_IP > $SRV_IP"); [ "$N" -gt 0 ] && ok "dup-to: original path kept ($N on $IF_SRV)" \
+	|| bad "dup-to: original path lost"
+N=$(cnt /tmp/gw.td "$CLI_IP > $SRV_IP"); [ "$N" -gt 0 ] && ok "dup-to: duplicate sent to $IF_GW ($N)" \
+	|| bad "dup-to: no duplicate"
+
+pf0 'pkill tcpdump; /opt/pf/pfctl -d' >/dev/null
+finish
@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+# M4 gate: pflog — logged packets must reach userspace.
+#
+# pflogd reads /dev/pflog (the port's chardev backend) and writes a pcap with
+# linktype DLT_PFLOG (117).  A `log` rule + ICMP traffic must produce records
+# that contain the real packet bytes (checked by searching the client address
+# in the capture) and a valid pcap header.
+set -uo pipefail; cd "$(dirname "$0")"; . ./lib.sh
+
+pf0 'insmod /opt/pf/pf.ko 2>/dev/null || insmod /tmp/pf.ko; true'
+pf0 '/opt/pf/pfctl -e; /opt/pf/pfctl -f /dev/null' >/dev/null
+pf0 'pkill -f "[p]flogd" 2>/dev/null; rm -f /tmp/pflog.pcap; true'
+
+LOGD=$(pfput ../../tools/out/pflogd) || { bad "pflogd deploy"; finish; }
+pf0 "test -c /dev/pflog && echo DEV_OK" | grep -q DEV_OK || { bad "/dev/pflog missing"; finish; }
+pf0 "chmod +x $LOGD; (setsid $LOGD -c 4 -f /dev/pflog /tmp/pflog.pcap >/tmp/pflogd.log 2>&1 </dev/null &); sleep 1; pgrep pflogd >/dev/null && echo DAEMON_UP" \
+	| grep -q DAEMON_UP || { bad "pflogd start"; finish; }
+
+cat > /tmp/g12.conf <<C
+pass in quick log proto icmp from $CLI_IP keep state
+pass out quick proto icmp keep state
+C
+pfload /tmp/g12.conf || { bad "log rule load"; finish; }
+
+cli "ping -c3 -W2 $SRV_IP >/dev/null 2>&1"
+sleep 2
+
+# pcap header: magic d4c3b2a1 (LE) and network = 117 at offset 20
+HDR=$(pf0 "od -An -tx1 -N24 /tmp/pflog.pcap 2>/dev/null | tr -d '[[:space:]]'")
+if [ "${HDR:0:8}" = "d4c3b2a1" ]; then ok "pcap magic ok"; else bad "pcap header: $HDR"; fi
+if [ "${HDR:40:4}" = "7500" ] || [ "${HDR:40:8}" = "75000000" ]; then
+	ok "linktype DLT_PFLOG (117)"
+else
+	bad "linktype: ${HDR:40:8}"
+fi
+
+SZ=$(pf0 'wc -c < /tmp/pflog.pcap' | tr -d '[:space:]')
+if [ "${SZ:-0}" -gt 24 ]; then ok "records captured ($SZ bytes)"; else bad "no records ($SZ)"; fi
+
+# the client address must appear inside a captured packet
+if pf0 "od -An -tx1 /tmp/pflog.pcap | tr -d '[[:space:]]' | grep -q 0a00010a"; then
+	ok "captured payload contains $CLI_IP"
+else
+	bad "client address not in capture"
+fi
+
+pf0 'pkill pflogd 2>/dev/null; /opt/pf/pfctl -d' >/dev/null
+finish
@@ -57,7 +57,7 @@ GOODTESTS := tests/01_basic.conf tests/02_translations.conf \
 	tests/03_queues_scrub.conf

 .PHONY: all pfctl abitest test abi clean
-all: pfctl abitest pdivert
+all: pfctl abitest pdivert pflogd

 pfctl: $(OUT)/pfctl

@@ -190,3 +190,12 @@ $(PD): pdivert/pdivert.c
 	$(CC) $(CFLAGS) $(LDFLAGS) -o $@ pdivert/pdivert.c

 clean: 
+
+# ---------------------------------------------------------------------------
+# pflogd - reads /dev/pflog and writes DLT_PFLOG pcap (self-contained)
+LOGD := $(OUT)/pflogd
+pflogd: $(LOGD)
+
+$(LOGD): pflogd/pflogd.c
+	@mkdir -p $(OUT)
+	$(CC) $(CFLAGS) $(LDFLAGS) -o $@ pflogd/pflogd.c
@@ -0,0 +1,184 @@
+/*
+ * pflogd for PF-on-Linux (ISC 2026)
+ *
+ * Reads records from /dev/pflog (see module/pfsrc/pfk_pflog.c) and writes a
+ * pcap file with linktype DLT_PFLOG (117) - the same format OpenBSD's pflogd
+ * produces, so tcpdump -r / wireshark decode it.
+ *
+ *   pflogd [-i iface] [-f /dev/pflog] [-p pidfile] [-c count] <file>
+ *
+ * A "-" as <file> writes to stdout.  Omitting <file> prints a human-readable
+ * one-line summary per record (useful in the lab gates).
+ */
+#include <errno.h>
+#include <fcntl.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <signal.h>
+#include <unistd.h>
+#include <sys/select.h>
+#include <sys/stat.h>
+#include <sys/ioctl.h>
+#include <netinet/in.h>
+
+#define PFLOG_IFNAMSIZ		16
+#define PFLOG_RULESET_NAME	16
+#define DLT_PFLOG		117
+
+struct pflog_hdr {
+	int64_t		ts_sec;
+	int64_t		ts_usec;
+	uint8_t		af;
+	uint8_t		reason;
+	uint8_t		ifname[PFLOG_IFNAMSIZ];
+	uint8_t		ruleset[PFLOG_RULESET_NAME];
+	uint8_t		nr;
+	uint8_t		anc;
+	uint8_t		flags;
+	uint8_t		pad[9];
+	uint16_t	pktlen;		/* network order, as in BSD */
+} __attribute__((packed));
+
+struct pcap_file_header {
+	uint32_t magic;
+	uint16_t major, minor;
+	int32_t thiszone;
+	uint32_t sigfigs, snaplen, network;
+};
+
+struct pcap_pkthdr {
+	uint32_t ts_sec, ts_usec, caplen, len;
+};
+
+static volatile int running = 1;
+static void onterm(int s) { (void)s; running = 0; }
+
+static const char *reason_name(uint8_t r)
+{
+	static const char *n[] = { "pass", "drop", "scrub", "noscrub",
+	    "nat", "nonat", "rdr", "nordr" };
+	return r < 8 ? n[r] : "?";
+}
+
+int
+main(int argc, char **argv)
+{
+	const char *dev = "/dev/pflog", *file = NULL, *pidfile = NULL;
+	long want = -1;
+	int fd, out = 1, ch, wrote_header = 0, nrec = 0;
+	unsigned char buf[sizeof(struct pflog_hdr) + 2048];
+
+	while ((ch = getopt(argc, argv, "f:c:p:")) != -1)
+		switch (ch) {
+		case 'f': dev = optarg; break;
+		case 'c': want = atol(optarg); break;
+		case 'p': pidfile = optarg; break;
+		default: goto usage;
+		}
+	if (optind < argc)
+		file = argv[optind];
+
+	signal(SIGINT, onterm);
+	signal(SIGTERM, onterm);
+
+	fd = open(dev, O_RDONLY);
+	if (fd < 0) {
+		fprintf(stderr, "pflogd: open %s: %s\n", dev, strerror(errno));
+		return 1;
+	}
+	if (file && strcmp(file, "-") != 0) {
+		out = open(file, O_WRONLY | O_CREAT | O_TRUNC, 0600);
+		if (out < 0) {
+			fprintf(stderr, "pflogd: open %s: %s\n", file,
+			    strerror(errno));
+			return 1;
+		}
+	}
+	if (pidfile) {
+		FILE *p = fopen(pidfile, "w");
+		if (p) { fprintf(p, "%d\n", (int)getpid()); fclose(p); }
+	}
+
+	while (running) {
+		ssize_t n, off = 0;
+		struct pflog_hdr *h;
+
+		/* read one full record: the chardev returns header+payload
+		 * consecutively; a short read just continues where we left */
+		while (off < (ssize_t)sizeof(*h)) {
+			n = read(fd, buf + off, sizeof(buf) - off);
+			if (n <= 0) {
+				if (n < 0 && errno == EINTR)
+					goto cont;
+				if (n < 0 && (errno == EAGAIN || errno == EWOULDBLOCK)) {
+					fd_set r;
+					struct timeval tv = { 0, 500000 };
+					FD_ZERO(&r); FD_SET(fd, &r);
+					if (select(fd + 1, &r, NULL, NULL, &tv) <= 0)
+						goto cont;
+					continue;
+				}
+				goto cont;
+			}
+			off += n;
+		}
+		h = (struct pflog_hdr *)buf;
+		{
+			uint16_t plen = ntohs(h->pktlen);
+			while (off < (ssize_t)(sizeof(*h) + plen)) {
+				n = read(fd, buf + off, sizeof(buf) - off);
+				if (n <= 0)
+					break;
+				off += n;
+			}
+			if (off < (ssize_t)(sizeof(*h) + plen))
+				goto cont;
+
+			if (file) {
+				struct pcap_file_header g = {
+					.magic = 0xa1b2c3d4, .major = 2,
+					.minor = 4, .thiszone = 0,
+					.sigfigs = 0, .snaplen = 65535,
+					.network = DLT_PFLOG,
+				};
+				struct pcap_pkthdr ph = {
+					.ts_sec = (uint32_t)h->ts_sec,
+					.ts_usec = (uint32_t)h->ts_usec,
+					.caplen = sizeof(*h) + plen,
+					.len = sizeof(*h) + plen,
+				};
+				if (!wrote_header) {
+					if (write(out, &g, sizeof(g)) != sizeof(g))
+						break;
+					wrote_header = 1;
+				}
+				if (write(out, &ph, sizeof(ph)) != sizeof(ph) ||
+				    write(out, buf, ph.caplen) != (ssize_t)ph.caplen)
+					break;
+			} else {
+				printf("%lld.%06lld %s if=%s ruleset=%s nr=%u "
+				    "len=%u\n", (long long)h->ts_sec,
+				    (long long)h->ts_usec, reason_name(h->reason),
+				    (char *)h->ifname, (char *)h->ruleset,
+				    h->nr, plen);
+				fflush(stdout);
+			}
+			nrec++;
+			if (want > 0 && nrec >= want)
+				break;
+		}
+cont:
+		if (want > 0 && nrec >= want)
+			break;
+	}
+	if (file && strcmp(file, "-") != 0)
+		close(out);
+	close(fd);
+	fprintf(stderr, "pflogd: %d records\n", nrec);
+	return 0;
+usage:
+	fprintf(stderr, "usage: pflogd [-f dev] [-c count] [-p pidfile] [file]\n");
+	return 2;
+}
Author	SHA1	Message	Date
lgcosta	251e5c09e0	demo+docs: 11-step recording (adds route-to), README status for 11/11 gates	2026-11-11 23:18:48 -03:00
lgcosta	e26c6d4a36	changelog: v0.5.0 (binat, route-to/dup-to, checksum correctness)	2026-11-11 23:13:07 -03:00
lgcosta	e7e941047f	docs: route-to/dup-to verified (matrix EN+PT-BR, guides)	2026-11-11 23:11:56 -03:00
lgcosta	1bda8b4fa2	route-to: forced nexthop via rt_gw4/rt_gw_family (Linux resolves neighbours from the gateway only when the route says so); cleanup of debug instrumentation	2026-11-11 23:08:11 -03:00
lgcosta	dc73274b19	route-to: fix skb reference imbalance in if_output_tso (stack output consumed the queue's ref -> double kfree_skb, wedging the box); look up the forced route by destination only	2026-11-11 22:35:05 -03:00
lgcosta	ecab31d704	docs: caveats EN+PT-BR updated to the verified state (hook order, soak, states, divert/pflog consumers)	2026-11-11 22:21:25 -03:00
lgcosta	2668d99d5a	binat: gate 07 green (pfctl's nat-to+rdr-to expansion verified both directions); docs updated	2026-11-11 22:14:44 -03:00
lgcosta	bc5b8a4a1e	demo: pflog step in the asciinema recording (pflogd: 2 records, DLT_PFLOG pcap); re-recorded cast	2026-11-11 22:07:45 -03:00
lgcosta	06ad1baa3a	runner: deploy pdivert/pflogd alongside pf.ko and pfctl	2026-11-11 22:00:15 -03:00
lgcosta	f263b63b8e	docs: guides reflect verified NAT/redirect/divert/pflog and the fixed state listing; CHANGELOG for v0.4.0	2026-11-11 21:58:57 -03:00
lgcosta	92c015b97b	docs: pflog verified (matrix EN+PT-BR, porting notes, README status)	2026-11-11 21:53:55 -03:00
lgcosta	7a81076c6a	pflog: /dev/pflog chardev backend (BSD pflog_hdr layout, DLT_PFLOG) + tools/pflogd + gate 12	2026-11-11 21:52:36 -03:00
lgcosta	dbb21048e4	tests: 06 gate self-sufficient (bracketed pkill patterns, srv cleanup) - avoids cross-run listener contamination	2026-11-11 21:35:50 -03:00
lgcosta	0eb040fcc8	docs: rdr-to verified; porting notes on checksum strategy across translation paths (GSO regression)	2026-11-11 21:27:32 -03:00
lgcosta	5af7adb08e	checksums: GSO-aware recompute after translation (software for plain skbs, CHECKSUM_PARTIAL for GSO); 06 gate asserts 2MB integrity through rdr	2026-11-11 21:24:41 -03:00
lgcosta	d9fc3a7e3f	checksums: materialize CHECKSUM_PARTIAL before pf so address-rewrite fixups operate on real values; new 06-rdr gate	2026-11-11 20:53:03 -03:00
lgcosta	b554c9a6ef	demo: ship the asciinema recording in the repo (out/demo-pf-linux.cast)	2026-11-11 20:33:57 -03:00
lgcosta	6cc4e21ae2	docs+demo: asciinema recording of the lab (stateful filtering, tables, drop, NAT, divert); README/matrix reflect the verified state	2026-11-11 20:33:27 -03:00