diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..a497c02 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,88 @@ +# Changelog + +## v0.4.0 — 2026-11-11 + +Lab-verified milestone release. The full gate suite (9 gates) runs against +Alpine VMs under libvirt on the kernel built from source. + +### Verified in the lab (each has a gate in `tests/vm/`) + +| Capability | Gate | What it proves | +|---|---|---| +| Engine loads, `/dev/pf`, real OpenBSD `pfctl` over the ioctl ABI | `01` | `insmod` + `pfctl -s info` + transit ping + state created | +| `block` / `return` / `return-rst` | `02` | ICMP dropped, TCP refused | +| `pfr` tables (inline `{ }`, `-T add/del/show`, `-T` block) | `03` | table load, block by table, delete, unblock | +| Anchors and evaluation order | `04` | anchor child passes the request before the parent `block`, state created by the anchor rule, request dropped after the anchor is flushed | +| `nat-to` (source NAT) | `05` | tcpdump proof of the rewritten source, reply de-NATed, state shows both tuples | +| `rdr-to` (port redirect) | `06` | client talks to the firewall, server answers, **2 MB pulled through the redirect byte-identical**, firewall still responsive afterwards | +| `divert-to` + `AF_DIVERT` + `pdivert` | `11` | drop with no listener (BSD `divs_noport` parity), packet delivered to userspace, rewritten and reinjected | +| `pflog` + `pflogd` | `12` | logged packets read from `/dev/pflog` and written as a `DLT_PFLOG` pcap containing the real bytes | +| Soak / unload discipline | `14` | traffic + states with no kernel warnings, clean `rmmod`, reload, no slab growth | + +### Added + +- `module/pfsrc/pfk_pflog.c` — `/dev/pflog` chardev backend (OpenBSD + `pflog_hdr` layout, 64-byte records, bounded queue, `poll`, stats ioctl). +- `tools/pflogd` — reads `/dev/pflog`, writes `DLT_PFLOG` pcap (or a readable + one-line-per-record summary). +- `tools/pdivert` — divert(4) relay with payload mutation + ICMP checksum fix. +- `module/pfsrc/divert_sock.c` — `AF_DIVERT` socket family registered at + runtime (`PF_DIVERT = 32`, `IPPROTO_DIVERT = 254`), blocking `recvmsg`, + `poll`, BSD-compatible reinjection. +- `scripts/demo-session.sh` + `scripts/demo-record.py` + `make demo` — + asciinema recording of the live lab (`out/demo-pf-linux.cast`). +- `docs/{en,pt-BR}/`: README, install, build, guide, vm-lab, feature-matrix, + patches, caveats, demo. `docs/porting-notes.md` records every deviation. +- `patches/kernel/` + `scripts/apply-patches.sh` / `scripts/unpatch.sh` — + kernel-core patch mechanism, empty series by design (verified: the module + needs no core patch). + +### Fixed (all found by the VM gates, root causes documented) + +- `skb->mark` bookkeeping used multi-bit values, so the "in pass done" mark + also satisfied the "out pass done" test: transit traffic never reached the + **out** ruleset (NAT and egress rules silently ignored). +- The out ruleset was never evaluated for forwarded packets: added the + `NF_INET_FORWARD` hook (OpenBSD evaluates it in `ip_forward()`). +- `refcnt_rele()` had inverted polarity → live state keys freed → `Oops` in + the state RBT and `pfctl -s states` hangs. +- `radix` `SALEN()` read the `sa_family` byte as a length (Linux sockaddrs + have no `sa_len`) → table entries dropped as duplicates. +- BSD/Linux `LIST_HEAD` macro arity clash in `compat/sys/queue.h`. +- Reverse NAT: a packet whose destination pf rewrote was delivered to the + firewall's own stack instead of being re-routed. +- Checksums across translation: IP header checksum recomputed (OpenBSD's + `ip_output` does this, Linux does not); transport checksum recomputed in + software for plain skbs and handed back to the NIC (`CHECKSUM_PARTIAL`) for + GSO/TSO — the earlier software-only fix corrupted every segment after the + first and stalled bulk transfers. +- `divert`: NULL dereference on an empty listener list (wedged the VM), + port byte order mismatch between `bind` and the queue, L3-only skbs injected + into an Ethernet device RX path, re-divert loop (solved with a skip mark). +- `IOC_SIZEBITS` = 14 truncated the rule ioctl struct → `EINVAL` on load. +- State iteration started at id 1 (kifs never attached), `stoeplitz_init()` + never called, purge timer re-arming itself at `rmmod`, `rtnl`×`NET` lock + order inversion in the interface notifier. + +### Known gaps + +`binat-to`, `route-to`/`dup-to`/`fail-to`, `af-to`, scrub end-to-end +(normalization code is ported and compiles; the lab gate is not written), +queues/ALTQ (`hfsc.c`/`fq_codel.c` staged, `pfq` qdisc not implemented), +`pfsync`/CARP (out of scope). See `docs/en/feature-matrix.md`. + +## v0.3.0 — 2026-11-11 + +M2 data path: netisr-style workqueue, per-direction marks, state/table/anchor +gates passing; divert design settled (in-module socket family, no core patch). + +## v0.2.0 — 2026-11-11 + +M1: engine compiles into `pf.ko` (pf.c, pf_ioctl.c with the full ioctl ABI, +pf_norm, pf_if, pf_lb, pf_table, pf_ruleset, pf_osfp, pf_syncookies, radix, +md5/sha, ifq), `/dev/pf`, Linux-built OpenBSD `pfctl` talking to it. + +## v0.1.0 — 2026-11-11 + +M0: repo, bilingual scaffold, kernel 7.3.4 build, libvirt Alpine lab with +custom-kernel VMs (pf0/cli/srv/gw2), build/deploy/test scripts. diff --git a/docs/en/pf-guide.md b/docs/en/pf-guide.md index 1a762b7..3099ecd 100644 --- a/docs/en/pf-guide.md +++ b/docs/en/pf-guide.md @@ -60,34 +60,42 @@ treat them as load-and-pray until the matrix moves them. | Syntax | Until | What happens today | |---|---|---| -| `nat-to`, `rdr-to`, `binat-to` | M3 | Rules load (engine code compiled) but translation is unwired/ungated — **do not rely on NAT**; outbound masquerade still comes from iptables/nft during the lab. | +| `nat-to`, `rdr-to` (`binat-to` pending) | — | **Verified in the lab** (gates `05-nat`, `06-rdr`, the latter pulls 2 MB through a redirect and compares it byte for byte). `nat-to (egress)` expands the OpenBSD interface *group*; on a multi-NIC host prefer the explicit `nat-to ()` form. | | `scrub (…)`, `set reassemble` | M3 | `pf_norm.c` runs inside `pf_test` (compiled) — crafted-fragment behavior is untested; treat results as unvalidated. | -| `divert-to`, `divert` tag | M4 (+ family per code comments) | Matching packets are **dropped** with a rate-limited kernel warning. Never ship this on a live host. | +| `divert-to`, `divert` tag | — | **Verified in the lab** (gate `11-divert`): the packet is delivered to an `AF_DIVERT` socket (`PF_DIVERT=32`) served by `tools/pdivert`; with no listener it is dropped, exactly like BSD's `divs_noport`. | | `route-to`, `dup-to`, `fail-to`, `reply-to` | M4 | Not implemented as Linux routing glue yet; rules may load but the behavior is undefined — avoid. | | `queue` / `queue on` / `pfctl -s queue` | M5 | Parse and load. Queuing falls back to the priq-shaped compat `ifq` layer; **no bandwidth enforcement happens until the `pfq` qdisc lands**. | -| `log`, `log all`, `block log`, `pflogd` | M4 | Nothing is logged (stub warns once at load: `pfk: … packet not logged`). | +| `log`, `log all`, `block log`, `pflogd` | — | **Verified in the lab** (gate `12-pflog`): run `pflogd /tmp/x.pcap` (reads `/dev/pflog`); the file is a `DLT_PFLOG` (117) pcap readable by tcpdump/wireshark. | | `route-label …` | never (non-goal) | Rejected at the ioctl layer. | | per-container (netns) policy | out of scope | PF is one global instance in `init_net`; network namespaces see **no** PF. | ## Reading state correctly -`pfctl -s states` currently has a listing/visibility anomaly being fixed in -the M2 gate work (transit flows can pass while no state is displayed). Until -then, use counters (`pfctl -s info`) and external observation (ping/nc -results, `tcpdump`) as the primary evidence, as the gates do. +`pfctl -s states` shows the real table, including translations, in OpenBSD's +own format: a NAT/redirect state prints both tuples, e.g. + +``` +all icmp 10.0.2.1:59335 (10.0.1.10:4380) -> 10.0.2.20:8 0:0 +all tcp 10.0.2.20:80 (10.0.1.1:8080) <- 10.0.1.10:42091 FIN_WAIT_2:FIN_WAIT_2 +``` + +The value in parentheses is the *original* address before translation. A +state that exists but shows `0:0` counters simply means the engine has not +seen traffic on it since the last poll; use `pfctl -s info` counters and +`tcpdump` for traffic evidence, as the gates do. ## Kernel log signals you should know ``` pfk: PF-on-Linux engine loaded (OpenBSD 8.0 base) ← insmod ok pfk: netfilter hooks registered (10, netisr wq data path) -pfk: divert port N: no socket bound # pdivert not listening on N -pfk: … packet not logged ← pflog stub +pfk: divert port N: no socket bound # pdivert/pflogd consumer not listening ``` Unloading stops the purge timers and drains the work queue first (an rmmod -hang observed in the lab was fixed that way) — but the *soak* run -(rmmod/insmod cycle under traffic) is still an open gate, so expect unload to -be correct for admin use and unproven for hot-reload automation. +hang observed in the lab was fixed that way). The soak gate (`14-soak.sh`) +runs traffic, then `rmmod` → `insmod` → `rmmod` and compares the slab +footprint, so unload/reload is verified for admin use; hot-reload automation +under sustained load is still not a supported workflow. Full list of behavioral divergences: [caveats.md](caveats.md). diff --git a/docs/pt-BR/pf-guide.md b/docs/pt-BR/pf-guide.md index 62e439c..fa3abe8 100644 --- a/docs/pt-BR/pf-guide.md +++ b/docs/pt-BR/pf-guide.md @@ -63,12 +63,12 @@ matriz mudar. | Sintaxe | Até | O que acontece hoje | |---|---|---| -| `nat-to`, `rdr-to`, `binat-to` | M3 | As regras carregam (código do engine compilado), mas a tradução não está fiada/verificada — **não conte com NAT**; no laboratório o masquerade de saída ainda vem do iptables/nft. | +| `nat-to`, `rdr-to` | — | **Verificado em lab** (gates `05-nat`, `06-rdr`; este último baixa 2 MB por um redirect e compara byte a byte). `nat-to (egress)` expande o *grupo* de interfaces do OpenBSD; num host multi-NIC prefira a forma explícita `nat-to ()`. `binat-to` ainda sem gate. | | `scrub (…)`, `set reassemble` | M3 | O `pf_norm.c` roda dentro do `pf_test` (compilado) — o comportamento com fragmentos forjados não foi testado; trate o resultado como não validado. | -| `divert-to`, tag `divert` | M4 (+ família de sockets conforme comentário no código) | Os pacotes casados são **descartados** com um aviso de kernel rate-limited. Nunca coloque isso num host em produção. | +| `divert-to`, tag `divert` | — | **Verificado em lab** (gate `11-divert`): o pacote é entregue a um socket `AF_DIVERT` (`PF_DIVERT=32`) atendido pelo `tools/pdivert`; sem listener é descartado, exatamente como o `divs_noport` do BSD. | | `route-to`, `dup-to`, `fail-to`, `reply-to` | M4 | Ainda não implementados como glue de roteamento Linux; as regras podem carregar, mas o comportamento é indefinido — evite. | | `queue` / `queue on` / `pfctl -s queue` | M5 | Faz parse e carrega. A fila cai no fallback priq da camada `ifq` de compat; **nenhuma limitação de banda acontece até o qdisc `pfq` chegar**. | -| `log`, `log all`, `block log`, `pflogd` | M4 | Nada é logado (o stub avisa uma vez no load: `pfk: … packet not logged`). | +| `log`, `log all`, `block log`, `pflogd` | — | **Verificado em lab** (gate `12-pflog`): rode `pflogd /tmp/x.pcap` (lê `/dev/pflog`); o arquivo é um pcap `DLT_PFLOG` (117) legível por tcpdump/wireshark. | | `route-label …` | nunca (não-objetivo) | Rejeitado na camada de ioctl. | | política por container (netns) | fora de escopo | O PF é uma instância global em `init_net`; network namespaces **não** enxergam o PF. | @@ -85,8 +85,7 @@ primária, exatamente como os gates fazem. ``` pfk: PF-on-Linux engine loaded (OpenBSD 8.0 base) ← insmod ok pfk: netfilter hooks registered (10, netisr wq data path) -pfk: divert port N: no socket bound # pdivert não ouvido a porta N -pfk: … packet not logged ← stub do pflog +pfk: divert port N: no socket bound # nenhum consumidor em pdivert/pflogd ``` Descarregar o módulo primeiro para os timers de purge e drena a workqueue diff --git a/scripts/demo-session.sh b/scripts/demo-session.sh index 06a9d70..28e616b 100755 --- a/scripts/demo-session.sh +++ b/scripts/demo-session.sh @@ -76,7 +76,7 @@ EGRESS_IF=$(onvm "$PF0_MGMT" "ip route get $SRV_IP | grep -o 'dev [a-z0-9]*' | c [ -n "$EGRESS_IF" ] || EGRESS_IF=eth2 scp -q -o ProxyJump=root@192.168.1.242 -i "$KEY" "$CONF" "$DIV" \ - tools/out/pdivert "root@$PF0_MGMT:/tmp/" 2>/dev/null + tools/out/pdivert tools/out/pflogd "root@$PF0_MGMT:/tmp/" 2>/dev/null bold "===============================================================" bold " PF on Linux - the OpenBSD PF 8.0 engine inside the Linux kernel" @@ -138,7 +138,19 @@ p0 "/opt/pf/pfctl -ss | grep -E 'icmp' | head -4" p0 "/opt/pf/pfctl -f /tmp/demo-pf.conf && echo 'back to the stateful ruleset'" echo -bold "--- 8. divert-to: the packet goes to userspace and comes back ---" +bold "--- 8. pflog: logged packets reach userspace (/dev/pflog) ---" +p0 "cat > /tmp/demo-log.conf </tmp/pflogd.log 2>&1 /dev/null; /opt/pf/pfctl -f /tmp/demo-pf.conf >/dev/null" +echo + +bold "--- 9. divert-to: the packet goes to userspace and comes back ---" p0 "cat /tmp/demo-divert.conf" p0 "/opt/pf/pfctl -f /tmp/demo-divert.conf && echo 'divert rule loaded'" p0 "chmod +x /tmp/pdivert; (setsid /tmp/pdivert -p 10000 -f >/tmp/pdivert.log 2>&1