docs: guides reflect verified NAT/redirect/divert/pflog and the fixed state listing; CHANGELOG for v0.4.0
This commit is contained in:
lgcosta committed
1 parent
92c015b97b
commit
f263b63b8e
4 files changed
+127
-20
No files matched your search
@@ -0,0 +1,88 @@
|
||||
# Changelog
|
||||
|
||||
## v0.4.0 — 2026-11-11
|
||||
|
||||
Lab-verified milestone release. The full gate suite (9 gates) runs against
|
||||
Alpine VMs under libvirt on the kernel built from source.
|
||||
|
||||
### Verified in the lab (each has a gate in `tests/vm/`)
|
||||
|
||||
| Capability | Gate | What it proves |
|
||||
|---|---|---|
|
||||
| Engine loads, `/dev/pf`, real OpenBSD `pfctl` over the ioctl ABI | `01` | `insmod` + `pfctl -s info` + transit ping + state created |
|
||||
| `block` / `return` / `return-rst` | `02` | ICMP dropped, TCP refused |
|
||||
| `pfr` tables (inline `{ }`, `-T add/del/show`, `-T` block) | `03` | table load, block by table, delete, unblock |
|
||||
| Anchors and evaluation order | `04` | anchor child passes the request before the parent `block`, state created by the anchor rule, request dropped after the anchor is flushed |
|
||||
| `nat-to` (source NAT) | `05` | tcpdump proof of the rewritten source, reply de-NATed, state shows both tuples |
|
||||
| `rdr-to` (port redirect) | `06` | client talks to the firewall, server answers, **2 MB pulled through the redirect byte-identical**, firewall still responsive afterwards |
|
||||
| `divert-to` + `AF_DIVERT` + `pdivert` | `11` | drop with no listener (BSD `divs_noport` parity), packet delivered to userspace, rewritten and reinjected |
|
||||
| `pflog` + `pflogd` | `12` | logged packets read from `/dev/pflog` and written as a `DLT_PFLOG` pcap containing the real bytes |
|
||||
| Soak / unload discipline | `14` | traffic + states with no kernel warnings, clean `rmmod`, reload, no slab growth |
|
||||
|
||||
### Added
|
||||
|
||||
- `module/pfsrc/pfk_pflog.c` — `/dev/pflog` chardev backend (OpenBSD
|
||||
`pflog_hdr` layout, 64-byte records, bounded queue, `poll`, stats ioctl).
|
||||
- `tools/pflogd` — reads `/dev/pflog`, writes `DLT_PFLOG` pcap (or a readable
|
||||
one-line-per-record summary).
|
||||
- `tools/pdivert` — divert(4) relay with payload mutation + ICMP checksum fix.
|
||||
- `module/pfsrc/divert_sock.c` — `AF_DIVERT` socket family registered at
|
||||
runtime (`PF_DIVERT = 32`, `IPPROTO_DIVERT = 254`), blocking `recvmsg`,
|
||||
`poll`, BSD-compatible reinjection.
|
||||
- `scripts/demo-session.sh` + `scripts/demo-record.py` + `make demo` —
|
||||
asciinema recording of the live lab (`out/demo-pf-linux.cast`).
|
||||
- `docs/{en,pt-BR}/`: README, install, build, guide, vm-lab, feature-matrix,
|
||||
patches, caveats, demo. `docs/porting-notes.md` records every deviation.
|
||||
- `patches/kernel/` + `scripts/apply-patches.sh` / `scripts/unpatch.sh` —
|
||||
kernel-core patch mechanism, empty series by design (verified: the module
|
||||
needs no core patch).
|
||||
|
||||
### Fixed (all found by the VM gates, root causes documented)
|
||||
|
||||
- `skb->mark` bookkeeping used multi-bit values, so the "in pass done" mark
|
||||
also satisfied the "out pass done" test: transit traffic never reached the
|
||||
**out** ruleset (NAT and egress rules silently ignored).
|
||||
- The out ruleset was never evaluated for forwarded packets: added the
|
||||
`NF_INET_FORWARD` hook (OpenBSD evaluates it in `ip_forward()`).
|
||||
- `refcnt_rele()` had inverted polarity → live state keys freed → `Oops` in
|
||||
the state RBT and `pfctl -s states` hangs.
|
||||
- `radix` `SALEN()` read the `sa_family` byte as a length (Linux sockaddrs
|
||||
have no `sa_len`) → table entries dropped as duplicates.
|
||||
- BSD/Linux `LIST_HEAD` macro arity clash in `compat/sys/queue.h`.
|
||||
- Reverse NAT: a packet whose destination pf rewrote was delivered to the
|
||||
firewall's own stack instead of being re-routed.
|
||||
- Checksums across translation: IP header checksum recomputed (OpenBSD's
|
||||
`ip_output` does this, Linux does not); transport checksum recomputed in
|
||||
software for plain skbs and handed back to the NIC (`CHECKSUM_PARTIAL`) for
|
||||
GSO/TSO — the earlier software-only fix corrupted every segment after the
|
||||
first and stalled bulk transfers.
|
||||
- `divert`: NULL dereference on an empty listener list (wedged the VM),
|
||||
port byte order mismatch between `bind` and the queue, L3-only skbs injected
|
||||
into an Ethernet device RX path, re-divert loop (solved with a skip mark).
|
||||
- `IOC_SIZEBITS` = 14 truncated the rule ioctl struct → `EINVAL` on load.
|
||||
- State iteration started at id 1 (kifs never attached), `stoeplitz_init()`
|
||||
never called, purge timer re-arming itself at `rmmod`, `rtnl`×`NET` lock
|
||||
order inversion in the interface notifier.
|
||||
|
||||
### Known gaps
|
||||
|
||||
`binat-to`, `route-to`/`dup-to`/`fail-to`, `af-to`, scrub end-to-end
|
||||
(normalization code is ported and compiles; the lab gate is not written),
|
||||
queues/ALTQ (`hfsc.c`/`fq_codel.c` staged, `pfq` qdisc not implemented),
|
||||
`pfsync`/CARP (out of scope). See `docs/en/feature-matrix.md`.
|
||||
|
||||
## v0.3.0 — 2026-11-11
|
||||
|
||||
M2 data path: netisr-style workqueue, per-direction marks, state/table/anchor
|
||||
gates passing; divert design settled (in-module socket family, no core patch).
|
||||
|
||||
## v0.2.0 — 2026-11-11
|
||||
|
||||
M1: engine compiles into `pf.ko` (pf.c, pf_ioctl.c with the full ioctl ABI,
|
||||
pf_norm, pf_if, pf_lb, pf_table, pf_ruleset, pf_osfp, pf_syncookies, radix,
|
||||
md5/sha, ifq), `/dev/pf`, Linux-built OpenBSD `pfctl` talking to it.
|
||||
|
||||
## v0.1.0 — 2026-11-11
|
||||
|
||||
M0: repo, bilingual scaffold, kernel 7.3.4 build, libvirt Alpine lab with
|
||||
custom-kernel VMs (pf0/cli/srv/gw2), build/deploy/test scripts.
|
||||
+20
-12
@@ -60,34 +60,42 @@ treat them as load-and-pray until the matrix moves them.
|
||||
|
||||
| Syntax | Until | What happens today |
|
||||
|---|---|---|
|
||||
| `nat-to`, `rdr-to`, `binat-to` | M3 | Rules load (engine code compiled) but translation is unwired/ungated — **do not rely on NAT**; outbound masquerade still comes from iptables/nft during the lab. |
|
||||
| `nat-to`, `rdr-to` (`binat-to` pending) | — | **Verified in the lab** (gates `05-nat`, `06-rdr`, the latter pulls 2 MB through a redirect and compares it byte for byte). `nat-to (egress)` expands the OpenBSD interface *group*; on a multi-NIC host prefer the explicit `nat-to (<iface>)` form. |
|
||||
| `scrub (…)`, `set reassemble` | M3 | `pf_norm.c` runs inside `pf_test` (compiled) — crafted-fragment behavior is untested; treat results as unvalidated. |
|
||||
| `divert-to`, `divert` tag | M4 (+ family per code comments) | Matching packets are **dropped** with a rate-limited kernel warning. Never ship this on a live host. |
|
||||
| `divert-to`, `divert` tag | — | **Verified in the lab** (gate `11-divert`): the packet is delivered to an `AF_DIVERT` socket (`PF_DIVERT=32`) served by `tools/pdivert`; with no listener it is dropped, exactly like BSD's `divs_noport`. |
|
||||
| `route-to`, `dup-to`, `fail-to`, `reply-to` | M4 | Not implemented as Linux routing glue yet; rules may load but the behavior is undefined — avoid. |
|
||||
| `queue` / `queue on` / `pfctl -s queue` | M5 | Parse and load. Queuing falls back to the priq-shaped compat `ifq` layer; **no bandwidth enforcement happens until the `pfq` qdisc lands**. |
|
||||
| `log`, `log all`, `block log`, `pflogd` | M4 | Nothing is logged (stub warns once at load: `pfk: … packet not logged`). |
|
||||
| `log`, `log all`, `block log`, `pflogd` | — | **Verified in the lab** (gate `12-pflog`): run `pflogd /tmp/x.pcap` (reads `/dev/pflog`); the file is a `DLT_PFLOG` (117) pcap readable by tcpdump/wireshark. |
|
||||
| `route-label …` | never (non-goal) | Rejected at the ioctl layer. |
|
||||
| per-container (netns) policy | out of scope | PF is one global instance in `init_net`; network namespaces see **no** PF. |
|
||||
|
||||
## Reading state correctly
|
||||
|
||||
`pfctl -s states` currently has a listing/visibility anomaly being fixed in
|
||||
the M2 gate work (transit flows can pass while no state is displayed). Until
|
||||
then, use counters (`pfctl -s info`) and external observation (ping/nc
|
||||
results, `tcpdump`) as the primary evidence, as the gates do.
|
||||
`pfctl -s states` shows the real table, including translations, in OpenBSD's
|
||||
own format: a NAT/redirect state prints both tuples, e.g.
|
||||
|
||||
```
|
||||
all icmp 10.0.2.1:59335 (10.0.1.10:4380) -> 10.0.2.20:8 0:0
|
||||
all tcp 10.0.2.20:80 (10.0.1.1:8080) <- 10.0.1.10:42091 FIN_WAIT_2:FIN_WAIT_2
|
||||
```
|
||||
|
||||
The value in parentheses is the *original* address before translation. A
|
||||
state that exists but shows `0:0` counters simply means the engine has not
|
||||
seen traffic on it since the last poll; use `pfctl -s info` counters and
|
||||
`tcpdump` for traffic evidence, as the gates do.
|
||||
|
||||
## Kernel log signals you should know
|
||||
|
||||
```
|
||||
pfk: PF-on-Linux engine loaded (OpenBSD 8.0 base) ← insmod ok
|
||||
pfk: netfilter hooks registered (10, netisr wq data path)
|
||||
pfk: divert port N: no socket bound # pdivert not listening on N
|
||||
pfk: … packet not logged ← pflog stub
|
||||
pfk: divert port N: no socket bound # pdivert/pflogd consumer not listening
|
||||
```
|
||||
|
||||
Unloading stops the purge timers and drains the work queue first (an rmmod
|
||||
hang observed in the lab was fixed that way) — but the *soak* run
|
||||
(rmmod/insmod cycle under traffic) is still an open gate, so expect unload to
|
||||
be correct for admin use and unproven for hot-reload automation.
|
||||
hang observed in the lab was fixed that way). The soak gate (`14-soak.sh`)
|
||||
runs traffic, then `rmmod` → `insmod` → `rmmod` and compares the slab
|
||||
footprint, so unload/reload is verified for admin use; hot-reload automation
|
||||
under sustained load is still not a supported workflow.
|
||||
|
||||
Full list of behavioral divergences: [caveats.md](caveats.md).
|
||||
@@ -63,12 +63,12 @@ matriz mudar.
|
||||
|
||||
| Sintaxe | Até | O que acontece hoje |
|
||||
|---|---|---|
|
||||
| `nat-to`, `rdr-to`, `binat-to` | M3 | As regras carregam (código do engine compilado), mas a tradução não está fiada/verificada — **não conte com NAT**; no laboratório o masquerade de saída ainda vem do iptables/nft. |
|
||||
| `nat-to`, `rdr-to` | — | **Verificado em lab** (gates `05-nat`, `06-rdr`; este último baixa 2 MB por um redirect e compara byte a byte). `nat-to (egress)` expande o *grupo* de interfaces do OpenBSD; num host multi-NIC prefira a forma explícita `nat-to (<iface>)`. `binat-to` ainda sem gate. |
|
||||
| `scrub (…)`, `set reassemble` | M3 | O `pf_norm.c` roda dentro do `pf_test` (compilado) — o comportamento com fragmentos forjados não foi testado; trate o resultado como não validado. |
|
||||
| `divert-to`, tag `divert` | M4 (+ família de sockets conforme comentário no código) | Os pacotes casados são **descartados** com um aviso de kernel rate-limited. Nunca coloque isso num host em produção. |
|
||||
| `divert-to`, tag `divert` | — | **Verificado em lab** (gate `11-divert`): o pacote é entregue a um socket `AF_DIVERT` (`PF_DIVERT=32`) atendido pelo `tools/pdivert`; sem listener é descartado, exatamente como o `divs_noport` do BSD. |
|
||||
| `route-to`, `dup-to`, `fail-to`, `reply-to` | M4 | Ainda não implementados como glue de roteamento Linux; as regras podem carregar, mas o comportamento é indefinido — evite. |
|
||||
| `queue` / `queue on` / `pfctl -s queue` | M5 | Faz parse e carrega. A fila cai no fallback priq da camada `ifq` de compat; **nenhuma limitação de banda acontece até o qdisc `pfq` chegar**. |
|
||||
| `log`, `log all`, `block log`, `pflogd` | M4 | Nada é logado (o stub avisa uma vez no load: `pfk: … packet not logged`). |
|
||||
| `log`, `log all`, `block log`, `pflogd` | — | **Verificado em lab** (gate `12-pflog`): rode `pflogd /tmp/x.pcap` (lê `/dev/pflog`); o arquivo é um pcap `DLT_PFLOG` (117) legível por tcpdump/wireshark. |
|
||||
| `route-label …` | nunca (não-objetivo) | Rejeitado na camada de ioctl. |
|
||||
| política por container (netns) | fora de escopo | O PF é uma instância global em `init_net`; network namespaces **não** enxergam o PF. |
|
||||
|
||||
@@ -85,8 +85,7 @@ primária, exatamente como os gates fazem.
|
||||
```
|
||||
pfk: PF-on-Linux engine loaded (OpenBSD 8.0 base) ← insmod ok
|
||||
pfk: netfilter hooks registered (10, netisr wq data path)
|
||||
pfk: divert port N: no socket bound # pdivert não ouvido a porta N
|
||||
pfk: … packet not logged ← stub do pflog
|
||||
pfk: divert port N: no socket bound # nenhum consumidor em pdivert/pflogd
|
||||
```
|
||||
|
||||
Descarregar o módulo primeiro para os timers de purge e drena a workqueue
|
||||
|
||||
+15
-3
@@ -76,7 +76,7 @@ EGRESS_IF=$(onvm "$PF0_MGMT" "ip route get $SRV_IP | grep -o 'dev [a-z0-9]*' | c
|
||||
[ -n "$EGRESS_IF" ] || EGRESS_IF=eth2
|
||||
|
||||
scp -q -o ProxyJump=root@192.168.1.242 -i "$KEY" "$CONF" "$DIV" \
|
||||
tools/out/pdivert "root@$PF0_MGMT:/tmp/" 2>/dev/null
|
||||
tools/out/pdivert tools/out/pflogd "root@$PF0_MGMT:/tmp/" 2>/dev/null
|
||||
|
||||
bold "==============================================================="
|
||||
bold " PF on Linux - the OpenBSD PF 8.0 engine inside the Linux kernel"
|
||||
@@ -138,7 +138,19 @@ p0 "/opt/pf/pfctl -ss | grep -E 'icmp' | head -4"
|
||||
p0 "/opt/pf/pfctl -f /tmp/demo-pf.conf && echo 'back to the stateful ruleset'"
|
||||
echo
|
||||
|
||||
bold "--- 8. divert-to: the packet goes to userspace and comes back ---"
|
||||
bold "--- 8. pflog: logged packets reach userspace (/dev/pflog) ---"
|
||||
p0 "cat > /tmp/demo-log.conf <<EOF
|
||||
pass in quick log proto icmp from $CLI_IP keep state
|
||||
pass out quick
|
||||
EOF"
|
||||
p0 "/opt/pf/pfctl -f /tmp/demo-log.conf && echo 'log rule loaded'"
|
||||
p0 "chmod +x /tmp/pflogd; (setsid /tmp/pflogd -c 3 /tmp/pflog.pcap >/tmp/pflogd.log 2>&1 </dev/null &); sleep 1; pgrep pflogd"
|
||||
vmc "ping -c2 -W2 $SRV_IP | tail -2"
|
||||
p0 "ls -l /tmp/pflog.pcap; echo 'pcap header (magic d4c3b2a1, linktype 75=DLT_PFLOG):'; od -An -tx1 -N24 /tmp/pflog.pcap"
|
||||
p0 "pkill pflogd 2>/dev/null; /opt/pf/pfctl -f /tmp/demo-pf.conf >/dev/null"
|
||||
echo
|
||||
|
||||
bold "--- 9. divert-to: the packet goes to userspace and comes back ---"
|
||||
p0 "cat /tmp/demo-divert.conf"
|
||||
p0 "/opt/pf/pfctl -f /tmp/demo-divert.conf && echo 'divert rule loaded'"
|
||||
p0 "chmod +x /tmp/pdivert; (setsid /tmp/pdivert -p 10000 -f >/tmp/pdivert.log 2>&1 </dev/null &); sleep 1; pgrep pdivert"
|
||||
@@ -147,7 +159,7 @@ p0 "head -3 /tmp/pdivert.log"
|
||||
echo " (pdivert flipped the last payload byte and fixed the ICMP checksum)"
|
||||
echo
|
||||
|
||||
bold "--- 9. shutdown ---"
|
||||
bold "--- 10. shutdown ---"
|
||||
p0 "pkill pdivert; /opt/pf/pfctl -f /dev/null; /opt/pf/pfctl -d; echo 'pf disabled'"
|
||||
p0 "lsmod | grep '^pf '"
|
||||
echo
|
||||
|
||||
Reference in new issue
Block a user