diff --git a/docs/en/feature-matrix.md b/docs/en/feature-matrix.md index 50af286..acc827d 100644 --- a/docs/en/feature-matrix.md +++ b/docs/en/feature-matrix.md @@ -59,7 +59,7 @@ As of 2026-11-11; source of per-row evidence: `out/test-logs/`, | Feature | Upstream syntax | Milestone | Status | |---|---|---|---| | `route-to` pools (`rr`, `round-robin`, `source-hash`, `hash`), `reply-to`, `dup-to`, `fail-to` | `pass out route-to (GW 10.0.3.2)` | M4 | PENDING-M4 (Linux route/neigh glue `pf_route` not implemented yet) | -| `divert-to` / `divert` tag | `pass in … divert-to ` | M4 e2e pending | **COMPILES (untested)** — AF_DIVERT family (PF_DIVERT=32) implemented in-module (`module/pfsrc/divert_sock.c`: bind/recvmsg/sendmsg/poll + BSD `divert_output` reinjection semantics); `pdivert` relay in `tools/pdivert/`. Packet-level proof is gate `11-divert.sh` (M4) — lab-only until then. | +| `divert-to` / `divert` tag | `pass in … divert-to port ` | VERIFIED in lab | **Works end-to-end.** The module registers an `AF_DIVERT` socket family (`PF_DIVERT=32`) at runtime — no kernel-tree patch. A `divert-to` rule hands the packet to that family; with no listener the packet is dropped (`divs_noport`, BSD parity); with `tools/pdivert` running the packet is delivered to userspace, rewritten and reinjected (verified with a payload mutation + checksum fix, ping 0% loss). | | Queues, default scheduler (priq) | `queue root on wan`, `pfctl -s queue -v` | M5 | COMPILES-untested — `compat/pfk_ifq.c` implements the ifq API with **priq fallback semantics**; `queue` rules parse and load, but packets are not actually shunted through a classful qdisc until M5 | | HFSC | `queue on wan { hfsc … }` | M5 | COMPILES-untested — **priq fallback until M5**; `pfsrc/hfsc.c` staged (real OpenBSD 8.0 code) but **not in the module object list** | | FQ-CoDel | `queue on wan { fq_codel … }` | M5 | COMPILES-untested — same as HFSC: staged, not built | diff --git a/docs/porting-notes.md b/docs/porting-notes.md index a31a6f0..93148c4 100644 --- a/docs/porting-notes.md +++ b/docs/porting-notes.md @@ -860,3 +860,25 @@ ruleset's block line and the resulting state carries the reply). at all) on pf0, which looked like a PF failure. Both now retry and verify the copy, and `vm-run-tests.sh` disables pf on pf0 on exit (trap) so a failed gate never leaves the lab filtered. + +## Divert (M4) — deviations from the OpenBSD implementation + +The OpenBSD flow is: `pf.c` tags the mbuf (`PACKET_TAG_PF_DIVERT`, +`PF_TAG_DIVERTED`) and `netinet/ip_input.c` hands the tagged packet to the +divert socket. Linux has no `ip_input` layer to hook, and this port does not +implement the BSD `divert(4)` `inpcb` plumbing (`ip_divert.c` is kept in the +tree as reference but is not built). + +Decisions taken (all verified in the libvirt lab): + +| Topic | OpenBSD | This port | +|---|---|---| +| Rule verdict | tag the mbuf, divert later in `ip_input` | hand the packet to the divert family directly at the IN verdict (`pf.c`, `divert_packet()`); the mbuf is verdict-exclusive there | +| Socket family | `divert(4)` via `divert_attach()`/`inpcb` | `AF_DIVERT` (`PF_DIVERT=32`, `IPPROTO_DIVERT=254`) registered at runtime with `sock_register()` — no kernel-tree patch | +| Port byte order | network order (`inp_lport`) | network order on both sides (`sin_port` stored raw, `pf_rule.divert.port` is `htons()`ed by pfctl) | +| Reinjection | `divert_output()` → `ip_output()` | local delivery: synthesised Ethernet header + `netif_rx()` into the ingress device (the device RX path demultiplexes by ethertype) | +| Re-divert loop | `PF_TAG_DIVERTED_PACKET` | `skb->mark |= PFK_SKIP_PF` — the hook bypasses pf for reinjected packets | +| Payload mutation | userspace must fix checksums | `pdivert -f` flips the last payload byte and recomputes the ICMP checksum | + +`pdivert` (userspace relay, `tools/pdivert/`) is the reference client; it +demonstrates the BSD "receive -> rewrite -> reinject" cycle. diff --git a/docs/pt-BR/feature-matrix.md b/docs/pt-BR/feature-matrix.md index 06737f0..3ff86a9 100644 --- a/docs/pt-BR/feature-matrix.md +++ b/docs/pt-BR/feature-matrix.md @@ -59,7 +59,7 @@ Estado em 2026-11-11; evidência por linha: `out/test-logs/`, | Feature | Sintaxe de upstream | Milestone | Status | |---|---|---|---| | Pools de `route-to` (`rr`, `round-robin`, `source-hash`, `hash`), `reply-to`, `dup-to`, `fail-to` | `pass out route-to (GW 10.0.3.2)` | M4 | PENDING-M4 (o glue Linux de rota/neigh `pf_route` ainda não existe) | -| `divert-to` / tag `divert` | `pass in … divert-to ` | e2e pendente (M4) | **COMPILA (não testado)** — família AF_DIVERT (PF_DIVERT=32) implementada no módulo (`module/pfsrc/divert_sock.c`: bind/recvmsg/sendmsg/poll + reinjeção com semântica `divert_output` do BSD); relay `pdivert` em `tools/pdivert/`. A prova fim-a-fim é o gate `11-divert.sh` (M4) — até lá, apenas em lab. | +| `divert-to` / tag `divert` | `pass in … divert-to port ` | VERIFICADO em lab | **Funciona ponta a ponta.** O módulo registra em runtime a família de sockets `AF_DIVERT` (`PF_DIVERT=32`) — sem patch no kernel. A regra `divert-to` entrega o pacote a essa família; sem listener o pacote é descartado (`divs_noport`, paridade BSD); com `tools/pdivert` rodando o pacote vai ao userspace, é reescrito e reinjetado (verificado com mutação de payload + correção de checksum, ping 0% loss). | | Filas, agendador default (priq) | `queue root on wan`, `pfctl -s queue -v` | M5 | COMPILES-untested — `compat/pfk_ifq.c` implementa a API ifq com **semântica de fallback priq**; regras `queue` fazem parse e carregam, mas pacotes ainda não são desviados por um qdisc classful até o M5 | | HFSC | `queue on wan { hfsc … }` | M5 | COMPILES-untested — **fallback priq até o M5**; `pfsrc/hfsc.c` staged (código real do OpenBSD 8.0) mas **fora da lista de objetos** do módulo | | FQ-CoDel | `queue on wan { fq_codel … }` | M5 | COMPILES-untested — igual ao HFSC: staged, não compilado | diff --git a/module/pfsrc/pfk_hooks.c b/module/pfsrc/pfk_hooks.c index 0d41a20..90f7a4c 100644 --- a/module/pfsrc/pfk_hooks.c +++ b/module/pfsrc/pfk_hooks.c @@ -138,14 +138,24 @@ pfk_enqueue_verdict(int af, struct sk_buff *skb, switch (st->hook) { case NF_INET_PRE_ROUTING: + /* OpenBSD evaluates the IN ruleset in ip_input(); every + * arriving packet passes it. */ dir = PF_IN; break; + case NF_INET_FORWARD: + /* OpenBSD evaluates the OUT ruleset of a forwarded packet in + * ip_forward() (pf_test(PF_FWD)); the IN ruleset was already + * applied at input time. Without this, transit traffic never + * sees the "out" rules and state matching for replies is + * impossible. */ + dir = PF_OUT; + break; case NF_INET_LOCAL_OUT: case NF_INET_POST_ROUTING: dir = PF_OUT; break; default: - return NF_ACCEPT; /* LOCAL_IN/FORWARD */ + return NF_ACCEPT; /* LOCAL_IN */ } if (skb->mark & PFK_SKIP_PF) return NF_ACCEPT; /* reinjected by divert: don't re-filter */ @@ -160,7 +170,8 @@ pfk_enqueue_verdict(int af, struct sk_buff *skb, if (dir == PF_IN) ifp = pfk_ifp_bydev(st->in ? st->in : skb->dev); else - ifp = pfk_ifp_bydev(skb->dev ? skb->dev : st->out); + ifp = pfk_ifp_bydev(st->out ? st->out : + (skb->dev ? skb->dev : NULL)); if (!ifp) return NF_ACCEPT; diff --git a/scripts/demo-record.py b/scripts/demo-record.py index 09537e8..d8eb846 100755 --- a/scripts/demo-record.py +++ b/scripts/demo-record.py @@ -1,141 +1,161 @@ #!/usr/bin/env python3 """ -PF-on-Linux demo recorder — produces an asciicast v2 (.cast) from a REAL -session driven against the lab VMs (no simulated output: every byte shown -comes from the live systems; only pacing is scripted). +PF-on-Linux demo recorder -> asciicast v2 (.cast) -Usage: ./scripts/demo-record.py [--out out/demo-pf-linux.cast] -Requires: lab running (pftest-pf0/cli/srv), repo built, ssh alias `pf0` -(ProxyJump) and lib.sh-style nested access working. +Every byte shown comes from the REAL lab (Alpine VMs under libvirt, kernel +7.3.4 built from source with our pf.ko). Only the pacing is scripted. -Edit DEMO below to change the narration script. Timing: each line is a -(real command, real output) pair; delays are cosmetic only. +Usage: + ./scripts/demo-record.py [--out out/demo-pf-linux.cast] [--quick] + +Play back with: asciinema play out/demo-pf-linux.cast +Upload with: asciinema upload out/demo-pf-linux.cast """ -import argparse, json, os, pty, re, shlex, subprocess, sys, time +import argparse +import json +import os +import shlex +import subprocess +import sys +import time REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) -OUT_DEFAULT = os.path.join(REPO, "out", "demo-pf-linux.cast") - DEB = os.environ.get("DEB", "root@192.168.1.242") -PF0_IP = os.environ.get("PF0_IP", "192.168.122.8") +PF0 = os.environ.get("PF0_IP", "192.168.122.8") +CLI = os.environ.get("CLI_IP", "10.0.1.10") +SRV = os.environ.get("SRV_IP", "10.0.2.20") +KEY = os.path.join(os.environ.get("HOME", "/root"), ".ssh/pflab_ed") + +CONF = """# PF-on-Linux demo ruleset (pf.conf syntax, real OpenBSD pfctl) +set block-policy return +table { 10.0.1.0/24 } + +# transit traffic: stateful, table-driven +pass in quick on ingress proto icmp from keep state +pass out quick on egress proto icmp keep state + +# deny a specific destination +block drop in quick proto icmp to 10.0.9.99 + +# everything else +pass in quick +pass out quick +""" + +DIVERT_CONF = """pass in quick proto icmp from 10.0.1.10 to 10.0.1.1 divert-to "10.0.1.1" port 10000 keep state +pass out quick +""" -def deb_run(cmd: str, timeout=30) -> str: - """run a command on the libvirt host (which has direct keys to VMs)""" - p = subprocess.run(["ssh", "-o", "ConnectTimeout=10", DEB, cmd], - capture_output=True, text=True, timeout=timeout) +def sh(cmd, timeout=40): + p = subprocess.run(cmd, shell=True, capture_output=True, text=True, + timeout=timeout) return (p.stdout or "") + (p.stderr or "") -def vm0(cmd: str, timeout=30) -> str: - return deb_run( - f"ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=8 " - f"-i /root/.ssh/pflab_ed root@{PF0_IP} {shlex.quote(cmd)}", - timeout) +def on_pf0(cmd, timeout=40): + return sh(f"ssh -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new " + f"-i {KEY} root@{PF0} {shlex.quote(cmd)}", timeout) -def vm_ip(ip: str, cmd: str, timeout=30) -> str: - return deb_run( - f"ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=8 " - f"-i /root/.ssh/pflab_ed root@{ip} {shlex.quote(cmd)}", - timeout) +def on_vm(ip, cmd, timeout=40): + return sh(f"ssh -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new " + f"-i {KEY} root@{ip} {shlex.quote(cmd)}", timeout) -CLI, SRV = "10.0.1.10", "10.0.2.20" +def on_deb(cmd, timeout=40): + return sh(f"ssh -o ConnectTimeout=8 {DEB} {shlex.quote(cmd)}", timeout) -def scene(title): - return ("#banner", title) +def stage(): + """put the demo ruleset + pdivert on pf0 (through the lab host)""" + with open("/tmp/demo-pf.conf", "w") as f: + f.write(CONF) + with open("/tmp/demo-divert.conf", "w") as f: + f.write(DIVERT_CONF) + sh(f"scp -q /tmp/demo-pf.conf /tmp/demo-divert.conf " + f"{os.path.join(REPO, 'tools/out/pdivert')} {DEB}:/tmp/", timeout=90) + on_deb(f"scp -q -o StrictHostKeyChecking=accept-new -i {KEY} " + f"/tmp/demo-pf.conf /tmp/demo-divert.conf /tmp/pdivert " + f"root@{PF0}:/tmp/ 2>&1", timeout=90) -# (shell-line, runner, command) — runners: pf0 / cli / srv / local -DEMO = [ - ("local", "echo 'PF-on-Linux — OpenBSD PF engine on Linux 7.3.4 (lab: Alpine VMs)'"), - ("local", "uname -r | sed 's/^/build host kernel: /'"), - ("pf0", "uname -r"), - ("pf0", "lsmod | grep -E '^pf ' || echo 'pf: not loaded yet'"), - ("pf0", "insmod /opt/pf/pf.ko && dmesg | tail -2"), - ("pf0", "/opt/pf/pfctl -s info | head -3"), - ("pf0", "/opt/pf/pfctl -e && /opt/pf/pfctl -s info | head -1"), - ("pf0", "cat /tmp/pfdemo.conf"), - ("pf0", "/opt/pf/pfctl -f /tmp/pfdemo.conf && /opt/pf/pfctl -sr"), - ("cli", f"ping -c 2 -W 2 {SRV} 2>&1 | tail -2"), - ("pf0", "/opt/pf/pfctl -s states"), - ("pf0", "/opt/pf/pfctl -s Tables"), - ("pf0", "/opt/pf/pfctl -t demo -T show"), - ("cli", "ping -c 2 -W 2 10.0.9.99 2>&1 | tail -2"), - ("pf0", "/opt/pf/pfctl -s info | grep -E 'match|current'"), - ("pf0", "/opt/pf/pfctl -ss | head -2"), + +SCENARIO = [ + ("local", "echo '=== PF on Linux — OpenBSD PF 8.0 engine running inside the Linux 7.3.4 kernel ==='"), + ("local", "echo 'lab: 3 Alpine VMs under libvirt (pf0 = firewall, cli/srv = transit endpoints)'"), + ("pf0", "uname -r"), + ("pf0", "lsmod | grep '^pf ' || echo 'pf: not loaded'"), + ("pf0", "insmod /opt/pf/pf.ko && dmesg | tail -3"), + ("pf0", "/opt/pf/pfctl -s info | head -4"), + ("pf0", "/opt/pf/pfctl -e && /opt/pf/pfctl -s info | head -1"), + ("pf0", "cat /tmp/demo-pf.conf"), + ("pf0", "/opt/pf/pfctl -f /tmp/demo-pf.conf && echo 'rules loaded' && /opt/pf/pfctl -sr"), + ("pf0", "/opt/pf/pfctl -sr -vv | head -8"), + ("cli", f"ping -c2 -W2 {SRV} | tail -3"), + ("pf0", "/opt/pf/pfctl -ss"), + ("pf0", "/opt/pf/pfctl -t trusted -T show"), + ("pf0", "/opt/pf/pfctl -s Tables"), + ("cli", "ping -c2 -W2 10.0.9.99 | tail -3"), + ("pf0", "/opt/pf/pfctl -sr -vv | head -6"), + ("pf0", "/opt/pf/pfctl -f /tmp/demo-divert.conf && echo 'divert rule loaded'"), + ("pf0", "(setsid /tmp/pdivert -p 10000 -f >/tmp/pdivert.log 2>&1 { 10.0.1.0/24 } -match in quick on ingress proto icmp from tag DEMO -pass in quick on ingress proto icmp keep state -pass out quick proto icmp keep state -block drop in quick proto icmp from any to 10.0.9.99 -""" - -def prep(): - # stage the demo ruleset into pf0 - import base64 - b64 = base64.b64encode(CONF.encode()).decode() - vm0(f"echo {b64} | base64 -d > /tmp/pfdemo.conf; echo staged") - - -def record(out_path: str, fps_pad=0.4): - os.makedirs(os.path.dirname(out_path), exist_ok=True) - casts = [] +def record(out_path, quick=False): + os.makedirs(os.path.dirname(out_path) or ".", exist_ok=True) + events = [] t = 0.0 def emit(text, delay): nonlocal t t += delay - casts.append([round(t, 3), "o", text]) + events.append([round(t, 3), "o", text]) - header = {"version": 2, "width": 100, "height": 24, - "timestamp": int(time.time()), "title": - "PF on Linux — port demo (OpenBSD PF 8.0 engine on Linux 7.3.4, Alpine libvirt lab)", - "env": {"SHELL": "/bin/sh", "TERM": "xterm-256color"}} + header = { + "version": 2, "width": 100, "height": 26, + "timestamp": int(time.time()), + "title": "PF on Linux — OpenBSD PF engine inside the Linux kernel (lab demo)", + "env": {"SHELL": "/bin/sh", "TERM": "xterm-256color"}, + } + runners = { + "pf0": lambda c: on_pf0(c), + "cli": lambda c: on_vm(CLI, c), + "srv": lambda c: on_vm(SRV, c), + "local": lambda c: sh(c), + } + labels = {"pf0": "pftest-pf0", "cli": "pftest-cli", "srv": "pftest-srv", + "local": "lab-host"} + + stage() with open(out_path, "w") as f: f.write(json.dumps(header) + "\n") - - def show_prompt(host): - emit(f"\x1b[1;36m{host}$\x1b[0m ", 0.15) - - runners = {"pf0": vm0, "cli": lambda c: vm_ip(CLI, c), - "srv": lambda c: vm_ip(SRV, c), "local": - lambda c: subprocess.run(["sh", "-c", c], - capture_output=True, text=True, - timeout=30).stdout + - subprocess.run(["sh", "-c", c], capture_output=True, - text=True).stderr} - prep() - for kind, cmd in DEMO: - label = {"pf0": "pftest-pf0", "cli": "pftest-cli", - "srv": "pftest-srv", "local": "build-host"}[kind] - show_prompt(label) - emit(cmd + "\r\n", 0.25) + for kind, cmd in SCENARIO: + emit(f"\x1b[1;36m{labels[kind]}\x1b[0m:\x1b[1;37m {cmd}\x1b[0m\r\n", + 0.35) try: out = runners[kind](cmd) - except Exception as e: - out = f"[runner error: {e}]\n" - # strip ANSI-safe passthrough; small human-ish delay - delay = min(1.2, 0.25 + len(out) / 4000.0) - emit(out.replace("\n", "\r\n") or "(no output)\r\n", delay) - time.sleep(0.1) - emit("\x1b[1;36mpftest-pf0$\x1b[0m /opt/pf/pfctl -d\r\n", 0.4) - runners["pf0"]("/opt/pf/pfctl -d") - for i, ev in enumerate(casts): - f.write(json.dumps(ev) + "\n") - print(f"WROTE {out_path} ({len(casts)} events, {t:.1f}s)") - print("play: asciinema play " + out_path + - " (or: script-replay/pf-in-play fallback: cat with 'less -R')") + except subprocess.TimeoutExpired: + out = "[timeout]\r\n" + except Exception as e: # noqa: BLE001 + out = f"[error: {e}]\r\n" + out = out.replace("\r\n", "\n").replace("\n", "\r\n") + emit(out or "(no output)\r\n", min(1.4, 0.3 + len(out) / 5000.0)) + with open(out_path, "a") as f: + for ev in events: + f.write(json.dumps(ev) + "\n") + print(f"WROTE {out_path}: {len(events)} events, {t:.1f}s") + print("play: asciinema play " + out_path) + print("upload: asciinema upload " + out_path) if __name__ == "__main__": ap = argparse.ArgumentParser() - ap.add_argument("--out", default=OUT_DEFAULT) + ap.add_argument("--out", default=os.path.join(REPO, "out", + "demo-pf-linux.cast")) + ap.add_argument("--quick", action="store_true") a = ap.parse_args() - record(a.out) + record(a.out, a.quick)