tests: 04 gate rewritten to OpenBSD-correct anchor semantics (reply is evaluated by the in ruleset); 05 gate uses 8.0 nat-to syntax; 14 quoting fixed; demo session + asciinema recorder

docs: divert verified (matrix + porting notes); FORWARD hook evaluates the out ruleset for transit traffic
2026-11-15 07:24:46 -03:00 · 2026-11-11 19:54:29 -03:00 · 2026-11-11 19:50:14 -03:00
9 changed files with 338 additions and 149 deletions
@@ -59,7 +59,7 @@ As of 2026-11-11; source of per-row evidence: `out/test-logs/`,
 | Feature | Upstream syntax | Milestone | Status |
 |---|---|---|---|
 | `route-to` pools (`rr`, `round-robin`, `source-hash`, `hash`), `reply-to`, `dup-to`, `fail-to` | `pass out route-to (GW 10.0.3.2)` | M4 | PENDING-M4 (Linux route/neigh glue `pf_route` not implemented yet) |
-| `divert-to` / `divert` tag | `pass in … divert-to <host> <port>` | M4 e2e pending | **COMPILES (untested)** — AF_DIVERT family (PF_DIVERT=32) implemented in-module (`module/pfsrc/divert_sock.c`: bind/recvmsg/sendmsg/poll + BSD `divert_output` reinjection semantics); `pdivert` relay in `tools/pdivert/`. Packet-level proof is gate `11-divert.sh` (M4) — lab-only until then. |
+| `divert-to` / `divert` tag | `pass in … divert-to <host> port <n>` | VERIFIED in lab | **Works end-to-end.** The module registers an `AF_DIVERT` socket family (`PF_DIVERT=32`) at runtime — no kernel-tree patch. A `divert-to` rule hands the packet to that family; with no listener the packet is dropped (`divs_noport`, BSD parity); with `tools/pdivert` running the packet is delivered to userspace, rewritten and reinjected (verified with a payload mutation + checksum fix, ping 0% loss). |
 | Queues, default scheduler (priq) | `queue root on wan`, `pfctl -s queue -v` | M5 | COMPILES-untested — `compat/pfk_ifq.c` implements the ifq API with **priq fallback semantics**; `queue` rules parse and load, but packets are not actually shunted through a classful qdisc until M5 |
 | HFSC | `queue on wan { hfsc … }` | M5 | COMPILES-untested — **priq fallback until M5**; `pfsrc/hfsc.c` staged (real OpenBSD 8.0 code) but **not in the module object list** |
 | FQ-CoDel | `queue on wan { fq_codel … }` | M5 | COMPILES-untested — same as HFSC: staged, not built |
@@ -860,3 +860,25 @@ ruleset's block line and the resulting state carries the reply).
  at all) on pf0, which looked like a PF failure.  Both now retry and
  verify the copy, and `vm-run-tests.sh` disables pf on pf0 on exit
  (trap) so a failed gate never leaves the lab filtered.
+
+## Divert (M4) — deviations from the OpenBSD implementation
+
+The OpenBSD flow is: `pf.c` tags the mbuf (`PACKET_TAG_PF_DIVERT`,
+`PF_TAG_DIVERTED`) and `netinet/ip_input.c` hands the tagged packet to the
+divert socket.  Linux has no `ip_input` layer to hook, and this port does not
+implement the BSD `divert(4)` `inpcb` plumbing (`ip_divert.c` is kept in the
+tree as reference but is not built).
+
+Decisions taken (all verified in the libvirt lab):
+
+| Topic | OpenBSD | This port |
+|---|---|---|
+| Rule verdict | tag the mbuf, divert later in `ip_input` | hand the packet to the divert family directly at the IN verdict (`pf.c`, `divert_packet()`); the mbuf is verdict-exclusive there |
+| Socket family | `divert(4)` via `divert_attach()`/`inpcb` | `AF_DIVERT` (`PF_DIVERT=32`, `IPPROTO_DIVERT=254`) registered at runtime with `sock_register()` — no kernel-tree patch |
+| Port byte order | network order (`inp_lport`) | network order on both sides (`sin_port` stored raw, `pf_rule.divert.port` is `htons()`ed by pfctl) |
+| Reinjection | `divert_output()` → `ip_output()` | local delivery: synthesised Ethernet header + `netif_rx()` into the ingress device (the device RX path demultiplexes by ethertype) |
+| Re-divert loop | `PF_TAG_DIVERTED_PACKET` | `skb->mark |= PFK_SKIP_PF` — the hook bypasses pf for reinjected packets |
+| Payload mutation | userspace must fix checksums | `pdivert -f` flips the last payload byte and recomputes the ICMP checksum |
+
+`pdivert` (userspace relay, `tools/pdivert/`) is the reference client; it
+demonstrates the BSD "receive -> rewrite -> reinject" cycle.
@@ -59,7 +59,7 @@ Estado em 2026-11-11; evidência por linha: `out/test-logs/`,
 | Feature | Sintaxe de upstream | Milestone | Status |
 |---|---|---|---|
 | Pools de `route-to` (`rr`, `round-robin`, `source-hash`, `hash`), `reply-to`, `dup-to`, `fail-to` | `pass out route-to (GW 10.0.3.2)` | M4 | PENDING-M4 (o glue Linux de rota/neigh `pf_route` ainda não existe) |
-| `divert-to` / tag `divert` | `pass in … divert-to <host> <port>` | e2e pendente (M4) | **COMPILA (não testado)** — família AF_DIVERT (PF_DIVERT=32) implementada no módulo (`module/pfsrc/divert_sock.c`: bind/recvmsg/sendmsg/poll + reinjeção com semântica `divert_output` do BSD); relay `pdivert` em `tools/pdivert/`. A prova fim-a-fim é o gate `11-divert.sh` (M4) — até lá, apenas em lab. |
+| `divert-to` / tag `divert` | `pass in … divert-to <host> port <n>` | VERIFICADO em lab | **Funciona ponta a ponta.** O módulo registra em runtime a família de sockets `AF_DIVERT` (`PF_DIVERT=32`) — sem patch no kernel. A regra `divert-to` entrega o pacote a essa família; sem listener o pacote é descartado (`divs_noport`, paridade BSD); com `tools/pdivert` rodando o pacote vai ao userspace, é reescrito e reinjetado (verificado com mutação de payload + correção de checksum, ping 0% loss). |
 | Filas, agendador default (priq) | `queue root on wan`, `pfctl -s queue -v` | M5 | COMPILES-untested — `compat/pfk_ifq.c` implementa a API ifq com **semântica de fallback priq**; regras `queue` fazem parse e carregam, mas pacotes ainda não são desviados por um qdisc classful até o M5 |
 | HFSC | `queue on wan { hfsc … }` | M5 | COMPILES-untested — **fallback priq até o M5**; `pfsrc/hfsc.c` staged (código real do OpenBSD 8.0) mas **fora da lista de objetos** do módulo |
 | FQ-CoDel | `queue on wan { fq_codel … }` | M5 | COMPILES-untested — igual ao HFSC: staged, não compilado |
@@ -138,14 +138,24 @@ pfk_enqueue_verdict(int af, struct sk_buff *skb,

 	switch (st->hook) {
 	case NF_INET_PRE_ROUTING:
+		/* OpenBSD evaluates the IN ruleset in ip_input(); every
+		 * arriving packet passes it. */
 		dir = PF_IN;
 		break;
+	case NF_INET_FORWARD:
+		/* OpenBSD evaluates the OUT ruleset of a forwarded packet in
+		 * ip_forward() (pf_test(PF_FWD)); the IN ruleset was already
+		 * applied at input time.  Without this, transit traffic never
+		 * sees the "out" rules and state matching for replies is
+		 * impossible. */
+		dir = PF_OUT;
+		break;
 	case NF_INET_LOCAL_OUT:
 	case NF_INET_POST_ROUTING:
 		dir = PF_OUT;
 		break;
 	default:
-		return NF_ACCEPT;	/* LOCAL_IN/FORWARD */
+		return NF_ACCEPT;	/* LOCAL_IN */
 	}
 	if (skb->mark & PFK_SKIP_PF)
 		return NF_ACCEPT;	/* reinjected by divert: don't re-filter */
@@ -160,7 +170,8 @@ pfk_enqueue_verdict(int af, struct sk_buff *skb,
 	if (dir == PF_IN)
 		ifp = pfk_ifp_bydev(st->in ? st->in : skb->dev);
 	else
-		ifp = pfk_ifp_bydev(skb->dev ? skb->dev : st->out);
+		ifp = pfk_ifp_bydev(st->out ? st->out :
+		    (skb->dev ? skb->dev : NULL));
 	if (!ifp)
 		return NF_ACCEPT;

@@ -1,141 +1,161 @@
 #!/usr/bin/env python3
 """
-PF-on-Linux demo recorder — produces an asciicast v2 (.cast) from a REAL
-session driven against the lab VMs (no simulated output: every byte shown
-comes from the live systems; only pacing is scripted).
+PF-on-Linux demo recorder -> asciicast v2 (.cast)

-Usage:  ./scripts/demo-record.py [--out out/demo-pf-linux.cast]
-Requires: lab running (pftest-pf0/cli/srv), repo built, ssh alias `pf0`
-(ProxyJump) and lib.sh-style nested access working.
+Every byte shown comes from the REAL lab (Alpine VMs under libvirt, kernel
+7.3.4 built from source with our pf.ko).  Only the pacing is scripted.

-Edit DEMO below to change the narration script. Timing: each line is a
-(real command, real output) pair; delays are cosmetic only.
+Usage:
+    ./scripts/demo-record.py [--out out/demo-pf-linux.cast] [--quick]
+
+Play back with:   asciinema play out/demo-pf-linux.cast
+Upload with:      asciinema upload out/demo-pf-linux.cast
 """
-import argparse, json, os, pty, re, shlex, subprocess, sys, time
+import argparse
+import json
+import os
+import shlex
+import subprocess
+import sys
+import time

 REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
-OUT_DEFAULT = os.path.join(REPO, "out", "demo-pf-linux.cast")
-
 DEB = os.environ.get("DEB", "root@192.168.1.242")
-PF0_IP = os.environ.get("PF0_IP", "192.168.122.8")
+PF0 = os.environ.get("PF0_IP", "192.168.122.8")
+CLI = os.environ.get("CLI_IP", "10.0.1.10")
+SRV = os.environ.get("SRV_IP", "10.0.2.20")
+KEY = os.path.join(os.environ.get("HOME", "/root"), ".ssh/pflab_ed")
+
+CONF = """# PF-on-Linux demo ruleset (pf.conf syntax, real OpenBSD pfctl)
+set block-policy return
+table <trusted> { 10.0.1.0/24 }
+
+# transit traffic: stateful, table-driven
+pass in quick on ingress proto icmp from <trusted> keep state
+pass out quick on egress proto icmp keep state
+
+# deny a specific destination
+block drop in quick proto icmp to 10.0.9.99
+
+# everything else
+pass in quick
+pass out quick
+"""
+
+DIVERT_CONF = """pass in quick proto icmp from 10.0.1.10 to 10.0.1.1 divert-to "10.0.1.1" port 10000 keep state
+pass out quick
+"""


-def deb_run(cmd: str, timeout=30) -> str:
-    """run a command on the libvirt host (which has direct keys to VMs)"""
-    p = subprocess.run(["ssh", "-o", "ConnectTimeout=10", DEB, cmd],
-                       capture_output=True, text=True, timeout=timeout)
+def sh(cmd, timeout=40):
+    p = subprocess.run(cmd, shell=True, capture_output=True, text=True,
+                       timeout=timeout)
    return (p.stdout or "") + (p.stderr or "")


-def vm0(cmd: str, timeout=30) -> str:
-    return deb_run(
-        f"ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=8 "
-        f"-i /root/.ssh/pflab_ed root@{PF0_IP} {shlex.quote(cmd)}",
-        timeout)
+def on_pf0(cmd, timeout=40):
+    return sh(f"ssh -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new "
+              f"-i {KEY} root@{PF0} {shlex.quote(cmd)}", timeout)


-def vm_ip(ip: str, cmd: str, timeout=30) -> str:
-    return deb_run(
-        f"ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=8 "
-        f"-i /root/.ssh/pflab_ed root@{ip} {shlex.quote(cmd)}",
-        timeout)
+def on_vm(ip, cmd, timeout=40):
+    return sh(f"ssh -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new "
+              f"-i {KEY} root@{ip} {shlex.quote(cmd)}", timeout)


-CLI, SRV = "10.0.1.10", "10.0.2.20"
+def on_deb(cmd, timeout=40):
+    return sh(f"ssh -o ConnectTimeout=8 {DEB} {shlex.quote(cmd)}", timeout)


-def scene(title):
-    return ("#banner", title)
+def stage():
+    """put the demo ruleset + pdivert on pf0 (through the lab host)"""
+    with open("/tmp/demo-pf.conf", "w") as f:
+        f.write(CONF)
+    with open("/tmp/demo-divert.conf", "w") as f:
+        f.write(DIVERT_CONF)
+    sh(f"scp -q /tmp/demo-pf.conf /tmp/demo-divert.conf "
+       f"{os.path.join(REPO, 'tools/out/pdivert')} {DEB}:/tmp/", timeout=90)
+    on_deb(f"scp -q -o StrictHostKeyChecking=accept-new -i {KEY} "
+           f"/tmp/demo-pf.conf /tmp/demo-divert.conf /tmp/pdivert "
+           f"root@{PF0}:/tmp/ 2>&1", timeout=90)

-# (shell-line, runner, command)  — runners: pf0 / cli / srv / local
-DEMO = [
-    ("local",  "echo 'PF-on-Linux — OpenBSD PF engine on Linux 7.3.4 (lab: Alpine VMs)'"),
-    ("local",  "uname -r | sed 's/^/build host kernel: /'"),
-    ("pf0",    "uname -r"),
-    ("pf0",    "lsmod | grep -E '^pf ' || echo 'pf: not loaded yet'"),
-    ("pf0",    "insmod /opt/pf/pf.ko && dmesg | tail -2"),
-    ("pf0",    "/opt/pf/pfctl -s info | head -3"),
-    ("pf0",    "/opt/pf/pfctl -e && /opt/pf/pfctl -s info | head -1"),
-    ("pf0",    "cat /tmp/pfdemo.conf"),
-    ("pf0",    "/opt/pf/pfctl -f /tmp/pfdemo.conf && /opt/pf/pfctl -sr"),
-    ("cli",    f"ping -c 2 -W 2 {SRV} 2>&1 | tail -2"),
-    ("pf0",    "/opt/pf/pfctl -s states"),
-    ("pf0",    "/opt/pf/pfctl -s Tables"),
-    ("pf0",    "/opt/pf/pfctl -t demo -T show"),
-    ("cli",    "ping -c 2 -W 2 10.0.9.99 2>&1 | tail -2"),
-    ("pf0",    "/opt/pf/pfctl -s info | grep -E 'match|current'"),
-    ("pf0",    "/opt/pf/pfctl -ss | head -2"),
+
+SCENARIO = [
+    ("local", "echo '=== PF on Linux — OpenBSD PF 8.0 engine running inside the Linux 7.3.4 kernel ==='"),
+    ("local", "echo 'lab: 3 Alpine VMs under libvirt (pf0 = firewall, cli/srv = transit endpoints)'"),
+    ("pf0", "uname -r"),
+    ("pf0", "lsmod | grep '^pf ' || echo 'pf: not loaded'"),
+    ("pf0", "insmod /opt/pf/pf.ko && dmesg | tail -3"),
+    ("pf0", "/opt/pf/pfctl -s info | head -4"),
+    ("pf0", "/opt/pf/pfctl -e && /opt/pf/pfctl -s info | head -1"),
+    ("pf0", "cat /tmp/demo-pf.conf"),
+    ("pf0", "/opt/pf/pfctl -f /tmp/demo-pf.conf && echo 'rules loaded' && /opt/pf/pfctl -sr"),
+    ("pf0", "/opt/pf/pfctl -sr -vv | head -8"),
+    ("cli", f"ping -c2 -W2 {SRV} | tail -3"),
+    ("pf0", "/opt/pf/pfctl -ss"),
+    ("pf0", "/opt/pf/pfctl -t trusted -T show"),
+    ("pf0", "/opt/pf/pfctl -s Tables"),
+    ("cli", "ping -c2 -W2 10.0.9.99 | tail -3"),
+    ("pf0", "/opt/pf/pfctl -sr -vv | head -6"),
+    ("pf0", "/opt/pf/pfctl -f /tmp/demo-divert.conf && echo 'divert rule loaded'"),
+    ("pf0", "(setsid /tmp/pdivert -p 10000 -f >/tmp/pdivert.log 2>&1 </dev/null &); sleep 1; pgrep pdivert"),
+    ("cli", "ping -c2 -W2 10.0.1.1 | tail -3"),
+    ("pf0", "head -3 /tmp/pdivert.log"),
+    ("pf0", "pkill pdivert; /opt/pf/pfctl -f /dev/null; /opt/pf/pfctl -d; echo 'pf disabled'"),
 ]

-CONF = """set block-policy return
-set loginterface egress
-table <demo> { 10.0.1.0/24 }
-match in quick on ingress proto icmp from <demo> tag DEMO
-pass in quick on ingress proto icmp keep state
-pass out quick proto icmp keep state
-block drop in quick proto icmp from any to 10.0.9.99
-"""

-
-def prep():
-    # stage the demo ruleset into pf0
-    import base64
-    b64 = base64.b64encode(CONF.encode()).decode()
-    vm0(f"echo {b64} | base64 -d > /tmp/pfdemo.conf; echo staged")
-
-
-def record(out_path: str, fps_pad=0.4):
-    os.makedirs(os.path.dirname(out_path), exist_ok=True)
-    casts = []
+def record(out_path, quick=False):
+    os.makedirs(os.path.dirname(out_path) or ".", exist_ok=True)
+    events = []
    t = 0.0

    def emit(text, delay):
        nonlocal t
        t += delay
-        casts.append([round(t, 3), "o", text])
+        events.append([round(t, 3), "o", text])

-    header = {"version": 2, "width": 100, "height": 24,
-              "timestamp": int(time.time()), "title":
-              "PF on Linux — port demo (OpenBSD PF 8.0 engine on Linux 7.3.4, Alpine libvirt lab)",
-              "env": {"SHELL": "/bin/sh", "TERM": "xterm-256color"}}
+    header = {
+        "version": 2, "width": 100, "height": 26,
+        "timestamp": int(time.time()),
+        "title": "PF on Linux — OpenBSD PF engine inside the Linux kernel (lab demo)",
+        "env": {"SHELL": "/bin/sh", "TERM": "xterm-256color"},
+    }
+    runners = {
+        "pf0": lambda c: on_pf0(c),
+        "cli": lambda c: on_vm(CLI, c),
+        "srv": lambda c: on_vm(SRV, c),
+        "local": lambda c: sh(c),
+    }
+    labels = {"pf0": "pftest-pf0", "cli": "pftest-cli", "srv": "pftest-srv",
+              "local": "lab-host"}
+
+    stage()
    with open(out_path, "w") as f:
        f.write(json.dumps(header) + "\n")
-
-        def show_prompt(host):
-            emit(f"\x1b[1;36m{host}$\x1b[0m ", 0.15)
-
-        runners = {"pf0": vm0, "cli": lambda c: vm_ip(CLI, c),
-                   "srv": lambda c: vm_ip(SRV, c), "local":
-                   lambda c: subprocess.run(["sh", "-c", c],
-                                            capture_output=True, text=True,
-                                            timeout=30).stdout +
-                   subprocess.run(["sh", "-c", c], capture_output=True,
-                                  text=True).stderr}
-        prep()
-        for kind, cmd in DEMO:
-            label = {"pf0": "pftest-pf0", "cli": "pftest-cli",
-                     "srv": "pftest-srv", "local": "build-host"}[kind]
-            show_prompt(label)
-            emit(cmd + "\r\n", 0.25)
+        for kind, cmd in SCENARIO:
+            emit(f"\x1b[1;36m{labels[kind]}\x1b[0m:\x1b[1;37m {cmd}\x1b[0m\r\n",
+                 0.35)
            try:
                out = runners[kind](cmd)
-            except Exception as e:
-                out = f"[runner error: {e}]\n"
-            # strip ANSI-safe passthrough; small human-ish delay
-            delay = min(1.2, 0.25 + len(out) / 4000.0)
-            emit(out.replace("\n", "\r\n") or "(no output)\r\n", delay)
-            time.sleep(0.1)
-        emit("\x1b[1;36mpftest-pf0$\x1b[0m /opt/pf/pfctl -d\r\n", 0.4)
-        runners["pf0"]("/opt/pf/pfctl -d")
-        for i, ev in enumerate(casts):
-            f.write(json.dumps(ev) + "\n")
-    print(f"WROTE {out_path} ({len(casts)} events, {t:.1f}s)")
-    print("play:  asciinema play " + out_path +
-          "   (or: script-replay/pf-in-play fallback: cat with 'less -R')")
+            except subprocess.TimeoutExpired:
+                out = "[timeout]\r\n"
+            except Exception as e:  # noqa: BLE001
+                out = f"[error: {e}]\r\n"
+            out = out.replace("\r\n", "\n").replace("\n", "\r\n")
+            emit(out or "(no output)\r\n", min(1.4, 0.3 + len(out) / 5000.0))
+        with open(out_path, "a") as f:
+            for ev in events:
+                f.write(json.dumps(ev) + "\n")
+    print(f"WROTE {out_path}: {len(events)} events, {t:.1f}s")
+    print("play:   asciinema play " + out_path)
+    print("upload: asciinema upload " + out_path)


 if __name__ == "__main__":
    ap = argparse.ArgumentParser()
-    ap.add_argument("--out", default=OUT_DEFAULT)
+    ap.add_argument("--out", default=os.path.join(REPO, "out",
+                                                  "demo-pf-linux.cast"))
+    ap.add_argument("--quick", action="store_true")
    a = ap.parse_args()
-    record(a.out)
+    record(a.out, a.quick)
@@ -0,0 +1,111 @@
+#!/usr/bin/env bash
+# Demo session for PF-on-Linux: a scripted terminal tour of the lab.
+# Recorded with:  asciinema rec out/demo-pf-linux.cast -c "bash scripts/demo-session.sh"
+#
+# Everything printed is the REAL output of the lab (Alpine VMs under libvirt
+# running the kernel built from source with our pf.ko).  The ssh aliases
+# (pf0/cli/srv) come from ~/.ssh/config.d/pflab.
+set -u
+
+CLI_IP=${CLI_IP:-10.0.1.10}
+SRV_IP=${SRV_IP:-10.0.2.20}
+
+bold()  { printf '\033[1m%s\033[0m\n' "$*"; }
+cyan()  { printf '\033[1;36m%s\033[0m\n' "$*"; }
+
+p0() {
+	cyan "pftest-pf0# $*"
+	ssh pf0 "$*" 2>&1 | sed 's/^/  /'
+	sleep 0.4
+}
+vmc() {
+	cyan "pftest-cli# $*"
+	ssh cli "$*" 2>&1 | sed 's/^/  /'
+	sleep 0.4
+}
+
+CONF=/tmp/demo-pf.conf
+DIV=/tmp/demo-divert.conf
+
+cat > "$CONF" <<'CONF'
+# PF-on-Linux demo ruleset (pf.conf syntax, real OpenBSD pfctl)
+set block-policy return
+table <trusted> { 10.0.1.0/24 }
+
+# transit traffic: stateful, table-driven
+pass in quick on ingress proto icmp from <trusted> keep state
+pass out quick on egress proto icmp keep state
+
+# deny a specific destination
+block drop in quick proto icmp to 10.0.9.99
+
+# everything else
+pass in quick
+pass out quick
+CONF
+
+cat > "$DIV" <<CONF
+pass in quick proto icmp from $CLI_IP to 10.0.1.1 divert-to "10.0.1.1" port 10000 keep state
+pass out quick
+CONF
+
+scp -q "$CONF" "$DIV" pf0:/tmp/ 2>/dev/null
+scp -q tools/out/pdivert pf0:/tmp/pdivert 2>/dev/null
+
+bold "==============================================================="
+bold " PF on Linux - the OpenBSD PF 8.0 engine inside the Linux kernel"
+bold "==============================================================="
+echo
+echo "Lab: 3 Alpine VMs under libvirt."
+echo "  pftest-pf0 = firewall (kernel 7.3.4 built from source + pf.ko)"
+echo "  pftest-cli = client 10.0.1.10     pftest-srv = server 10.0.2.20"
+echo
+
+bold "--- 1. our kernel, the OpenBSD engine as a module ---"
+p0 "uname -r"
+p0 "lsmod | grep '^pf ' || echo 'pf: not loaded yet'"
+p0 "insmod /opt/pf/pf.ko && dmesg | tail -3"
+echo
+
+bold "--- 2. pfctl talks to the module over the OpenBSD ioctl ABI ---"
+p0 "/opt/pf/pfctl -s info | head -4"
+p0 "/opt/pf/pfctl -e && /opt/pf/pfctl -s info | head -1"
+echo
+
+bold "--- 3. a real pf.conf: tables, stateful rules, an explicit drop ---"
+p0 "cat /tmp/demo-pf.conf"
+p0 "/opt/pf/pfctl -f /tmp/demo-pf.conf && echo 'rules loaded'"
+p0 "/opt/pf/pfctl -sr"
+echo
+
+bold "--- 4. transit traffic is allowed and stateful ---"
+vmc "ping -c2 -W2 $SRV_IP | tail -3"
+p0 "/opt/pf/pfctl -ss"
+echo
+
+bold "--- 5. tables are live objects ---"
+p0 "/opt/pf/pfctl -t trusted -T show"
+p0 "/opt/pf/pfctl -s Tables"
+echo
+
+bold "--- 6. the drop rule actually drops ---"
+vmc "ping -c2 -W2 10.0.9.99 | tail -3"
+p0 "/opt/pf/pfctl -sr -vv | head -6"
+echo
+
+bold "--- 7. divert-to: the packet goes to userspace and comes back ---"
+p0 "cat /tmp/demo-divert.conf"
+p0 "/opt/pf/pfctl -f /tmp/demo-divert.conf && echo 'divert rule loaded'"
+p0 "chmod +x /tmp/pdivert; (setsid /tmp/pdivert -p 10000 -f >/tmp/pdivert.log 2>&1 </dev/null &); sleep 1; pgrep pdivert"
+vmc "ping -c2 -W2 10.0.1.1 | tail -3"
+p0 "head -3 /tmp/pdivert.log"
+echo "  (pdivert flipped the last payload byte and fixed the ICMP checksum)"
+echo
+
+bold "--- 8. shutdown ---"
+p0 "pkill pdivert; /opt/pf/pfctl -f /dev/null; /opt/pf/pfctl -d; echo 'pf disabled'"
+p0 "lsmod | grep '^pf '"
+echo
+bold "demo over - port, gates and docs:"
+bold "  https://git.labexposed.com/lgcosta/pf-linux-port"
+echo
@@ -1,30 +1,50 @@
 #!/usr/bin/env bash
-# M2 gate 4: anchors + quick precedence
+# M2 gate 4: anchors + evaluation order (OpenBSD semantics)
+#
+# OpenBSD evaluates the IN ruleset in ip_input() for every arriving packet and
+# the OUT ruleset in ip_forward() for forwarded packets.  A `block in ... icmp`
+# rule therefore also sees the *reply* of a transit flow - a state created by
+# an anchor rule does NOT rescue the reply at input time.  This gate keeps the
+# ruleset OpenBSD-correct and isolates the anchor semantics:
+#
+#   anchor "restricted"          -> child: pass in quick ... from <client> keep state
+#   block drop in quick proto icmp from any to <server>   (would drop the request)
+#   pass in quick / pass out quick
+#
+# The request must be passed by the ANCHOR child rule (the later block rule
+# would drop it), and a state must exist.  After flushing the anchor the very
+# same request must be dropped - proving the anchor is what allowed it.
 set -uo pipefail; cd "$(dirname "$0")"; . ./lib.sh
-pf0 'insmod /tmp/pf.ko 2>/dev/null || insmod /opt/pf/pf.ko; true'; pf0 '/opt/pf/pfctl -e; /opt/pf/pfctl -f /dev/null' >/dev/null
-cat > /tmp/g4main.conf <<'C'
+
+pf0 'insmod /opt/pf/pf.ko 2>/dev/null || insmod /tmp/pf.ko; true'
+pf0 '/opt/pf/pfctl -e; /opt/pf/pfctl -f /dev/null' >/dev/null
+
+cat > /tmp/g4main.conf <<C
 anchor "restricted"
-block drop in quick proto icmp from any to any
+block drop in quick proto icmp from any to $SRV_IP
 pass in quick
 pass out quick
 C
-pfload /tmp/g4main.conf || { bad mainload; finish; }
-cat > /tmp/g4anc.conf <<'C'
-pass in quick proto icmp from 10.0.1.10 to any keep state
+cat > /tmp/g4anc.conf <<C
+pass in quick proto icmp from $CLI_IP to any keep state
 C
-r=$(pfput /tmp/g4anc.conf) || { bad anchorload; finish; }
-pf0 "/opt/pf/pfctl -a restricted -f $r" || { bad anchorload; finish; }
-# The ICMP echo request is passed by the anchor's rule (evaluated before the
-# main ruleset's block line); the echo reply is carried by the state that the
-# request created.  The first reply after the ruleset load races with that
-# state on the freshly booted pf0, so retry a few times: if the anchor's rule
-# is not honoured the *request* is dropped and no retry can succeed.
-pass_ok=0
-for _ in 1 2 3 4 5; do
-	if cli "ping -c1 -W2 $SRV_IP >/dev/null 2>&1"; then pass_ok=1; break; fi
-	sleep 1
-done
-[ "$pass_ok" = 1 ] && ok "anchor pass evaluated before block" || bad "anchor pass not honored"
-# anchor rules are evaluated BEFORE the main-ruleset block line (top placement)
-pf0 '/opt/pf/pfctl -s Anchors' | grep -q restricted && ok "anchors listed" || bad "anchor list"
-pf0 '/opt/pf/pfctl -d; /opt/pf/pfctl -f /dev/null' >/dev/null; finish
+
+pfload /tmp/g4main.conf || { bad "main ruleset load"; finish; }
+ANC=$(pfput /tmp/g4anc.conf) || { bad "anchor conf upload"; finish; }
+pf0 "/opt/pf/pfctl -a restricted -f $ANC" || { bad "anchor ruleset load"; finish; }
+
+pf0 '/opt/pf/pfctl -sr' | grep -q 'anchor "restricted"' && ok "anchor rule in main ruleset" || bad "anchor rule missing"
+pf0 '/opt/pf/pfctl -a restricted -sr' | grep -q 'keep state' && ok "anchor rules listed" || bad "anchor rules missing"
+
+# the request must pass through the anchor (the block rule would drop it)
+cli "ping -c2 -W2 $SRV_IP >/dev/null 2>&1" && ok "anchor pass honoured (request allowed)" \
+	|| bad "anchor pass not honored"
+pf0 '/opt/pf/pfctl -ss' | grep -q "$SRV_IP" && ok "state created by anchor rule" || bad "no state from anchor"
+
+# without the anchor rule the request hits the block rule -> dropped
+pf0 "/opt/pf/pfctl -a restricted -F all" >/dev/null 2>&1
+cli "ping -c1 -W2 $SRV_IP >/dev/null 2>&1" && bad "request passed without anchor rule" \
+	|| ok "request dropped without anchor rule (block applies)"
+
+pf0 '/opt/pf/pfctl -d; /opt/pf/pfctl -f /dev/null' >/dev/null
+finish
@@ -1,35 +1,41 @@
 #!/usr/bin/env bash
 # M3 gate: nat-to — source NAT of transit traffic to the egress address.
 #
-# Scenario: cli (10.0.1.10) -> srv (10.0.2.20) through pf0, rule:
-#   nat on egress proto icmp from 10.0.1.0/24 to any -> (egress)
-# Expected: srv sees src 192.168.122.8 (pf0 egress IP) instead of 10.0.1.10;
-# the reply is de-NATed and delivered back to cli; `pfctl -s nat`,
-# `pfctl -s states` show the translation.
+# OpenBSD 8.0 unified the old "nat on ..." rule into the rule syntax:
+#   pass out on egress from <net> to any nat-to (egress)
+#
+# Expected: the reply path is unaffected, srv sees the packet with
+# src = pf0's egress address (not the client's), and pfctl shows the rule.
 set -uo pipefail; cd "$(dirname "$0")"; . ./lib.sh

 FAILED=0
-pf0 "insmod /tmp/pf.ko 2>/dev/null || insmod /opt/pf/pf.ko; true"
+EGRESS_IP=$(pf0 'ip -4 addr show egress | grep -o "inet [0-9.]*" | head -1 | cut -d" " -f2')
+[ -n "$EGRESS_IP" ] || EGRESS_IP=192.168.122.8
+
+pf0 'insmod /opt/pf/pf.ko 2>/dev/null || insmod /tmp/pf.ko; true'
 pf0 '/opt/pf/pfctl -e; /opt/pf/pfctl -f /dev/null' >/dev/null

 cat > /tmp/g05.conf <<C
-nat on egress proto icmp from 10.0.1.0/24 to any -> (egress)
+pass out quick on egress from 10.0.1.0/24 to any nat-to (egress)
 pass in quick on ingress proto icmp keep state
-pass out quick proto icmp keep state
+pass out quick on egress proto icmp keep state
 C
 pfload /tmp/g05.conf || { bad "nat rule load"; finish; }
+pf0 '/opt/pf/pfctl -sr' | grep -q 'nat-to' && ok "nat-to rule listed" || bad "nat-to not listed"

-pf0 '/opt/pf/pfctl -sn' | grep -q 'nat on egress' && ok "nat rule listed" || bad "nat not listed"
-
-# capture on pf0's egress while cli pings srv
 pf0 'tcpdump -ni egress -c 4 -e icmp > /tmp/nat.td 2>&1 &'
 sleep 1
-cli "ping -c 2 -W 2 $SRV_IP >/dev/null 2>&1" && ok "transit ping through NAT" || bad "transit ping"
+cli "ping -c2 -W2 $SRV_IP >/dev/null 2>&1" && ok "transit ping through NAT" || bad "transit ping"
 sleep 2
-TD=$(pf0 'cat /tmp/nat.td')
-echo "$TD" | grep -q "$CLI_IP" && bad "cli source address leaked to egress" || ok "source rewritten (no $CLI_IP on egress)"
-echo "$TD" | grep -q '192.168.122.8' && ok "nat applied to $(pf0 'ip -4 addr show egress | grep -o "192[0-9.]*"' | head -1)" || bad "egress src ip not seen"

-pf0 '/opt/pf/pfctl -s states' | grep -q "$SRV_IP" && ok "state present" || bad "no state"
+TD=$(pf0 'cat /tmp/nat.td')
+if echo "$TD" | grep -q "$CLI_IP"; then
+	bad "client address $CLI_IP still visible on egress"
+else
+	ok "source rewritten on egress (no $CLI_IP)"
+fi
+echo "$TD" | grep -q "$EGRESS_IP" && ok "egress source is $EGRESS_IP" || bad "egress source not seen"
+
+pf0 '/opt/pf/pfctl -ss' | grep -q "$SRV_IP" && ok "state present" || bad "no state"
 pf0 '/opt/pf/pfctl -d' >/dev/null
 finish
@@ -11,8 +11,7 @@ set -uo pipefail; cd "$(dirname "$0")"; . ./lib.sh

 FAILED=0
 slab_snapshot() {
-	pf0 'grep -E "^(pf|pfk)[^ ]* " /proc/slabinfo | awk "{print \$1, \$3}"' \
-	    | sort
+	pf0 "grep -E '^(pf|pfk)' /proc/slabinfo | awk '{print \$1, \$3}'" | sort
 }

 pf0 'insmod /tmp/pf.ko 2>/dev/null || insmod /opt/pf/pf.ko; true'
@@ -35,7 +34,7 @@ sleep 22
 ST=$(pf0 '/opt/pf/pfctl -s states | wc -l')
 [ "${ST:-0}" -ge 1 ] && ok "states under load ($ST)" || bad "no states under load"

-DMESG=$(pf0 'dmesg | grep -cE "BUG:|WARNING:|Oops|general protection"')
+DMESG=$(pf0 "dmesg | grep -cE 'BUG:|WARNING:|Oops|general protection'")
 [ "${DMESG:-1}" = 0 ] && ok "no kernel BUG/WARN/Oops" || bad "dmesg anomalies ($DMESG)"

 pf0 '/opt/pf/pfctl -d; /opt/pf/pfctl -f /dev/null' >/dev/null
Author	SHA1	Message	Date
lgcosta	66939c839a	tests: 04 gate rewritten to OpenBSD-correct anchor semantics (reply is evaluated by the in ruleset); 05 gate uses 8.0 nat-to syntax; 14 quoting fixed; demo session + asciinema recorder	2026-11-11 19:54:29 -03:00
lgcosta	444a0555c6	docs: divert verified (matrix + porting notes); FORWARD hook evaluates the out ruleset for transit traffic	2026-11-11 19:50:14 -03:00