docs: divert verified (matrix + porting notes); FORWARD hook evaluates the out ruleset for transit traffic
This commit is contained in:
lgcosta committed
1 parent
bdf9db7a21
commit
444a0555c6
5 files changed
+160
-107
No files matched your search
@@ -59,7 +59,7 @@ As of 2026-11-11; source of per-row evidence: `out/test-logs/`,
|
||||
| Feature | Upstream syntax | Milestone | Status |
|
||||
|---|---|---|---|
|
||||
| `route-to` pools (`rr`, `round-robin`, `source-hash`, `hash`), `reply-to`, `dup-to`, `fail-to` | `pass out route-to (GW 10.0.3.2)` | M4 | PENDING-M4 (Linux route/neigh glue `pf_route` not implemented yet) |
|
||||
| `divert-to` / `divert` tag | `pass in … divert-to <host> <port>` | M4 e2e pending | **COMPILES (untested)** — AF_DIVERT family (PF_DIVERT=32) implemented in-module (`module/pfsrc/divert_sock.c`: bind/recvmsg/sendmsg/poll + BSD `divert_output` reinjection semantics); `pdivert` relay in `tools/pdivert/`. Packet-level proof is gate `11-divert.sh` (M4) — lab-only until then. |
|
||||
| `divert-to` / `divert` tag | `pass in … divert-to <host> port <n>` | VERIFIED in lab | **Works end-to-end.** The module registers an `AF_DIVERT` socket family (`PF_DIVERT=32`) at runtime — no kernel-tree patch. A `divert-to` rule hands the packet to that family; with no listener the packet is dropped (`divs_noport`, BSD parity); with `tools/pdivert` running the packet is delivered to userspace, rewritten and reinjected (verified with a payload mutation + checksum fix, ping 0% loss). |
|
||||
| Queues, default scheduler (priq) | `queue root on wan`, `pfctl -s queue -v` | M5 | COMPILES-untested — `compat/pfk_ifq.c` implements the ifq API with **priq fallback semantics**; `queue` rules parse and load, but packets are not actually shunted through a classful qdisc until M5 |
|
||||
| HFSC | `queue on wan { hfsc … }` | M5 | COMPILES-untested — **priq fallback until M5**; `pfsrc/hfsc.c` staged (real OpenBSD 8.0 code) but **not in the module object list** |
|
||||
| FQ-CoDel | `queue on wan { fq_codel … }` | M5 | COMPILES-untested — same as HFSC: staged, not built |
|
||||
|
||||
@@ -860,3 +860,25 @@ ruleset's block line and the resulting state carries the reply).
|
||||
at all) on pf0, which looked like a PF failure. Both now retry and
|
||||
verify the copy, and `vm-run-tests.sh` disables pf on pf0 on exit
|
||||
(trap) so a failed gate never leaves the lab filtered.
|
||||
|
||||
## Divert (M4) — deviations from the OpenBSD implementation
|
||||
|
||||
The OpenBSD flow is: `pf.c` tags the mbuf (`PACKET_TAG_PF_DIVERT`,
|
||||
`PF_TAG_DIVERTED`) and `netinet/ip_input.c` hands the tagged packet to the
|
||||
divert socket. Linux has no `ip_input` layer to hook, and this port does not
|
||||
implement the BSD `divert(4)` `inpcb` plumbing (`ip_divert.c` is kept in the
|
||||
tree as reference but is not built).
|
||||
|
||||
Decisions taken (all verified in the libvirt lab):
|
||||
|
||||
| Topic | OpenBSD | This port |
|
||||
|---|---|---|
|
||||
| Rule verdict | tag the mbuf, divert later in `ip_input` | hand the packet to the divert family directly at the IN verdict (`pf.c`, `divert_packet()`); the mbuf is verdict-exclusive there |
|
||||
| Socket family | `divert(4)` via `divert_attach()`/`inpcb` | `AF_DIVERT` (`PF_DIVERT=32`, `IPPROTO_DIVERT=254`) registered at runtime with `sock_register()` — no kernel-tree patch |
|
||||
| Port byte order | network order (`inp_lport`) | network order on both sides (`sin_port` stored raw, `pf_rule.divert.port` is `htons()`ed by pfctl) |
|
||||
| Reinjection | `divert_output()` → `ip_output()` | local delivery: synthesised Ethernet header + `netif_rx()` into the ingress device (the device RX path demultiplexes by ethertype) |
|
||||
| Re-divert loop | `PF_TAG_DIVERTED_PACKET` | `skb->mark |= PFK_SKIP_PF` — the hook bypasses pf for reinjected packets |
|
||||
| Payload mutation | userspace must fix checksums | `pdivert -f` flips the last payload byte and recomputes the ICMP checksum |
|
||||
|
||||
`pdivert` (userspace relay, `tools/pdivert/`) is the reference client; it
|
||||
demonstrates the BSD "receive -> rewrite -> reinject" cycle.
|
||||
@@ -59,7 +59,7 @@ Estado em 2026-11-11; evidência por linha: `out/test-logs/`,
|
||||
| Feature | Sintaxe de upstream | Milestone | Status |
|
||||
|---|---|---|---|
|
||||
| Pools de `route-to` (`rr`, `round-robin`, `source-hash`, `hash`), `reply-to`, `dup-to`, `fail-to` | `pass out route-to (GW 10.0.3.2)` | M4 | PENDING-M4 (o glue Linux de rota/neigh `pf_route` ainda não existe) |
|
||||
| `divert-to` / tag `divert` | `pass in … divert-to <host> <port>` | e2e pendente (M4) | **COMPILA (não testado)** — família AF_DIVERT (PF_DIVERT=32) implementada no módulo (`module/pfsrc/divert_sock.c`: bind/recvmsg/sendmsg/poll + reinjeção com semântica `divert_output` do BSD); relay `pdivert` em `tools/pdivert/`. A prova fim-a-fim é o gate `11-divert.sh` (M4) — até lá, apenas em lab. |
|
||||
| `divert-to` / tag `divert` | `pass in … divert-to <host> port <n>` | VERIFICADO em lab | **Funciona ponta a ponta.** O módulo registra em runtime a família de sockets `AF_DIVERT` (`PF_DIVERT=32`) — sem patch no kernel. A regra `divert-to` entrega o pacote a essa família; sem listener o pacote é descartado (`divs_noport`, paridade BSD); com `tools/pdivert` rodando o pacote vai ao userspace, é reescrito e reinjetado (verificado com mutação de payload + correção de checksum, ping 0% loss). |
|
||||
| Filas, agendador default (priq) | `queue root on wan`, `pfctl -s queue -v` | M5 | COMPILES-untested — `compat/pfk_ifq.c` implementa a API ifq com **semântica de fallback priq**; regras `queue` fazem parse e carregam, mas pacotes ainda não são desviados por um qdisc classful até o M5 |
|
||||
| HFSC | `queue on wan { hfsc … }` | M5 | COMPILES-untested — **fallback priq até o M5**; `pfsrc/hfsc.c` staged (código real do OpenBSD 8.0) mas **fora da lista de objetos** do módulo |
|
||||
| FQ-CoDel | `queue on wan { fq_codel … }` | M5 | COMPILES-untested — igual ao HFSC: staged, não compilado |
|
||||
|
||||
@@ -138,14 +138,24 @@ pfk_enqueue_verdict(int af, struct sk_buff *skb,
|
||||
|
||||
switch (st->hook) {
|
||||
case NF_INET_PRE_ROUTING:
|
||||
/* OpenBSD evaluates the IN ruleset in ip_input(); every
|
||||
* arriving packet passes it. */
|
||||
dir = PF_IN;
|
||||
break;
|
||||
case NF_INET_FORWARD:
|
||||
/* OpenBSD evaluates the OUT ruleset of a forwarded packet in
|
||||
* ip_forward() (pf_test(PF_FWD)); the IN ruleset was already
|
||||
* applied at input time. Without this, transit traffic never
|
||||
* sees the "out" rules and state matching for replies is
|
||||
* impossible. */
|
||||
dir = PF_OUT;
|
||||
break;
|
||||
case NF_INET_LOCAL_OUT:
|
||||
case NF_INET_POST_ROUTING:
|
||||
dir = PF_OUT;
|
||||
break;
|
||||
default:
|
||||
return NF_ACCEPT; /* LOCAL_IN/FORWARD */
|
||||
return NF_ACCEPT; /* LOCAL_IN */
|
||||
}
|
||||
if (skb->mark & PFK_SKIP_PF)
|
||||
return NF_ACCEPT; /* reinjected by divert: don't re-filter */
|
||||
@@ -160,7 +170,8 @@ pfk_enqueue_verdict(int af, struct sk_buff *skb,
|
||||
if (dir == PF_IN)
|
||||
ifp = pfk_ifp_bydev(st->in ? st->in : skb->dev);
|
||||
else
|
||||
ifp = pfk_ifp_bydev(skb->dev ? skb->dev : st->out);
|
||||
ifp = pfk_ifp_bydev(st->out ? st->out :
|
||||
(skb->dev ? skb->dev : NULL));
|
||||
if (!ifp)
|
||||
return NF_ACCEPT;
|
||||
|
||||
|
||||
+123
-103
@@ -1,141 +1,161 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
PF-on-Linux demo recorder — produces an asciicast v2 (.cast) from a REAL
|
||||
session driven against the lab VMs (no simulated output: every byte shown
|
||||
comes from the live systems; only pacing is scripted).
|
||||
PF-on-Linux demo recorder -> asciicast v2 (.cast)
|
||||
|
||||
Usage: ./scripts/demo-record.py [--out out/demo-pf-linux.cast]
|
||||
Requires: lab running (pftest-pf0/cli/srv), repo built, ssh alias `pf0`
|
||||
(ProxyJump) and lib.sh-style nested access working.
|
||||
Every byte shown comes from the REAL lab (Alpine VMs under libvirt, kernel
|
||||
7.3.4 built from source with our pf.ko). Only the pacing is scripted.
|
||||
|
||||
Edit DEMO below to change the narration script. Timing: each line is a
|
||||
(real command, real output) pair; delays are cosmetic only.
|
||||
Usage:
|
||||
./scripts/demo-record.py [--out out/demo-pf-linux.cast] [--quick]
|
||||
|
||||
Play back with: asciinema play out/demo-pf-linux.cast
|
||||
Upload with: asciinema upload out/demo-pf-linux.cast
|
||||
"""
|
||||
import argparse, json, os, pty, re, shlex, subprocess, sys, time
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
OUT_DEFAULT = os.path.join(REPO, "out", "demo-pf-linux.cast")
|
||||
|
||||
DEB = os.environ.get("DEB", "root@192.168.1.242")
|
||||
PF0_IP = os.environ.get("PF0_IP", "192.168.122.8")
|
||||
PF0 = os.environ.get("PF0_IP", "192.168.122.8")
|
||||
CLI = os.environ.get("CLI_IP", "10.0.1.10")
|
||||
SRV = os.environ.get("SRV_IP", "10.0.2.20")
|
||||
KEY = os.path.join(os.environ.get("HOME", "/root"), ".ssh/pflab_ed")
|
||||
|
||||
CONF = """# PF-on-Linux demo ruleset (pf.conf syntax, real OpenBSD pfctl)
|
||||
set block-policy return
|
||||
table <trusted> { 10.0.1.0/24 }
|
||||
|
||||
# transit traffic: stateful, table-driven
|
||||
pass in quick on ingress proto icmp from <trusted> keep state
|
||||
pass out quick on egress proto icmp keep state
|
||||
|
||||
# deny a specific destination
|
||||
block drop in quick proto icmp to 10.0.9.99
|
||||
|
||||
# everything else
|
||||
pass in quick
|
||||
pass out quick
|
||||
"""
|
||||
|
||||
DIVERT_CONF = """pass in quick proto icmp from 10.0.1.10 to 10.0.1.1 divert-to "10.0.1.1" port 10000 keep state
|
||||
pass out quick
|
||||
"""
|
||||
|
||||
|
||||
def deb_run(cmd: str, timeout=30) -> str:
|
||||
"""run a command on the libvirt host (which has direct keys to VMs)"""
|
||||
p = subprocess.run(["ssh", "-o", "ConnectTimeout=10", DEB, cmd],
|
||||
capture_output=True, text=True, timeout=timeout)
|
||||
def sh(cmd, timeout=40):
|
||||
p = subprocess.run(cmd, shell=True, capture_output=True, text=True,
|
||||
timeout=timeout)
|
||||
return (p.stdout or "") + (p.stderr or "")
|
||||
|
||||
|
||||
def vm0(cmd: str, timeout=30) -> str:
|
||||
return deb_run(
|
||||
f"ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=8 "
|
||||
f"-i /root/.ssh/pflab_ed root@{PF0_IP} {shlex.quote(cmd)}",
|
||||
timeout)
|
||||
def on_pf0(cmd, timeout=40):
|
||||
return sh(f"ssh -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new "
|
||||
f"-i {KEY} root@{PF0} {shlex.quote(cmd)}", timeout)
|
||||
|
||||
|
||||
def vm_ip(ip: str, cmd: str, timeout=30) -> str:
|
||||
return deb_run(
|
||||
f"ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=8 "
|
||||
f"-i /root/.ssh/pflab_ed root@{ip} {shlex.quote(cmd)}",
|
||||
timeout)
|
||||
def on_vm(ip, cmd, timeout=40):
|
||||
return sh(f"ssh -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new "
|
||||
f"-i {KEY} root@{ip} {shlex.quote(cmd)}", timeout)
|
||||
|
||||
|
||||
CLI, SRV = "10.0.1.10", "10.0.2.20"
|
||||
def on_deb(cmd, timeout=40):
|
||||
return sh(f"ssh -o ConnectTimeout=8 {DEB} {shlex.quote(cmd)}", timeout)
|
||||
|
||||
|
||||
def scene(title):
|
||||
return ("#banner", title)
|
||||
def stage():
|
||||
"""put the demo ruleset + pdivert on pf0 (through the lab host)"""
|
||||
with open("/tmp/demo-pf.conf", "w") as f:
|
||||
f.write(CONF)
|
||||
with open("/tmp/demo-divert.conf", "w") as f:
|
||||
f.write(DIVERT_CONF)
|
||||
sh(f"scp -q /tmp/demo-pf.conf /tmp/demo-divert.conf "
|
||||
f"{os.path.join(REPO, 'tools/out/pdivert')} {DEB}:/tmp/", timeout=90)
|
||||
on_deb(f"scp -q -o StrictHostKeyChecking=accept-new -i {KEY} "
|
||||
f"/tmp/demo-pf.conf /tmp/demo-divert.conf /tmp/pdivert "
|
||||
f"root@{PF0}:/tmp/ 2>&1", timeout=90)
|
||||
|
||||
# (shell-line, runner, command) — runners: pf0 / cli / srv / local
|
||||
DEMO = [
|
||||
("local", "echo 'PF-on-Linux — OpenBSD PF engine on Linux 7.3.4 (lab: Alpine VMs)'"),
|
||||
("local", "uname -r | sed 's/^/build host kernel: /'"),
|
||||
("pf0", "uname -r"),
|
||||
("pf0", "lsmod | grep -E '^pf ' || echo 'pf: not loaded yet'"),
|
||||
("pf0", "insmod /opt/pf/pf.ko && dmesg | tail -2"),
|
||||
("pf0", "/opt/pf/pfctl -s info | head -3"),
|
||||
("pf0", "/opt/pf/pfctl -e && /opt/pf/pfctl -s info | head -1"),
|
||||
("pf0", "cat /tmp/pfdemo.conf"),
|
||||
("pf0", "/opt/pf/pfctl -f /tmp/pfdemo.conf && /opt/pf/pfctl -sr"),
|
||||
("cli", f"ping -c 2 -W 2 {SRV} 2>&1 | tail -2"),
|
||||
("pf0", "/opt/pf/pfctl -s states"),
|
||||
("pf0", "/opt/pf/pfctl -s Tables"),
|
||||
("pf0", "/opt/pf/pfctl -t demo -T show"),
|
||||
("cli", "ping -c 2 -W 2 10.0.9.99 2>&1 | tail -2"),
|
||||
("pf0", "/opt/pf/pfctl -s info | grep -E 'match|current'"),
|
||||
("pf0", "/opt/pf/pfctl -ss | head -2"),
|
||||
|
||||
SCENARIO = [
|
||||
("local", "echo '=== PF on Linux — OpenBSD PF 8.0 engine running inside the Linux 7.3.4 kernel ==='"),
|
||||
("local", "echo 'lab: 3 Alpine VMs under libvirt (pf0 = firewall, cli/srv = transit endpoints)'"),
|
||||
("pf0", "uname -r"),
|
||||
("pf0", "lsmod | grep '^pf ' || echo 'pf: not loaded'"),
|
||||
("pf0", "insmod /opt/pf/pf.ko && dmesg | tail -3"),
|
||||
("pf0", "/opt/pf/pfctl -s info | head -4"),
|
||||
("pf0", "/opt/pf/pfctl -e && /opt/pf/pfctl -s info | head -1"),
|
||||
("pf0", "cat /tmp/demo-pf.conf"),
|
||||
("pf0", "/opt/pf/pfctl -f /tmp/demo-pf.conf && echo 'rules loaded' && /opt/pf/pfctl -sr"),
|
||||
("pf0", "/opt/pf/pfctl -sr -vv | head -8"),
|
||||
("cli", f"ping -c2 -W2 {SRV} | tail -3"),
|
||||
("pf0", "/opt/pf/pfctl -ss"),
|
||||
("pf0", "/opt/pf/pfctl -t trusted -T show"),
|
||||
("pf0", "/opt/pf/pfctl -s Tables"),
|
||||
("cli", "ping -c2 -W2 10.0.9.99 | tail -3"),
|
||||
("pf0", "/opt/pf/pfctl -sr -vv | head -6"),
|
||||
("pf0", "/opt/pf/pfctl -f /tmp/demo-divert.conf && echo 'divert rule loaded'"),
|
||||
("pf0", "(setsid /tmp/pdivert -p 10000 -f >/tmp/pdivert.log 2>&1 </dev/null &); sleep 1; pgrep pdivert"),
|
||||
("cli", "ping -c2 -W2 10.0.1.1 | tail -3"),
|
||||
("pf0", "head -3 /tmp/pdivert.log"),
|
||||
("pf0", "pkill pdivert; /opt/pf/pfctl -f /dev/null; /opt/pf/pfctl -d; echo 'pf disabled'"),
|
||||
]
|
||||
|
||||
CONF = """set block-policy return
|
||||
set loginterface egress
|
||||
table <demo> { 10.0.1.0/24 }
|
||||
match in quick on ingress proto icmp from <demo> tag DEMO
|
||||
pass in quick on ingress proto icmp keep state
|
||||
pass out quick proto icmp keep state
|
||||
block drop in quick proto icmp from any to 10.0.9.99
|
||||
"""
|
||||
|
||||
|
||||
def prep():
|
||||
# stage the demo ruleset into pf0
|
||||
import base64
|
||||
b64 = base64.b64encode(CONF.encode()).decode()
|
||||
vm0(f"echo {b64} | base64 -d > /tmp/pfdemo.conf; echo staged")
|
||||
|
||||
|
||||
def record(out_path: str, fps_pad=0.4):
|
||||
os.makedirs(os.path.dirname(out_path), exist_ok=True)
|
||||
casts = []
|
||||
def record(out_path, quick=False):
|
||||
os.makedirs(os.path.dirname(out_path) or ".", exist_ok=True)
|
||||
events = []
|
||||
t = 0.0
|
||||
|
||||
def emit(text, delay):
|
||||
nonlocal t
|
||||
t += delay
|
||||
casts.append([round(t, 3), "o", text])
|
||||
events.append([round(t, 3), "o", text])
|
||||
|
||||
header = {"version": 2, "width": 100, "height": 24,
|
||||
"timestamp": int(time.time()), "title":
|
||||
"PF on Linux — port demo (OpenBSD PF 8.0 engine on Linux 7.3.4, Alpine libvirt lab)",
|
||||
"env": {"SHELL": "/bin/sh", "TERM": "xterm-256color"}}
|
||||
header = {
|
||||
"version": 2, "width": 100, "height": 26,
|
||||
"timestamp": int(time.time()),
|
||||
"title": "PF on Linux — OpenBSD PF engine inside the Linux kernel (lab demo)",
|
||||
"env": {"SHELL": "/bin/sh", "TERM": "xterm-256color"},
|
||||
}
|
||||
runners = {
|
||||
"pf0": lambda c: on_pf0(c),
|
||||
"cli": lambda c: on_vm(CLI, c),
|
||||
"srv": lambda c: on_vm(SRV, c),
|
||||
"local": lambda c: sh(c),
|
||||
}
|
||||
labels = {"pf0": "pftest-pf0", "cli": "pftest-cli", "srv": "pftest-srv",
|
||||
"local": "lab-host"}
|
||||
|
||||
stage()
|
||||
with open(out_path, "w") as f:
|
||||
f.write(json.dumps(header) + "\n")
|
||||
|
||||
def show_prompt(host):
|
||||
emit(f"\x1b[1;36m{host}$\x1b[0m ", 0.15)
|
||||
|
||||
runners = {"pf0": vm0, "cli": lambda c: vm_ip(CLI, c),
|
||||
"srv": lambda c: vm_ip(SRV, c), "local":
|
||||
lambda c: subprocess.run(["sh", "-c", c],
|
||||
capture_output=True, text=True,
|
||||
timeout=30).stdout +
|
||||
subprocess.run(["sh", "-c", c], capture_output=True,
|
||||
text=True).stderr}
|
||||
prep()
|
||||
for kind, cmd in DEMO:
|
||||
label = {"pf0": "pftest-pf0", "cli": "pftest-cli",
|
||||
"srv": "pftest-srv", "local": "build-host"}[kind]
|
||||
show_prompt(label)
|
||||
emit(cmd + "\r\n", 0.25)
|
||||
for kind, cmd in SCENARIO:
|
||||
emit(f"\x1b[1;36m{labels[kind]}\x1b[0m:\x1b[1;37m {cmd}\x1b[0m\r\n",
|
||||
0.35)
|
||||
try:
|
||||
out = runners[kind](cmd)
|
||||
except Exception as e:
|
||||
out = f"[runner error: {e}]\n"
|
||||
# strip ANSI-safe passthrough; small human-ish delay
|
||||
delay = min(1.2, 0.25 + len(out) / 4000.0)
|
||||
emit(out.replace("\n", "\r\n") or "(no output)\r\n", delay)
|
||||
time.sleep(0.1)
|
||||
emit("\x1b[1;36mpftest-pf0$\x1b[0m /opt/pf/pfctl -d\r\n", 0.4)
|
||||
runners["pf0"]("/opt/pf/pfctl -d")
|
||||
for i, ev in enumerate(casts):
|
||||
f.write(json.dumps(ev) + "\n")
|
||||
print(f"WROTE {out_path} ({len(casts)} events, {t:.1f}s)")
|
||||
print("play: asciinema play " + out_path +
|
||||
" (or: script-replay/pf-in-play fallback: cat with 'less -R')")
|
||||
except subprocess.TimeoutExpired:
|
||||
out = "[timeout]\r\n"
|
||||
except Exception as e: # noqa: BLE001
|
||||
out = f"[error: {e}]\r\n"
|
||||
out = out.replace("\r\n", "\n").replace("\n", "\r\n")
|
||||
emit(out or "(no output)\r\n", min(1.4, 0.3 + len(out) / 5000.0))
|
||||
with open(out_path, "a") as f:
|
||||
for ev in events:
|
||||
f.write(json.dumps(ev) + "\n")
|
||||
print(f"WROTE {out_path}: {len(events)} events, {t:.1f}s")
|
||||
print("play: asciinema play " + out_path)
|
||||
print("upload: asciinema upload " + out_path)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--out", default=OUT_DEFAULT)
|
||||
ap.add_argument("--out", default=os.path.join(REPO, "out",
|
||||
"demo-pf-linux.cast"))
|
||||
ap.add_argument("--quick", action="store_true")
|
||||
a = ap.parse_args()
|
||||
record(a.out)
|
||||
record(a.out, a.quick)
|
||||
Reference in new issue
Block a user