pf-linux-port/docs/superpowers/specs/2026-11-11-pf-linux-port-design.md

# PF on Linux — Port Design / Design do Port

Date/2026-11-11 · Status: awaiting approval/aguardando aprovação · Repo: https://git.labexposed.com/lgcosta/pf-linux-port (public)

## 1. Intent / Intenção

Run OpenBSD's PF packet filter inside Linux 7.3.4 — the real PF engine, driven by the
real `pfctl` speaking the real `/dev/pf` ioctl ABI — as an out-of-tree module with a
documented, minimal set of kernel core patches, validated end-to-end in a libvirt
Alpine VM. Docs bilingual (EN + pt-BR).

Rodar o PF do OpenBSD no Linux 7.3.4 — engine PF real, controlado pelo `pfctl` real
falando a ABI ioctl `/dev/pf` — como módulo out-of-tree com um conjunto mínimo e
documentado de patches no core, validado numa VM Alpine via libvirt.

## 2. Facts found / Fatos levantados

- OpenBSD **replaced classic ALTQ by ≥7.7** (verified: no `sys/net/altq` dir in
  7.7/7.8/8.0) — **but PF queueing lives on in 8.0**, rearchitected into the core
  `ifq` framework: `sys/net/ifq.c` (priq ops), `sys/net/hfsc.c`,
  `sys/net/fq_codel.c`, ioctls `DIOCADDQUEUE/DIOCGETQUEUES/DIOCGETQUEUE/
  DIOCGETQSTATS` (`pfvar.h:1707+`), `pf_rule.queue` glue all present; `pfctl`
  still parses/installs `queue on … hfsc|fq_codel|default`. Older ALTQ releases
  (≤7.6) are no longer mirrored anywhere (cdn/ftp/mirrors/GitHub-raw/Wayback all
  404), making a 7.x base infeasible/undesirable anyway.
  ⇒ **port base = OpenBSD 8.0** (exactly matches the user's provided userland
  tree, pfctl.c v1.406 2026-08-30; sources fetched from the user's C3SL mirror).
  The ALTQ requirement is met as **OpenBSD-8.0 queueing (priq + HFSC + fq_codel)**
  — the framework modern OpenBSD itself ships; classic cbq/wfq/red are
  non-goals *because they don't exist in the base*, honestly matrixed.
- Kernel side ≈ 24.6k LOC (`pf.c` 9.2k, `pf_ioctl.c` 4.6k, `pf_norm/if/lb/table/
  ruleset/osfp/syncookies`, `pfvar(_priv).h`) + queueing core (`ifq/hfsc/fq_codel`
  ≈ 2.4k LOC) + `ip_divert.c` reference (~300 LOC).
- Userland: single `pfctl` (~20k LOC incl. `parse.y`) — **libpfctl was folded back
  into pfctl** in this era (Makefile `LDADD+= -lm` only; no libpfctl in tree).
- Linux 7.3.4 tree clean; toolchain gcc 16/flex/bison/bc; libvirt 12 + KVM usable
  (existing `sms-vm` untouched); sudo works; Alpine cloud images available on dl-cdn.
- Module-visible APIs confirmed exported: `neigh_lookup`, `pskb_expand_head`,
  `skb_clone`, `register_qdisc`, `qdisc_tree_reduce_backlog`, `tcf_classify`,
  `__dev_queue_xmit`, `netif_rx`, `inet_addr_type`, `fib_table_lookup` (GPL).
- Confirmed **not exported** in 7.3.4: `neigh_output`, `dev_queue_xmit` (inline
  wrapper over `__dev_queue_xmit`), `ip_route_output_key`, `ip6_route_output`,
  `netlink_kernel_create` ⇒ core patch **P1 (mandatory): export symbols required
  by the PF module** (additive `EXPORT_SYMBOL_*`, zero behavior change) — allowed
  by user, fully documented in `patches/kernel/` + `docs/*/patches.md`.

## 3. Goals / Non-goals

**Goals (G)**
1. PF engine in-tree-equivalent quality, delivered as module `pf.ko` built from
   `pf_port/` sources: rules (anchors, quick, match/block/pass/drop, return variants,
   keep state), states + syncookies, tables (pfr: radix/hash, `pfctl -t`),
   NAT (nat-to, rdr-to incl. sticky-port/bitmap basics, binat-to, route-to),
   **scrub/normalize** (fragment reassembly, DF/TTL actions, min-ttl, max-mss),
   **PF queues/ALTQ semantics of modern OpenBSD 8.0** (`queue`, `queue on` with
   `default`(priq), `hfsc`, `fq_codel` schedulers from `sys/net/ifq.c`,
   `hfsc.c`, `fq_codel.c`; `pfctl -s queue -v` stats), **divert** (`divert-to`,
   `divert` tag; userspace channel modeled on `sys/netinet/ip_divert.c`),
   **route-to** (pools + rr/rr-source/rr-any, hash-source/state;
   `reply-to`, `dup-to`, `fail-to`).
2. Control plane = OpenBSD ioctl ABI on `/dev/pf`; Linux-built `pfctl` (8.0,
   single binary, no libpfctl) + `pflogd`-lite + `pdivert` tool.
3. Patchset + scripts in the repo; kernel core patches only when unavoidable, each
   documented (what/why/where/impact).
4. VM validation: kernel 7.3.4 built with patches, module + tools deployed into an
   Alpine VM booted with that kernel; a scripted suite exercises every goal-1 area
   with real traffic across a 4-VM libvirt lab (§7) and checks PASS/FAIL.
5. Full docs EN + pt-BR: apply patches, build, install, feature guide, VM validation
   how-to, feature matrix incl. exclusions.

**Non-goals (NG, explicit to user)**: pfsync/carp (no failover), full IPsec/pfkey,
classic-ALTQ-only schedulers (cbq/wfq/red — **do not exist in the OpenBSD 8.0 base**,
matrixed as "removed upstream"), queues on interfaces whose root qdisc is not ours,
per-netns PF instances (faithful port = single global instance like OpenBSD; netns
isolation documented as out of scope), OS-fingerprinting beyond `pfctl -o` (osfp is
bonus if cheap), `ftp-proxy` and other userland daemons.

**Anti-goals**: silently faking a feature (stub verdicts), nftables-as-substitute
claims, tests that only check exit codes without inspecting packets/state.

## 4. Approach (chosen) / Abordagem

**B′ — faithful port with a compat shim** (alternative C rejected: user wants PF in
kernel; A rejected: infeasible). Port the actual OpenBSD 8.0 `pf*.c` + `ifq/hfsc/
fq_codel` + `ip_divert`-referenced divert under a compatibility layer, hook into
Linux via netfilter, keep upstream ABI for userland.
(Rejected variant: rewrite of PF semantics on nf APIs = not a port.)

### 4.1 Compat layer (`module/compat/`)
- **Packet object**: `struct mbuf` shim over `sk_buff` (single-segment: linearize on
  entry to PF; `m_copym/m_adj/m_pull/m_copydata/m_cat/m_clump/m_free` → skb ops;
  `m_pkthdr` ↔ `skb->protocol/tc/index`, address family, ifindex via
  `skb->dev`). This mirrors how OpenBSD's own pf code treats mbufs and keeps diffs
  localized to packet I/O, not to matching logic.
- **APIs**: `kern_*` glue: M_* malloc→`kmalloc/kfree` (pf's malloc flags → GFP),
  `timeout()`→timer/workqueue (state purger), `rwlock_*`→RW-lock,
  `pool`→kmem_cache, epoch→RCU read side + refcounts on rulesets (matching how
  conntrack already models this), `time_second`→`ktime_get_seconds`,
  `syncache` untouched, `radix.h`/`tree.h` (BSD) vendored with Linux tweaks,
  `imsg` not needed in kernel.
- **Netlink/ioctl**: `misc_register` `/dev/pf`, all DIOC* handled exactly as
  `pf_ioctl.c` (port file with shim includes). **ioctl number encoding**: define
  `_IOR/_IOW/_IOWR(PFRDR,'n',type)` OpenBSD-style in our shared `pfvar.h` so kernel
  and Linux-`pfctl` agree bit-for-bit.
- **IPv4/IPv6**: `in_proto`/`in6_proto` glue for checksums (`ip_sum_eq`,
  pseudo-header sum via `csum_tcpudp_magic`, `tcp_input.c`'s in_cksum ported to
  `pf_compat_cksum.c`); raw socket `sosend` paths replaced by
  `icmp_send`/`icmp6_send` (RST via `tcp_v4_send_reset` equivalents kept in-pf to
  preserve PF's own return logic).
- **Interface/addresses**: `ifnet`→`net_device` + `in_ifaddr`→`net_ifa` walks via
  `net_if_FOREACH`/`in_ifa_lookup` helpers over `if_addr_list` + `fib` (replaces
  `ifa_ifwithroute` for route-to); `pf_getifaddr` ported with Linux address lists.

### 4.2 Hooking
Register netfilter hooks per family:
- `PREROUTING (NF_IP_PRI_FIRST+)`: pf input path = scrub/normalize → state lookup →
  nat/rdr (pf's own NAT, **independent of conntrack**; deployment docs warn about
  overlap zones), filter INPUT.
- `FORWARD`: filter + route-to/dup-to execution.
- `LOCAL_OUT`: filter (OUTPUT) + nat-to (locally generated).
- `POSTROUTING (pri low)`: state insert, queue classify (skb→pfq class id),
  route-to final xmit.
Verdict mapping: pass→`NF_ACCEPT`, block/normalization-drop→`NF_DROP` (with PF's
return-RST/ICMP generated by pf itself), `divert`→`NF_STOLEN` + divert channel,
`nat/rdr` rewrite → `NF_ACCEPT` after `skb_cow` + checksum fix. Coexistence with
iptables/nft: PF hooks run before conntrack (unregister-safe ordering); documented.

### 4.3 Queues (OpenBSD 8.0 `ifq`) ⇒ `pfq` qdisc shim
Port `sys/net/ifq.c/ifq.h` + `sys/net/hfsc.c` + `sys/net/fq_codel.c` (the real 8.0
queueing framework: priq glue, HFSC, FQ-CoDel as `ifq_ops`) into `module/ifq/`.
`struct ifqueue` instances are backed by a classful Linux qdisc **`pfq`**
(`register_qdisc`): `pfq_enqueue` → `ifq_enqueue` (matched class via pf tags),
`pfq_dequeue` → `ifq_deq_begin/commit` → `__dev_queue_xmit`. `tc qdisc replace dev X
root pfq` attaches; `scripts/pf-qdisc.sh` auto-attaches when a loaded ruleset
declares `queue on` (pfq handles both schedulers + default priq via `pf_qblocks`
priority bits). `DIOCADDQUEUE/DIOCGETQUEUES/DIOCGETQUEUE/DIOCGETQSTATS` (the real
8.0 names) served against the pfq/ifq registry so `pfctl -s queue -v` and
`pfctl -s ifq -v` work. Tested mandatory schedulers: **default(priq), hfsc,
fq_codel** with rate gates (§7); class visibility via `tc -s qdisc show` is a
stretch goal, counters via `pfctl` are the gate.

### 4.4 divert — real AF_DIVERT in the kernel (no netlink substitute)
OpenBSD's `divert(4)` socket family is ported as a **runtime-registered
socket family inside pf.ko**: Linux 7.3 registers protocol families with
`sock_register()` and `NPROTO = AF_MAX = 45`, so BSD's `PF_DIVERT = 32`
(the unused legacy PF_IUCV slot) fits with **zero core changes**. Implemented
in `module/pfsrc/divert_sock.c`: `socket(PF_DIVERT, SOCK_RAW, IPPROTO_DIVERT)`
+ `bind(port)` + `recvmsg/sendmsg`, packet metadata via cmsg (ifindex, dir,
divert type, original dst). `divert_packet()/divert6_packet()` (called by the
ported pf.c) queue to the bound socket; `sendmsg` re-injects with OpenBSD
semantics: `PF_DIVERT_TO` → local delivery (`netif_rx` on the recorded
interface), plain `divert` tag → re-route (`dst->output`), mirroring
`ip_divert.c:divert_output()`. The BSD `divert(8)` workflow (upstream
`regress/sys/net/pf_divert`) drives this socket unchanged; our shipped
`pdivert` is a thin Linux port. Wire format documented in
`docs/en/divert-wireformat.md`. pflog keeps its netlink multicast channel
(slot PFLOG_PROTO=1 on `NETLINK_USERSOCK`).

### 4.5 route-to / dup-to / fail-to
`pf_route6`/`pf_route` port: pool selection via `pf_lb.c` (as-is), next-hop
resolution `ifa_ifwithroute`→`pf_route4gw()`: build `flowi4/flowi6`,
`ip_route_output_key` to **gw**, then L2-redirect via `neigh_lookup(arp_tbl/nd_tbl,
&gw, dev)` + `neigh_output` (or `dev_queue_xmit` after `dev_hard_header`); TTL--
before xmit, state key = translated, `reply-to` via reverse state's stored answer
route. `dup-to` = `skb_clone` to original path. If a needed route/neigh helper
isn't exported → kernel core export patch, documented.

### 4.6 pflog (small, included)
`if_pflog.c` → netlink multicast channel (`NETLINK_PF` group) + `pflogd` userland
that writes pcap file (OpenBSD pflogd logic), so `tcpdump -r` verifies passes;
enables state/queue tests to assert headers (e.g., NAT rewrites, scrub).

### 4.7 Userland build (`tools/`)
Linux build of OpenBSD 8.0 `sbin/pfctl` (single binary — libpfctl is folded in),
`parse.y` via flex/bison, plus `pdivert` + `pflogd`, with a `compat-user/` shim
(err/warn, strlcpy, pledge stub, arc4random, sysctl subset, OpenBSD-shaped
`_IOR/_IOW/_IOWR`). Shared ABI header `include/net/pfvar.h` (ours, identical to
the kernel's) guarantees ioctl bit-parity; static asserts on every ioctl struct.

## 5. Kernel core patches
Target **zero behavior changes** to Linux; the patchset ships:
- **P1 (mandatory)** `net: export symbols needed by the PF module`
  (`neigh_output`, `ip_route_output_key`, `ip6_route_output`,
  `netlink_kernel_create`, + any added during M1–M5 builds) — additive
  `EXPORT_SYMBOL*()` lines, no logic touched. Confirmed missing in 7.3.4.
- No other core changes by default: module builds out-of-tree (`make -C
  linux-7.3.4 M=pf_port/module`), tree otherwise untouched; if a future step
  needs more (e.g., a netlink protocol reservation), each lands as its own
  `patches/kernel/000N-*.patch` appended by `scripts/apply-patches.sh`, with a
  per-patch entry in `docs/{en,pt-BR}/patches.md`: what/why/impact/how to revert.

## 6. Repo layout
```
pf_port/            (= repo root, git → pf-linux-port)
  README.md / README.pt-BR.md
  docs/{en,pt-BR}/  guide, install, apply-patches, feature-matrix, validation,
                    divert-wireformat, patches/*.md
  docs/superpowers/specs/            this file
  patches/kernel/   000N-*.patch + series + README
  module/           pf.ko + pfq.ko sources: pf/, ifq/, compat/, include/
  tools/            pfctl/, pdivert/, pflogd/, compat-user/
  scripts/          fetch-sources.sh  apply-patches.sh  unpatch.sh
                    build-module.sh   build-tools.sh    vm-create.sh
                    vm-deploy.sh      vm-run-tests.sh   bench-basic.sh
  tests/vm/         01-state.sh … 14-altq.sh, scapy/python helpers, expect-files
  _vendor/          OpenBSD 7.8/8.0 tarballs (gitignored; refetch script)
```

## 7. Validation (libvirt on the 192.168.1.242 host, Alpine) — "done" definition
`scripts/vm-create.sh` clones the Alpine 3.22 cloud qcow2 (verified: 840 MiB,
sha256 e83786b1…), NoCloud seed (user/network config), and defines **4 libvirt
domains on an isolated 192.168.122.x-free bridge lab (`pf-lan`, `pf-wan`, `pf-wan2`
networks, no NAT to the outside needed for gates)**:
`pftest-pf0` (**boots the self-built 7.3.4 kernel + initrd via direct kernel boot**;
3 NICs: lan/wan/wan2), `pftest-cli`, `pftest-srv`, `pftest-gw2`.
Decision (builder-check #1): PF stays a **single global instance bound to
`init_net`**, exactly like OpenBSD — no per-netns rulesets; hence endpoints must be
real hosts (VMs), not namespaces. The `deb` host (12 cores, libvirt 11.3 validated)
runs the lab; the laptop keeps building sources.
Suite (`vm-run-tests.sh`, ssh-driven, PASS/FAIL per feature): stateful icmp/tcp/udp
(+IPv6 smoke), block drop/return, tables (`pfctl -t -T add/del/test`),
anchors+quick, nat-to (asserted on srv: src=pf0's addr), rdr-to (srv httpd reached
via pf0 DNAT), binat-to, scrub (crafted frags via scapy on cli: overlap drop,
DF-set, max-mss clamp), **route-to pools rr + hash-source (asserted: wan vs wan2
tcpdump counts; fail-to when gw2 down; reply-to symmetry; dup-to capture)**,
**divert (`pdivert` rewrites payload end-to-end; timeout drop without server)**,
**queues: `queue on … { hfsc … }` and `fq_codel` and default priq — iperf
throughput within ±15% of rate on the bottleneck link + `pfctl -s queue -v`
counter growth + pfq attaches/detaches cleanly**, pflog capture sanity (passes
appear in pcap), ruleset reload under traffic (soak loop), rmmod/load leak check
(kmemleak + module refcounts). IPv6 policy (builder-check #4): engine is v6-complete
by construction (single pf.c); **gates are v4 for everything + v6 smoke for
state/block/nat66/scrub**; v6-only landmines (route-to6 pool failover) matrixed
"ported, smoke-validated". Performance (builder-check #2): **functional gates are
the bar; `bench-basic.sh` is informational** (reports pps vs iptables baseline, no
numeric gate) — a perf contract would force a non-faithful rewrite-first shim.
Artifacts: run log + per-test pcap attached to the Gitea release. **All gates must
pass**; anything failing ships as explicit FAIL in the feature matrix, not silently.

## 8. Milestones & delivery
M0 infra: repo+CI-ish scripts, kernel baseline build OK, VM boots custom kernel.
M1 engine-boot: compat layer + module loads, `/dev/pf`, pfctl `-f` empty + `-s info`.
M2 filter core: rules/states/tables/anchors pass VM tests 1–4.
M3 NAT suite (nat/rdr/binat) + scrub + pflog.
M4 route-to/dup-to/fail-to + divert.
M5 queues: ifq/hfsc/fq_codel port + pfq qdisc + `queue` semantics + counters.
M6 docs bilingual complete, patchset tidy, full suite re-run green, tags v0.x in repo.
Commits per milestone to Gitea (public), final release with validation artifacts.

## 9. Builder-decisions (my calls, reversible)
- Base **OpenBSD 8.0** (matches the provided userland tree exactly; ALTQ delivered
  as 8.0's real queue framework ifq{priq,hfsc,fq_codel}; older ALTQ releases are
  unrecoverable from public mirrors anyway).
- **Single global PF instance on `init_net`** (faithful to OpenBSD); validation via
  multi-VM lab, not netns (builder-check #1).
- sk_buff-backed mbuf shim instead of rewriting pf.c to skb (keeps upstream logic
  line-proximate; enables future rebase to newer OpenBSD).
- divert via netlink channel instead of inventing AF_DIVERT (no core socket-family
  patch).
- pflog via netlink not pseudo-if tap (keeps "no extra interface" promise to users).
- Module name `pf.ko` (no Linux collision).
- **No numeric perf gate** (bench informational) and **v4-gated + v6-smoked** suite
  (builder-checks #2/#4).

## 10. Risks
- `pf.c` porting surface largest; queue scheduler coverage limited to what exists
  upstream in 8.0 (mitigation: 3 schedulers gated, matrix explicit).
- ioctl-ABI bit parity between two toolchains (mitigation: single shared header,
  struct-size static asserts both sides).
- Kernel 7.3 symbol drift (mitigation: P1 export patches allowed+documented).
- Netfilter ordering vs conntrack for transit flows (mitigation: docs + test with
  conntrack module loaded/unloaded).
- Multi-segment/GSO packets through mbuf shim (mitigation: linearize-on-entry for
  v1, document perf cost; bench script quantifies).