pf-linux-port/docs/en/caveats.md

Caveats — divergences from OpenBSD behavior

Português: docs/pt-BR/caveats.md.

These are the ways this port differs from running the same ruleset on OpenBSD 8.0, or is not yet production-shaped. Everything here is grounded in the code (module/pfsrc/pfk_hooks.c, module/pfsrc/pfk_mod.c, docs/porting-notes.md) and in lab observations; nothing is speculative.

Data path

1. PF runs in process context via a workqueue (netisr model), not in the packet fast path

OpenBSD evaluates pf_test() on the CPU that received the packet, at IPL_NET, with a non-sleeping NET read lock. Linux gives us softirq context and sleepable compat locks (rw-semaphores), so the hook never calls pf_test() inline: it steals the skb (NF_STOLEN) onto an ordered workqueue (pfk_netisr, WQ_MEM_RECLAIM) and runs the engine in process context, then re-injects (netif_rx() for input, dst->output() for output/transit).

Consequences, all observable:

• Verdicts are asynchronous. Anything chained after our hook on the same netfilter chain (other iptables/nft rules at higher priorities) does not see the packet until the reinjected pass comes through. Counters and log interleaving differ accordingly.
• Latency + throughput: every packet takes a work-queue round trip serialized on one ordered worker. Functional gates are the current bar; the planned bench-basic.sh is informational only.
• Reordering: the ordered queue preserves arrival order of queued work items, but the stack around it (RPS, multi-NAPI) can still hand us packets whose original completion order differs from reinjection order.

2. Re-entry protection is a per-direction tag in skb->mark

The reinjected survivor carries skb->mark |= PFK_DONE_IN (0x50460010) or PFK_DONE_OUT (0x50460020); our hooks accept any packet already tagged for that direction instead of re-queuing it (otherwise PF_IN → netif_rx → PF_IN… would loop). skb->mark is a shared, writable 32-bit field: any subsystem that overwrites it — nft meta mark set, CONNMARK --restore, tc filters, some VPN stacks — can either (a) clear our tag and cause the packet to be re-queued through PF again, or (b) coincidentally set those bits and let a packet skip a PF pass. Practical rule on a PF-on-Linux host: do not use fwmark mangling on the same traffic; if you must, mask the PF tag bits out of your mark space.

3. No NET-lock on the data path (M2 interim)

pf_test() currently runs without the compat NET read lock (comment in pfk_hooks.c): with the workqueue model, a parked control writer (ruleset change, purge ioctl) starved the packet side badly enough to be counter-productive. Rule-set mutation and packet walk therefore run concurrently, protected by what upstream pf.c itself uses (PF_LOCK/RCU-ish patterns) — a deliberate interim deviation being reworked with the lock model in M5+. Until then: expect correctness under steady state, not under aggressive concurrent reload; the reload-under-traffic soak is an open gate.

4. Fail-open under memory pressure

The queue item is kmalloc(GFP_ATOMIC) in softirq. If it fails, the hook returns NF_ACCEPT — the packet bypasses PF. This is a deliberate availability choice (mirrors dropping into a full queue bypassing the qdisc) but it is a security trade-off worth knowing: PF verdicts are best-effort exactly when the host is out of atomic memory.

5. Hook ordering vs conntrack differs by direction

• Ingress (PREROUTING, NF_IP_PRI_FIRST): PF runs before defrag and conntrack — as designed (spec §4.2).
• Output (LOCAL_OUT at NF_IP_PRI_CONNTRACK+1): PF sees locally generated traffic after conntrack has already committed the entry.
• Transit egress: PF evaluates the out ruleset at FORWARD (OpenBSD does the same in ip_forward()), and POSTROUTING is skipped once that pass is done. nat-to/rdr-to were validated against this ordering (gates 05/06).
• The "overlapping policy with conntrack" warning from the design still applies: if iptables/nft SNAT also runs, PF's translation happens at the layer it is hooked into, and double translation is yours to avoid.

Control plane / lifecycle

6. Unload is hardened and soak-verified

rmmod pf deregisters /dev/pf, zeroes pf_status.running under the NET lock, deletes and syncs both purge timers (they self-rearm — a real rmmod hang was observed and fixed by stopping sources before flushing the workqueue), unregisters hooks and flushes pfk_netisr. Gate 14-soak.sh drives ICMP+TCP traffic, checks dmesg for faults, then runs rmmod → insmod → rmmod and compares the slab footprint before and after: clean unload, successful reload, no growth. Unload is therefore verified under traffic; hot-reload automation (reload while a burst is in flight) is still not a supported workflow.

Behavioral notes verified today

7. pfctl -s states lists real states, including translations

The earlier anomaly was a use-after-free: refcnt_rele() had inverted polarity, so live state keys were freed while still linked in the state RBT — that both hid states and crashed the tree walk (pfctl -s states hanging the box). After the fix, gates 01/04/05/06/07 assert on the listed states, and translated flows print both tuples (10.0.2.1:59335 (10.0.1.10:4380) -> 10.0.2.20:8).

8. Consumers still required for two features

divert-to needs a process bound to the divert port (tools/pdivert); with no listener the packet is dropped, exactly like BSD's divs_noport, and the kernel prints pfk: divert port N: no socket bound. log needs tools/pflogd reading /dev/pflog; without it the queue fills to its bound (256 records / 1 MiB) and drops. Neither is a stub any more, but neither does anything until its consumer runs (see pf-guide.md).

9. Queues: path verified; shaping semantics are HFSC's, not a bug

15-queue.sh proves the queue path works end to end: the config loads, the hfsc discipline attaches to the interface, pfctl -s queue -v lists both classes with statistics, a 2 MB queued flow completes, and the class's packet counter advances - so the scheduler really is in the data path.

What the lab cannot show is a rate cap, and that is HFSC semantics rather than a port defect: OpenBSD 8.0's pf.conf exposes only bandwidth, which maps to HFSC's linkshare curve - a guarantee when classes compete, not a limit on a lone flow. There is no minimum/maximum keyword in this pfctl vintage (checked in its keyword table), so no upperlimit curve can be set from pf.conf. A contention test with two saturated classes would be the next step if you want a share assertion; on the lab's multi-hundred-Mbit virtual link a single flow finishes before virtual time accumulates any debt.

Bugs found and fixed while getting here (details in docs/porting-notes.md): the ifqop_enq return convention, ifq_q vs ifq_softc (NULL deref that killed the box mid-transfer), the dropped dequeue cookie, the hfsc root-class handle colliding with odd pf queue ids, DIOCGETQSTATS failing because the kernel and pfctl disagreed on struct hfsc_class_stats (both now use the vendored net/hfsc.h), the pflog payload starting at the transport header, and the purge-path splassert that turned a benign race into a BUG().

Compat-layer deviations (from docs/porting-notes.md, M1 record)

These are deliberate, marked-in-code divergences of the compat layer; they matter when reasoning about rule semantics:

• Socket-owner matching (user/pid on rules): uid only — Linux struct sock carries no egid/cpid, so gid degrades to (gid_t)-1, pid to NO_PID.
• Checksums: driver offload is not trusted on ingress (M1 decision) — TCP checksums are recomputed in software; output pseudo-checksums are done in the port's own path.
• IP options / record-route / timestamps: not processed on the PF output path; ICMP redirects are not generated by PF's route code (the Linux FIB emits those).
• Routing labels: non-goal — the ioctl layer rejects labeled rules; rtable_iterate() always returns NULL (no multipath uRPF iteration).
• Interface groups: Linux has none; group … matches behave like a missing group on OpenBSD.
• GSO/multi-segment skbs: handled by linearization on entry to the mbuf shim (perf cost documented in spec §10; gates will quantify).
• Single instance: one PF world in init_net, like OpenBSD has one kernel. Containers/namespaces are not visible to PF and cannot be filtered with it in this design.

Screencast pipeline (2026-11-12)

10. MiniMax video generation is not enabled on this account

Every MiniMax video model (T2V-01, T2V-01-Director, MiniMax-Hailuo-02) answers with API error 2068 — "The current Token Plan tier does not support this model". The narration (TTS) and the title artwork work fine; only the video models are blocked. The phase transitions are therefore rendered locally with ffmpeg (a crossfade over a chapter card) instead of AI-generated clips. Upgrading the plan is the only thing needed to change that — the extension point is scripts/screencast/join.py, which builds the cards.

11. The narration is a separate track, not a dubbing of the video

Each phase is recorded once and serves both languages: the terminal session is language-neutral, and the two narrated versions differ only in the audio track and the burned-in subtitles. The recorder holds each beat on screen for as long as the longest narration segment (EN or PT) needs, so the voice and the terminal action stay in step in both.

12. What the videos do not show

The videos are a demonstration, not a test run: the lab's own gate suite (scripts/vm-run-tests.sh) is the authority for PASS/FAIL, and its logs live in out/test-logs/. The screencasts show the same commands, but a video frame is not a verdict.

Capacity findings (2026-11-12, from scripts/stress/run-stress.sh)

13. The workqueue backlog grows with the offered rate: above ~1 Gbit/s the guest OOM-panics

This is the sharpest consequence of §1 and it is reproducible, not theoretical. PF steals every skb onto one ordered workqueue and runs the engine in process context, so the number of packets in flight is bounded only by the memory the backlog takes.

Measured on the lab (Alpine guest, 1 GB RAM, 4 TCP streams through a stateful ruleset):

Offered rate (per stream × 4)	Result
50 Mbit/s (200 Mbit/s total)	stable
100 Mbit/s (400 Mbit/s total)	stable
250 Mbit/s (1 Gbit/s total)	kernel panic: "System is deadlocked on memory"

The panic is an OOM with no killable process; the slab at that moment shows kmalloc-8k ≈ 50 MB, skbuff_head_cache ≈ 18 MB and files_cache ≈ 40 MB, i.e. the backlog is what ate the 1 GB. The guest reboots (120 s watchdog delay) and every connection through the firewall is lost with it.

Practical consequences:

• Rate-limit what you point at PF, and size the box for the backlog: memory needed grows with the offered rate, not with the number of concurrent flows.
• The stress test (tests/vm/16-stress.sh) therefore runs its traffic phases at a bounded rate (default 100 Mbit/s per stream = 400 Mbit/s total) — the regime this port survives — and reports the ceiling separately instead of OOM-killing the firewall under test.
• The functional gates (01-15) are unaffected: they move megabytes, not gigabits per second.

What would fix it is a real fast path (evaluate pf_test() inline in softirq, as OpenBSD does) or a bounded queue with a drop policy — both are design changes beyond this port's scope.

What is not a caveat

ARP and other non-IPv4/IPv6 EtherTypes never enter the netfilter IPv4/IPv6 hooks — they pass through the box untouched, exactly as PF on OpenBSD ignores non-IP. /dev/pf is 0600 root-only by design (matches OpenBSD's securelevel-era default for pfctl write access being root).