Caveats — divergences from OpenBSD behavior

Português: docs/pt-BR/caveats.md.

These are the ways this port differs from running the same ruleset on OpenBSD 8.0, or is not yet production-shaped. Everything here is grounded in the code (module/pfsrc/pfk_hooks.c, module/pfsrc/pfk_mod.c, docs/porting-notes.md) and in lab observations; nothing is speculative.

Data path

1. PF runs in process context via a workqueue (netisr model), not in the packet fast path

OpenBSD evaluates pf_test() on the CPU that received the packet, at IPL_NET, with a non-sleeping NET read lock. Linux gives us softirq context and sleepable compat locks (rw-semaphores), so the hook never calls pf_test() inline: it steals the skb (NF_STOLEN) onto an ordered workqueue (pfk_netisr, WQ_MEM_RECLAIM) and runs the engine in process context, then re-injects (netif_rx() for input, dst->output() for output/transit).

Consequences, all observable:

Verdicts are asynchronous. Anything chained after our hook on the same netfilter chain (other iptables/nft rules at higher priorities) does not see the packet until the reinjected pass comes through. Counters and log interleaving differ accordingly.
Latency + throughput: every packet takes a work-queue round trip serialized on one ordered worker. Functional gates are the current bar; the planned bench-basic.sh is informational only.
Reordering: the ordered queue preserves arrival order of queued work items, but the stack around it (RPS, multi-NAPI) can still hand us packets whose original completion order differs from reinjection order.

2. Re-entry protection is a per-direction tag in `skb->mark`

The reinjected survivor carries skb->mark |= PFK_DONE_IN (0x50460010) or PFK_DONE_OUT (0x50460020); our hooks accept any packet already tagged for that direction instead of re-queuing it (otherwise PF_IN → netif_rx → PF_IN… would loop). skb->mark is a shared, writable 32-bit field: any subsystem that overwrites it — nft meta mark set, CONNMARK --restore, tc filters, some VPN stacks — can either (a) clear our tag and cause the packet to be re-queued through PF again, or (b) coincidentally set those bits and let a packet skip a PF pass. Practical rule on a PF-on-Linux host: do not use fwmark mangling on the same traffic; if you must, mask the PF tag bits out of your mark space.

3. No NET-lock on the data path (M2 interim)

pf_test() currently runs without the compat NET read lock (comment in pfk_hooks.c): with the workqueue model, a parked control writer (ruleset change, purge ioctl) starved the packet side badly enough to be counter-productive. Rule-set mutation and packet walk therefore run concurrently, protected by what upstream pf.c itself uses (PF_LOCK/RCU-ish patterns) — a deliberate interim deviation being reworked with the lock model in M5+. Until then: expect correctness under steady state, not under aggressive concurrent reload; the reload-under-traffic soak is an open gate.

4. Fail-open under memory pressure

The queue item is kmalloc(GFP_ATOMIC) in softirq. If it fails, the hook returns NF_ACCEPT — the packet bypasses PF. This is a deliberate availability choice (mirrors dropping into a full queue bypassing the qdisc) but it is a security trade-off worth knowing: PF verdicts are best-effort exactly when the host is out of atomic memory.

5. Hook ordering vs conntrack differs by direction

Ingress (PREROUTING, NF_IP_PRI_FIRST): PF runs before defrag and conntrack — as designed (spec §4.2).
Output (LOCAL_OUT at NF_IP_PRI_CONNTRACK+1): PF sees locally generated traffic after conntrack has already committed the entry.
Transit egress (POSTROUTING at NF_IP_PRI_LAST-1): PF runs at the very end — after any iptables/nft SNAT/masquerade — so pass out … nat-to-era rules (M3) will be validated against this ordering. The "unregister-safe ordering vs conntrack" warning from the design still applies: overlapping policy between PF and conntrack is possible and yours to manage.

Control plane / lifecycle

6. Unload is hardened; the soak is still pending

rmmod pf deregisters /dev/pf, zeroes pf_status.running under the NET lock, deletes and syncs both purge timers (they self-rearm — a real rmmod hang was observed and fixed by stopping sources before flushing the workqueue), unregisters hooks and flushes pfk_netisr. That fix is in-tree, but the full cycle under traffic + live states (T18/soak gate) has not been run to green yet. For now treat unload as admin-safe, automation- unproven.

Behavioral / rendering issues open today

7. `pfctl -s states` visibility anomaly (M2 gate work)

In the most recent captured run (gate 01): transit ping passed under keep state, but no icmp state was listed afterwards. Fixing this is the open item of the M2 gate work; until a green run lands, the state-listing row is the honest status. Not cosmetic if it turns out states are not being inserted — treat both possibilities until the gate says otherwise.

8. Silent-but-warned stubs

divert verdicts drop with a rate-limited warning; log rules log nothing with a one-time warning. Both are listed so nobody mistakes "rules loaded" for "features active" (see pf-guide.md).

Compat-layer deviations (from `docs/porting-notes.md`, M1 record)

These are deliberate, marked-in-code divergences of the compat layer; they matter when reasoning about rule semantics:

Socket-owner matching (user/pid on rules): uid only — Linux struct sock carries no egid/cpid, so gid degrades to (gid_t)-1, pid to NO_PID.
Checksums: driver offload is not trusted on ingress (M1 decision) — TCP checksums are recomputed in software; output pseudo-checksums are done in the port's own path.
IP options / record-route / timestamps: not processed on the PF output path; ICMP redirects are not generated by PF's route code (the Linux FIB emits those).
Routing labels: non-goal — the ioctl layer rejects labeled rules; rtable_iterate() always returns NULL (no multipath uRPF iteration).
Interface groups: Linux has none; group … matches behave like a missing group on OpenBSD.
GSO/multi-segment skbs: handled by linearization on entry to the mbuf shim (perf cost documented in spec §10; gates will quantify).
Single instance: one PF world in init_net, like OpenBSD has one kernel. Containers/namespaces are not visible to PF and cannot be filtered with it in this design.

What is not a caveat

ARP and other non-IPv4/IPv6 EtherTypes never enter the netfilter IPv4/IPv6 hooks — they pass through the box untouched, exactly as PF on OpenBSD ignores non-IP. /dev/pf is 0600 root-only by design (matches OpenBSD's securelevel-era default for pfctl write access being root).

6.9 KiB Raw Blame History