docs: queue status honest - attach path reached, purge lock assert documented as the blocker (caveats #9)

CHANGELOG.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## Unreleased
+
+- **M5 (first step)**: OpenBSD's `hfsc.c` and `fq_codel.c` now compile and link
+  into `pf.ko` (157 scheduler symbols reachable from `pf_ioctl`).  Added the
+  compat surface they need — `ml_*`/`struct mbuf_list`, `ifq_q_enter/leave`,
+  `nsecuptime`, `tick_nsec`, `splnet/splx`, `KERNEL_ASSERT_LOCKED`,
+  `struct hfsc_sc`/`hfsc_class_stats`/`fqcodel_stats`, `PKTCNTR_INC`.  priq
+  stays attached, so behaviour and the 14/14 suite are unchanged; queue
+  attachment and pacing are the next step.
+
 ## v0.6.0 — 2026-11-11
 
 Adds scrub, route pools and `reply-to` verification.  Suite: 14 gates, all green.

README.md

@@ -9,7 +9,7 @@ real OpenBSD `pfctl` over the `/dev/pf` ioctl ABI. (A second module for the
 **Status:** the filter core, stateful filtering, tables, anchors, NAT
 (`nat-to`/`rdr-to`/`binat-to`), forced routing (`route-to`/`dup-to`), `divert`
 and `pflog` are **verified in a real libvirt lab** (14/14 gates green).  The
-remaining features (ALTQ/pfq, af-to) are still pending — honest per-feature status:
+remaining features (queue attachment/pacing, af-to) are still pending — honest per-feature status:
 [`docs/en/feature-matrix.md`](docs/en/feature-matrix.md).
 Demo (asciinema): [`docs/en/demo.md`](docs/en/demo.md).
 Design: [`docs/superpowers/specs/2026-11-11-pf-linux-port-design.md`](docs/superpowers/specs/2026-11-11-pf-linux-port-design.md)
@@ -24,7 +24,7 @@ Design: [`docs/superpowers/specs/2026-11-11-pf-linux-port-design.md`](docs/super
 | M2 | filter core: rules/states/tables/anchors, VM gates 01–04 | **done** — gates 01/02/03/04 green |
 | M3 | NAT (`nat-to`/`rdr-to`/`binat-to`), scrub | **done** — gates 05/06/07/09 (incl. 2 MB integrity through a redirect, min-ttl/max-mss measured) |
 | M4 | route-to/dup-to/reply-to, divert e2e, pflog | **done** — divert (11), pflog (12), route-to/dup-to (08), pools (10), reply-to (13) verified |
-| M5 | queues: HFSC/FQ-CoDel back ends + `pfq` qdisc | pending |
+| M5 | queues: HFSC/FQ-CoDel back ends + queue attachment | **partial** — both back ends now compile and link into `pf.ko` (suite still 14/14); priq remains attached, no bandwidth enforced yet |
 | M6 | docs complete, patchset tidy, full suite green, tagged release | **done** — docs EN+PT-BR, 11/11 suite green, tagged (v0.5.0) |
 
 ## What you get (planned)

docs/en/caveats.md

@@ -111,6 +111,30 @@ kernel prints `pfk: divert port N: no socket bound`. `log` needs
 (256 records / 1 MiB) and drops. Neither is a stub any more, but neither does
 anything until its consumer runs (see [pf-guide.md](pf-guide.md)).
 
+### 9. Queue configuration hits a lock assertion in the purge path (open)
+
+Loading a `queue` rule now reaches the scheduler (`pfq_alloc` succeeds, the
+hfsc discipline is created), and `pf_commit_queues()` is exercised end to end.
+Two bugs were found and fixed while getting there — the root-class handle
+colliding with odd pf queue ids (`pfq_addqueue` returned EBUSY), and a NULL
+discipline not being checked.
+
+What still blocks queues: the port's purge timer can run while an ioctl holds
+the write lock, and `pf_purge_expired_states()` asserts the opposite:
+
+```
+splassert: pf_purge_expired_states: expected 0, got 2
+kernel BUG at compat/pfk_kern.c:51
+```
+
+OpenBSD serialises this with `splnet()`; the port replaced the spl layer with
+rw-semas and a work queue, so the assertion is now a real race rather than a
+formality. Until it is fixed (release the lock around the call, or give the
+state tree its own lock as upstream does for `pf_state_lock`), a host that
+configures queues can wedge the module in `Unloading`. **Do not enable
+`queue` rules on a live host yet** — the 14 lab gates never configure queues,
+which is why they stay green.
+
 ## Compat-layer deviations (from `docs/porting-notes.md`, M1 record)
 
 These are deliberate, marked-in-code divergences of the compat layer; they

docs/en/feature-matrix.md

@@ -60,7 +60,7 @@ As of 2026-11-11; source of per-row evidence: `out/test-logs/`,
 |---|---|---|---|
 | `route-to` / `dup-to` / `reply-to` | `pass in … route-to { 10.0.3.2 } keep state`, `reply-to { gw }` | M4 | **VERIFIED in lab** — gates `08-routeto.sh` (route-to leaves via the gateway's link, dup-to copies while keeping the flow) and `13-replyto.sh` (the reply of a flow is forced onto the gateway link instead of the client's). Deviation: the forced nexthop is installed by setting `rt_gw4`/`rt_gw_family` on the looked-up route (Linux only ARPs the gateway when the route declares one). `fail-to` is **not a keyword in OpenBSD 8.0** (removed upstream), so it is out of scope. |
 | `divert-to` / `divert` tag | `pass in … divert-to <host> port <n>` | VERIFIED in lab | **Works end-to-end.** The module registers an `AF_DIVERT` socket family (`PF_DIVERT=32`) at runtime — no kernel-tree patch. A `divert-to` rule hands the packet to that family; with no listener the packet is dropped (`divs_noport`, BSD parity); with `tools/pdivert` running the packet is delivered to userspace, rewritten and reinjected (verified with a payload mutation + checksum fix, ping 0% loss). |
-| Queues, default scheduler (priq) | `queue root on wan`, `pfctl -s queue -v` | M5 | COMPILES-untested — `compat/pfk_ifq.c` implements the ifq API with **priq fallback semantics**; `queue` rules parse and load, but packets are not actually shunted through a classful qdisc until M5 |
+| Queues (`queue`, `queue on`, `pfctl -s queue -v`) | `queue root q1 bandwidth 1Mbit; queue on egress all ...` | M5 | **Back ends built into `pf.ko`, attachment pending** — OpenBSD's `hfsc.c` and `fq_codel.c` now compile and link (157 scheduler symbols, `hfsc_initialize`/`hfsc_pf_addqueue`/`fqcodel_pf_addqueue` reachable from `pf_ioctl`), and the full suite still passes with them in. The data path can now hand packets to an attached scheduler (`pf_kifq_input()` + a real `ifq_start()` that drains and transmits), but queues are **not safe to enable yet**: the purge path hits a lock assertion (see caveats #9), so no bandwidth is enforced and no gate covers it. |
 | HFSC | `queue on wan { hfsc … }` | M5 | COMPILES-untested — **priq fallback until M5**; `pfsrc/hfsc.c` staged (real OpenBSD 8.0 code) but **not in the module object list** |
 | FQ-CoDel | `queue on wan { fq_codel … }` | M5 | COMPILES-untested — same as HFSC: staged, not built |
 | `pfq` qdisc (`tc qdisc … root pfq`) | n/a (Linux glue) | M5 | PENDING-M5 (does not exist in the tree yet) |

docs/en/guide.md

@@ -92,7 +92,7 @@ rebuilt.
 | M2 | filter core: rules/states/tables/anchors VM gates | **done** — gates 01–04 green |
 | M3 | NAT (`nat-to`/`rdr-to`/`binat-to`), scrub, pflog | **done** — gates 05/06/07/09 (incl. 2 MB integrity through a redirect) and pflog gate 12 |
 | M4 | route-to/dup-to/reply-to, pools, divert e2e | **done** — gates 08/10/13 and divert gate 11 |
-| M5 | queues: ifq back ends (hfsc/fq_codel) + `pfq` qdisc | pending (scheduler code ported, qdisc missing) |
+| M5 | queues: ifq back ends (hfsc/fq_codel) | **partial** — back ends compiled into `pf.ko`; queue attachment/pacing pending |
 | M6 | docs complete, patchset tidy, full suite green, tagged releases | in progress — docs EN+PT-BR complete, 14/14 suite green, tagged |
 
 ## Next reads

docs/porting-notes.md

@@ -939,3 +939,33 @@ Rationale: a single consumer is the real use case, and Linux gives no bpf hook
 for a non-`net_device` virtual interface without pulling in a much larger
 amount of the BSD if layer.  The on-disk format is unchanged, so existing
 tooling works.
+
+## Scheduler back ends in the build (M5, first step)
+
+`pfsrc/hfsc.c` and `pfsrc/fq_codel.c` are now compiled into `pf.ko` (the
+`pf_ioctl` queue manager already references `hfsc_initialize`,
+`hfsc_pf_addqueue`, `ifq_hfsc_ops`, `fqcodel_pf_addqueue`, `ifq_fqcodel_ops`).
+The priq shim in `compat/pfk_ifq.c` remains the *attached* scheduler, so
+behaviour is unchanged until the data path enqueues into a queue.
+
+Compat surface added for the back ends (all in `module/include/pfk.h` /
+`pfk_if.h` / `compat/pfk_ifq.c`):
+
+| OpenBSD | This port |
+|---|---|
+| `struct mbuf_list` + `ml_init/ml_enqueue/ml_dequeue/ml_dequeue_tail/ml_enlist/ml_len/ml_empty/ml_purge`, `MBUF_LIST_FIRST/INITIALIZER` | inline helpers in `pfk_if.h` (`ml_purge` in `compat/pfk_mbuf.c`) |
+| `ifq_q_enter()/ifq_q_leave()` (scheduler ownership handshake, guarded by the ifq mutex) | ownership check against `ifq->ifq_ops`, returning `ifq_softc`; `leave` is a no-op (the port has no ifq mutex — the data path is one ordered work queue) |
+| `nsecuptime()` | `getnsecuptime()` (`ktime_get_ns`) |
+| `tick_nsec` (`HFSC_CLK_PER_TICK`) | `1000000000 / HZ` |
+| `splnet()/splx()` | no-ops, kept as markers of the critical regions |
+| `KERNEL_ASSERT_LOCKED()` | no-op (structural, not runtime) |
+| `struct hfsc_sc`, `struct hfsc_class_stats`, `struct fqcodel_stats`, `PKTCNTR_INC`, `HFSC_*` limits | declared in `pfk.h` (OpenBSD keeps them in `net/if_hfsc.h` / `net/if_fq_codel.h`) |
+
+Two port fixes worth recording:
+
+- `min(cd->drops - 1, nitems(...) - 1)` → `min_t(unsigned int, ...)`: Linux
+  type-checks `min()` and `nitems()` is `size_t`.
+- `hfsc_more_slots(u_int current)` → renamed the parameter: **`current` is a
+  kernel macro** (`get_current()`), which turned the definition into a K&R
+  function and produced a baffling "makes pointer from integer" error at the
+  call site.

docs/pt-BR/caveats.md

@@ -111,6 +111,30 @@ lendo `/dev/pflog`; sem ele a fila atinge o limite (256 registros / 1 MiB) e
 descarta. Nenhum dos dois é mais stub, mas nenhum faz algo sem o consumidor
 rodar (ver [pf-guide.md](pf-guide.md)).
 
+### 9. Configurar filas esbarra numa asserção de lock no caminho de purge (aberto)
+
+Carregar uma regra `queue` agora alcança o escalonador (`pfq_alloc` tem sucesso,
+a disciplina hfsc é criada) e `pf_commit_queues()` é exercitada de ponta a ponta.
+Dois bugs foram encontrados e corrigidos no caminho: o handle da classe raiz
+colidia com qids ímpares do pf (`pfq_addqueue` devolvia EBUSY) e a disciplina
+NULL não era verificada.
+
+O que ainda bloqueia as filas: o timer de purge do port pode rodar enquanto um
+ioctl segura o lock de escrita, e `pf_purge_expired_states()` afirma o contrário:
+
+```
+splassert: pf_purge_expired_states: expected 0, got 2
+kernel BUG at compat/pfk_kern.c:51
+```
+
+O OpenBSD serializa isso com `splnet()`; o port substituiu a camada spl por
+rw-semáforos e work queue, então a asserção passou a ser uma corrida real em vez
+de formalidade. Até corrigi-la (liberar o lock em torno da chamada, ou dar ao
+estado o seu próprio lock como o upstream faz com `pf_state_lock`), um host que
+configure filas pode travar o módulo em `Unloading`. **Não habilite regras
+`queue` em host real ainda** — os 14 gates do lab nunca configuram filas, e é
+por isso que continuam verdes.
+
 ## Desvios da camada de compat (de `docs/porting-notes.md`, registro do M1)
 
 Estes são desvios deliberados e marcados no código; importam ao raciocinar

docs/pt-BR/feature-matrix.md

@@ -60,7 +60,7 @@ Estado em 2026-11-11; evidência por linha: `out/test-logs/`,
 |---|---|---|---|
 | Pools de `route-to` (`rr`, `round-robin`, `source-hash`, `hash`), `reply-to`, `dup-to` | `pass in … route-to { 10.0.3.2, 10.0.2.2 } round-robin keep state` | M4 | **`round-robin` e `source-hash` VERIFICADOS em lab** (gate `10-pools.sh`): com dois gateways em dois enlaces, o round-robin usa ambos e o source-hash fixa todo fluxo de uma origem em um único enlace. `dup-to` verificado no gate 08; `reply-to` ainda sem gate. `fail-to` **não existe no OpenBSD 8.0** (removido upstream), então está fora de escopo. |
 | `divert-to` / tag `divert` | `pass in … divert-to <host> port <n>` | VERIFICADO em lab | **Funciona ponta a ponta.** O módulo registra em runtime a família de sockets `AF_DIVERT` (`PF_DIVERT=32`) — sem patch no kernel. A regra `divert-to` entrega o pacote a essa família; sem listener o pacote é descartado (`divs_noport`, paridade BSD); com `tools/pdivert` rodando o pacote vai ao userspace, é reescrito e reinjetado (verificado com mutação de payload + correção de checksum, ping 0% loss). |
-| Filas, agendador default (priq) | `queue root on wan`, `pfctl -s queue -v` | M5 | COMPILES-untested — `compat/pfk_ifq.c` implementa a API ifq com **semântica de fallback priq**; regras `queue` fazem parse e carregam, mas pacotes ainda não são desviados por um qdisc classful até o M5 |
+| Filas (`queue`, `queue on`, `pfctl -s queue -v`) | `queue root q1 bandwidth 1Mbit; queue on egress all ...` | M5 | **Back ends compilados no `pf.ko`, anexação pendente** — `hfsc.c` e `fq_codel.c` do OpenBSD agora compilam e linkam (157 símbolos do escalonador; `hfsc_initialize`/`hfsc_pf_addqueue`/`fqcodel_pf_addqueue` alcançáveis pelo `pf_ioctl`), e a suíte inteira continua passando com eles dentro. O caminho de dados já pode entregar pacotes a um escalonador anexado (`pf_kifq_input()` + um `ifq_start()` real que drena e transmite), mas as filas **ainda não são seguras para habilitar**: o caminho de purge esbarra numa asserção de lock (ver caveats #9), então nenhuma banda é aplicada e não há gate para isso. |
 | HFSC | `queue on wan { hfsc … }` | M5 | COMPILES-untested — **fallback priq até o M5**; `pfsrc/hfsc.c` staged (código real do OpenBSD 8.0) mas **fora da lista de objetos** do módulo |
 | FQ-CoDel | `queue on wan { fq_codel … }` | M5 | COMPILES-untested — igual ao HFSC: staged, não compilado |
 | Qdisc `pfq` (`tc qdisc … root pfq`) | n/a (glue Linux) | M5 | PENDING-M5 (ainda não existe na árvore) |

docs/pt-BR/guide.md

@@ -94,7 +94,7 @@ tamanho embutido na codificação `_IOR/_IOW` OpenBSD e despacha para o
 | M2 | núcleo do filtro: regras/estados/tabelas/âncoras, gates VM | **pronto** — gates 01–04 verdes |
 | M3 | NAT (`nat-to`/`rdr-to`/`binat-to`), scrub, pflog | **pronto** — gates 05/06/07/09 (incl. integridade de 2 MB via redirect) e pflog gate 12 |
 | M4 | route-to/dup-to/reply-to, pools, divert e2e | **pronto** — gates 08/10/13 e divert gate 11 |
-| M5 | filas: back ends ifq (hfsc/fq_codel) + qdisc `pfq` | pendente (código dos escalonadores portado, qdisc faltando) |
+| M5 | filas: back ends ifq (hfsc/fq_codel) | **parcial** — back ends compilados no `pf.ko`; anexação/ritmo pendentes |
 | M6 | docs completas, patchset limpo, suíte verde, releases com tag | em andamento — docs EN+PT-BR completas, suíte 14/14 verde, com tag |
 
 ## Próximas leituras

module/.pf.mod.cmd

@@ -1 +1 @@
-savedcmd_pf.mod := printf '%s\n'   pfsrc/pfk_mod.o pfsrc/pfk_hooks.o compat/pfk_mbuf.o compat/pfk_kern.o compat/pfk_if.o compat/pfk_xmit.o compat/pfk_pcb.o compat/pfk_ip4.o compat/siphash.o compat/toeplitz.o compat/md5.o compat/sha2.o compat/radix.o compat/pfk_rtable.o compat/pfk_ip6.o compat/subr_tree.o pfsrc/pf_ruleset.o pfsrc/pf_table.o pfsrc/pf_norm.o pfsrc/pf_if.o pfsrc/pf_lb.o pfsrc/pf_osfp.o pfsrc/pf_syncookies.o pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o pfsrc/pfk_pflog.o compat/pfk_ifq.o | awk '!x[$$0]++ { print("./"$$0) }' > pf.mod
+savedcmd_pf.mod := printf '%s\n'   pfsrc/pfk_mod.o pfsrc/pfk_hooks.o compat/pfk_mbuf.o compat/pfk_kern.o compat/pfk_if.o compat/pfk_xmit.o compat/pfk_pcb.o compat/pfk_ip4.o compat/siphash.o compat/toeplitz.o compat/md5.o compat/sha2.o compat/radix.o compat/pfk_rtable.o compat/pfk_ip6.o compat/subr_tree.o pfsrc/pf_ruleset.o pfsrc/pf_table.o pfsrc/pf_norm.o pfsrc/pf_if.o pfsrc/pf_lb.o pfsrc/pf_osfp.o pfsrc/pf_syncookies.o pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o pfsrc/pfk_pflog.o pfsrc/fq_codel.o pfsrc/hfsc.o compat/pfk_ifq.o | awk '!x[$$0]++ { print("./"$$0) }' > pf.mod

module/Makefile

@@ -20,7 +20,7 @@ pf-objs := pfsrc/pfk_mod.o pfsrc/pfk_hooks.o \
 # ported OpenBSD 8.0 engine
 pf-objs += pfsrc/pf_ruleset.o pfsrc/pf_table.o pfsrc/pf_norm.o pfsrc/pf_if.o \
            pfsrc/pf_lb.o pfsrc/pf_osfp.o pfsrc/pf_syncookies.o \
-           pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o pfsrc/pfk_pflog.o
+           pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o pfsrc/pfk_pflog.o pfsrc/fq_codel.o pfsrc/hfsc.o
 
 # queueing framework (compat/pfk_ifq.c implements the ifq API with priq
 # semantics; the OpenBSD hfsc/fq_codel back ends land in milestone M5)

module/compat/pfk_ifq.c

@@ -273,10 +273,11 @@ static const struct pfq_ops pfk_priq_pfq_ops = {
  * packets are still queued, but the traffic shaping curves are not yet
  * applied — recorded in the feature matrix as pending M5).
  */
-const struct ifq_ops * const ifq_hfsc_ops = &pfk_priq_ifq_ops;
+/*
-const struct ifq_ops * const ifq_fqcodel_ops = &pfk_priq_ifq_ops;
+ * M5: the real scheduler back ends (pfsrc/hfsc.c, pfsrc/fq_codel.c) define
-const struct pfq_ops * const pfq_hfsc_ops = &pfk_priq_pfq_ops;
+ * ifq_hfsc_ops/pfq_hfsc_ops/ifq_fqcodel_ops/pfq_fqcodel_ops themselves; the
-const struct pfq_ops * const pfq_fqcodel_ops = &pfk_priq_pfq_ops;
+ * priq shim above stays as the default for interfaces without a queue rule.
+ */
 
 /* ---------------- framework ---------------- */
 void
@@ -305,6 +306,29 @@ ifq_attach(struct ifqueue *ifq, const struct ifq_ops *ops, void *opsarg)
 	ifq->ifq_softc = opsarg;
 }
 
+/*
+ * Scheduler hand-off used by the ported back ends (hfsc/fq_codel) when they
+ * need their per-interface state.  PFLINUX: the port has no ifq mutex (the
+ * data path is a single ordered work queue), so "enter" is a plain ownership
+ * check and "leave" is a no-op - the pair exists to mirror OpenBSD's API.
+ */
+void *
+ifq_q_enter(struct ifqueue *ifq, const struct ifq_ops *ops)
+{
+	if (ifq == NULL || ops == NULL || ifq->ifq_ops != ops)
+		return NULL;			/* another scheduler owns it */
+	if (ifq->ifq_softc == NULL && ops->ifqop_alloc)
+		ifq->ifq_softc = ops->ifqop_alloc(0, NULL);
+	return ifq->ifq_softc;
+}
+
+void
+ifq_q_leave(struct ifqueue *ifq, void *q)
+{
+	(void)ifq;
+	(void)q;
+}
+
 void
 ifq_destroy(struct ifqueue *ifq)
 {
@@ -336,14 +360,16 @@ ifq_deq_begin(struct ifqueue *ifq)
 	if (!ifq->ifq_ops || !ifq->ifq_ops->ifqop_deq_begin)
 		return NULL;
 	m = ifq->ifq_ops->ifqop_deq_begin(ifq, &cookie);
+	ifq->ifq_dequeue = cookie;
 	return m;
 }
 
 void
 ifq_deq_commit(struct ifqueue *ifq, struct mbuf *m)
 {
-	void *cookie = NULL;
+	void *cookie = ifq->ifq_dequeue;
 
+	ifq->ifq_dequeue = NULL;
 	if (ifq->ifq_ops && ifq->ifq_ops->ifqop_deq_commit)
 		ifq->ifq_ops->ifqop_deq_commit(ifq, m, cookie);
 }
@@ -365,12 +391,47 @@ ifq_dequeue(struct ifqueue *ifq)
 	return m;
 }
 
+/*
+ * Drain the queue as far as the scheduler allows and hand each packet to the
+ * stack's output path.  OpenBSD calls if_start from softint context; here it
+ * runs either from the pf work queue (enqueue side) or from the scheduler's
+ * deferred timeout (pacing side), which is what makes hfsc's bandwidth
+ * limits take effect.
+ */
 void
 ifq_start(struct ifqueue *ifq)
 {
+	struct mbuf *m;
+
+	ifq->ifq_oactive = 1;
+	while ((m = ifq_dequeue(ifq)) != NULL) {
+		if (pfk_ifq_transmit(ifq, m))
+			break;			/* device busy: packet kept */
+	}
 	ifq->ifq_oactive = 0;
 }
 
+/*
+ * pf data-path entry: hand a packet to the interface's scheduler when one is
+ * attached (a pf `queue' rule is configured).  Returns 1 when the queue took
+ * ownership of m, 0 when the caller must keep going (priq/no scheduler).
+ */
+int
+pf_kifq_input(struct pfk_ifnet *ifp, struct mbuf *m)
+{
+	struct ifqueue *ifq;
+
+	if (ifp == NULL || m == NULL)
+		return 0;
+	ifq = &ifp->if_snd;
+	if (ifq->ifq_ops == NULL || ifq->ifq_ops == ifq_priq_ops)
+		return 0;			/* no scheduler: fast path */
+	if (ifq_enqueue(ifq, m) != 0)
+		return 1;			/* dropped by the queue: owned */
+	ifq_start(ifq);
+	return 1;
+}
+
 unsigned int
 ifq_purge(struct ifqueue *ifq)
 {
@@ -423,19 +484,3 @@ ifq_hdatalen(struct ifqueue *ifq)
 	return 0;
 }
 
-/* ---------------- scheduler back ends (milestone M5) ---------------- */
-/* The staged OpenBSD hfsc.c/fq_codel.c are not in the build yet; the priq
- * ops above are the active scheduler.  These wrappers keep pf_ioctl's
- * initialisation calls valid until M5 replaces them with the ported
- * schedulers (docs/porting-notes.md). */
-void
-hfsc_initialize(void)
-{
-	pr_info("pfk: hfsc back end pending M5 (priq active)\n");
-}
-
-void
-fq_codel_initialize(void)
-{
-	pr_info("pfk: fq_codel back end pending M5 (priq active)\n");
-}

module/compat/pfk_mbuf.c

@@ -607,3 +607,14 @@ pfk_mbuf_exit(void)
 {
 	kmem_cache_destroy(pfk_mbuf_cache);
 }
+
+void
+ml_purge(struct mbuf_list *ml)
+{
+	struct mbuf *m;
+
+	while ((m = ml_dequeue(ml)) != NULL)
+		m_freem(m);
+	ml->ml_first = ml->ml_last = NULL;
+	ml->ml_len = 0;
+}

module/compat/pfk_xmit.c

@@ -270,3 +270,40 @@ pfk_if_output(struct pfk_ifnet *ifp, struct mbuf *m, void *dst,
 		m_freem(mp);
 	return err;
 }
+
+/*
+ * Transmit one queued packet.  Called from ifq_start() - i.e. from the pf
+ * work queue or from a scheduler's deferred timeout - and always consumes the
+ * mbuf.  Returns 0 (keep draining); a scheduler that wants to defer does so by
+ * simply not returning a packet from ifqop_deq_begin().
+ */
+int
+pfk_ifq_transmit(struct ifqueue *ifq, struct mbuf *m)
+{
+	struct sk_buff *skb = m_to_skb(m);
+	struct dst_entry *d;
+
+	(void)ifq;
+	if (skb == NULL) {
+		m_freem(m);
+		return 0;
+	}
+	d = skb_dst(skb);
+	if (d == NULL || IS_ERR(d) || d->dev == NULL || !netif_running(d->dev)) {
+		m_freem(m);
+		return 0;
+	}
+	skb->dev = d->dev;
+	skb_reset_network_header(skb);
+	if (skb->protocol == 0)
+		skb->protocol = htons(ip_hdr(skb)->protocol == IPPROTO_IPV6 ?
+		    ETH_P_IPV6 : ETH_P_IP);
+	/* the stack owns skb->cb (IPCB) from here; pf already tested it */
+	memset(skb->cb, 0, sizeof(skb->cb));
+	skb->mark |= PFK_DONE_OUT;
+	skb_get(skb);		/* the output path consumes one reference */
+	m->skb = NULL;		/* the mbuf's reference moves to the stack */
+	m_freem(m);
+	d->output(&init_net, NULL, skb);
+	return 0;
+}

module/include/compat/net/ifq.h

@@ -27,6 +27,7 @@ struct ifqueue {
 	struct mbuf_list	*ifq_free;	/* free list (unused in the port) */
 	unsigned int		 ifq_len;
 	unsigned int		 ifq_oactive;
+	void			*ifq_dequeue;	/* cookie between deq_begin/commit */
 
 	/* statistics */
 	u_int64_t		 ifq_packets;

module/include/pfk.h

@@ -1212,5 +1212,103 @@ struct pfk_pflog_stat {
 };
 #define PFKIOCGETPFLOGSTAT	_IOR('L', 1, struct pfk_pflog_stat)
 
+
+/*
+ * Pieces the staged scheduler back ends (fq_codel(4)/hfsc(4) ports) need from
+ * OpenBSD <net/if_var.h>, <net/if_fq_codel.h> and <net/if_hfsc.h>.
+ */
+#ifndef nsecuptime
+#define nsecuptime()	 getnsecuptime()
+#endif
+
+struct fqcodel_pktcntr { uint64_t packets; uint64_t bytes; };
+struct hfsc_pktcntr	 { uint64_t packets; uint64_t bytes; };
+
+struct fqcodel_stats {
+	struct fqcodel_pktcntr xmit_cnt;
+	struct fqcodel_pktcntr drop_cnt;
+	uint32_t	 target, interval, tc_interval_drop;
+	uint32_t	 rcvtime, mtu, qlimit, qlength;
+	uint64_t	 delaysum, delaysumsq;
+	uint8_t		 ecn, limit, flows, lscroll, flags;
+};
+
+/* ifq scheduler hand-off (see pfsrc/ifq.c): enter returns the scheduler's
+ * queue when it owns the interface queue, NULL otherwise. */
+void	*ifq_q_enter(struct ifqueue *, const struct ifq_ops *);
+void	 ifq_q_leave(struct ifqueue *, void *);
+int	 pfk_ifq_transmit(struct ifqueue *, struct mbuf *);
+int	 pf_kifq_input(struct pfk_ifnet *, struct mbuf *);
+
+
+/*
+ * OpenBSD <net/if_hfsc.h> pieces the staged hfsc(4) back end needs.
+ * PFLINUX: the port keeps the BSD scheduler sources verbatim, so the
+ * user-visible service curve, the class-stats record and the class-table
+ * limits live here instead of in a separate kernel header.
+ */
+struct hfsc_sc {				/* user-supplied service curve */
+	u_int64_t	m1;			/* slope1 (bytes/sec) */
+	u_int64_t	d;			/* ordinate of the knee (us) */
+	u_int64_t	m2;			/* slope2 (bytes/sec) */
+};
+
+struct hfsc_class_stats {
+	u_int		class_id;
+	u_int32_t	class_handle;
+	struct hfsc_sc	rsc, fsc, usc;
+	u_int64_t	total, cumul;
+	u_int64_t	d, e, vt, f;
+	u_int64_t	initvt, vtperiod, parentperiod;
+	int		nactive;
+	u_int64_t	vtoff, cvtmax, myf, cfmin, cvtmin, myfadj, vtadj;
+	u_int		period;
+	struct hfsc_pktcntr xmit_cnt, drop_cnt;
+	u_int64_t	cur_time;
+	u_int64_t	machclk_freq;
+	int		qlength;
+	int		qlimit;
+	int		qtype;
+};
+
+/* OpenBSD's global nanoseconds-per-clocktick (HFSC_CLK_PER_TICK).  The port
+ * has no clock_rate machinery, so derive it from the kernel's HZ. */
+#ifndef tick_nsec
+#define tick_nsec		(1000000000LL / HZ)
+#endif
+
+/*
+ * Class-handle namespace.  pf queue ids are u_int16_t, so the root class gets
+ * a high bit that can never collide with a queue handle (a collision made
+ * hfsc_pf_addqueue() see the root class as a duplicate and refuse the queue
+ * with EBUSY).  HFSC_DEFAULTCLASS must equal PFQS_DEFAULT because hfsc.c
+ * receives the pf queue flags verbatim.
+ */
+#define HFSC_ROOT_CLASS		0x80000000u
+#define HFSC_DEFAULTCLASS	0x1000
+#define HFSC_MAX_CLASSES	64
+#define HFSC_DEFAULT_CLASSES	8
+#define HFSC_DEFAULT_QLIMIT	1000
+
+/*
+ * PFLINUX notes on the scheduler back ends:
+ *
+ * - splnet()/splx(): OpenBSD raises the software interrupt level around
+ *   class-table surgery.  This port runs the whole data path in one ordered
+ *   work queue and guards the queue with ifq_mtx, so these are no-ops; they
+ *   are kept as markers of the critical regions.
+ * - KERNEL_ASSERT_LOCKED(): the schedulers are only reached from ifq/pf paths
+ *   that already hold the queue mutex; asserted structurally, not by spl.
+ * - HFSC_ENABLED(): the scheduler owns the interface queue once it attached
+ *   its private state through ifq_q_enter().
+ */
+#define splnet()		0
+#define splx(s)			((void)(s))
+#define KERNEL_ASSERT_LOCKED()	do { } while (0)
+#define HFSC_ENABLED(_ifq)	((_ifq)->ifq_q != NULL)
+
+#define PKTCNTR_INC(_c, _len)	do { (_c)->packets++; (_c)->bytes += (_len); } \
+				  while (0)
+
 #endif /* _PFK_H */
 

module/include/pfk_if.h

@@ -74,6 +74,79 @@ ml_dequeue(struct mbuf_list *ml)
 	return m;
 }
 
+
+/* BSD <net/if_var.h> mbuf_list helpers (net/if_var.h ml_*) */
+static inline void
+ml_init(struct mbuf_list *ml)
+{
+	ml->ml_first = ml->ml_last = NULL;
+	ml->ml_len = 0;
+}
+
+static inline void
+ml_enqueue(struct mbuf_list *ml, struct mbuf *m)
+{
+	m->m_next = NULL;
+	if (ml->ml_last == NULL)
+		ml->ml_first = m;
+	else
+		ml->ml_last->m_next = m;
+	ml->ml_last = m;
+	ml->ml_len++;
+}
+
+static inline struct mbuf *
+ml_dequeue_tail(struct mbuf_list *ml)
+{
+	struct mbuf *m, *prev = NULL;
+
+	if (ml->ml_first == NULL)
+		return NULL;
+	for (m = ml->ml_first; m->m_next; m = m->m_next)
+		prev = m;
+	if (prev == NULL)			/* single element */
+		ml->ml_first = NULL;
+	else
+		prev->m_next = NULL;
+	ml->ml_last = prev;
+	ml->ml_len--;
+	m->m_next = NULL;
+	return m;
+}
+
+/* append every packet in m2 to m1 (m2 becomes empty) */
+static inline void
+ml_enlist(struct mbuf_list *m1, struct mbuf_list *m2)
+{
+	if (m2->ml_first == NULL)
+		return;
+	if (m1->ml_last == NULL)
+		m1->ml_first = m2->ml_first;
+	else
+		m1->ml_last->m_next = m2->ml_first;
+	m1->ml_last = m2->ml_last;
+	m1->ml_len += m2->ml_len;
+	m2->ml_first = m2->ml_last = NULL;
+	m2->ml_len = 0;
+}
+
+static inline unsigned int
+ml_len(const struct mbuf_list *ml)
+{
+	return (unsigned int)ml->ml_len;
+}
+
+static inline int
+ml_empty(const struct mbuf_list *ml)
+{
+	return ml->ml_first == NULL;
+}
+
+void	ml_purge(struct mbuf_list *ml);	/* compat/pfk_mbuf.c (needs m_freem) */
+
+#define MBUF_LIST_FIRST(_ml)		((_ml)->ml_first)
+#define MBUF_LIST_INITIALIZER()		{ NULL, NULL, 0 }
+
 /* egress interface (default-route dev), maintained on fib/net events */
 struct pfk_ifnet *pfk_egress_if(void);
 void pfk_egress_refresh(void);

module/include/pfk_mbuf.h

@@ -73,7 +73,7 @@ struct pkthdr {
 	u_int32_t		 ph_flowid;
 	u_int			 ph_ifidx;
 	u_int			 ph_rtableid;
-	struct pfk_timeval	 ph_timestamp;
+	int64_t			 ph_timestamp;	/* OpenBSD 8.0: nanoseconds */
 	struct pfk_ifnet	*ifp;
 	struct pfk_ifnet	*rcvif;
 	u_short			 ph;		/* tso segs: unused, kept as pad */

module/pfsrc/fq_codel.c

@@ -332,7 +332,9 @@ control_law(struct codel *cd, struct codel_params *cp, int64_t rts)
 	unsigned int idx;
 
 	KASSERT(cd->drops > 0);
-	idx = min(cd->drops - 1, nitems(codel_intervals) - 1);
+	/* PFLINUX: Linux' min() type-checks its arguments; nitems() is size_t */
+	idx = min_t(unsigned int, cd->drops - 1,
+	    (unsigned int)(nitems(codel_intervals) - 1));
 	cd->next = rts + cp->intervals[idx];
 }
 

module/pfsrc/hfsc.c

@@ -335,9 +335,9 @@ hfsc_class_purge(struct hfsc_class *cl, struct mbuf_list *ml)
 }
 
 static inline u_int
-hfsc_more_slots(u_int current)
+hfsc_more_slots(u_int cur)	/* PFLINUX: 'current' is a kernel macro */
 {
-	u_int want = current * 2;
+	u_int want = cur * 2;
 
 	return (want > HFSC_MAX_CLASSES ? HFSC_MAX_CLASSES : want);
 }

module/pfsrc/pf_ioctl.c

@@ -662,6 +662,11 @@ pf_create_queues(void)
 		}
 
 		qif->disc = qif->pfqops->pfq_alloc(ifp);
+		if (qif->disc == NULL) {
+			error = ENOBUFS;
+			free(qif, M_PF, sizeof(*qif));
+			goto error;
+		}
 
 		qif->next = list;
 		list = qif;

module/pfsrc/pfk_hooks.c

@@ -204,6 +204,22 @@ pfk_pkt_work(struct work_struct *w)
 
 
 	if (m == m0) {
+		/*
+		 * Queueing: when a pf `queue' rule attached a scheduler to the
+		 * egress interface, hand the packet to it instead of sending it
+		 * straight out - this is what makes bandwidth limits real.
+		 */
+		if (action == PF_PASS && pk->dir == PF_OUT &&
+		    pskb_may_pull(pk->skb, sizeof(struct iphdr)) &&
+		    skb_dst(pk->skb) && !IS_ERR(skb_dst(pk->skb)) &&
+		    pf_kifq_input(skb_dst(pk->skb)->dev ?
+			pfk_ifp_bydev(skb_dst(pk->skb)->dev) : NULL, m)) {
+			orig = pk->skb;
+			pk->skb = NULL;		/* the queue owns the packet */
+			kfree_skb(orig);
+			kfree(pk);
+			return;
+		}
 		/* same buffer, possibly rewritten in place: our wrapper
 		 * held one extra skb reference — release it */
 		m_freem(m);