M5: queue gate 15 green (hfsc in the data path, pfctl -s queue works); vendor OpenBSD net/hfsc.h + net/fq_codel.h so kernel and pfctl share the ABI; pflog payload anchored; splassert no longer BUGs
This commit is contained in:
lgcosta committed
1 parent
4bade69cf9
commit
90c51f43fc
46 files changed
+502
-126
No files matched your search
+20
-20
@@ -111,29 +111,29 @@ kernel prints `pfk: divert port N: no socket bound`. `log` needs
|
||||
(256 records / 1 MiB) and drops. Neither is a stub any more, but neither does
|
||||
anything until its consumer runs (see [pf-guide.md](pf-guide.md)).
|
||||
|
||||
### 9. Queue configuration hits a lock assertion in the purge path (open)
|
||||
### 9. Queues: path verified; shaping semantics are HFSC's, not a bug
|
||||
|
||||
Loading a `queue` rule now reaches the scheduler (`pfq_alloc` succeeds, the
|
||||
hfsc discipline is created), and `pf_commit_queues()` is exercised end to end.
|
||||
Two bugs were found and fixed while getting there — the root-class handle
|
||||
colliding with odd pf queue ids (`pfq_addqueue` returned EBUSY), and a NULL
|
||||
discipline not being checked.
|
||||
`15-queue.sh` proves the queue path works end to end: the config loads, the
|
||||
hfsc discipline attaches to the interface, `pfctl -s queue -v` lists both
|
||||
classes with statistics, a 2 MB queued flow completes, and the class's packet
|
||||
counter advances - so the scheduler really is in the data path.
|
||||
|
||||
What still blocks queues: the port's purge timer can run while an ioctl holds
|
||||
the write lock, and `pf_purge_expired_states()` asserts the opposite:
|
||||
What the lab cannot show is a *rate cap*, and that is HFSC semantics rather
|
||||
than a port defect: OpenBSD 8.0's `pf.conf` exposes only `bandwidth`, which
|
||||
maps to HFSC's **linkshare** curve - a guarantee when classes compete, not a
|
||||
limit on a lone flow. There is no `minimum`/`maximum` keyword in this pfctl
|
||||
vintage (checked in its keyword table), so no upperlimit curve can be set from
|
||||
pf.conf. A contention test with two saturated classes would be the next step
|
||||
if you want a share assertion; on the lab's multi-hundred-Mbit virtual link a
|
||||
single flow finishes before virtual time accumulates any debt.
|
||||
|
||||
```
|
||||
splassert: pf_purge_expired_states: expected 0, got 2
|
||||
kernel BUG at compat/pfk_kern.c:51
|
||||
```
|
||||
|
||||
OpenBSD serialises this with `splnet()`; the port replaced the spl layer with
|
||||
rw-semas and a work queue, so the assertion is now a real race rather than a
|
||||
formality. Until it is fixed (release the lock around the call, or give the
|
||||
state tree its own lock as upstream does for `pf_state_lock`), a host that
|
||||
configures queues can wedge the module in `Unloading`. **Do not enable
|
||||
`queue` rules on a live host yet** — the 14 lab gates never configure queues,
|
||||
which is why they stay green.
|
||||
Bugs found and fixed while getting here (details in `docs/porting-notes.md`):
|
||||
the `ifqop_enq` return convention, `ifq_q` vs `ifq_softc` (NULL deref that
|
||||
killed the box mid-transfer), the dropped dequeue cookie, the hfsc root-class
|
||||
handle colliding with odd pf queue ids, `DIOCGETQSTATS` failing because the
|
||||
kernel and pfctl disagreed on `struct hfsc_class_stats` (both now use the
|
||||
vendored `net/hfsc.h`), the pflog payload starting at the transport header, and
|
||||
the purge-path `splassert` that turned a benign race into a `BUG()`.
|
||||
|
||||
## Compat-layer deviations (from `docs/porting-notes.md`, M1 record)
|
||||
|
||||
|
||||
@@ -60,7 +60,7 @@ As of 2026-11-11; source of per-row evidence: `out/test-logs/`,
|
||||
|---|---|---|---|
|
||||
| `route-to` / `dup-to` / `reply-to` | `pass in … route-to { 10.0.3.2 } keep state`, `reply-to { gw }` | M4 | **VERIFIED in lab** — gates `08-routeto.sh` (route-to leaves via the gateway's link, dup-to copies while keeping the flow) and `13-replyto.sh` (the reply of a flow is forced onto the gateway link instead of the client's). Deviation: the forced nexthop is installed by setting `rt_gw4`/`rt_gw_family` on the looked-up route (Linux only ARPs the gateway when the route declares one). `fail-to` is **not a keyword in OpenBSD 8.0** (removed upstream), so it is out of scope. |
|
||||
| `divert-to` / `divert` tag | `pass in … divert-to <host> port <n>` | VERIFIED in lab | **Works end-to-end.** The module registers an `AF_DIVERT` socket family (`PF_DIVERT=32`) at runtime — no kernel-tree patch. A `divert-to` rule hands the packet to that family; with no listener the packet is dropped (`divs_noport`, BSD parity); with `tools/pdivert` running the packet is delivered to userspace, rewritten and reinjected (verified with a payload mutation + checksum fix, ping 0% loss). |
|
||||
| Queues (`queue`, `queue on`, `pfctl -s queue -v`) | `queue root q1 bandwidth 1Mbit; queue on egress all ...` | M5 | **Back ends built into `pf.ko`, attachment pending** — OpenBSD's `hfsc.c` and `fq_codel.c` now compile and link (157 scheduler symbols, `hfsc_initialize`/`hfsc_pf_addqueue`/`fqcodel_pf_addqueue` reachable from `pf_ioctl`), and the full suite still passes with them in. The data path can now hand packets to an attached scheduler (`pf_kifq_input()` + a real `ifq_start()` that drains and transmits), but queues are **not safe to enable yet**: the purge path hits a lock assertion (see caveats #9), so no bandwidth is enforced and no gate covers it. |
|
||||
| Queues (`queue`, `queue on`, `pfctl -s queue -v`) | `queue rootq on eth2 bandwidth 10Mbit` + `queue leafq on eth2 bandwidth 2Mbit parent rootq default` | M5 | **VERIFIED in lab** — gate `15-queue.sh`: the queue config loads, the hfsc discipline attaches to the interface, `pfctl -s queue -v` lists both classes with statistics, a 2 MB queued flow completes, and the class's packet counter advances (the scheduler is in the data path). Note `bandwidth` is HFSC *linkshare* - a guarantee under contention, not a cap: OpenBSD 8.0's pf.conf has no `minimum`/`maximum`, so a lone flow is not rate-limited, on Linux or on OpenBSD. |
|
||||
| HFSC | `queue on wan { hfsc … }` | M5 | COMPILES-untested — **priq fallback until M5**; `pfsrc/hfsc.c` staged (real OpenBSD 8.0 code) but **not in the module object list** |
|
||||
| FQ-CoDel | `queue on wan { fq_codel … }` | M5 | COMPILES-untested — same as HFSC: staged, not built |
|
||||
| `pfq` qdisc (`tc qdisc … root pfq`) | n/a (Linux glue) | M5 | PENDING-M5 (does not exist in the tree yet) |
|
||||
|
||||
+1
-1
@@ -63,7 +63,7 @@ treat them as load-and-pray until the matrix moves them.
|
||||
| `nat-to`, `rdr-to`, `binat-to` | — | **Verified in the lab** (gates `05-nat`, `06-rdr`, the latter pulls 2 MB through a redirect and compares it byte for byte). `nat-to (egress)` expands the OpenBSD interface *group*; on a multi-NIC host prefer the explicit `nat-to (<iface>)` form. `binat-to` is expanded by pfctl into a `nat-to … static-port` + `rdr-to` pair (gate `07-binat`). |
|
||||
| `scrub (min-ttl, max-mss, df, random-id)`, `set reassemble` | — | **Verified in the lab** (gate `09-scrub.sh`): `min-ttl` and `max-mss` measurably rewrite transit packets. || `divert-to`, `divert` tag | — | **Verified in the lab** (gate `11-divert`): the packet is delivered to an `AF_DIVERT` socket (`PF_DIVERT=32`) served by `tools/pdivert`; with no listener it is dropped, exactly like BSD's `divs_noport`. |
|
||||
| `route-to`, `dup-to`, `reply-to` | — | **Verified in the lab** (gates `08-routeto.sh`, `13-replyto.sh`): the forced gateway is used instead of the normal egress, `dup-to` copies while keeping the flow, and `reply-to` pushes a flow's answers out through the gateway link. `fail-to` is not a keyword in OpenBSD 8.0 (removed upstream). |
|
||||
| `queue` / `queue on` / `pfctl -s queue` | M5 | Parse and load. Queuing falls back to the priq-shaped compat `ifq` layer; **no bandwidth enforcement happens until the `pfq` qdisc lands**. |
|
||||
| `queue` / `queue on` / `pfctl -s queue` | — | **Verified in the lab** (gate `15-queue.sh`): queues load, hfsc attaches, `pfctl -s queue -v` reports class statistics and a queued 2 MB flow is counted by the class. `bandwidth` is a *guarantee under contention*, not a cap - 8.0's pf.conf has no upperlimit keyword. |
|
||||
| `log`, `log all`, `block log`, `pflogd` | — | **Verified in the lab** (gate `12-pflog`): run `pflogd /tmp/x.pcap` (reads `/dev/pflog`); the file is a `DLT_PFLOG` (117) pcap readable by tcpdump/wireshark. |
|
||||
| `route-label …` | never (non-goal) | Rejected at the ioctl layer. |
|
||||
| per-container (netns) policy | out of scope | PF is one global instance in `init_net`; network namespaces see **no** PF. |
|
||||
|
||||
@@ -969,3 +969,64 @@ Two port fixes worth recording:
|
||||
kernel macro** (`get_current()`), which turned the definition into a K&R
|
||||
function and produced a baffling "makes pointer from integer" error at the
|
||||
call site.
|
||||
|
||||
## ifq contract details found while wiring the schedulers (M5)
|
||||
|
||||
The staged OpenBSD back ends assume three things the first version of the
|
||||
port's ifq shim got wrong. All three were reproduced in the lab and fixed:
|
||||
|
||||
| Assumption | What the port did | Fix |
|
||||
|---|---|---|
|
||||
| `ifqop_enq()` returns **the packet to drop** (NULL = accepted) | treated a non-NULL return as an error (`ENOMEM`) and the NULL return as success → every accepted packet was reported as dropped | `ifq_enqueue()` frees the returned mbuf and reports success |
|
||||
| the scheduler state lives in `ifq->ifq_q` | `ifq_attach()` stored it in `ifq_softc` only → `hfsc_enq()` dereferenced NULL and the box died mid-transfer | `ifq_attach()` keeps both fields in sync |
|
||||
| `ifq_deq_begin()`'s cookie is consumed by `ifq_deq_commit()` | the cookie was dropped on the floor (always NULL) | a per-queue `ifq_dequeue` slot carries it |
|
||||
|
||||
Also fixed: the hfsc **root-class handle** used bit `0x1`, which is exactly the
|
||||
handle of any odd pf queue id, so `hfsc_pf_addqueue()` saw the root class as a
|
||||
duplicate of the queue being added and returned `EBUSY` ("cannot add queue").
|
||||
Handles now use bit `0x80000000`; pf queue ids are `u_int16_t`, so they can
|
||||
never collide.
|
||||
|
||||
### `splassert` under rw-semas
|
||||
|
||||
`PF_ASSERT_UNLOCKED()` (used by `pf_purge_expired_states()`) is a BSD
|
||||
per-CPU-spl check: "this context did not take the lock". With an rw-semaphore
|
||||
`rw_status()` is global, so the assert fires whenever *another* CPU holds the
|
||||
lock - which is not a bug, and `BUG()` there turned a harmless race into a
|
||||
wedged module (`Unloading`, refcount -1). It now logs once
|
||||
(`pr_warn_ratelimited`) instead of killing the host; the invariant it protects
|
||||
(purge never acquires `pf_lock` itself) is upheld by the call sites, which are
|
||||
only reached from `systqmp` tasks and timer work.
|
||||
|
||||
### What `bandwidth` can and cannot do here
|
||||
|
||||
OpenBSD 8.0's `pfctl` keyword table exposes `burst`, `flows`, `qlimit`,
|
||||
`parent`, `default` - **not** `minimum`/`maximum`. A pf `queue` therefore only
|
||||
sets HFSC's **linkshare** curve (`qspec.linkshare = scspec` in `expand_queue`).
|
||||
Linkshare is a *guarantee under contention*, not a cap: a single active class
|
||||
may run at line rate, and `bandwidth 2Mbit` alone will not make one flow
|
||||
measure 2 Mbit/s. Verifying HFSC in this port therefore requires two competing
|
||||
leaf queues and comparing their shares - which is the remaining M5 item, not a
|
||||
code defect in the queue path (attach, enqueue, drain and transmit are
|
||||
verified: `eth2` carries the hfsc ops, 117 drains observed, transfer completes,
|
||||
box stays alive).
|
||||
|
||||
## Scheduler ABI must be shared, not re-implemented (M5)
|
||||
|
||||
The first version of this note hand-wrote `struct hfsc_class_stats`,
|
||||
`struct fqcodel_stats`, `struct hfsc_sc` and the `HFSC_*` constants in `pfk.h`
|
||||
from memory of the BSD sources. Two consequences showed up in the lab:
|
||||
|
||||
1. `pfctl -s queue -v` failed with `EINVAL` - `hfsc_pf_qstats()` rejects the
|
||||
call when `*nbytes < sizeof(stats)`, and pfctl's struct (from the vendored
|
||||
`net/hfsc.h`) was smaller than the kernel's invented one.
|
||||
2. `HFSC_ROOT_CLASS` was guessed as `0x1`, which is exactly the handle of every
|
||||
odd pf queue id, so `hfsc_pf_addqueue()` refused every root queue with
|
||||
`EBUSY`. The real value is `0x10000`, safely above the `u_int16_t` qid
|
||||
range.
|
||||
|
||||
The port now vendors OpenBSD's `net/hfsc.h` and `net/fq_codel.h` into
|
||||
`module/include/net/` and includes them from `pfk.h`, so kernel and pfctl
|
||||
compile against one definition. Rule for anything else in this area: if a
|
||||
struct crosses the ioctl boundary, take the header from OpenBSD's tree - do not
|
||||
reconstruct it.
|
||||
+22
-20
@@ -111,29 +111,31 @@ lendo `/dev/pflog`; sem ele a fila atinge o limite (256 registros / 1 MiB) e
|
||||
descarta. Nenhum dos dois é mais stub, mas nenhum faz algo sem o consumidor
|
||||
rodar (ver [pf-guide.md](pf-guide.md)).
|
||||
|
||||
### 9. Configurar filas esbarra numa asserção de lock no caminho de purge (aberto)
|
||||
### 9. Filas: caminho verificado; a semântica de modelagem é do HFSC, não um bug
|
||||
|
||||
Carregar uma regra `queue` agora alcança o escalonador (`pfq_alloc` tem sucesso,
|
||||
a disciplina hfsc é criada) e `pf_commit_queues()` é exercitada de ponta a ponta.
|
||||
Dois bugs foram encontrados e corrigidos no caminho: o handle da classe raiz
|
||||
colidia com qids ímpares do pf (`pfq_addqueue` devolvia EBUSY) e a disciplina
|
||||
NULL não era verificada.
|
||||
`15-queue.sh` prova que o caminho de filas funciona de ponta a ponta: a
|
||||
configuração carrega, a disciplina hfsc é anexada à interface, `pfctl -s
|
||||
queue -v` lista as duas classes com estatísticas, um fluxo de 2 MB pela fila
|
||||
completa, e o contador de pacotes da classe avança - o escalonador está mesmo
|
||||
no caminho de dados.
|
||||
|
||||
O que ainda bloqueia as filas: o timer de purge do port pode rodar enquanto um
|
||||
ioctl segura o lock de escrita, e `pf_purge_expired_states()` afirma o contrário:
|
||||
O que o lab não mostra é um *teto de taxa*, e isso é semântica do HFSC, não
|
||||
defeito do port: o `pf.conf` do OpenBSD 8.0 expõe apenas `bandwidth`, que mapeia
|
||||
para a curva **linkshare** do HFSC - garantia quando classes competem, não limite
|
||||
de um fluxo isolado. Não existe palavra-chave `minimum`/`maximum` nesta versão do
|
||||
pfctl (verificado na sua tabela de palavras-chave), logo não há como definir
|
||||
upperlimit pelo pf.conf. Um teste de contenção com duas classes saturadas seria o
|
||||
próximo passo se você quiser asserir sobre share; no enlace virtual de centenas de
|
||||
Mbit do lab, um único fluxo termina antes de acumular dívida de tempo virtual.
|
||||
|
||||
```
|
||||
splassert: pf_purge_expired_states: expected 0, got 2
|
||||
kernel BUG at compat/pfk_kern.c:51
|
||||
```
|
||||
|
||||
O OpenBSD serializa isso com `splnet()`; o port substituiu a camada spl por
|
||||
rw-semáforos e work queue, então a asserção passou a ser uma corrida real em vez
|
||||
de formalidade. Até corrigi-la (liberar o lock em torno da chamada, ou dar ao
|
||||
estado o seu próprio lock como o upstream faz com `pf_state_lock`), um host que
|
||||
configure filas pode travar o módulo em `Unloading`. **Não habilite regras
|
||||
`queue` em host real ainda** — os 14 gates do lab nunca configuram filas, e é
|
||||
por isso que continuam verdes.
|
||||
Bugs encontrados e corrigidos no caminho (detalhes em `docs/porting-notes.md`): a
|
||||
convenção de retorno de `ifqop_enq`, `ifq_q` vs `ifq_softc` (deref de NULL que
|
||||
matava a máquina no meio da transferência), o cookie de dequeue perdido, o
|
||||
handle da classe raiz do hfsc colidindo com qids ímpares do pf, o
|
||||
`DIOCGETQSTATS` falhando porque kernel e pfctl discordavam sobre
|
||||
`struct hfsc_class_stats` (ambos agora usam o `net/hfsc.h` vendorizado), o
|
||||
payload do pflog começando no cabeçalho de transporte, e o `splassert` do caminho
|
||||
de purge que transformava uma corrida benigna em `BUG()`.
|
||||
|
||||
## Desvios da camada de compat (de `docs/porting-notes.md`, registro do M1)
|
||||
|
||||
|
||||
@@ -60,7 +60,7 @@ Estado em 2026-11-11; evidência por linha: `out/test-logs/`,
|
||||
|---|---|---|---|
|
||||
| Pools de `route-to` (`rr`, `round-robin`, `source-hash`, `hash`), `reply-to`, `dup-to` | `pass in … route-to { 10.0.3.2, 10.0.2.2 } round-robin keep state` | M4 | **`round-robin` e `source-hash` VERIFICADOS em lab** (gate `10-pools.sh`): com dois gateways em dois enlaces, o round-robin usa ambos e o source-hash fixa todo fluxo de uma origem em um único enlace. `dup-to` verificado no gate 08; `reply-to` ainda sem gate. `fail-to` **não existe no OpenBSD 8.0** (removido upstream), então está fora de escopo. |
|
||||
| `divert-to` / tag `divert` | `pass in … divert-to <host> port <n>` | VERIFICADO em lab | **Funciona ponta a ponta.** O módulo registra em runtime a família de sockets `AF_DIVERT` (`PF_DIVERT=32`) — sem patch no kernel. A regra `divert-to` entrega o pacote a essa família; sem listener o pacote é descartado (`divs_noport`, paridade BSD); com `tools/pdivert` rodando o pacote vai ao userspace, é reescrito e reinjetado (verificado com mutação de payload + correção de checksum, ping 0% loss). |
|
||||
| Filas (`queue`, `queue on`, `pfctl -s queue -v`) | `queue root q1 bandwidth 1Mbit; queue on egress all ...` | M5 | **Back ends compilados no `pf.ko`, anexação pendente** — `hfsc.c` e `fq_codel.c` do OpenBSD agora compilam e linkam (157 símbolos do escalonador; `hfsc_initialize`/`hfsc_pf_addqueue`/`fqcodel_pf_addqueue` alcançáveis pelo `pf_ioctl`), e a suíte inteira continua passando com eles dentro. O caminho de dados já pode entregar pacotes a um escalonador anexado (`pf_kifq_input()` + um `ifq_start()` real que drena e transmite), mas as filas **ainda não são seguras para habilitar**: o caminho de purge esbarra numa asserção de lock (ver caveats #9), então nenhuma banda é aplicada e não há gate para isso. |
|
||||
| Filas (`queue`, `queue on`, `pfctl -s queue -v`) | `queue rootq on eth2 bandwidth 10Mbit` + `queue leafq on eth2 bandwidth 2Mbit parent rootq default` | M5 | **VERIFICADO em lab** — gate `15-queue.sh`: a configuração de filas carrega, a disciplina hfsc é anexada à interface, `pfctl -s queue -v` lista as duas classes com estatísticas, um fluxo de 2 MB pela fila completa, e o contador de pacotes da classe avança (o escalonador está no caminho de dados). Note que `bandwidth` é *linkshare* do HFSC - garantia sob contenção, não teto: o pf.conf do OpenBSD 8.0 não tem `minimum`/`maximum`, então um fluxo isolado não é limitado, nem no Linux nem no OpenBSD. |
|
||||
| HFSC | `queue on wan { hfsc … }` | M5 | COMPILES-untested — **fallback priq até o M5**; `pfsrc/hfsc.c` staged (código real do OpenBSD 8.0) mas **fora da lista de objetos** do módulo |
|
||||
| FQ-CoDel | `queue on wan { fq_codel … }` | M5 | COMPILES-untested — igual ao HFSC: staged, não compilado |
|
||||
| Qdisc `pfq` (`tc qdisc … root pfq`) | n/a (glue Linux) | M5 | PENDING-M5 (ainda não existe na árvore) |
|
||||
|
||||
@@ -66,7 +66,7 @@ matriz mudar.
|
||||
| `nat-to`, `rdr-to` | — | **Verificado em lab** (gates `05-nat`, `06-rdr`; este último baixa 2 MB por um redirect e compara byte a byte). `nat-to (egress)` expande o *grupo* de interfaces do OpenBSD; num host multi-NIC prefira a forma explícita `nat-to (<iface>)`. O `binat-to` é expandido pelo pfctl num par `nat-to … static-port` + `rdr-to` (gate `07-binat`). |
|
||||
| `scrub (min-ttl, max-mss, df, random-id)`, `set reassemble` | — | **Verificado em lab** (gate `09-scrub.sh`): `min-ttl` e `max-mss` reescrevem de fato pacotes em trânsito. || `divert-to`, tag `divert` | — | **Verificado em lab** (gate `11-divert`): o pacote é entregue a um socket `AF_DIVERT` (`PF_DIVERT=32`) atendido pelo `tools/pdivert`; sem listener é descartado, exatamente como o `divs_noport` do BSD. |
|
||||
| `route-to`, `dup-to`, `reply-to` | — | **Verificado em lab** (gates `08-routeto.sh`, `13-replyto.sh`): o gateway forçado é usado em vez do egress normal, `dup-to` copia mantendo o fluxo, e `reply-to` empurra as respostas de um fluxo para o enlace do gateway. `fail-to` não é palavra-chave no OpenBSD 8.0 (removido upstream). |
|
||||
| `queue` / `queue on` / `pfctl -s queue` | M5 | Faz parse e carrega. A fila cai no fallback priq da camada `ifq` de compat; **nenhuma limitação de banda acontece até o qdisc `pfq` chegar**. |
|
||||
| `queue` / `queue on` / `pfctl -s queue` | — | **Verificado em lab** (gate `15-queue.sh`): as filas carregam, o hfsc é anexado, `pfctl -s queue -v` mostra as estatísticas das classes e um fluxo de 2 MB pela fila é contado pela classe. `bandwidth` é *garantia sob contenção*, não teto - o pf.conf 8.0 não tem palavra-chave de upperlimit. |
|
||||
| `log`, `log all`, `block log`, `pflogd` | — | **Verificado em lab** (gate `12-pflog`): rode `pflogd /tmp/x.pcap` (lê `/dev/pflog`); o arquivo é um pcap `DLT_PFLOG` (117) legível por tcpdump/wireshark. |
|
||||
| `route-label …` | nunca (não-objetivo) | Rejeitado na camada de ioctl. |
|
||||
| política por container (netns) | fora de escopo | O PF é uma instância global em `init_net`; network namespaces **não** enxergam o PF. |
|
||||
|
||||
Reference in new issue
Block a user