7 Commits
Author SHA1 Message Date
lgcosta cc67531602 Screencasts v2: real shell sessions, MiniMax narration (EN/PT), lab diagram
O problema da primeira serie era a sensacao de 'quadro a quadro': a captura
era a saida de um script despejada em blocos.  Agora cada video e uma sessao
shell de verdade.

- record.py: abre um pty, faz SSH no firewall e digita os comandos tecla a
  tecla (~34 c/s, com jitter).  O tty remoto ecoa, o kernel responde ao vivo;
  a captura e asciicast v2 com timestamps.  'bash --noediting' evita que o
  editor de linha redesenhe o comando e o tty o repita (a linha aparecia
  duplicada).
- record.py --pace: segura a tela enquanto a voz fala - cada batida dura o
  trecho de audio que a acompanha, de modo que uma unica captura serve as
  duas versoes de idioma.
- record.py --ensure-pf: garante pf.ko carregado antes de gravar.  Sem isso
  uma fase que roda 'rmmod' (ativacao) pode derrubar o /dev/pf de outra
  gravacao concorrentes - foi exatamente o que quebrou a fase de roteamento
  na primeira tentativa.
- make_audio.py: geracao da narracao pelo MiniMax TTS, uma voz feminina por
  idioma (English_ConfidentWoman / Portuguese_ConfidentWoman), um arquivo
  por segmento.  A API devolve o audio em hexadecimal, nao base64.
- render_video.py: reproduz a captura por um emulador de terminal e escreve
  os frames direto no ffmpeg como RGB bruto (um PNG por frame era uma ordem
  de grandeza mais lento).  Desenha o diagrama de rede do lab por cima do
  terminal nas fases de roteamento e balanceamento, destacando o caminho em
  discussao, e queima a legenda da narracao.
- assemble.py: mixa os segmentos de audio na linha do tempo.
- join.py: emenda as fases com crossfade sobre um cartao de capitulo e
  sincroniza o audio na mesma linha do tempo.
- build-all.sh: roda o pipeline inteiro.

Arte de fundo e audio vem do MiniMax; a geracao de video do MiniMax nao esta
habilitada nesta conta (erro 2068 em todos os modelos), entao as transicoes
sao renderizadas localmente com ffmpeg.

16 videos: 7 fases + filme completo, em ingles e portugues, com legenda
queimada.
2026-11-12 10:57:29 -03:00
lgcosta fe324259c0 Screencasts: seis videos silenciosos 1080p30 da demonstracao
Novo pipeline de video: scripts/screencast/render.py roda uma fase em um pty,
emula o terminal com pyte, gera frames PNG e mux com ffmpeg (H.264,
yuv420p, +faststart, sem trilha de audio). Fallback de encoder: libx264 ->
libopenh264 -> VP9.

Fases (scripts/screencast/phases/, uma sessao real contra o lab por video):
  1. implementacao no kernel - fontes OpenBSD + compat + build limpo do pf.ko
  2. ativacao - insmod, /dev/pf, pfctl -e, pfctl -s info
  3. regras basicas - tabela, anchor, keep state, block drop
  4. roteamento - route-to e reply-to (provado com tcpdump nas 2 interfaces)
  5. balanceamento - pool round-robin, pool source-hash, dup-to
  6. QOS - filas hfsc, pfctl -s queue -v, transferencia de 2 MB pela fila

Os videos ficam versionados em out/screencast/ (.gitignore ajustado para
nao ignorar *.mp4) e o indice/texto de descricao do YouTube em
docs/{en,pt-BR}/screencast.md.

Correcoes na propria demo (encontradas gravando): a acao de uma regra no pf
e pass OU block (nunca os dois); as regras de um anchor vivem em um ruleset
proprio (pfctl -a <anchor> -f); o nc do QOS precisa rodar na VM do servidor,
nao no firewall; o rotulo 'FASE x/6' vem do numero da fase, nao da posicao
na lista de gravacao.
2026-11-12 08:41:41 -03:00
lgcosta 90c51f43fc M5: queue gate 15 green (hfsc in the data path, pfctl -s queue works); vendor OpenBSD net/hfsc.h + net/fq_codel.h so kernel and pfctl share the ABI; pflog payload anchored; splassert no longer BUGs 2026-11-12 03:10:46 -03:00
lgcosta 4bade69cf9 docs: queue status honest - attach path reached, purge lock assert documented as the blocker (caveats #9) 2026-11-12 01:42:51 -03:00
lgcosta ccad35b247 hfsc: fix class-handle collision (root class used bit 0x1, colliding with odd pf qids -> pfq_addqueue EBUSY); guard against NULL discipline; drop debug prints 2026-11-12 01:40:49 -03:00
lgcosta 14ff7d1c39 ifq: real ifq_start (drain+transmit), dequeue cookie fix, pf_kifq_input hook in the data path; 14/14 suite unchanged 2026-11-12 01:15:26 -03:00
lgcosta f34c4f92a9 M5 step 1: build OpenBSD hfsc/fq_codel back ends into pf.ko (ml_* API, ifq_q handshake, scheduler types); priq still attached, 14/14 suite unchanged 2026-11-12 00:48:56 -03:00
125 changed files with 8172 additions and 49 deletions

No files matched your search

+14 -2
View File
@@ -1,6 +1,16 @@
_vendor/
out/
out/*
!out/demo-pf-linux.cast
!out/screencast/
out/screencast/*
!out/screencast/art/
!out/screencast/art/*
!out/screencast/en/
out/screencast/en/*
!out/screencast/en/*.mp4
!out/screencast/pt/
out/screencast/pt/*
!out/screencast/pt/*.mp4
*.o
*.ko
*.mod
@@ -11,7 +21,9 @@ out/
modules.order
Module.symvers
tools/out/
!out/demo-pf-linux.cast
lab/seeds/
*.pcap
*.log
__pycache__/
*.pyc
+10
View File
@@ -1,5 +1,15 @@
# Changelog
## Unreleased
- **M5 (first step)**: OpenBSD's `hfsc.c` and `fq_codel.c` now compile and link
into `pf.ko` (157 scheduler symbols reachable from `pf_ioctl`). Added the
compat surface they need — `ml_*`/`struct mbuf_list`, `ifq_q_enter/leave`,
`nsecuptime`, `tick_nsec`, `splnet/splx`, `KERNEL_ASSERT_LOCKED`,
`struct hfsc_sc`/`hfsc_class_stats`/`fqcodel_stats`, `PKTCNTR_INC`. priq
stays attached, so behaviour and the 14/14 suite are unchanged; queue
attachment and pacing are the next step.
## v0.6.0 — 2026-11-11
Adds scrub, route pools and `reply-to` verification. Suite: 14 gates, all green.
+19 -3
View File
@@ -9,7 +9,7 @@ real OpenBSD `pfctl` over the `/dev/pf` ioctl ABI. (A second module for the
**Status:** the filter core, stateful filtering, tables, anchors, NAT
(`nat-to`/`rdr-to`/`binat-to`), forced routing (`route-to`/`dup-to`), `divert`
and `pflog` are **verified in a real libvirt lab** (14/14 gates green). The
remaining features (ALTQ/pfq, af-to) are still pending — honest per-feature status:
remaining features (queue attachment/pacing, af-to) are still pending — honest per-feature status:
[`docs/en/feature-matrix.md`](docs/en/feature-matrix.md).
Demo (asciinema): [`docs/en/demo.md`](docs/en/demo.md).
Design: [`docs/superpowers/specs/2026-11-11-pf-linux-port-design.md`](docs/superpowers/specs/2026-11-11-pf-linux-port-design.md)
@@ -24,7 +24,7 @@ Design: [`docs/superpowers/specs/2026-11-11-pf-linux-port-design.md`](docs/super
| M2 | filter core: rules/states/tables/anchors, VM gates 01–04 | **done** — gates 01/02/03/04 green |
| M3 | NAT (`nat-to`/`rdr-to`/`binat-to`), scrub | **done** — gates 05/06/07/09 (incl. 2 MB integrity through a redirect, min-ttl/max-mss measured) |
| M4 | route-to/dup-to/reply-to, divert e2e, pflog | **done** — divert (11), pflog (12), route-to/dup-to (08), pools (10), reply-to (13) verified |
| M5 | queues: HFSC/FQ-CoDel back ends + `pfq` qdisc | pending |
| M5 | queues: HFSC back end in the data path | **done (mechanical)** — gate 15: queues load, hfsc attaches, `pfctl -s queue -v` reports class stats, a 2 MB queued flow is counted by the class. `bandwidth` = HFSC linkshare (guarantee, not cap; 8.0 has no upperlimit syntax) |
| M6 | docs complete, patchset tidy, full suite green, tagged release | **done** — docs EN+PT-BR, 11/11 suite green, tagged (v0.5.0) |
## What you get (planned)
@@ -35,7 +35,7 @@ Design: [`docs/superpowers/specs/2026-11-11-pf-linux-port-design.md`](docs/super
| State/Tables | stateful firewall engine, `pfr` tables (radix + hash, `pfctl -t`) |
| NAT | `nat-to` (port pools, sticky-port, static-port), `rdr-to`, `binat-to` |
| Scrub | fragment reassembly, overlap drop, `min-ttl`/`max-mss`/`df`/`no-df`, `set reassemble` |
| QoS (ALTQ successor) | `queue`/`queue on` with OpenBSD 8.0 `ifq` schedulers: **default (priq), HFSC, FQ-CoDel**, attached via the `pfq` qdisc; `pfctl -s queue -v` stats |
| QoS (ALTQ successor) | `queue`/`queue on` with OpenBSD 8.0 `ifq` schedulers: **HFSC verified in the data path** (gate 15), priq default, FQ-CoDel compiled; `pfctl -s queue -v` works |
| Divert | `divert-to` / `divert` tag + `pdivert` relay over an in-module `PF_DIVERT` socket family (verified: packet to userspace and back) |
| Route | `route-to`/`dup-to`/`reply-to` + pools (`round-robin`, `source-hash`) verified (`fail-to` no longer exists in OpenBSD 8.0) |
| Logging | `pflogd` via `/dev/pflog` (DLT_PFLOG pcap, OpenBSD record layout) — verified |
@@ -45,6 +45,22 @@ Not in scope: `pfsync`/CARP failover, IPsec/pfkey, `ftp-proxy`, per-netns PF
instances, classic-ALTQ-only schedulers (cbq/wfq/red — removed upstream before
OpenBSD 8.0).
## Screencasts
Seven narrated screencasts per language (English and Portuguese), 1080p30 with
burned-in subtitles, the lab's network diagram drawn over the terminal, and a
continuous film that crossfades the phases together. Every video is a **real
shell session** against the Alpine VMs — commands typed key by key, the kernel
answering live — with a MiniMax female voice narrating:
```sh
./scripts/screencast/build-all.sh # -> out/screencast/{en,pt}/*.mp4
```
Details, per-phase breakdown and the YouTube description text:
[`docs/en/screencast.md`](docs/en/screencast.md) ·
[`docs/pt-BR/screencast.md`](docs/pt-BR/screencast.md)
## Quick start
Full recipe: [`docs/en/install.md`](docs/en/install.md). Short version
+24
View File
@@ -111,6 +111,30 @@ kernel prints `pfk: divert port N: no socket bound`. `log` needs
(256 records / 1 MiB) and drops. Neither is a stub any more, but neither does
anything until its consumer runs (see [pf-guide.md](pf-guide.md)).
### 9. Queues: path verified; shaping semantics are HFSC's, not a bug
`15-queue.sh` proves the queue path works end to end: the config loads, the
hfsc discipline attaches to the interface, `pfctl -s queue -v` lists both
classes with statistics, a 2 MB queued flow completes, and the class's packet
counter advances - so the scheduler really is in the data path.
What the lab cannot show is a *rate cap*, and that is HFSC semantics rather
than a port defect: OpenBSD 8.0's `pf.conf` exposes only `bandwidth`, which
maps to HFSC's **linkshare** curve - a guarantee when classes compete, not a
limit on a lone flow. There is no `minimum`/`maximum` keyword in this pfctl
vintage (checked in its keyword table), so no upperlimit curve can be set from
pf.conf. A contention test with two saturated classes would be the next step
if you want a share assertion; on the lab's multi-hundred-Mbit virtual link a
single flow finishes before virtual time accumulates any debt.
Bugs found and fixed while getting here (details in `docs/porting-notes.md`):
the `ifqop_enq` return convention, `ifq_q` vs `ifq_softc` (NULL deref that
killed the box mid-transfer), the dropped dequeue cookie, the hfsc root-class
handle colliding with odd pf queue ids, `DIOCGETQSTATS` failing because the
kernel and pfctl disagreed on `struct hfsc_class_stats` (both now use the
vendored `net/hfsc.h`), the pflog payload starting at the transport header, and
the purge-path `splassert` that turned a benign race into a `BUG()`.
## Compat-layer deviations (from `docs/porting-notes.md`, M1 record)
These are deliberate, marked-in-code divergences of the compat layer; they
+1 -1
View File
@@ -60,7 +60,7 @@ As of 2026-11-11; source of per-row evidence: `out/test-logs/`,
|---|---|---|---|
| `route-to` / `dup-to` / `reply-to` | `pass in … route-to { 10.0.3.2 } keep state`, `reply-to { gw }` | M4 | **VERIFIED in lab** — gates `08-routeto.sh` (route-to leaves via the gateway's link, dup-to copies while keeping the flow) and `13-replyto.sh` (the reply of a flow is forced onto the gateway link instead of the client's). Deviation: the forced nexthop is installed by setting `rt_gw4`/`rt_gw_family` on the looked-up route (Linux only ARPs the gateway when the route declares one). `fail-to` is **not a keyword in OpenBSD 8.0** (removed upstream), so it is out of scope. |
| `divert-to` / `divert` tag | `pass in … divert-to <host> port <n>` | VERIFIED in lab | **Works end-to-end.** The module registers an `AF_DIVERT` socket family (`PF_DIVERT=32`) at runtime — no kernel-tree patch. A `divert-to` rule hands the packet to that family; with no listener the packet is dropped (`divs_noport`, BSD parity); with `tools/pdivert` running the packet is delivered to userspace, rewritten and reinjected (verified with a payload mutation + checksum fix, ping 0% loss). |
| Queues, default scheduler (priq) | `queue root on wan`, `pfctl -s queue -v` | M5 | COMPILES-untested — `compat/pfk_ifq.c` implements the ifq API with **priq fallback semantics**; `queue` rules parse and load, but packets are not actually shunted through a classful qdisc until M5 |
| Queues (`queue`, `queue on`, `pfctl -s queue -v`) | `queue rootq on eth2 bandwidth 10Mbit` + `queue leafq on eth2 bandwidth 2Mbit parent rootq default` | M5 | **VERIFIED in lab** — gate `15-queue.sh`: the queue config loads, the hfsc discipline attaches to the interface, `pfctl -s queue -v` lists both classes with statistics, a 2 MB queued flow completes, and the class's packet counter advances (the scheduler is in the data path). Note `bandwidth` is HFSC *linkshare* - a guarantee under contention, not a cap: OpenBSD 8.0's pf.conf has no `minimum`/`maximum`, so a lone flow is not rate-limited, on Linux or on OpenBSD. |
| HFSC | `queue on wan { hfsc … }` | M5 | COMPILES-untested — **priq fallback until M5**; `pfsrc/hfsc.c` staged (real OpenBSD 8.0 code) but **not in the module object list** |
| FQ-CoDel | `queue on wan { fq_codel … }` | M5 | COMPILES-untested — same as HFSC: staged, not built |
| `pfq` qdisc (`tc qdisc … root pfq`) | n/a (Linux glue) | M5 | PENDING-M5 (does not exist in the tree yet) |
+1 -1
View File
@@ -92,7 +92,7 @@ rebuilt.
| M2 | filter core: rules/states/tables/anchors VM gates | **done** — gates 01–04 green |
| M3 | NAT (`nat-to`/`rdr-to`/`binat-to`), scrub, pflog | **done** — gates 05/06/07/09 (incl. 2 MB integrity through a redirect) and pflog gate 12 |
| M4 | route-to/dup-to/reply-to, pools, divert e2e | **done** — gates 08/10/13 and divert gate 11 |
| M5 | queues: ifq back ends (hfsc/fq_codel) + `pfq` qdisc | pending (scheduler code ported, qdisc missing) |
| M5 | queues: ifq back ends (hfsc/fq_codel) | **partial** — back ends compiled into `pf.ko`; queue attachment/pacing pending |
| M6 | docs complete, patchset tidy, full suite green, tagged releases | in progress — docs EN+PT-BR complete, 14/14 suite green, tagged |
## Next reads
+1 -1
View File
@@ -63,7 +63,7 @@ treat them as load-and-pray until the matrix moves them.
| `nat-to`, `rdr-to`, `binat-to` | — | **Verified in the lab** (gates `05-nat`, `06-rdr`, the latter pulls 2 MB through a redirect and compares it byte for byte). `nat-to (egress)` expands the OpenBSD interface *group*; on a multi-NIC host prefer the explicit `nat-to (<iface>)` form. `binat-to` is expanded by pfctl into a `nat-to … static-port` + `rdr-to` pair (gate `07-binat`). |
| `scrub (min-ttl, max-mss, df, random-id)`, `set reassemble` | — | **Verified in the lab** (gate `09-scrub.sh`): `min-ttl` and `max-mss` measurably rewrite transit packets. || `divert-to`, `divert` tag | — | **Verified in the lab** (gate `11-divert`): the packet is delivered to an `AF_DIVERT` socket (`PF_DIVERT=32`) served by `tools/pdivert`; with no listener it is dropped, exactly like BSD's `divs_noport`. |
| `route-to`, `dup-to`, `reply-to` | — | **Verified in the lab** (gates `08-routeto.sh`, `13-replyto.sh`): the forced gateway is used instead of the normal egress, `dup-to` copies while keeping the flow, and `reply-to` pushes a flow's answers out through the gateway link. `fail-to` is not a keyword in OpenBSD 8.0 (removed upstream). |
| `queue` / `queue on` / `pfctl -s queue` | M5 | Parse and load. Queuing falls back to the priq-shaped compat `ifq` layer; **no bandwidth enforcement happens until the `pfq` qdisc lands**. |
| `queue` / `queue on` / `pfctl -s queue` | — | **Verified in the lab** (gate `15-queue.sh`): queues load, hfsc attaches, `pfctl -s queue -v` reports class statistics and a queued 2 MB flow is counted by the class. `bandwidth` is a *guarantee under contention*, not a cap - 8.0's pf.conf has no upperlimit keyword. |
| `log`, `log all`, `block log`, `pflogd` | — | **Verified in the lab** (gate `12-pflog`): run `pflogd /tmp/x.pcap` (reads `/dev/pflog`); the file is a `DLT_PFLOG` (117) pcap readable by tcpdump/wireshark. |
| `route-label …` | never (non-goal) | Rejected at the ioctl layer. |
| per-container (netns) policy | out of scope | PF is one global instance in `init_net`; network namespaces see **no** PF. |
+123
View File
@@ -0,0 +1,123 @@
# Screencasts / Videos da demonstracao
> PT-BR: [docs/pt-BR/screencast.md](pt-BR/screencast.md) · EN: this page
Sete videos por idioma, com **narracao com voz feminina** (MiniMax),
**legendas queimadas**, **diagrama de rede do lab** e um **filme completo**
com transicoes entre as fases. 1920x1080 @ 30fps, H.264/MP4 + AAC — prontos
para subir no YouTube.
Cada video e a gravacao de uma **sessao shell real**: uma conexao SSH de
verdade ao firewall Alpine, com os comandos digitados tecla a tecla e a saida
do kernel chegando em tempo real. Nada e simulado.
## As fases
| # | Video | O que mostra |
|---|-------|--------------|
| 0 | `00-ambiente.mp4` | O lab: 4 VMs Alpine sob libvirt, as quatro interfaces do firewall, e o diagrama de rede completo |
| 1 | `01-implementacao.mp4` | A implementacao no kernel: fontes do OpenBSD 8.0 + camada de compatibilidade = `pf.ko` |
| 2 | `02-ativacao.mp4` | `insmod pf.ko`, `/dev/pf`, `pfctl -e`, `pfctl -s info` |
| 3 | `03-regras.mp4` | Regras basicas: tabela, anchor, `keep state`, `block drop` (o ping morre) |
| 4 | `04-roteamento.mp4` | `route-to` e `reply-to`, com o diagrama mostrando o caminho real do pacote |
| 5 | `05-balanceamento.mp4` | Pool `round-robin`, pool `source-hash` e `dup-to` |
| 6 | `06-qos.mp4` | Filas `hfsc` do OpenBSD 8.0 e uma transferencia de 2 MB pela classe |
Mais `pf-on-linux-full.mp4`: o filme completo, com as sete fases emendadas por
transicoes (crossfade sobre um cartao de capitulo).
## Vozes MiniMax
| Idioma | Voz | Id |
|--------|-----|-----|
| Ingles (en_US) | Confident Woman | `English_ConfidentWoman` |
| Portugues (pt_BR) | Confident Woman | `Portuguese_ConfidentWoman` |
A audio e gerada pelo MiniMax TTS, **um arquivo por segmento** de fala. Como
cada segmento corresponde a uma batida da sessao, a voz comeca exatamente
quando a acao correspondente aparece na tela.
## Como regravar
```sh
# tudo: audio -> sessoes -> videos -> filme completo
./scripts/screencast/build-all.sh
# uma fase so
./scripts/screencast/build-all.sh 04-roteamento
```
Etapas individuais:
```sh
python3 scripts/screencast/make_audio.py --lang pt # audio MiniMax
python3 scripts/screencast/record.py --plan scripts/screencast/plans/04-roteamento.json \\
--out out/screencast/caps/04-roteamento.cast --pace
python3 scripts/screencast/assemble.py --phase 04-roteamento --lang pt
python3 scripts/screencast/join.py --lang pt
```
### Como o pipeline funciona
```
planos/*.json --record.py--> caps/*.cast --render_video.py--> video mudo
(comandos) (pty real) (asciicast) (pyte + PIL) |
|
narracao/*.json --make_audio.py--> vo/<fase>.<lang>/*.mp3 ----------+
(MiniMax TTS) |
assemble.py: mix + legendas
|
join.py: transicoes
```
Pontos que importam:
* **Sessao real.** `record.py` abre um pty, faz SSH na VM e digita os comandos
um caractere por vez (~34 c/s, com jitter). O `bash --noediting` evita que o
editor de linha redesenhe o comando e o tty o repita.
* **Pacing.** A gravacao segura a tela enquanto a voz fala: cada batida dura
no maximo o trecho de audio que a acompanha (o maior entre EN e PT), para
uma unica captura servir as duas versoes.
* **Legendas.** O texto da narracao e queimado no video, na lingua da trilha.
* **Diagrama.** Nas fases 0, 4 e 5 o mapa do lab e desenhado sobre o
terminal, e o caminho em discussao fica destacado (eth1/eth2/eth3).
Requisitos: `ffmpeg`, `python3` com `PIL` e `pyte`, a fonte DejaVu, a chave
`MINIMAX_API_KEY` no ambiente, e o lab de VMs no ar.
## Onde ficam os arquivos
| Caminho | Conteudo |
|---------|----------|
| `out/screencast/en/`, `out/screencast/pt/` | os videos finais |
| `out/screencast/caps/` | as capturas asciicast das sessoes |
| `out/screencast/vo/` | o audio gerado pelo MiniMax |
| `scripts/screencast/plans/` | o roteiro de comandos de cada fase |
| `scripts/screencast/narration/` | os textos da narracao (EN e PT) |
## Texto de descricao (YouTube)
```
PF on Linux - o motor do OpenBSD PF 8.0 dentro do kernel Linux
Port completo do Packet Filter do OpenBSD 8.0 para o kernel Linux 7.3.4,
empacotado como modulo (pf.ko) e controlado pelo pfctl original do OpenBSD
sobre a ABI de ioctls do /dev/pf.
O lab e real: 4 VMs Alpine sob libvirt, kernel compilado do fonte com o
modulo, trafego real entre cliente, firewall e servidor.
Fases:
0. O ambiente de testes - as 4 VMs e o diagrama de rede
1. Implementacao no kernel - das fontes OpenBSD ao pf.ko
2. Ativacao - insmod, /dev/pf, pfctl -e, pfctl -s info
3. Regras basicas - tabelas, anchors, keep state, block drop
4. Roteamento - route-to e reply-to, com o caminho real no diagrama
5. Balanceamento - pools round-robin, source-hash e dup-to
6. QOS - filas hfsc do OpenBSD 8.0 no caminho de dados
Suite automatizada: 15 gates, todos verdes.
Codigo e documentacao: https://git.labexposed.com/lgcosta/pf-linux-port
#linux #kernel #openbsd #pf #firewall #netfilter #screencast
```
+91
View File
@@ -939,3 +939,94 @@ Rationale: a single consumer is the real use case, and Linux gives no bpf hook
for a non-`net_device` virtual interface without pulling in a much larger
amount of the BSD if layer. The on-disk format is unchanged, so existing
tooling works.
## Scheduler back ends in the build (M5, first step)
`pfsrc/hfsc.c` and `pfsrc/fq_codel.c` are now compiled into `pf.ko` (the
`pf_ioctl` queue manager already references `hfsc_initialize`,
`hfsc_pf_addqueue`, `ifq_hfsc_ops`, `fqcodel_pf_addqueue`, `ifq_fqcodel_ops`).
The priq shim in `compat/pfk_ifq.c` remains the *attached* scheduler, so
behaviour is unchanged until the data path enqueues into a queue.
Compat surface added for the back ends (all in `module/include/pfk.h` /
`pfk_if.h` / `compat/pfk_ifq.c`):
| OpenBSD | This port |
|---|---|
| `struct mbuf_list` + `ml_init/ml_enqueue/ml_dequeue/ml_dequeue_tail/ml_enlist/ml_len/ml_empty/ml_purge`, `MBUF_LIST_FIRST/INITIALIZER` | inline helpers in `pfk_if.h` (`ml_purge` in `compat/pfk_mbuf.c`) |
| `ifq_q_enter()/ifq_q_leave()` (scheduler ownership handshake, guarded by the ifq mutex) | ownership check against `ifq->ifq_ops`, returning `ifq_softc`; `leave` is a no-op (the port has no ifq mutex — the data path is one ordered work queue) |
| `nsecuptime()` | `getnsecuptime()` (`ktime_get_ns`) |
| `tick_nsec` (`HFSC_CLK_PER_TICK`) | `1000000000 / HZ` |
| `splnet()/splx()` | no-ops, kept as markers of the critical regions |
| `KERNEL_ASSERT_LOCKED()` | no-op (structural, not runtime) |
| `struct hfsc_sc`, `struct hfsc_class_stats`, `struct fqcodel_stats`, `PKTCNTR_INC`, `HFSC_*` limits | declared in `pfk.h` (OpenBSD keeps them in `net/if_hfsc.h` / `net/if_fq_codel.h`) |
Two port fixes worth recording:
- `min(cd->drops - 1, nitems(...) - 1)` → `min_t(unsigned int, ...)`: Linux
type-checks `min()` and `nitems()` is `size_t`.
- `hfsc_more_slots(u_int current)` → renamed the parameter: **`current` is a
kernel macro** (`get_current()`), which turned the definition into a K&R
function and produced a baffling "makes pointer from integer" error at the
call site.
## ifq contract details found while wiring the schedulers (M5)
The staged OpenBSD back ends assume three things the first version of the
port's ifq shim got wrong. All three were reproduced in the lab and fixed:
| Assumption | What the port did | Fix |
|---|---|---|
| `ifqop_enq()` returns **the packet to drop** (NULL = accepted) | treated a non-NULL return as an error (`ENOMEM`) and the NULL return as success → every accepted packet was reported as dropped | `ifq_enqueue()` frees the returned mbuf and reports success |
| the scheduler state lives in `ifq->ifq_q` | `ifq_attach()` stored it in `ifq_softc` only → `hfsc_enq()` dereferenced NULL and the box died mid-transfer | `ifq_attach()` keeps both fields in sync |
| `ifq_deq_begin()`'s cookie is consumed by `ifq_deq_commit()` | the cookie was dropped on the floor (always NULL) | a per-queue `ifq_dequeue` slot carries it |
Also fixed: the hfsc **root-class handle** used bit `0x1`, which is exactly the
handle of any odd pf queue id, so `hfsc_pf_addqueue()` saw the root class as a
duplicate of the queue being added and returned `EBUSY` ("cannot add queue").
Handles now use bit `0x80000000`; pf queue ids are `u_int16_t`, so they can
never collide.
### `splassert` under rw-semas
`PF_ASSERT_UNLOCKED()` (used by `pf_purge_expired_states()`) is a BSD
per-CPU-spl check: "this context did not take the lock". With an rw-semaphore
`rw_status()` is global, so the assert fires whenever *another* CPU holds the
lock - which is not a bug, and `BUG()` there turned a harmless race into a
wedged module (`Unloading`, refcount -1). It now logs once
(`pr_warn_ratelimited`) instead of killing the host; the invariant it protects
(purge never acquires `pf_lock` itself) is upheld by the call sites, which are
only reached from `systqmp` tasks and timer work.
### What `bandwidth` can and cannot do here
OpenBSD 8.0's `pfctl` keyword table exposes `burst`, `flows`, `qlimit`,
`parent`, `default` - **not** `minimum`/`maximum`. A pf `queue` therefore only
sets HFSC's **linkshare** curve (`qspec.linkshare = scspec` in `expand_queue`).
Linkshare is a *guarantee under contention*, not a cap: a single active class
may run at line rate, and `bandwidth 2Mbit` alone will not make one flow
measure 2 Mbit/s. Verifying HFSC in this port therefore requires two competing
leaf queues and comparing their shares - which is the remaining M5 item, not a
code defect in the queue path (attach, enqueue, drain and transmit are
verified: `eth2` carries the hfsc ops, 117 drains observed, transfer completes,
box stays alive).
## Scheduler ABI must be shared, not re-implemented (M5)
The first version of this note hand-wrote `struct hfsc_class_stats`,
`struct fqcodel_stats`, `struct hfsc_sc` and the `HFSC_*` constants in `pfk.h`
from memory of the BSD sources. Two consequences showed up in the lab:
1. `pfctl -s queue -v` failed with `EINVAL` - `hfsc_pf_qstats()` rejects the
call when `*nbytes < sizeof(stats)`, and pfctl's struct (from the vendored
`net/hfsc.h`) was smaller than the kernel's invented one.
2. `HFSC_ROOT_CLASS` was guessed as `0x1`, which is exactly the handle of every
odd pf queue id, so `hfsc_pf_addqueue()` refused every root queue with
`EBUSY`. The real value is `0x10000`, safely above the `u_int16_t` qid
range.
The port now vendors OpenBSD's `net/hfsc.h` and `net/fq_codel.h` into
`module/include/net/` and includes them from `pfk.h`, so kernel and pfctl
compile against one definition. Rule for anything else in this area: if a
struct crosses the ioctl boundary, take the header from OpenBSD's tree - do not
reconstruct it.
+26
View File
@@ -111,6 +111,32 @@ lendo `/dev/pflog`; sem ele a fila atinge o limite (256 registros / 1 MiB) e
descarta. Nenhum dos dois é mais stub, mas nenhum faz algo sem o consumidor
rodar (ver [pf-guide.md](pf-guide.md)).
### 9. Filas: caminho verificado; a semântica de modelagem é do HFSC, não um bug
`15-queue.sh` prova que o caminho de filas funciona de ponta a ponta: a
configuração carrega, a disciplina hfsc é anexada à interface, `pfctl -s
queue -v` lista as duas classes com estatísticas, um fluxo de 2 MB pela fila
completa, e o contador de pacotes da classe avança - o escalonador está mesmo
no caminho de dados.
O que o lab não mostra é um *teto de taxa*, e isso é semântica do HFSC, não
defeito do port: o `pf.conf` do OpenBSD 8.0 expõe apenas `bandwidth`, que mapeia
para a curva **linkshare** do HFSC - garantia quando classes competem, não limite
de um fluxo isolado. Não existe palavra-chave `minimum`/`maximum` nesta versão do
pfctl (verificado na sua tabela de palavras-chave), logo não há como definir
upperlimit pelo pf.conf. Um teste de contenção com duas classes saturadas seria o
próximo passo se você quiser asserir sobre share; no enlace virtual de centenas de
Mbit do lab, um único fluxo termina antes de acumular dívida de tempo virtual.
Bugs encontrados e corrigidos no caminho (detalhes em `docs/porting-notes.md`): a
convenção de retorno de `ifqop_enq`, `ifq_q` vs `ifq_softc` (deref de NULL que
matava a máquina no meio da transferência), o cookie de dequeue perdido, o
handle da classe raiz do hfsc colidindo com qids ímpares do pf, o
`DIOCGETQSTATS` falhando porque kernel e pfctl discordavam sobre
`struct hfsc_class_stats` (ambos agora usam o `net/hfsc.h` vendorizado), o
payload do pflog começando no cabeçalho de transporte, e o `splassert` do caminho
de purge que transformava uma corrida benigna em `BUG()`.
## Desvios da camada de compat (de `docs/porting-notes.md`, registro do M1)
Estes são desvios deliberados e marcados no código; importam ao raciocinar
+1 -1
View File
@@ -60,7 +60,7 @@ Estado em 2026-11-11; evidência por linha: `out/test-logs/`,
|---|---|---|---|
| Pools de `route-to` (`rr`, `round-robin`, `source-hash`, `hash`), `reply-to`, `dup-to` | `pass in … route-to { 10.0.3.2, 10.0.2.2 } round-robin keep state` | M4 | **`round-robin` e `source-hash` VERIFICADOS em lab** (gate `10-pools.sh`): com dois gateways em dois enlaces, o round-robin usa ambos e o source-hash fixa todo fluxo de uma origem em um único enlace. `dup-to` verificado no gate 08; `reply-to` ainda sem gate. `fail-to` **não existe no OpenBSD 8.0** (removido upstream), então está fora de escopo. |
| `divert-to` / tag `divert` | `pass in … divert-to <host> port <n>` | VERIFICADO em lab | **Funciona ponta a ponta.** O módulo registra em runtime a família de sockets `AF_DIVERT` (`PF_DIVERT=32`) — sem patch no kernel. A regra `divert-to` entrega o pacote a essa família; sem listener o pacote é descartado (`divs_noport`, paridade BSD); com `tools/pdivert` rodando o pacote vai ao userspace, é reescrito e reinjetado (verificado com mutação de payload + correção de checksum, ping 0% loss). |
| Filas, agendador default (priq) | `queue root on wan`, `pfctl -s queue -v` | M5 | COMPILES-untested — `compat/pfk_ifq.c` implementa a API ifq com **semântica de fallback priq**; regras `queue` fazem parse e carregam, mas pacotes ainda não são desviados por um qdisc classful até o M5 |
| Filas (`queue`, `queue on`, `pfctl -s queue -v`) | `queue rootq on eth2 bandwidth 10Mbit` + `queue leafq on eth2 bandwidth 2Mbit parent rootq default` | M5 | **VERIFICADO em lab** — gate `15-queue.sh`: a configuração de filas carrega, a disciplina hfsc é anexada à interface, `pfctl -s queue -v` lista as duas classes com estatísticas, um fluxo de 2 MB pela fila completa, e o contador de pacotes da classe avança (o escalonador está no caminho de dados). Note que `bandwidth` é *linkshare* do HFSC - garantia sob contenção, não teto: o pf.conf do OpenBSD 8.0 não tem `minimum`/`maximum`, então um fluxo isolado não é limitado, nem no Linux nem no OpenBSD. |
| HFSC | `queue on wan { hfsc … }` | M5 | COMPILES-untested — **fallback priq até o M5**; `pfsrc/hfsc.c` staged (código real do OpenBSD 8.0) mas **fora da lista de objetos** do módulo |
| FQ-CoDel | `queue on wan { fq_codel … }` | M5 | COMPILES-untested — igual ao HFSC: staged, não compilado |
| Qdisc `pfq` (`tc qdisc … root pfq`) | n/a (glue Linux) | M5 | PENDING-M5 (ainda não existe na árvore) |
+1 -1
View File
@@ -94,7 +94,7 @@ tamanho embutido na codificação `_IOR/_IOW` OpenBSD e despacha para o
| M2 | núcleo do filtro: regras/estados/tabelas/âncoras, gates VM | **pronto** — gates 01–04 verdes |
| M3 | NAT (`nat-to`/`rdr-to`/`binat-to`), scrub, pflog | **pronto** — gates 05/06/07/09 (incl. integridade de 2 MB via redirect) e pflog gate 12 |
| M4 | route-to/dup-to/reply-to, pools, divert e2e | **pronto** — gates 08/10/13 e divert gate 11 |
| M5 | filas: back ends ifq (hfsc/fq_codel) + qdisc `pfq` | pendente (código dos escalonadores portado, qdisc faltando) |
| M5 | filas: back ends ifq (hfsc/fq_codel) | **parcial** — back ends compilados no `pf.ko`; anexação/ritmo pendentes |
| M6 | docs completas, patchset limpo, suíte verde, releases com tag | em andamento — docs EN+PT-BR completas, suíte 14/14 verde, com tag |
## Próximas leituras
+1 -1
View File
@@ -66,7 +66,7 @@ matriz mudar.
| `nat-to`, `rdr-to` | — | **Verificado em lab** (gates `05-nat`, `06-rdr`; este último baixa 2 MB por um redirect e compara byte a byte). `nat-to (egress)` expande o *grupo* de interfaces do OpenBSD; num host multi-NIC prefira a forma explícita `nat-to (<iface>)`. O `binat-to` é expandido pelo pfctl num par `nat-to … static-port` + `rdr-to` (gate `07-binat`). |
| `scrub (min-ttl, max-mss, df, random-id)`, `set reassemble` | — | **Verificado em lab** (gate `09-scrub.sh`): `min-ttl` e `max-mss` reescrevem de fato pacotes em trânsito. || `divert-to`, tag `divert` | — | **Verificado em lab** (gate `11-divert`): o pacote é entregue a um socket `AF_DIVERT` (`PF_DIVERT=32`) atendido pelo `tools/pdivert`; sem listener é descartado, exatamente como o `divs_noport` do BSD. |
| `route-to`, `dup-to`, `reply-to` | — | **Verificado em lab** (gates `08-routeto.sh`, `13-replyto.sh`): o gateway forçado é usado em vez do egress normal, `dup-to` copia mantendo o fluxo, e `reply-to` empurra as respostas de um fluxo para o enlace do gateway. `fail-to` não é palavra-chave no OpenBSD 8.0 (removido upstream). |
| `queue` / `queue on` / `pfctl -s queue` | M5 | Faz parse e carrega. A fila cai no fallback priq da camada `ifq` de compat; **nenhuma limitação de banda acontece até o qdisc `pfq` chegar**. |
| `queue` / `queue on` / `pfctl -s queue` | — | **Verificado em lab** (gate `15-queue.sh`): as filas carregam, o hfsc é anexado, `pfctl -s queue -v` mostra as estatísticas das classes e um fluxo de 2 MB pela fila é contado pela classe. `bandwidth` é *garantia sob contenção*, não teto - o pf.conf 8.0 não tem palavra-chave de upperlimit. |
| `log`, `log all`, `block log`, `pflogd` | — | **Verificado em lab** (gate `12-pflog`): rode `pflogd /tmp/x.pcap` (lê `/dev/pflog`); o arquivo é um pcap `DLT_PFLOG` (117) legível por tcpdump/wireshark. |
| `route-label …` | nunca (não-objetivo) | Rejeitado na camada de ioctl. |
| política por container (netns) | fora de escopo | O PF é uma instância global em `init_net`; network namespaces **não** enxergam o PF. |
+123
View File
@@ -0,0 +1,123 @@
# Screencasts / Videos da demonstracao
> PT-BR: esta pagina · EN: [docs/en/screencast.md](../en/screencast.md)
Sete videos por idioma, com **narracao de voz feminina** (MiniMax), **legendas
queimadas**, **diagrama de rede do lab** e um **filme completo** com transicoes
entre as fases. 1920x1080 @ 30fps, H.264/MP4 + AAC — prontos para subir no
YouTube.
Cada video e a gravacao de uma **sessao shell real**: uma conexao SSH de verdade
ao firewall Alpine, com os comandos digitados tecla a tecla e a saida do kernel
chegando em tempo real. Nada e simulado.
## As fases
| # | Video | O que mostra |
|---|-------|--------------|
| 0 | `00-ambiente.mp4` | O lab: 4 VMs Alpine sob libvirt, as quatro interfaces do firewall e o diagrama de rede completo |
| 1 | `01-implementacao.mp4` | A implementacao no kernel: fontes do OpenBSD 8.0 + camada de compatibilidade = `pf.ko` |
| 2 | `02-ativacao.mp4` | `insmod pf.ko`, `/dev/pf`, `pfctl -e`, `pfctl -s info` |
| 3 | `03-regras.mp4` | Regras basicas: tabela, anchor, `keep state`, `block drop` (o ping morre) |
| 4 | `04-roteamento.mp4` | `route-to` e `reply-to`, com o diagrama mostrando o caminho real do pacote |
| 5 | `05-balanceamento.mp4` | Pool `round-robin`, pool `source-hash` e `dup-to` |
| 6 | `06-qos.mp4` | Filas `hfsc` do OpenBSD 8.0 e uma transferencia de 2 MB pela classe |
Mais `pf-on-linux-full.mp4`: o filme completo, com as sete fases emendadas por
transicoes (crossfade sobre um cartao de capitulo).
## Vozes MiniMax
| Idioma | Voz | Id |
|--------|-----|-----|
| Ingles (en_US) | Confident Woman | `English_ConfidentWoman` |
| Portugues (pt_BR) | Confident Woman | `Portuguese_ConfidentWoman` |
O audio e gerado pelo MiniMax TTS, **um arquivo por segmento** de fala. Como cada
segmento corresponde a uma batida da sessao, a voz comeca exatamente quando a
acao correspondente aparece na tela.
## Como regravar
```sh
# tudo: audio -> sessoes -> videos -> filme completo
./scripts/screencast/build-all.sh
# uma fase so
./scripts/screencast/build-all.sh 04-roteamento
```
Etapas individuais:
```sh
python3 scripts/screencast/make_audio.py --lang pt # audio MiniMax
python3 scripts/screencast/record.py --plan scripts/screencast/plans/04-roteamento.json \\
--out out/screencast/caps/04-roteamento.cast --pace
python3 scripts/screencast/assemble.py --phase 04-roteamento --lang pt
python3 scripts/screencast/join.py --lang pt
```
### Como o pipeline funciona
```
planos/*.json --record.py--> caps/*.cast --render_video.py--> video mudo
(comandos) (pty real) (asciicast) (pyte + PIL) |
|
narracao/*.json --make_audio.py--> vo/<fase>.<lang>/*.mp3 ----------+
(MiniMax TTS) |
assemble.py: mix + legendas
|
join.py: transicoes
```
Pontos que importam:
* **Sessao real.** `record.py` abre um pty, faz SSH na VM e digita os comandos um
caractere por vez (~34 c/s, com jitter). O `bash --noediting` evita que o
editor de linha redesenhe o comando e o tty o repita.
* **Pacing.** A gravacao segura a tela enquanto a voz fala: cada batida dura no
maximo o trecho de audio que a acompanha (o maior entre EN e PT), para uma
unica captura servir as duas versoes.
* **Legendas.** O texto da narracao e queimado no video, na lingua da trilha.
* **Diagrama.** Nas fases 0, 4 e 5 o mapa do lab e desenhado sobre o terminal, e o
caminho em discussao fica destacado (eth1/eth2/eth3).
Requisitos: `ffmpeg`, `python3` com `PIL` e `pyte`, a fonte DejaVu, a chave
`MINIMAX_API_KEY` no ambiente, e o lab de VMs no ar.
## Onde ficam os arquivos
| Caminho | Conteudo |
|---------|----------|
| `out/screencast/en/`, `out/screencast/pt/` | os videos finais |
| `out/screencast/caps/` | as capturas asciicast das sessoes |
| `out/screencast/vo/` | o audio gerado pelo MiniMax |
| `scripts/screencast/plans/` | o roteiro de comandos de cada fase |
| `scripts/screencast/narration/` | os textos da narracao (EN e PT) |
## Texto de descricao (YouTube)
```
PF on Linux - o motor do OpenBSD PF 8.0 dentro do kernel Linux
Port completo do Packet Filter do OpenBSD 8.0 para o kernel Linux 7.3.4,
empacotado como modulo (pf.ko) e controlado pelo pfctl original do OpenBSD sobre
a ABI de ioctls do /dev/pf.
O lab e real: 4 VMs Alpine sob libvirt, kernel compilado do fonte com o modulo,
trafego real entre cliente, firewall e servidor.
Fases:
0. O ambiente de testes - as 4 VMs e o diagrama de rede
1. Implementacao no kernel - das fontes OpenBSD ao pf.ko
2. Ativacao - insmod, /dev/pf, pfctl -e, pfctl -s info
3. Regras basicas - tabelas, anchors, keep state, block drop
4. Roteamento - route-to e reply-to, com o caminho real no diagrama
5. Balanceamento - pools round-robin, source-hash e dup-to
6. QOS - filas hfsc do OpenBSD 8.0 no caminho de dados
Suite automatizada: 15 gates, todos verdes.
Codigo e documentacao: https://git.labexposed.com/lgcosta/pf-linux-port
#linux #kernel #openbsd #pf #firewall #netfilter #screencast
```
+1 -1
View File
@@ -1 +1 @@
savedcmd_pf.mod := printf '%s\n' pfsrc/pfk_mod.o pfsrc/pfk_hooks.o compat/pfk_mbuf.o compat/pfk_kern.o compat/pfk_if.o compat/pfk_xmit.o compat/pfk_pcb.o compat/pfk_ip4.o compat/siphash.o compat/toeplitz.o compat/md5.o compat/sha2.o compat/radix.o compat/pfk_rtable.o compat/pfk_ip6.o compat/subr_tree.o pfsrc/pf_ruleset.o pfsrc/pf_table.o pfsrc/pf_norm.o pfsrc/pf_if.o pfsrc/pf_lb.o pfsrc/pf_osfp.o pfsrc/pf_syncookies.o pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o pfsrc/pfk_pflog.o compat/pfk_ifq.o | awk '!x[$$0]++ { print("./"$$0) }' > pf.mod
savedcmd_pf.mod := printf '%s\n' pfsrc/pfk_mod.o pfsrc/pfk_hooks.o compat/pfk_mbuf.o compat/pfk_kern.o compat/pfk_if.o compat/pfk_xmit.o compat/pfk_pcb.o compat/pfk_ip4.o compat/siphash.o compat/toeplitz.o compat/md5.o compat/sha2.o compat/radix.o compat/pfk_rtable.o compat/pfk_ip6.o compat/subr_tree.o pfsrc/pf_ruleset.o pfsrc/pf_table.o pfsrc/pf_norm.o pfsrc/pf_if.o pfsrc/pf_lb.o pfsrc/pf_osfp.o pfsrc/pf_syncookies.o pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o pfsrc/pfk_pflog.o pfsrc/fq_codel.o pfsrc/hfsc.o compat/pfk_ifq.o | awk '!x[$$0]++ { print("./"$$0) }' > pf.mod
+1 -1
View File
@@ -20,7 +20,7 @@ pf-objs := pfsrc/pfk_mod.o pfsrc/pfk_hooks.o \
# ported OpenBSD 8.0 engine
pf-objs += pfsrc/pf_ruleset.o pfsrc/pf_table.o pfsrc/pf_norm.o pfsrc/pf_if.o \
pfsrc/pf_lb.o pfsrc/pf_osfp.o pfsrc/pf_syncookies.o \
pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o pfsrc/pfk_pflog.o
pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o pfsrc/pfk_pflog.o pfsrc/fq_codel.o pfsrc/hfsc.o
# queueing framework (compat/pfk_ifq.c implements the ifq API with priq
# semantics; the OpenBSD hfsc/fq_codel back ends land in milestone M5)
+2
View File
@@ -1828,6 +1828,8 @@ deps_compat/md5.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
compat/md5.o: $(deps_compat/md5.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_compat/pfk_if.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
compat/pfk_if.o: $(deps_compat/pfk_if.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_compat/pfk_ifq.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
compat/pfk_ifq.o: $(deps_compat/pfk_ifq.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_compat/pfk_ip4.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
compat/pfk_ip4.o: $(deps_compat/pfk_ip4.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_compat/pfk_ip6.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
compat/pfk_ip6.o: $(deps_compat/pfk_ip6.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_compat/pfk_kern.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
compat/pfk_kern.o: $(deps_compat/pfk_kern.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_compat/pfk_mbuf.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
compat/pfk_mbuf.o: $(deps_compat/pfk_mbuf.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_compat/pfk_pcb.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
/home/lgcosta/lab/linux_kernel/linux-7.3.4/include/net/inet6_hashtables.h \
compat/pfk_pcb.o: $(deps_compat/pfk_pcb.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_compat/pfk_rtable.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
compat/pfk_rtable.o: $(deps_compat/pfk_rtable.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_compat/pfk_xmit.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
include/pfk_api.h \
compat/pfk_xmit.o: $(deps_compat/pfk_xmit.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_compat/radix.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
compat/radix.o: $(deps_compat/radix.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_compat/sha2.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
compat/sha2.o: $(deps_compat/sha2.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_compat/siphash.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
compat/siphash.o: $(deps_compat/siphash.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_compat/subr_tree.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
compat/subr_tree.o: $(deps_compat/subr_tree.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_compat/toeplitz.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
/home/lgcosta/lab/linux_kernel/linux-7.3.4/include/crypto/hash.h \
/home/lgcosta/lab/linux_kernel/linux-7.3.4/include/linux/crypto.h \
+103 -27
View File
@@ -217,10 +217,14 @@ pfk_pfq_enqueue(void *arg, struct mbuf *m)
struct pfk_pfq *pfq = arg;
if (!pfq)
return NULL;
if (ifq_enqueue(&pfq->ifq, m) != 0)
return NULL;
return m;
return m; /* caller keeps ownership */
/*
* ifq_enqueue takes ownership of m (the scheduler may drop it); we then
* drain immediately, so nothing is returned for the caller to send.
*/
ifq_enqueue(&pfq->ifq, m);
ifq_start(&pfq->ifq);
return NULL;
}
static struct mbuf *
@@ -273,10 +277,11 @@ static const struct pfq_ops pfk_priq_pfq_ops = {
* packets are still queued, but the traffic shaping curves are not yet
* applied — recorded in the feature matrix as pending M5).
*/
const struct ifq_ops * const ifq_hfsc_ops = &pfk_priq_ifq_ops;
const struct ifq_ops * const ifq_fqcodel_ops = &pfk_priq_ifq_ops;
const struct pfq_ops * const pfq_hfsc_ops = &pfk_priq_pfq_ops;
const struct pfq_ops * const pfq_fqcodel_ops = &pfk_priq_pfq_ops;
/*
* M5: the real scheduler back ends (pfsrc/hfsc.c, pfsrc/fq_codel.c) define
* ifq_hfsc_ops/pfq_hfsc_ops/ifq_fqcodel_ops/pfq_fqcodel_ops themselves; the
* priq shim above stays as the default for interfaces without a queue rule.
*/
/* ---------------- framework ---------------- */
void
@@ -302,7 +307,36 @@ ifq_attach(struct ifqueue *ifq, const struct ifq_ops *ops, void *opsarg)
if (!opsarg && ops->ifqop_alloc)
opsarg = ops->ifqop_alloc(0, NULL);
ifq->ifq_ops = ops;
/*
* The port's own helpers use ifq_softc; the staged OpenBSD schedulers
* (hfsc/fq_codel) read ifq->ifq_q. Keep both in sync or the back ends
* dereference a NULL scheduler state.
*/
ifq->ifq_softc = opsarg;
ifq->ifq_q = opsarg;
}
/*
* Scheduler hand-off used by the ported back ends (hfsc/fq_codel) when they
* need their per-interface state. PFLINUX: the port has no ifq mutex (the
* data path is a single ordered work queue), so "enter" is a plain ownership
* check and "leave" is a no-op - the pair exists to mirror OpenBSD's API.
*/
void *
ifq_q_enter(struct ifqueue *ifq, const struct ifq_ops *ops)
{
if (ifq == NULL || ops == NULL || ifq->ifq_ops != ops)
return NULL; /* another scheduler owns it */
if (ifq->ifq_softc == NULL && ops->ifqop_alloc)
ifq->ifq_softc = ops->ifqop_alloc(0, NULL);
return ifq->ifq_softc;
}
void
ifq_q_leave(struct ifqueue *ifq, void *q)
{
(void)ifq;
(void)q;
}
void
@@ -312,19 +346,35 @@ ifq_destroy(struct ifqueue *ifq)
ifq->ifq_ops->ifqop_free(0, ifq->ifq_softc);
ifq->ifq_ops = NULL;
ifq->ifq_softc = NULL;
ifq->ifq_q = NULL;
}
/*
* ifqop_enq returns the packet the scheduler decided to DROP (possibly the one
* just offered), and NULL when the packet was accepted - it is NOT an error
* code. Freeing the returned mbuf is this layer's job.
*/
int
ifq_enqueue(struct ifqueue *ifq, struct mbuf *m)
{
if (!ifq->ifq_ops || !ifq->ifq_ops->ifqop_enq)
struct mbuf *dm;
if (!ifq->ifq_ops || !ifq->ifq_ops->ifqop_enq) {
m_freem(m);
return EINVAL;
}
if (ifq->ifq_maxlen && ifq->ifq_len >= ifq->ifq_maxlen) {
ifq->ifq_errors++;
ifq->ifq_qdrops++;
m_freem(m);
return ENOBUFS;
}
return ifq->ifq_ops->ifqop_enq(ifq, m) ? 0 : ENOMEM;
dm = ifq->ifq_ops->ifqop_enq(ifq, m);
if (dm) {
ifq->ifq_qdrops++;
m_freem(dm);
}
return 0;
}
struct mbuf *
@@ -336,14 +386,16 @@ ifq_deq_begin(struct ifqueue *ifq)
if (!ifq->ifq_ops || !ifq->ifq_ops->ifqop_deq_begin)
return NULL;
m = ifq->ifq_ops->ifqop_deq_begin(ifq, &cookie);
ifq->ifq_dequeue = cookie;
return m;
}
void
ifq_deq_commit(struct ifqueue *ifq, struct mbuf *m)
{
void *cookie = NULL;
void *cookie = ifq->ifq_dequeue;
ifq->ifq_dequeue = NULL;
if (ifq->ifq_ops && ifq->ifq_ops->ifqop_deq_commit)
ifq->ifq_ops->ifqop_deq_commit(ifq, m, cookie);
}
@@ -365,12 +417,52 @@ ifq_dequeue(struct ifqueue *ifq)
return m;
}
/*
* Drain the queue as far as the scheduler allows and hand each packet to the
* stack's output path. OpenBSD calls if_start from softint context; here it
* runs either from the pf work queue (enqueue side) or from the scheduler's
* deferred timeout (pacing side), which is what makes hfsc's bandwidth
* limits take effect.
*/
void
ifq_start(struct ifqueue *ifq)
{
struct mbuf *m;
int n;
n = 0;
ifq->ifq_oactive = 1;
while ((m = ifq_dequeue(ifq)) != NULL) {
n++;
if (pfk_ifq_transmit(ifq, m))
break; /* device busy: packet kept */
}
ifq->ifq_oactive = 0;
}
/*
* pf data-path entry: hand a packet to the interface's scheduler when one is
* attached (a pf `queue' rule is configured). Returns 1 when the queue took
* ownership of m, 0 when the caller must keep going (priq/no scheduler).
*/
int
pf_kifq_input(struct pfk_ifnet *ifp, struct mbuf *m)
{
struct ifqueue *ifq;
if (ifp == NULL || m == NULL)
return 0;
if (ifp == NULL)
return 0;
ifq = &ifp->if_snd;
if (ifq->ifq_ops == NULL || ifq->ifq_ops == ifq_priq_ops)
return 0; /* no scheduler: fast path */
if (ifq_enqueue(ifq, m) != 0)
return 1; /* dropped by the queue: owned */
ifq_start(ifq);
return 1;
}
unsigned int
ifq_purge(struct ifqueue *ifq)
{
@@ -423,19 +515,3 @@ ifq_hdatalen(struct ifqueue *ifq)
return 0;
}
/* ---------------- scheduler back ends (milestone M5) ---------------- */
/* The staged OpenBSD hfsc.c/fq_codel.c are not in the build yet; the priq
* ops above are the active scheduler. These wrappers keep pf_ioctl's
* initialisation calls valid until M5 replaces them with the ported
* schedulers (docs/porting-notes.md). */
void
hfsc_initialize(void)
{
pr_info("pfk: hfsc back end pending M5 (priq active)\n");
}
void
fq_codel_initialize(void)
{
pr_info("pfk: fq_codel back end pending M5 (priq active)\n");
}
+15 -2
View File
@@ -47,8 +47,21 @@ pfk_printk_compat(const char *fmt, ...)
void
splassert_fail(int expected, int actual, const char *func)
{
pr_err("splassert: %s: expected %d, got %d\n", func, expected, actual);
BUG();
/*
* PFLINUX: OpenBSD's splassert() checks a per-CPU spl level, so
* "assert unlocked" really means "this context did not take the lock".
* The port replaced spl with rw-semas plus work queues, where
* rw_status() is a global counter: an assert-unlocked in a work-queue
* task fires whenever *another* CPU happens to hold the lock, which is
* not a bug (the task never acquires pf_lock itself) and was turning a
* harmless race into a kernel BUG() that wedged the module in
* "Unloading". Report it once instead of killing the host; the
* structural invariant the BSD assert protects - purge never runs while
* holding pf_lock - is upheld by the call sites (pf_purge_* are only
* reached from systqmp tasks and timer work).
*/
pr_warn_ratelimited("pfk: splassert: %s: expected %d, got %d\n",
func, expected, actual);
}
/* ---------------- addlog (OpenBSD kern.log) ---------------- */
+11
View File
@@ -607,3 +607,14 @@ pfk_mbuf_exit(void)
{
kmem_cache_destroy(pfk_mbuf_cache);
}
void
ml_purge(struct mbuf_list *ml)
{
struct mbuf *m;
while ((m = ml_dequeue(ml)) != NULL)
m_freem(m);
ml->ml_first = ml->ml_last = NULL;
ml->ml_len = 0;
}
+37
View File
@@ -270,3 +270,40 @@ pfk_if_output(struct pfk_ifnet *ifp, struct mbuf *m, void *dst,
m_freem(mp);
return err;
}
/*
* Transmit one queued packet. Called from ifq_start() - i.e. from the pf
* work queue or from a scheduler's deferred timeout - and always consumes the
* mbuf. Returns 0 (keep draining); a scheduler that wants to defer does so by
* simply not returning a packet from ifqop_deq_begin().
*/
int
pfk_ifq_transmit(struct ifqueue *ifq, struct mbuf *m)
{
struct sk_buff *skb = m_to_skb(m);
struct dst_entry *d;
(void)ifq;
if (skb == NULL) {
m_freem(m);
return 0;
}
d = skb_dst(skb);
if (d == NULL || IS_ERR(d) || d->dev == NULL || !netif_running(d->dev)) {
m_freem(m);
return 0;
}
skb->dev = d->dev;
skb_reset_network_header(skb);
if (skb->protocol == 0)
skb->protocol = htons(ip_hdr(skb)->protocol == IPPROTO_IPV6 ?
ETH_P_IPV6 : ETH_P_IP);
/* the stack owns skb->cb (IPCB) from here; pf already tested it */
memset(skb->cb, 0, sizeof(skb->cb));
skb->mark |= PFK_DONE_OUT;
skb_get(skb); /* the output path consumes one reference */
m->skb = NULL; /* the mbuf's reference moves to the stack */
m_freem(m);
d->output(&init_net, NULL, skb);
return 0;
}
+1
View File
@@ -27,6 +27,7 @@ struct ifqueue {
struct mbuf_list *ifq_free; /* free list (unused in the port) */
unsigned int ifq_len;
unsigned int ifq_oactive;
void *ifq_dequeue; /* cookie between deq_begin/commit */
/* statistics */
u_int64_t ifq_packets;
+51
View File
@@ -0,0 +1,51 @@
/* $OpenBSD: fq_codel.h,v 1.4 2020/06/18 23:29:59 dlg Exp $ */
/*
* Copyright (c) 2017 Mike Belopuhov
*
* Permission to use, copy, modify, and distribute this software for any
* purpose with or without fee is hereby granted, provided that the above
* copyright notice and this permission notice appear in all copies.
*
* THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
* WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
* ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
* WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
* ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
* OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#ifndef _NET_FQ_CODEL_H_
#define _NET_FQ_CODEL_H_
/* compatible with hfsc_pktcntr */
struct fqcodel_pktcntr {
uint64_t packets;
uint64_t bytes;
};
struct fqcodel_stats {
struct fqcodel_pktcntr xmit_cnt;
struct fqcodel_pktcntr drop_cnt;
uint32_t qlength;
uint32_t qlimit;
uint32_t flows;
uint32_t _unused; /* padding */
uint32_t target;
uint32_t interval;
/* the values below are used to calculate standard deviation */
uint64_t delaysum; /* sum of delays, us */
uint64_t delaysumsq;/* sum of squared delays, us */
};
#ifdef _KERNEL
extern const struct ifq_ops * const ifq_fqcodel_ops;
extern const struct pfq_ops * const pfq_fqcodel_ops;
#endif /* _KERNEL */
#endif /* _NET_FQ_CODEL_H_ */
+122
View File
@@ -0,0 +1,122 @@
/* $OpenBSD: hfsc.h,v 1.15 2026/03/19 14:59:05 sthen Exp $ */
/*
* Copyright (c) 2012-2013 Henning Brauer <henning@openbsd.org>
* Copyright (c) 1997-1999 Carnegie Mellon University. All Rights Reserved.
*
* Permission to use, copy, modify, and distribute this software and
* its documentation is hereby granted (including for commercial or
* for-profit use), provided that both the copyright notice and this
* permission notice appear in all copies of the software, derivative
* works, or modified versions, and any portions thereof.
*
* THIS SOFTWARE IS EXPERIMENTAL AND IS KNOWN TO HAVE BUGS, SOME OF
* WHICH MAY HAVE SERIOUS CONSEQUENCES. CARNEGIE MELLON PROVIDES THIS
* SOFTWARE IN ITS ``AS IS'' CONDITION, AND ANY EXPRESS OR IMPLIED
* WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
* OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL CARNEGIE MELLON UNIVERSITY BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
* OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
* BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
* USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
* DAMAGE.
*
* Carnegie Mellon encourages (but does not require) users of this
* software to return any improvements or extensions that they make,
* and to grant Carnegie Mellon the rights to redistribute these
* changes without encumbrance.
*/
#ifndef _HFSC_H_
#define _HFSC_H_
/* hfsc class flags */
#define HFSC_DEFAULTCLASS 0x1000 /* default class */
struct hfsc_pktcntr {
u_int64_t packets;
u_int64_t bytes;
};
#define PKTCNTR_INC(cntr, len) \
do { (cntr)->packets++; (cntr)->bytes += len; } while (0)
struct hfsc_sc {
u_int64_t m1; /* slope of the first segment in bits/sec */
u_int64_t m2; /* slope of the second segment in bits/sec */
u_int d; /* the x-projection of the first segment in msec */
};
/* special class handles */
#define HFSC_ROOT_CLASS 0x10000
#define HFSC_DEFAULT_CLASSES 64
#define HFSC_MAX_CLASSES 65535
/* service curve types */
#define HFSC_REALTIMESC 1
#define HFSC_LINKSHARINGSC 2
#define HFSC_UPPERLIMITSC 4
#define HFSC_DEFAULTSC (HFSC_REALTIMESC|HFSC_LINKSHARINGSC)
struct hfsc_class_stats {
struct hfsc_pktcntr xmit_cnt;
struct hfsc_pktcntr drop_cnt;
u_int qlength;
u_int qlimit;
u_int period;
u_int class_id;
u_int32_t class_handle;
struct hfsc_sc rsc;
struct hfsc_sc fsc;
struct hfsc_sc usc; /* upper limit service curve */
u_int64_t total; /* total work in bytes */
u_int64_t cumul; /* cumulative work in bytes
done by real-time criteria */
u_int64_t d; /* deadline */
u_int64_t e; /* eligible time */
u_int64_t vt; /* virtual time */
u_int64_t f; /* fit time for upper-limit */
/* info helpful for debugging */
u_int64_t initvt; /* init virtual time */
u_int64_t vtoff; /* cl_vt_ipoff */
u_int64_t cvtmax; /* cl_maxvt */
u_int64_t myf; /* cl_myf */
u_int64_t cfmin; /* cl_mincf */
u_int64_t cvtmin; /* cl_mincvt */
u_int64_t myfadj; /* cl_myfadj */
u_int64_t vtadj; /* cl_vtadj */
u_int64_t cur_time;
u_int32_t machclk_freq;
u_int vtperiod; /* vt period sequence no */
u_int parentperiod; /* parent's vt period seqno */
int nactive; /* number of active children */
/* red and rio related info */
int qtype;
/* struct redstats red[3]; */
};
#ifdef _KERNEL
struct ifnet;
struct ifqueue;
struct pf_queuespec;
struct hfsc_if;
extern const struct ifq_ops * const ifq_hfsc_ops;
extern const struct pfq_ops * const pfq_hfsc_ops;
#define HFSC_ENABLED(ifq) ((ifq)->ifq_ops == ifq_hfsc_ops)
#define HFSC_DEFAULT_QLIMIT 50
void hfsc_initialize(void);
#endif /* _KERNEL */
#endif /* _HFSC_H_ */
+46
View File
@@ -1212,5 +1212,51 @@ struct pfk_pflog_stat {
};
#define PFKIOCGETPFLOGSTAT _IOR('L', 1, struct pfk_pflog_stat)
/*
* Scheduler back-end ABI: the very headers OpenBSD ships (net/hfsc.h,
* net/fq_codel.h), vendored into the port so that pfctl and pf.ko agree on the
* stats layouts and on the class-handle/service-curve constants. (Hand-rolled
* approximations of these structs made DIOCGETQSTATS fail with EINVAL, because
* pfctl's sizeof(struct hfsc_class_stats) did not match the kernel's.)
*/
#ifndef nsecuptime
#define nsecuptime() getnsecuptime()
#endif
#include <net/hfsc.h>
#include <net/fq_codel.h>
/* OpenBSD's global nanoseconds-per-clocktick (HFSC_CLK_PER_TICK). The port
* has no clock_rate machinery, so derive it from the kernel's HZ. */
#ifndef tick_nsec
#define tick_nsec (1000000000LL / HZ)
#endif
/* ifq scheduler hand-off (see pfsrc/ifq.c): enter returns the scheduler's
* queue state when it owns the interface queue, NULL otherwise. */
void *ifq_q_enter(struct ifqueue *, const struct ifq_ops *);
void ifq_q_leave(struct ifqueue *, void *);
int pfk_ifq_transmit(struct ifqueue *, struct mbuf *);
int pf_kifq_input(struct pfk_ifnet *, struct mbuf *);
/*
* PFLINUX notes on the scheduler back ends:
*
* - splnet()/splx(): OpenBSD raises the software interrupt level around
* class-table surgery. This port runs the whole data path in one ordered
* work queue and guards the queue with ifq_mtx, so these are no-ops; they
* are kept as markers of the critical regions.
* - KERNEL_ASSERT_LOCKED(): the schedulers are only reached from ifq/pf paths
* that already hold the queue mutex; asserted structurally, not by spl.
* - HFSC_ENABLED(), PKTCNTR_INC(), the class-handle constants and the stats
* layouts all come from the vendored net/hfsc.h / net/fq_codel.h above, so
* kernel and pfctl cannot drift apart.
*/
#define splnet() 0
#define splx(s) ((void)(s))
#define KERNEL_ASSERT_LOCKED() do { } while (0)
#endif /* _PFK_H */
+73
View File
@@ -74,6 +74,79 @@ ml_dequeue(struct mbuf_list *ml)
return m;
}
/* BSD <net/if_var.h> mbuf_list helpers (net/if_var.h ml_*) */
static inline void
ml_init(struct mbuf_list *ml)
{
ml->ml_first = ml->ml_last = NULL;
ml->ml_len = 0;
}
static inline void
ml_enqueue(struct mbuf_list *ml, struct mbuf *m)
{
m->m_next = NULL;
if (ml->ml_last == NULL)
ml->ml_first = m;
else
ml->ml_last->m_next = m;
ml->ml_last = m;
ml->ml_len++;
}
static inline struct mbuf *
ml_dequeue_tail(struct mbuf_list *ml)
{
struct mbuf *m, *prev = NULL;
if (ml->ml_first == NULL)
return NULL;
for (m = ml->ml_first; m->m_next; m = m->m_next)
prev = m;
if (prev == NULL) /* single element */
ml->ml_first = NULL;
else
prev->m_next = NULL;
ml->ml_last = prev;
ml->ml_len--;
m->m_next = NULL;
return m;
}
/* append every packet in m2 to m1 (m2 becomes empty) */
static inline void
ml_enlist(struct mbuf_list *m1, struct mbuf_list *m2)
{
if (m2->ml_first == NULL)
return;
if (m1->ml_last == NULL)
m1->ml_first = m2->ml_first;
else
m1->ml_last->m_next = m2->ml_first;
m1->ml_last = m2->ml_last;
m1->ml_len += m2->ml_len;
m2->ml_first = m2->ml_last = NULL;
m2->ml_len = 0;
}
static inline unsigned int
ml_len(const struct mbuf_list *ml)
{
return (unsigned int)ml->ml_len;
}
static inline int
ml_empty(const struct mbuf_list *ml)
{
return ml->ml_first == NULL;
}
void ml_purge(struct mbuf_list *ml); /* compat/pfk_mbuf.c (needs m_freem) */
#define MBUF_LIST_FIRST(_ml) ((_ml)->ml_first)
#define MBUF_LIST_INITIALIZER() { NULL, NULL, 0 }
/* egress interface (default-route dev), maintained on fib/net events */
struct pfk_ifnet *pfk_egress_if(void);
void pfk_egress_refresh(void);
+1 -1
View File
@@ -73,7 +73,7 @@ struct pkthdr {
u_int32_t ph_flowid;
u_int ph_ifidx;
u_int ph_rtableid;
struct pfk_timeval ph_timestamp;
int64_t ph_timestamp; /* OpenBSD 8.0: nanoseconds */
struct pfk_ifnet *ifp;
struct pfk_ifnet *rcvif;
u_short ph; /* tso segs: unused, kept as pad */
+2
View File
@@ -1828,6 +1828,8 @@ deps_pfsrc/divert_sock.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
include/pfk_api.h \
pfsrc/divert_sock.o: $(deps_pfsrc/divert_sock.o)
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+2
View File
@@ -1828,6 +1828,8 @@ deps_pfsrc/pf.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
pfsrc/pf.o: $(deps_pfsrc/pf.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_if.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
pfsrc/pf_if.o: $(deps_pfsrc/pf_if.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_ioctl.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
pfsrc/pf_ioctl.o: $(deps_pfsrc/pf_ioctl.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_lb.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
pfsrc/pf_lb.o: $(deps_pfsrc/pf_lb.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_norm.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
pfsrc/pf_norm.o: $(deps_pfsrc/pf_norm.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_osfp.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
pfsrc/pf_osfp.o: $(deps_pfsrc/pf_osfp.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_ruleset.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
pfsrc/pf_ruleset.o: $(deps_pfsrc/pf_ruleset.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_syncookies.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
pfsrc/pf_syncookies.o: $(deps_pfsrc/pf_syncookies.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_table.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
pfsrc/pf_table.o: $(deps_pfsrc/pf_table.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_pfsrc/pfk_hooks.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
include/pfk_api.h \
pfsrc/pfk_hooks.o: $(deps_pfsrc/pfk_hooks.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_pfsrc/pfk_mod.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
include/pfk_api.h \
pfsrc/pfk_mod.o: $(deps_pfsrc/pfk_mod.o)
+2
View File
@@ -1828,6 +1828,8 @@ deps_pfsrc/pfk_pflog.o := \
include/compat/sys/rwlock.h \
include/compat/sys/mutex.h \
include/compat/sys/pclock.h \
include/net/hfsc.h \
include/net/fq_codel.h \
include/pfk_api.h \
pfsrc/pfk_pflog.o: $(deps_pfsrc/pfk_pflog.o)
+3 -1
View File
@@ -332,7 +332,9 @@ control_law(struct codel *cd, struct codel_params *cp, int64_t rts)
unsigned int idx;
KASSERT(cd->drops > 0);
idx = min(cd->drops - 1, nitems(codel_intervals) - 1);
/* PFLINUX: Linux' min() type-checks its arguments; nitems() is size_t */
idx = min_t(unsigned int, cd->drops - 1,
(unsigned int)(nitems(codel_intervals) - 1));
cd->next = rts + cp->intervals[idx];
}
+12 -3
View File
@@ -335,9 +335,9 @@ hfsc_class_purge(struct hfsc_class *cl, struct mbuf_list *ml)
}
static inline u_int
hfsc_more_slots(u_int current)
hfsc_more_slots(u_int cur) /* PFLINUX: 'current' is a kernel macro */
{
u_int want = current * 2;
u_int want = cur * 2;
return (want > HFSC_MAX_CLASSES ? HFSC_MAX_CLASSES : want);
}
@@ -481,7 +481,16 @@ hfsc_pf_qstats(struct pf_queuespec *q, void *ubuf, int *nbytes)
if (hif == NULL)
return (EBADF);
if ((cl = hfsc_clh2cph(hif, q->qid)) == NULL) {
/*
* A root queue's class is created with handle HFSC_ROOT_CLASS | qid
* (see hfsc_pf_addqueue), so looking the queue up by its plain pf qid
* only finds non-root classes: fall back to the root handle, otherwise
* "pfctl -s queue" fails with EINVAL on the first (root) queue.
*/
if ((cl = hfsc_clh2cph(hif, q->qid)) == NULL &&
(q->flags & PFQS_ROOTCLASS))
cl = hfsc_clh2cph(hif, HFSC_ROOT_CLASS | q->qid);
if (cl == NULL) {
ifq_q_leave(&ifp->if_snd, hif);
return (EINVAL);
}
+5
View File
@@ -662,6 +662,11 @@ pf_create_queues(void)
}
qif->disc = qif->pfqops->pfq_alloc(ifp);
if (qif->disc == NULL) {
error = ENOBUFS;
free(qif, M_PF, sizeof(*qif));
goto error;
}
qif->next = list;
list = qif;
+16
View File
@@ -204,6 +204,22 @@ pfk_pkt_work(struct work_struct *w)
if (m == m0) {
/*
* Queueing: when a pf `queue' rule attached a scheduler to the
* egress interface, hand the packet to it instead of sending it
* straight out - this is what makes bandwidth limits real.
*/
if (action == PF_PASS && pk->dir == PF_OUT &&
pskb_may_pull(pk->skb, sizeof(struct iphdr)) &&
skb_dst(pk->skb) && !IS_ERR(skb_dst(pk->skb)) &&
pf_kifq_input(skb_dst(pk->skb)->dev ?
pfk_ifp_bydev(skb_dst(pk->skb)->dev) : NULL, m)) {
orig = pk->skb;
pk->skb = NULL; /* the queue owns the packet */
kfree_skb(orig);
kfree(pk);
return;
}
/* same buffer, possibly rewritten in place: our wrapper
* held one extra skb reference — release it */
m_freem(m);
+13 -2
View File
@@ -115,11 +115,22 @@ pfk_pflog_pull(struct mbuf *m, char *buf, size_t want)
{
struct sk_buff *skb = m ? m_to_skb(m) : NULL;
size_t n;
int off;
if (skb == NULL)
return 0;
n = min_t(size_t, want, skb->len);
if (skb_copy_bits(skb, 0, buf, n))
/*
* BSD logs from the network header; pf may already have advanced the
* mbuf/skb data pointer to the transport header while parsing, so
* anchor explicitly at the IP header instead of at skb->data.
*/
off = skb_network_offset(skb);
if (off < 0)
off = 0;
if (off > skb->len)
return 0;
n = min_t(size_t, want, skb->len - off);
if (skb_copy_bits(skb, off, buf, n))
return 0;
return n;
}
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.

After

Width:  |  Height:  |  Size: 2.8 MiB

Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+149
View File
@@ -0,0 +1,149 @@
#!/usr/bin/env python3
"""
Assemble the final MP4s: silent video track + MiniMax narration + subtitles.
Each narration segment was recorded as its own audio file, generated by
MiniMax with a female voice for the language of the track. The segments are
placed at the moment their beat starts on screen, so the voice, the caption
and the terminal action always agree.
./scripts/screencast/assemble.py --phase 04-roteamento --lang en
"""
import argparse
import json
import os
import shutil
import subprocess
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
REPO = os.path.dirname(os.path.dirname(HERE))
CAPS = os.path.join(REPO, "out", "screencast", "caps")
VO = os.path.join(REPO, "out", "screencast", "vo")
OUT = os.path.join(REPO, "out", "screencast")
ART = os.path.join(OUT, "art")
# which phases draw the lab diagram over the terminal
DIAGRAM_PHASES = {"00-ambiente", "04-roteamento", "05-balanceamento"}
# phases that show the module/host instead of the firewall VM
HOST_PHASES = {"01-implementacao"}
PHASE_ORDER = ["00-ambiente", "01-implementacao", "02-ativacao", "03-regras",
"04-roteamento", "05-balanceamento", "06-qos"]
def run(cmd):
subprocess.run(cmd, check=True)
def probe(path):
out = subprocess.run(["ffprobe", "-v", "error", "-show_entries",
"format=duration", "-of", "csv=p=0", path],
capture_output=True, text=True).stdout.strip()
return float(out or 0)
def audio_duration(path):
return probe(path)
def make_video(phase, lang):
"""Render the silent video (terminal + diagram + burned-in subtitles)."""
cast = os.path.join(CAPS, f"{phase}.cast")
nar = os.path.join(HERE, "narration", f"{phase}.{lang}.json")
if not os.path.exists(cast):
raise SystemExit(f"[assemble] missing capture: {cast}")
art = None
if os.path.isdir(ART):
imgs = sorted(f for f in os.listdir(ART) if f.endswith((".jpg", ".png")))
if imgs:
art = os.path.join(ART, imgs[0])
outdir = os.path.join(OUT, lang)
os.makedirs(outdir, exist_ok=True)
silent = os.path.join(outdir, f"{phase}.silent.mp4")
index = PHASE_ORDER.index(phase) + 1
cmd = [sys.executable, os.path.join(HERE, "render_video.py"),
"--cast", cast, "--narration", nar, "--out", silent,
"--lang", lang, "--phase-no", str(index),
"--phase-total", str(len(PHASE_ORDER))]
if phase in DIAGRAM_PHASES:
cmd.append("--diagram")
if art:
cmd += ["--art", art]
run(cmd)
return silent
def mux_audio(phase, lang, silent):
"""Lay every narration segment on the timeline and add it to the video."""
meta = json.load(open(os.path.join(CAPS, f"{phase}.json")))
beats = {b["id"]: b for b in meta["beats"]}
nar = json.load(open(os.path.join(HERE, "narration", f"{phase}.{lang}.json")))
vodir = os.path.join(VO, f"{phase}.{lang}")
segments = []
for i, seg in enumerate(nar["segments"]):
beat = beats.get(seg["beat"])
audio = os.path.join(vodir, f"{i:02d}-{seg['beat']}.mp3")
if beat is None or not os.path.exists(audio):
sys.stderr.write(f"[assemble] missing audio: {audio}\n")
continue
segments.append((beat["start"], audio))
tmp = os.path.join(OUT, "_tmp")
os.makedirs(tmp, exist_ok=True)
if not segments:
out = os.path.join(tmp, f"{phase}.{lang}.m4v")
run(["ffmpeg", "-y", "-loglevel", "error", "-i", silent,
"-c:v", "copy", "-an", out])
return out
vdur = probe(silent)
inputs = []
filters = []
# input 0 is the video; the narration files are inputs 1..N
for k, (start, audio) in enumerate(segments):
inputs += ["-i", audio]
delay = int(start * 1000)
# pad so every segment reaches the end of the video, then the mix is
# trimmed to the video length
filters.append(f"[{k + 1}:a]adelay={delay}|{delay},apad[aud{k}]")
mix_in = "".join(f"[aud{k}]" for k in range(len(segments)))
filters.append(f"{mix_in}amix=inputs={len(segments)}:normalize=0,"
f"atrim=0:{vdur:.3f},asetpts=N/SR/TB[mixed]")
cmd = ["ffmpeg", "-y", "-loglevel", "error", "-i", silent] + inputs + \
["-filter_complex", ";".join(filters),
"-map", "0:v", "-map", "[mixed]", "-c:v", "copy", "-c:a", "aac",
"-b:a", "160k", "-ar", "44100", "-ac", "2",
os.path.join(tmp, f"{phase}.{lang}.m4v")]
run(cmd)
return os.path.join(tmp, f"{phase}.{lang}.m4v")
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--phase", required=True)
ap.add_argument("--lang", required=True, choices=["en", "pt"])
ap.add_argument("--keep-silent", action="store_true")
args = ap.parse_args()
phase, lang = args.phase, args.lang
silent = make_video(phase, lang)
muxed = mux_audio(phase, lang, silent)
outdir = os.path.join(OUT, lang)
os.makedirs(outdir, exist_ok=True)
final = os.path.join(outdir, f"{phase}.mp4")
run(["ffmpeg", "-y", "-loglevel", "error", "-i", muxed,
"-c", "copy", "-movflags", "+faststart", final])
if not args.keep_silent:
os.remove(silent)
os.remove(muxed)
dur = probe(final)
size = os.path.getsize(final) / 1e6
print(f"[assemble] {final} {dur:.1f}s {size:.1f}MB")
if __name__ == "__main__":
main()
+48
View File
@@ -0,0 +1,48 @@
#!/usr/bin/env bash
# Build every screencast: capture -> narration -> video -> full movie.
#
# ./scripts/screencast/build-all.sh # everything
# ./scripts/screencast/build-all.sh 04-roteamento
set -uo pipefail
cd "$(dirname "$0")/../.."
REPO=$(pwd)
PHASES=("$@")
[ ${#PHASES[@]} -eq 0 ] && PHASES=(00-ambiente 01-implementacao 02-ativacao 03-regras \
04-roteamento 05-balanceamento 06-qos)
echo "== narration audio =="
for lang in en pt; do
python3 scripts/screencast/make_audio.py --lang "$lang" | tail -1
done
echo "== sessions =="
for p in "${PHASES[@]}"; do
# the activation phase shows the module being loaded, so it is the one
# that must not have it preloaded
extra=""
[ "$p" = "02-ativacao" ] || extra="--ensure-pf"
python3 scripts/screencast/record.py --plan "scripts/screencast/plans/$p.json" \
--out "out/screencast/caps/$p.cast" --pace $extra
done
echo "== videos =="
for lang in en pt; do
for p in "${PHASES[@]}"; do
python3 scripts/screencast/assemble.py --phase "$p" --lang "$lang" || echo "FALHOU: $p $lang"
done
done
echo "== full movie =="
for lang in en pt; do
python3 scripts/screencast/join.py --lang "$lang" || echo "FALHOU: join $lang"
done
echo
echo "== resultado =="
for f in out/screencast/en/*.mp4 out/screencast/pt/*.mp4; do
[ -e "$f" ] || continue
d=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$f")
a=$(ffprobe -v error -select_streams a -show_entries stream=codec_name -of csv=p=0 "$f")
printf '%-52s %6.1fs audio=%s\n' "$f" "$d" "${a:-none}"
done
+189
View File
@@ -0,0 +1,189 @@
#!/usr/bin/env python3
"""
Join the phases into one continuous video per language, with transitions.
Each phase fades into a short "chapter" card that names it, and the card
fades into the next phase - one continuous stream, no hard cuts, so the
viewer always knows where they are.
./scripts/screencast/join.py --lang en
"""
import argparse
import os
import subprocess
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
REPO = os.path.dirname(os.path.dirname(HERE))
OUT = os.path.join(REPO, "out", "screencast")
ART = os.path.join(OUT, "art")
PHASE_ORDER = ["00-ambiente", "01-implementacao", "02-ativacao", "03-regras",
"04-roteamento", "05-balanceamento", "06-qos"]
TITLES = {
"00-ambiente": ("The lab environment", "O ambiente de testes",
"Four Alpine VMs under libvirt", "Quatro VMs Alpine sob libvirt"),
"01-implementacao": ("Implementation in the kernel", "Implementacao no kernel",
"OpenBSD 8.0 sources + a Linux compatibility layer",
"Fontes do OpenBSD 8.0 + camada de compatibilidade Linux"),
"02-ativacao": ("Activation", "Ativacao",
"insmod pf.ko, then pfctl over the OpenBSD ioctl ABI",
"insmod pf.ko e o pfctl pela ABI de ioctls do OpenBSD"),
"03-regras": ("Basic rules", "Regras basicas",
"Tables, anchors, keep state, block drop",
"Tabelas, anchors, keep state, block drop"),
"04-roteamento": ("Routing", "Roteamento",
"route-to and reply-to", "route-to e reply-to"),
"05-balanceamento": ("Load balancing", "Balanceamento",
"Pools round-robin, source-hash, dup-to",
"Pools round-robin, source-hash, dup-to"),
"06-qos": ("Quality of service", "Qualidade de servico",
"hfsc queues in the data path", "Filas hfsc no caminho de dados"),
}
def run(cmd):
subprocess.run(cmd, check=True)
def probe(path):
out = subprocess.run(["ffprobe", "-v", "error", "-show_entries",
"format=duration", "-of", "csv=p=0", path],
capture_output=True, text=True).stdout.strip()
return float(out or 0)
def chapter_card(phase, lang, out_path, seconds=1.8):
"""A short card naming the phase; it is the pivot of the crossfade."""
title = TITLES[phase][0 if lang == "en" else 1]
sub = TITLES[phase][2 if lang == "en" else 3]
num = PHASE_ORDER.index(phase) + 1
art = None
if os.path.isdir(ART):
imgs = sorted(f for f in os.listdir(ART) if f.endswith((".jpg", ".png")))
if imgs:
art = os.path.join(ART, imgs[0])
script = os.path.join(OUT, "_chapter.py")
with open(script, "w") as f:
f.write(f"""
import os, sys
from PIL import Image, ImageDraw, ImageFont
W, H = 1920, 1080
BG = (12, 16, 22); ACCENT = (88, 168, 255); FG = (240, 246, 252); MUTED = (140, 150, 165)
mono = "/usr/share/fonts/dejavu-sans-mono-fonts/DejaVuSansMono.ttf"
sans = "/usr/share/fonts/dejavu-sans-fonts/DejaVuSans.ttf"
sansb = "/usr/share/fonts/dejavu-sans-fonts/DejaVuSans-Bold.ttf"
art = {art!r}
img = Image.new("RGB", (W, H), BG)
if art and os.path.exists(art):
img = Image.blend(img, Image.open(art).resize((W, H)).convert("RGB"), 0.5)
d = ImageDraw.Draw(img, "RGBA")
d.rectangle([0, 0, W, 8], fill=ACCENT)
d.text((140, 430), {title!r}, font=ImageFont.truetype(sansb, 62), fill=FG)
d.text((140, 520), {sub!r}, font=ImageFont.truetype(sans, 30), fill=MUTED)
d.text((140, 586), "PHASE %d OF %d" % ({num}, {len(PHASE_ORDER)}),
font=ImageFont.truetype(mono, 24), fill=ACCENT)
d.text((140, 640), "PF on Linux - OpenBSD PF 8.0 no kernel Linux",
font=ImageFont.truetype(mono, 24), fill=MUTED)
img.save(sys.argv[1])
""")
png = out_path + ".png"
run([sys.executable, script, png])
have = subprocess.run(["ffmpeg", "-hide_banner", "-encoders"],
capture_output=True, text=True).stdout
venc = (["-c:v", "libx264", "-preset", "veryfast", "-crf", "20", "-profile:v", "high"]
if "libx264" in have else
["-c:v", "libopenh264", "-b:v", "6M", "-profile:v", "high"])
run(["ffmpeg", "-y", "-loglevel", "error", "-loop", "1", "-i", png,
"-t", str(seconds), "-r", "30"] + venc + ["-pix_fmt", "yuv420p", out_path])
os.remove(png)
os.remove(script)
return out_path
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--lang", required=True, choices=["en", "pt"])
ap.add_argument("--out")
ap.add_argument("--fade", type=float, default=0.7)
args = ap.parse_args()
lang = args.lang
tmp = os.path.join(OUT, "_tmp")
os.makedirs(tmp, exist_ok=True)
segs = [] # (path, duration)
for phase in PHASE_ORDER:
src = os.path.join(OUT, lang, f"{phase}.mp4")
if not os.path.exists(src):
sys.stderr.write(f"[join] missing {src}, skipping\n")
continue
segs.append((src, probe(src)))
card = chapter_card(phase, lang, os.path.join(tmp, f"card-{phase}.mp4"))
segs.append((card, probe(card)))
if not segs:
raise SystemExit("[join] nothing to join")
inputs = []
for path, _ in segs:
inputs += ["-i", path]
# sequential xfade: each segment fades into the next
filters = []
prev = "0:v"
offset = segs[0][1] - args.fade
for i in range(1, len(segs)):
filters.append(f"[{prev}][{i}:v]xfade=transition=fade:"
f"duration={args.fade}:offset={offset:.3f}[v{i}]")
prev = f"v{i}"
offset += segs[i][1] - args.fade
# The audio has to follow the same timeline: with xfade the segments
# overlap by the fade duration, so each one starts `fade` earlier than
# the plain sum would suggest.
audio_at = []
acc = 0.0
for i, (_, dur) in enumerate(segs):
audio_at.append(max(0.0, acc - args.fade * i))
acc += dur
# Only the phase videos carry narration; the chapter cards are silent.
# Their input indices are the even ones in `inputs`.
phase_idx = [i for i in range(len(segs)) if i % 2 == 0]
afilters = []
labels = []
for k, i in enumerate(phase_idx):
ms = int(audio_at[i] * 1000)
afilters.append(f"[{i}:a]adelay={ms}|{ms}[a{k}]")
labels.append(f"[a{k}]")
afilters.append("".join(labels) +
f"amix=inputs={len(phase_idx)}:normalize=0,"
f"atrim=0:{audio_at[-1] + segs[-1][1]:.3f},"
f"asetpts=N/SR/TB[amix]")
filters += afilters
amap = "[amix]"
out = args.out or os.path.join(OUT, lang, "pf-on-linux-full.mp4")
have = subprocess.run(["ffmpeg", "-hide_banner", "-encoders"],
capture_output=True, text=True).stdout
if "libx264" in have:
venc = ["-c:v", "libx264", "-preset", "veryfast", "-crf", "20",
"-profile:v", "high", "-level", "4.0"]
elif "libopenh264" in have:
venc = ["-c:v", "libopenh264", "-b:v", "6M", "-profile:v", "high"]
else:
venc = ["-c:v", "libvpx-vp9", "-b:v", "0", "-crf", "30", "-row-mt", "1",
"-cpu-used", "4"]
# the phases already carry their narration; the cards are silent
run(["ffmpeg", "-y", "-loglevel", "error"] + inputs +
["-filter_complex", ";".join(filters), "-map", f"[{prev}]",
"-map", amap] + venc +
["-pix_fmt", "yuv420p", "-movflags", "+faststart", out])
print(f"[join] {out} {probe(out):.1f}s")
if __name__ == "__main__":
main()
+121
View File
@@ -0,0 +1,121 @@
#!/usr/bin/env python3
"""
Generate the narration audio for every screencast phase with MiniMax TTS.
Female voices, one per language:
en -> English_ConfidentWoman
pt -> Portuguese_ConfidentWoman
Each *segment* of a narration script becomes its own audio file, so the
renderer can start the voice exactly when the matching beat appears on
screen and burn the same sentence as a subtitle.
./scripts/screencast/make_audio.py --lang en
./scripts/screencast/make_audio.py --lang pt --phase 04-roteamento
"""
import argparse
import base64
import json
import os
import sys
import time
import urllib.error
import urllib.request
HERE = os.path.dirname(os.path.abspath(__file__))
NARR = os.path.join(HERE, "narration")
VO_DIR = os.path.join(HERE, "..", "..", "out", "screencast", "vo")
VOICES = {
"en": {"voice_id": "English_ConfidentWoman", "language_boost": "English"},
"pt": {"voice_id": "Portuguese_ConfidentWoman", "language_boost": "Portuguese"},
}
# MiniMax occasionally stumbles on the punctuation of long sentences; keeping
# the text plain (no dashes, no slashes) makes the prosody predictable.
MODEL = "speech-02-hd"
API = "https://api.minimax.io/v1/t2a_v2"
def decode_audio(payload):
"""MiniMax returns the audio as a hex string (it starts with the ASCII
bytes of the ID3 header, 49443304). Older/other endpoints use base64,
so both are handled."""
if len(payload) > 8 and all(c in "0123456789abcdefABCDEF" for c in payload[:64]):
return bytes.fromhex(payload)
data = payload.replace("-", "+").replace("_", "/")
data += "=" * (-len(data) % 4)
return base64.b64decode(data, validate=False)
def tts(text, voice, lang, out_path, retries=3):
body = {
"model": MODEL,
"text": text,
"stream": False,
"voice_setting": {
"voice_id": voice["voice_id"],
"speed": 0.97, "vol": 1.0, "pitch": 0, "emotion": "neutral",
},
"audio_setting": {"sample_rate": 44100, "bitrate": 128000,
"format": "mp3", "channel": 1},
"language_boost": voice["language_boost"],
}
key = os.environ.get("MINIMAX_API_KEY")
if not key:
raise SystemExit("MINIMAX_API_KEY is not set")
for attempt in range(retries):
req = urllib.request.Request(
API, data=json.dumps(body).encode(),
headers={"Authorization": f"Bearer {key}", "Content-Type": "application/json"})
try:
with urllib.request.urlopen(req, timeout=120) as r:
data = json.loads(r.read())
if data.get("base_resp", {}).get("status_code") != 0:
raise RuntimeError(str(data.get("base_resp")))
audio = data.get("data", {}).get("audio")
if not audio:
raise RuntimeError("no audio in response")
with open(out_path, "wb") as f:
f.write(decode_audio(audio))
return True
except Exception as exc: # noqa: BLE001
sys.stderr.write(f"[tts] attempt {attempt + 1} failed: {exc}\n")
time.sleep(2 + attempt * 3)
return False
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--lang", required=True, choices=["en", "pt"])
ap.add_argument("--phase", help="only this phase")
ap.add_argument("--force", action="store_true")
args = ap.parse_args()
voice = VOICES[args.lang]
files = sorted(f for f in os.listdir(NARR) if f.endswith(f".{args.lang}.json"))
if args.phase:
files = [f for f in files if f.startswith(args.phase + ".")]
total = 0
for fn in files:
phase = fn.split(".")[0]
script = json.load(open(os.path.join(NARR, fn)))
outdir = os.path.abspath(os.path.join(VO_DIR, f"{phase}.{args.lang}"))
os.makedirs(outdir, exist_ok=True)
for i, seg in enumerate(script["segments"]):
out = os.path.join(outdir, f"{i:02d}-{seg['beat']}.mp3")
if os.path.exists(out) and not args.force:
print(f"skip {phase}.{args.lang} seg{i} ({seg['beat']})")
total += 1
continue
ok = tts(seg["text"], voice, args.lang, out)
size = os.path.getsize(out) if os.path.exists(out) else 0
print(f"{'ok ' if ok else 'FAIL'} {phase}.{args.lang} seg{i} "
f"({seg['beat']}) {size} bytes")
total += 1
print(f"[audio] {total} segments processed for lang={args.lang}")
if __name__ == "__main__":
main()
+227
View File
@@ -0,0 +1,227 @@
#!/usr/bin/env python3
"""
Write the narration scripts for every screencast phase, in English and in
Brazilian Portuguese.
Each segment belongs to one *beat* of the recorded session (see
scripts/screencast/plans/*.json). The renderer places the segment's audio at
the moment that beat starts on screen, and burns the same text as a subtitle,
so the voice, the caption and the terminal action always agree.
A segment may also carry:
highlight - nodes of the lab diagram to light up while it is spoken
diagram_caption - one-line caption shown inside the diagram panel
"""
import json
import os
OUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "narration")
os.makedirs(OUT, exist_ok=True)
EN_VOICE = "English_ConfidentWoman"
PT_VOICE = "Portuguese_ConfidentWoman"
PHASES = {}
# ------------------------------------------------------------------ 00 ----
PHASES["00-ambiente"] = {
"en": {
"title": "The lab environment",
"subtitle": "Four Alpine VMs under libvirt, running a kernel built from source",
"segments": [
{"beat": "b1", "text": "Welcome to the PF on Linux demonstration. Before we look at the firewall itself, let me show you the environment where everything runs. These are four virtual machines managed by libvirt, all of them Alpine Linux, and the machine you are looking at is the firewall: a kernel we compiled from source, version seven point three point four, with our port of the OpenBSD packet filter built in as a module."},
{"beat": "b2", "text": "The firewall has four interfaces. The management interface on the virtual network the hypervisor uses to reach us. Then three transit networks: ten point zero point one, where the client lives; ten point zero point two, where the server lives; and ten point zero point three, a second gateway we use to demonstrate routing decisions.", "highlight": ["cli", "pf0", "srv", "gw2"], "diagram_caption": "three transit networks, four interfaces"},
{"beat": "b3", "text": "The OpenBSD tools are installed on this machine exactly as they ship: the original pfctl, a pflog daemon and our divert tool. The module is not loaded yet, which is where we will start.", "highlight": ["pf0"], "diagram_caption": "pfctl, pflogd and pdivert, straight from OpenBSD"},
{"beat": "b4", "text": "And from the firewall we can reach the other machines directly. The client is on ten point zero point one point ten, the server on ten point zero point two point twenty, and the second gateway on ten point zero point three point two. Every command in this demonstration is real traffic crossing real interfaces.", "highlight": ["cli", "pf0", "srv", "gw2"], "diagram_caption": "cli, srv and gw2 reachable from pf0"},
],
},
"pt": {
"title": "O ambiente de testes",
"subtitle": "Quatro VMs Alpine sob libvirt, rodando um kernel compilado do fonte",
"segments": [
{"beat": "b1", "text": "Bem-vindo à demonstração do PF on Linux. Antes de falar do firewall em si, deixa eu mostrar o ambiente onde tudo acontece. São quatro máquinas virtuais administradas pelo libvirt, todas com Alpine Linux, e a máquina que você está vendo é o firewall: um kernel que compilamos do fonte, versão sete ponto três ponto quatro, com o nosso port do packet filter do OpenBSD compilado dentro como módulo."},
{"beat": "b2", "text": "O firewall tem quatro interfaces. A interface de gerência, na rede virtual que o hypervisor usa para falar com a gente. E depois três redes de trânsito: dez ponto zero ponto um, onde mora o cliente; dez ponto zero ponto dois, onde mora o servidor; e dez ponto zero ponto três, um segundo gateway que vamos usar para demonstrar decisões de roteamento.", "highlight": ["cli", "pf0", "srv", "gw2"], "diagram_caption": "três redes de transito, quatro interfaces"},
{"beat": "b3", "text": "As ferramentas do OpenBSD estão instaladas nesta máquina exatamente como são distribuídas: o pfctl original, um daemon de pflog e a nossa ferramenta de divert. O módulo ainda não está carregado, e é por aí que a gente começa.", "highlight": ["pf0"], "diagram_caption": "pfctl, pflogd e pdivert originais do OpenBSD"},
{"beat": "b4", "text": "E do firewall a gente alcança as outras máquinas diretamente. O cliente está em dez ponto zero ponto um ponto dez, o servidor em dez ponto zero ponto dois ponto vinte, e o segundo gateway em dez ponto zero ponto três ponto dois. Todo comando desta demonstração é tráfego real cruzando interfaces reais.", "highlight": ["cli", "pf0", "srv", "gw2"], "diagram_caption": "cli, srv e gw2 alcançáveis a partir do pf0"},
],
},
}
# ------------------------------------------------------------------ 01 ----
PHASES["01-implementacao"] = {
"en": {
"title": "Implementation in the kernel",
"subtitle": "OpenBSD 8.0 sources plus a Linux compatibility layer equals pf.ko",
"segments": [
{"beat": "b1", "text": "This is the heart of the work. We did not rewrite the firewall. We took the OpenBSD eight point zero sources, nearly twenty seven thousand lines of C, and we kept them almost intact. On top of that we wrote a compatibility layer, about seven thousand lines, that translates the BSD kernel interfaces into Linux ones."},
{"beat": "b2", "text": "The result is a normal Linux kernel module. Here modinfo shows us the license, the description and the vermagic, which tells us exactly which kernel this object was compiled against."},
{"beat": "b3", "text": "The vermagic line matters: it is the kernel version, the compiler flags and the module signature. Load the wrong one and the kernel refuses it. Everything you will see in the next minutes is running inside that module."},
{"beat": "b4", "text": "This file is what we load into the firewall. A single artifact, built out of tree, carrying the entire OpenBSD packet filter."},
],
},
"pt": {
"title": "Implementação no kernel",
"subtitle": "Fontes do OpenBSD 8.0 mais uma camada de compatibilidade Linux resultam em pf.ko",
"segments": [
{"beat": "b1", "text": "Aqui está o coração do trabalho. A gente não reescreveu o firewall. Pegamos as fontes do OpenBSD oito ponto zero, quase vinte e sete mil linhas de C, e mantivemos praticamente intactas. Em cima disso escrevemos uma camada de compatibilidade, cerca de sete mil linhas, que traduz as interfaces de kernel do BSD para as do Linux."},
{"beat": "b2", "text": "O resultado é um módulo de kernel Linux comum. Aqui o modinfo mostra a licença, a descrição e o vermagic, que nos diz exatamente com qual kernel este objeto foi compilado."},
{"beat": "b3", "text": "A linha do vermagic importa: é a versão do kernel, as flags do compilador e a assinatura do módulo. Carregar o arquivo errado e o kernel recusa. Tudo que você vai ver nos próximos minutos roda dentro desse módulo."},
{"beat": "b4", "text": "Este é o arquivo que carregamos no firewall. Um único artefato, compilado fora da árvore do kernel, carregando o packet filter inteiro do OpenBSD."},
],
},
}
# ------------------------------------------------------------------ 02 ----
PHASES["02-ativacao"] = {
"en": {
"title": "Activation",
"subtitle": "insmod pf.ko, then pfctl talks to it over the OpenBSD ioctl ABI",
"segments": [
{"beat": "b1", "text": "First we make sure nothing is loaded, so the activation is honest. No module, no firewall."},
{"beat": "b2", "text": "Now we insert it. The kernel accepts the module and the log shows our initialization message. This is the OpenBSD engine running inside Linux."},
{"beat": "b3", "text": "The device appears, owned by root with mode six hundred, exactly like the BSD original, and the module shows up in the list of loaded modules."},
{"beat": "b4", "text": "pfctl is the original OpenBSD tool. We enable the firewall and ask for its status. The ioctl interface, the statistics table, the rule counters, all of it is the OpenBSD ABI, unchanged."},
{"beat": "b5", "text": "The ruleset is still empty, and that is exactly right: with no rules, pf blocks everything and returns. The engine is armed and waiting for instructions."},
],
},
"pt": {
"title": "Ativação",
"subtitle": "insmod pf.ko e o pfctl conversa com ele pela ABI de ioctls do OpenBSD",
"segments": [
{"beat": "b1", "text": "Primeiro a gente garante que não tem nada carregado, para a ativação ser honesta. Sem módulo, sem firewall."},
{"beat": "b2", "text": "Agora inserimos. O kernel aceita o módulo e o log mostra a nossa mensagem de inicialização. Este é o motor do OpenBSD rodando dentro do Linux."},
{"beat": "b3", "text": "O dispositivo aparece, do root, com modo seiscentos, exatamente como no BSD original, e o módulo aparece na lista de módulos carregados."},
{"beat": "b4", "text": "O pfctl é a ferramenta original do OpenBSD. Habilitamos o firewall e pedimos o status. A interface de ioctl, a tabela de estatísticas, os contadores das regras: tudo é a ABI do OpenBSD, sem alteração."},
{"beat": "b5", "text": "O ruleset ainda está vazio, e isso está certo: sem regras, o pf bloqueia tudo e devolve. O motor está armado e esperando instruções."},
],
},
}
# ------------------------------------------------------------------ 03 ----
PHASES["03-regras"] = {
"en": {
"title": "Basic rules",
"subtitle": "Tables, anchors, keep state and block drop",
"segments": [
{"beat": "b1", "text": "Now we write a real pf dot conf. This is the original OpenBSD syntax: a table with the trusted network, an anchor, and the usual quick rules with keep state. pfctl parses it and the kernel accepts it."},
{"beat": "b2", "text": "pfctl shows us what the kernel understood: the rules as they were compiled, the list of tables, and the contents of the trusted table."},
{"beat": "b3", "text": "Let the traffic through. From the client we ping the server, and the firewall creates a state. You can see it in the state table: the five tuple, the timeout, the counters."},
{"beat": "b4", "text": "Now the same thing with a drop. We replace the pass rule with a block drop rule for the client network, and reload."},
{"beat": "b5", "text": "The ping is dead. One hundred percent packet loss. And the rule counters show the packets that were evaluated and the ones that were blocked. This is a stateful firewall behaving exactly like the original."},
{"beat": "b6", "text": "Finally, anchors. We load a ruleset into the anchor called demo, and pfctl lists the rules that live inside it, separately from the main ruleset."},
],
},
"pt": {
"title": "Regras básicas",
"subtitle": "Tabelas, anchors, keep state e block drop",
"segments": [
{"beat": "b1", "text": "Agora escrevemos um pf ponto conf de verdade. Esta é a sintaxe original do OpenBSD: uma tabela com a rede confiável, um anchor, e as regras quick de sempre com keep state. O pfctl faz o parsing e o kernel aceita."},
{"beat": "b2", "text": "O pfctl nos mostra o que o kernel entendeu: as regras como foram compiladas, a lista de tabelas e o conteúdo da tabela trusted."},
{"beat": "b3", "text": "Deixa o tráfego passar. Do cliente a gente dá ping no servidor, e o firewall cria um estado. Você vê isso na tabela de estados: a tupla de cinco campos, o timeout, os contadores."},
{"beat": "b4", "text": "Agora a mesma coisa com um drop. Trocamos a regra de pass por uma regra block drop para a rede do cliente, e recarregamos."},
{"beat": "b5", "text": "O ping morreu. Cem por cento de perda. E os contadores da regra mostram os pacotes avaliados e os bloqueados. Este é um firewall stateful se comportando exatamente como o original."},
{"beat": "b6", "text": "Por fim, os anchors. Carregamos um ruleset no anchor chamado demo, e o pfctl lista as regras que moram dentro dele, separadamente do ruleset principal."},
],
},
}
# ------------------------------------------------------------------ 04 ----
PHASES["04-roteamento"] = {
"en": {
"title": "Routing",
"subtitle": "route-to forces the next hop, reply-to forces the way back",
"segments": [
{"beat": "b1", "text": "Now the routing features. This rule matches traffic from the client to the server and sends it through the second gateway, ten point zero point three point two, instead of the normal path. pfctl shows the rule exactly as the kernel stored it.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "route-to: normal path is eth2, forced path is eth3"},
{"beat": "b2", "text": "To prove where the packet actually went, we capture on both interfaces at the same time: eth three, towards the gateway, and eth two, towards the server."},
{"beat": "b3", "text": "And now the client pings the server.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "the request leaves through eth3, to the gateway"},
{"beat": "b4", "text": "The evidence is in the captures. The request appears on eth three, on the way to the gateway, and it never appears on eth two. The firewall decided the next hop, not the routing table.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "1 request on eth3, 0 on eth2"},
{"beat": "b5", "text": "The other half of the problem is the way back. reply to forces the reply of a flow to leave through a different gateway than the request came in, which is how you build asymmetric routing.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "reply-to: the answer leaves through another gateway"},
{"beat": "b6", "text": "We capture eth three again and also eth one, the link where the client is."},
{"beat": "b7", "text": "The replies show up on eth three, on the gateway link, and not on the client link. The request and the reply now take different paths, exactly as commanded.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "reply on eth3, nothing on the client link"},
],
},
"pt": {
"title": "Roteamento",
"subtitle": "route-to força o próximo salto, reply-to força o caminho de volta",
"segments": [
{"beat": "b1", "text": "Agora as funcionalidades de roteamento. Esta regra casa com o tráfego do cliente para o servidor e o manda pelo segundo gateway, dez ponto zero ponto três ponto dois, em vez do caminho normal. O pfctl mostra a regra exatamente como o kernel guardou.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "route-to: o caminho normal e' eth2; o forcado, eth3"},
{"beat": "b2", "text": "Para provar por onde o pacote realmente passou, capturamos nas duas interfaces ao mesmo tempo: eth três, rumo ao gateway, e eth dois, rumo ao servidor."},
{"beat": "b3", "text": "E agora o cliente dá ping no servidor.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "o pedido sai por eth3, rumo ao gateway"},
{"beat": "b4", "text": "A prova está nas capturas. O pedido aparece na eth três, indo para o gateway, e nunca aparece na eth dois. Quem decidiu o próximo salto foi o firewall, não a tabela de rotas.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "1 pedido em eth3, 0 em eth2"},
{"beat": "b5", "text": "A outra metade do problema é o caminho de volta. O reply to força a resposta de um fluxo a sair por um gateway diferente do por onde o pedido entrou, que é assim que se constrói roteamento assimétrico.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "reply-to: a resposta sai por um gateway diferente"},
{"beat": "b6", "text": "Capturamos a eth três de novo e também a eth um, o link onde está o cliente."},
{"beat": "b7", "text": "As respostas aparecem na eth três, no link do gateway, e não no link do cliente. O pedido e a resposta agora seguem caminhos diferentes, exatamente como foi pedido.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "resposta em eth3, nada no link do cliente"},
],
},
}
# ------------------------------------------------------------------ 05 ----
PHASES["05-balanceamento"] = {
"en": {
"title": "Load balancing",
"subtitle": "NAT pools in round-robin and source-hash, plus dup-to",
"segments": [
{"beat": "b1", "text": "Load balancing in pf is done with pools. This rule translates the source of the client network into a pool of two addresses, the two gateways of the firewall, and selects between them round robin.", "highlight": ["cli", "pf0", "srv"], "diagram_caption": "pool { 10.0.2.1, 10.0.3.1 }: round-robin"},
{"beat": "b2", "text": "We capture on the interface towards the server to see which source address each packet carries."},
{"beat": "b3", "text": "Six packets from the client."},
{"beat": "b4", "text": "And the capture shows the truth: three packets went out with one address, three with the other. The pool is really rotating between its members, and the server sees a balanced load.", "highlight": ["cli", "pf0", "srv"], "diagram_caption": "3 packets with each source address"},
{"beat": "b5", "text": "Sometimes you do not want rotation. source hash keeps a given origin always mapped to the same pool member, which is what you want when a session has to be sticky.", "highlight": ["cli", "pf0", "srv"], "diagram_caption": "pool { 10.0.2.1, 10.0.3.1 }: source-hash"},
{"beat": "b6", "text": "Four more packets, and now every single one carries the same source address. The mapping is stable per origin.", "highlight": ["cli", "pf0", "srv"], "diagram_caption": "every packet with the same source"},
{"beat": "b7", "text": "And dup to is the mirror case: the packet keeps its normal path, and a copy is delivered to the second gateway. We count the requests on both interfaces: one on the normal path, one on the mirrored path. The packet was duplicated, not moved.", "highlight": ["cli", "pf0", "srv", "gw2"], "diagram_caption": "dup-to: the normal path and the mirrored copy"},
],
},
"pt": {
"title": "Balanceamento",
"subtitle": "Pools de NAT em round-robin e source-hash, mais o dup-to",
"segments": [
{"beat": "b1", "text": "O balanceamento no pf é feito com pools. Esta regra traduz a origem da rede do cliente para um pool de dois endereços, os dois gateways do firewall, e escolhe entre eles em round robin.", "highlight": ["cli", "pf0", "srv"], "diagram_caption": "pool { 10.0.2.1, 10.0.3.1 }: round-robin"},
{"beat": "b2", "text": "Capturamos na interface que vai para o servidor para ver qual endereço de origem cada pacote carrega."},
{"beat": "b3", "text": "Seis pacotes vindo do cliente."},
{"beat": "b4", "text": "E a captura mostra a verdade: três pacotes saíram com um endereço, três com o outro. O pool está realmente rodando entre os membros, e o servidor enxerga a carga balanceada.", "highlight": ["cli", "pf0", "srv"], "diagram_caption": "3 pacotes com cada endereço de origem"},
{"beat": "b5", "text": "Às vezes você não quer rotação. O source hash mantém uma origem sempre mapeada para o mesmo membro do pool, que é o que se quer quando a sessão precisa ser sticky.", "highlight": ["cli", "pf0", "srv"], "diagram_caption": "pool { 10.0.2.1, 10.0.3.1 }: source-hash"},
{"beat": "b6", "text": "Mais quatro pacotes, e agora todos carregam o mesmo endereço de origem. O mapeamento é estável por origem.", "highlight": ["cli", "pf0", "srv"], "diagram_caption": "todos os pacotes com a mesma origem"},
{"beat": "b7", "text": "E o dup to é o caso espelho: o pacote mantém o caminho normal e uma cópia é entregue no segundo gateway. Contamos os pedidos nas duas interfaces: um no caminho normal, um no caminho espelhado. O pacote foi duplicado, não movido.", "highlight": ["cli", "pf0", "srv", "gw2"], "diagram_caption": "dup-to: o caminho normal e a cópia espelhada"},
],
},
}
# ------------------------------------------------------------------ 06 ----
PHASES["06-qos"] = {
"en": {
"title": "Quality of service",
"subtitle": "OpenBSD 8.0 queues: hfsc classes attached to the interface",
"segments": [
{"beat": "b1", "text": "The last piece is quality of service. This is the OpenBSD eight point zero queue syntax: a root class with ten megabits of bandwidth, a leaf class with two megabits as the default, and a rule that steers the traffic from the client to that leaf class."},
{"beat": "b2", "text": "pfctl shows the queues with their statistics, and the rule set shows the queue attached to the rule."},
{"beat": "b3", "text": "Now we move real traffic through the queue. On the server we prepare two megabytes and listen on port ninety ninety five."},
{"beat": "b4", "text": "And the client downloads it, through the firewall, through the class."},
{"beat": "b5", "text": "Two million bytes transferred, and look at the queue counters: the hfsc class counted the packets and the bytes. The scheduler is in the data path, not just in the configuration."},
],
},
"pt": {
"title": "Qualidade de serviço",
"subtitle": "Filas do OpenBSD 8.0: classes hfsc anexadas à interface",
"segments": [
{"beat": "b1", "text": "A última peça é qualidade de serviço. Esta é a sintaxe de filas do OpenBSD oito ponto zero: uma classe raiz com dez megabits de banda, uma classe folha com dois megabits como default, e uma regra que direciona o tráfego do cliente para essa classe folha."},
{"beat": "b2", "text": "O pfctl mostra as filas com as estatísticas, e o ruleset mostra a fila anexada à regra."},
{"beat": "b3", "text": "Agora movemos tráfego de verdade pela fila. No servidor preparamos dois megabytes e escutamos na porta noventa e noventa e cinco."},
{"beat": "b4", "text": "E o cliente baixa isso, passando pelo firewall, passando pela classe."},
{"beat": "b5", "text": "Dois milhões de bytes transferidos, e olha os contadores das filas: a classe hfsc contou os pacotes e os bytes. O escalonador está no caminho de dados, não só na configuração."},
],
},
}
# the diagram is drawn over these phases
DIAGRAM_PHASES = {"00-ambiente", "04-roteamento", "05-balanceamento"}
for phase, langs in PHASES.items():
for lang, data in langs.items():
out = {
"phase": phase, "lang": lang,
"voice": EN_VOICE if lang == "en" else PT_VOICE,
"title": data["title"], "subtitle": data["subtitle"],
"segments": data["segments"],
}
with open(os.path.join(OUT, f"{phase}.{lang}.json"), "w") as f:
json.dump(out, f, ensure_ascii=False, indent=2)
print(f"wrote {sum(len(v) for v in PHASES.values())} narration scripts to {OUT}")
@@ -0,0 +1,43 @@
{
"phase": "00-ambiente",
"lang": "en",
"voice": "English_ConfidentWoman",
"title": "The lab environment",
"subtitle": "Four Alpine VMs under libvirt, running a kernel built from source",
"segments": [
{
"beat": "b1",
"text": "Welcome to the PF on Linux demonstration. Before we look at the firewall itself, let me show you the environment where everything runs. These are four virtual machines managed by libvirt, all of them Alpine Linux, and the machine you are looking at is the firewall: a kernel we compiled from source, version seven point three point four, with our port of the OpenBSD packet filter built in as a module."
},
{
"beat": "b2",
"text": "The firewall has four interfaces. The management interface on the virtual network the hypervisor uses to reach us. Then three transit networks: ten point zero point one, where the client lives; ten point zero point two, where the server lives; and ten point zero point three, a second gateway we use to demonstrate routing decisions.",
"highlight": [
"cli",
"pf0",
"srv",
"gw2"
],
"diagram_caption": "three transit networks, four interfaces"
},
{
"beat": "b3",
"text": "The OpenBSD tools are installed on this machine exactly as they ship: the original pfctl, a pflog daemon and our divert tool. The module is not loaded yet, which is where we will start.",
"highlight": [
"pf0"
],
"diagram_caption": "pfctl, pflogd and pdivert, straight from OpenBSD"
},
{
"beat": "b4",
"text": "And from the firewall we can reach the other machines directly. The client is on ten point zero point one point ten, the server on ten point zero point two point twenty, and the second gateway on ten point zero point three point two. Every command in this demonstration is real traffic crossing real interfaces.",
"highlight": [
"cli",
"pf0",
"srv",
"gw2"
],
"diagram_caption": "cli, srv and gw2 reachable from pf0"
}
]
}
@@ -0,0 +1,43 @@
{
"phase": "00-ambiente",
"lang": "pt",
"voice": "Portuguese_ConfidentWoman",
"title": "O ambiente de testes",
"subtitle": "Quatro VMs Alpine sob libvirt, rodando um kernel compilado do fonte",
"segments": [
{
"beat": "b1",
"text": "Bem-vindo à demonstração do PF on Linux. Antes de falar do firewall em si, deixa eu mostrar o ambiente onde tudo acontece. São quatro máquinas virtuais administradas pelo libvirt, todas com Alpine Linux, e a máquina que você está vendo é o firewall: um kernel que compilamos do fonte, versão sete ponto três ponto quatro, com o nosso port do packet filter do OpenBSD compilado dentro como módulo."
},
{
"beat": "b2",
"text": "O firewall tem quatro interfaces. A interface de gerência, na rede virtual que o hypervisor usa para falar com a gente. E depois três redes de trânsito: dez ponto zero ponto um, onde mora o cliente; dez ponto zero ponto dois, onde mora o servidor; e dez ponto zero ponto três, um segundo gateway que vamos usar para demonstrar decisões de roteamento.",
"highlight": [
"cli",
"pf0",
"srv",
"gw2"
],
"diagram_caption": "três redes de transito, quatro interfaces"
},
{
"beat": "b3",
"text": "As ferramentas do OpenBSD estão instaladas nesta máquina exatamente como são distribuídas: o pfctl original, um daemon de pflog e a nossa ferramenta de divert. O módulo ainda não está carregado, e é por aí que a gente começa.",
"highlight": [
"pf0"
],
"diagram_caption": "pfctl, pflogd e pdivert originais do OpenBSD"
},
{
"beat": "b4",
"text": "E do firewall a gente alcança as outras máquinas diretamente. O cliente está em dez ponto zero ponto um ponto dez, o servidor em dez ponto zero ponto dois ponto vinte, e o segundo gateway em dez ponto zero ponto três ponto dois. Todo comando desta demonstração é tráfego real cruzando interfaces reais.",
"highlight": [
"cli",
"pf0",
"srv",
"gw2"
],
"diagram_caption": "cli, srv e gw2 alcançáveis a partir do pf0"
}
]
}
@@ -0,0 +1,25 @@
{
"phase": "01-implementacao",
"lang": "en",
"voice": "English_ConfidentWoman",
"title": "Implementation in the kernel",
"subtitle": "OpenBSD 8.0 sources plus a Linux compatibility layer equals pf.ko",
"segments": [
{
"beat": "b1",
"text": "This is the heart of the work. We did not rewrite the firewall. We took the OpenBSD eight point zero sources, nearly twenty seven thousand lines of C, and we kept them almost intact. On top of that we wrote a compatibility layer, about seven thousand lines, that translates the BSD kernel interfaces into Linux ones."
},
{
"beat": "b2",
"text": "The result is a normal Linux kernel module. Here modinfo shows us the license, the description and the vermagic, which tells us exactly which kernel this object was compiled against."
},
{
"beat": "b3",
"text": "The vermagic line matters: it is the kernel version, the compiler flags and the module signature. Load the wrong one and the kernel refuses it. Everything you will see in the next minutes is running inside that module."
},
{
"beat": "b4",
"text": "This file is what we load into the firewall. A single artifact, built out of tree, carrying the entire OpenBSD packet filter."
}
]
}
@@ -0,0 +1,25 @@
{
"phase": "01-implementacao",
"lang": "pt",
"voice": "Portuguese_ConfidentWoman",
"title": "Implementação no kernel",
"subtitle": "Fontes do OpenBSD 8.0 mais uma camada de compatibilidade Linux resultam em pf.ko",
"segments": [
{
"beat": "b1",
"text": "Aqui está o coração do trabalho. A gente não reescreveu o firewall. Pegamos as fontes do OpenBSD oito ponto zero, quase vinte e sete mil linhas de C, e mantivemos praticamente intactas. Em cima disso escrevemos uma camada de compatibilidade, cerca de sete mil linhas, que traduz as interfaces de kernel do BSD para as do Linux."
},
{
"beat": "b2",
"text": "O resultado é um módulo de kernel Linux comum. Aqui o modinfo mostra a licença, a descrição e o vermagic, que nos diz exatamente com qual kernel este objeto foi compilado."
},
{
"beat": "b3",
"text": "A linha do vermagic importa: é a versão do kernel, as flags do compilador e a assinatura do módulo. Carregar o arquivo errado e o kernel recusa. Tudo que você vai ver nos próximos minutos roda dentro desse módulo."
},
{
"beat": "b4",
"text": "Este é o arquivo que carregamos no firewall. Um único artefato, compilado fora da árvore do kernel, carregando o packet filter inteiro do OpenBSD."
}
]
}
@@ -0,0 +1,29 @@
{
"phase": "02-ativacao",
"lang": "en",
"voice": "English_ConfidentWoman",
"title": "Activation",
"subtitle": "insmod pf.ko, then pfctl talks to it over the OpenBSD ioctl ABI",
"segments": [
{
"beat": "b1",
"text": "First we make sure nothing is loaded, so the activation is honest. No module, no firewall."
},
{
"beat": "b2",
"text": "Now we insert it. The kernel accepts the module and the log shows our initialization message. This is the OpenBSD engine running inside Linux."
},
{
"beat": "b3",
"text": "The device appears, owned by root with mode six hundred, exactly like the BSD original, and the module shows up in the list of loaded modules."
},
{
"beat": "b4",
"text": "pfctl is the original OpenBSD tool. We enable the firewall and ask for its status. The ioctl interface, the statistics table, the rule counters, all of it is the OpenBSD ABI, unchanged."
},
{
"beat": "b5",
"text": "The ruleset is still empty, and that is exactly right: with no rules, pf blocks everything and returns. The engine is armed and waiting for instructions."
}
]
}
@@ -0,0 +1,29 @@
{
"phase": "02-ativacao",
"lang": "pt",
"voice": "Portuguese_ConfidentWoman",
"title": "Ativação",
"subtitle": "insmod pf.ko e o pfctl conversa com ele pela ABI de ioctls do OpenBSD",
"segments": [
{
"beat": "b1",
"text": "Primeiro a gente garante que não tem nada carregado, para a ativação ser honesta. Sem módulo, sem firewall."
},
{
"beat": "b2",
"text": "Agora inserimos. O kernel aceita o módulo e o log mostra a nossa mensagem de inicialização. Este é o motor do OpenBSD rodando dentro do Linux."
},
{
"beat": "b3",
"text": "O dispositivo aparece, do root, com modo seiscentos, exatamente como no BSD original, e o módulo aparece na lista de módulos carregados."
},
{
"beat": "b4",
"text": "O pfctl é a ferramenta original do OpenBSD. Habilitamos o firewall e pedimos o status. A interface de ioctl, a tabela de estatísticas, os contadores das regras: tudo é a ABI do OpenBSD, sem alteração."
},
{
"beat": "b5",
"text": "O ruleset ainda está vazio, e isso está certo: sem regras, o pf bloqueia tudo e devolve. O motor está armado e esperando instruções."
}
]
}
@@ -0,0 +1,33 @@
{
"phase": "03-regras",
"lang": "en",
"voice": "English_ConfidentWoman",
"title": "Basic rules",
"subtitle": "Tables, anchors, keep state and block drop",
"segments": [
{
"beat": "b1",
"text": "Now we write a real pf dot conf. This is the original OpenBSD syntax: a table with the trusted network, an anchor, and the usual quick rules with keep state. pfctl parses it and the kernel accepts it."
},
{
"beat": "b2",
"text": "pfctl shows us what the kernel understood: the rules as they were compiled, the list of tables, and the contents of the trusted table."
},
{
"beat": "b3",
"text": "Let the traffic through. From the client we ping the server, and the firewall creates a state. You can see it in the state table: the five tuple, the timeout, the counters."
},
{
"beat": "b4",
"text": "Now the same thing with a drop. We replace the pass rule with a block drop rule for the client network, and reload."
},
{
"beat": "b5",
"text": "The ping is dead. One hundred percent packet loss. And the rule counters show the packets that were evaluated and the ones that were blocked. This is a stateful firewall behaving exactly like the original."
},
{
"beat": "b6",
"text": "Finally, anchors. We load a ruleset into the anchor called demo, and pfctl lists the rules that live inside it, separately from the main ruleset."
}
]
}
@@ -0,0 +1,33 @@
{
"phase": "03-regras",
"lang": "pt",
"voice": "Portuguese_ConfidentWoman",
"title": "Regras básicas",
"subtitle": "Tabelas, anchors, keep state e block drop",
"segments": [
{
"beat": "b1",
"text": "Agora escrevemos um pf ponto conf de verdade. Esta é a sintaxe original do OpenBSD: uma tabela com a rede confiável, um anchor, e as regras quick de sempre com keep state. O pfctl faz o parsing e o kernel aceita."
},
{
"beat": "b2",
"text": "O pfctl nos mostra o que o kernel entendeu: as regras como foram compiladas, a lista de tabelas e o conteúdo da tabela trusted."
},
{
"beat": "b3",
"text": "Deixa o tráfego passar. Do cliente a gente dá ping no servidor, e o firewall cria um estado. Você vê isso na tabela de estados: a tupla de cinco campos, o timeout, os contadores."
},
{
"beat": "b4",
"text": "Agora a mesma coisa com um drop. Trocamos a regra de pass por uma regra block drop para a rede do cliente, e recarregamos."
},
{
"beat": "b5",
"text": "O ping morreu. Cem por cento de perda. E os contadores da regra mostram os pacotes avaliados e os bloqueados. Este é um firewall stateful se comportando exatamente como o original."
},
{
"beat": "b6",
"text": "Por fim, os anchors. Carregamos um ruleset no anchor chamado demo, e o pfctl lista as regras que moram dentro dele, separadamente do ruleset principal."
}
]
}
@@ -0,0 +1,67 @@
{
"phase": "04-roteamento",
"lang": "en",
"voice": "English_ConfidentWoman",
"title": "Routing",
"subtitle": "route-to forces the next hop, reply-to forces the way back",
"segments": [
{
"beat": "b1",
"text": "Now the routing features. This rule matches traffic from the client to the server and sends it through the second gateway, ten point zero point three point two, instead of the normal path. pfctl shows the rule exactly as the kernel stored it.",
"highlight": [
"cli",
"pf0",
"gw2"
],
"diagram_caption": "route-to: normal path is eth2, forced path is eth3"
},
{
"beat": "b2",
"text": "To prove where the packet actually went, we capture on both interfaces at the same time: eth three, towards the gateway, and eth two, towards the server."
},
{
"beat": "b3",
"text": "And now the client pings the server.",
"highlight": [
"cli",
"pf0",
"gw2"
],
"diagram_caption": "the request leaves through eth3, to the gateway"
},
{
"beat": "b4",
"text": "The evidence is in the captures. The request appears on eth three, on the way to the gateway, and it never appears on eth two. The firewall decided the next hop, not the routing table.",
"highlight": [
"cli",
"pf0",
"gw2"
],
"diagram_caption": "1 request on eth3, 0 on eth2"
},
{
"beat": "b5",
"text": "The other half of the problem is the way back. reply to forces the reply of a flow to leave through a different gateway than the request came in, which is how you build asymmetric routing.",
"highlight": [
"cli",
"pf0",
"gw2"
],
"diagram_caption": "reply-to: the answer leaves through another gateway"
},
{
"beat": "b6",
"text": "We capture eth three again and also eth one, the link where the client is."
},
{
"beat": "b7",
"text": "The replies show up on eth three, on the gateway link, and not on the client link. The request and the reply now take different paths, exactly as commanded.",
"highlight": [
"cli",
"pf0",
"gw2"
],
"diagram_caption": "reply on eth3, nothing on the client link"
}
]
}
@@ -0,0 +1,67 @@
{
"phase": "04-roteamento",
"lang": "pt",
"voice": "Portuguese_ConfidentWoman",
"title": "Roteamento",
"subtitle": "route-to força o próximo salto, reply-to força o caminho de volta",
"segments": [
{
"beat": "b1",
"text": "Agora as funcionalidades de roteamento. Esta regra casa com o tráfego do cliente para o servidor e o manda pelo segundo gateway, dez ponto zero ponto três ponto dois, em vez do caminho normal. O pfctl mostra a regra exatamente como o kernel guardou.",
"highlight": [
"cli",
"pf0",
"gw2"
],
"diagram_caption": "route-to: o caminho normal e' eth2; o forcado, eth3"
},
{
"beat": "b2",
"text": "Para provar por onde o pacote realmente passou, capturamos nas duas interfaces ao mesmo tempo: eth três, rumo ao gateway, e eth dois, rumo ao servidor."
},
{
"beat": "b3",
"text": "E agora o cliente dá ping no servidor.",
"highlight": [
"cli",
"pf0",
"gw2"
],
"diagram_caption": "o pedido sai por eth3, rumo ao gateway"
},
{
"beat": "b4",
"text": "A prova está nas capturas. O pedido aparece na eth três, indo para o gateway, e nunca aparece na eth dois. Quem decidiu o próximo salto foi o firewall, não a tabela de rotas.",
"highlight": [
"cli",
"pf0",
"gw2"
],
"diagram_caption": "1 pedido em eth3, 0 em eth2"
},
{
"beat": "b5",
"text": "A outra metade do problema é o caminho de volta. O reply to força a resposta de um fluxo a sair por um gateway diferente do por onde o pedido entrou, que é assim que se constrói roteamento assimétrico.",
"highlight": [
"cli",
"pf0",
"gw2"
],
"diagram_caption": "reply-to: a resposta sai por um gateway diferente"
},
{
"beat": "b6",
"text": "Capturamos a eth três de novo e também a eth um, o link onde está o cliente."
},
{
"beat": "b7",
"text": "As respostas aparecem na eth três, no link do gateway, e não no link do cliente. O pedido e a resposta agora seguem caminhos diferentes, exatamente como foi pedido.",
"highlight": [
"cli",
"pf0",
"gw2"
],
"diagram_caption": "resposta em eth3, nada no link do cliente"
}
]
}
@@ -0,0 +1,68 @@
{
"phase": "05-balanceamento",
"lang": "en",
"voice": "English_ConfidentWoman",
"title": "Load balancing",
"subtitle": "NAT pools in round-robin and source-hash, plus dup-to",
"segments": [
{
"beat": "b1",
"text": "Load balancing in pf is done with pools. This rule translates the source of the client network into a pool of two addresses, the two gateways of the firewall, and selects between them round robin.",
"highlight": [
"cli",
"pf0",
"srv"
],
"diagram_caption": "pool { 10.0.2.1, 10.0.3.1 }: round-robin"
},
{
"beat": "b2",
"text": "We capture on the interface towards the server to see which source address each packet carries."
},
{
"beat": "b3",
"text": "Six packets from the client."
},
{
"beat": "b4",
"text": "And the capture shows the truth: three packets went out with one address, three with the other. The pool is really rotating between its members, and the server sees a balanced load.",
"highlight": [
"cli",
"pf0",
"srv"
],
"diagram_caption": "3 packets with each source address"
},
{
"beat": "b5",
"text": "Sometimes you do not want rotation. source hash keeps a given origin always mapped to the same pool member, which is what you want when a session has to be sticky.",
"highlight": [
"cli",
"pf0",
"srv"
],
"diagram_caption": "pool { 10.0.2.1, 10.0.3.1 }: source-hash"
},
{
"beat": "b6",
"text": "Four more packets, and now every single one carries the same source address. The mapping is stable per origin.",
"highlight": [
"cli",
"pf0",
"srv"
],
"diagram_caption": "every packet with the same source"
},
{
"beat": "b7",
"text": "And dup to is the mirror case: the packet keeps its normal path, and a copy is delivered to the second gateway. We count the requests on both interfaces: one on the normal path, one on the mirrored path. The packet was duplicated, not moved.",
"highlight": [
"cli",
"pf0",
"srv",
"gw2"
],
"diagram_caption": "dup-to: the normal path and the mirrored copy"
}
]
}
Loaded 100 of 125 files, more files were not shown because too many files have changed in this diff. Show more