Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
cc67531602 | ||
|
|
fe324259c0 | ||
|
|
90c51f43fc | ||
|
|
4bade69cf9 | ||
|
|
ccad35b247 | ||
|
|
14ff7d1c39 | ||
|
|
f34c4f92a9 |
No files matched your search
+14
-2
@@ -1,6 +1,16 @@
|
||||
_vendor/
|
||||
out/
|
||||
out/*
|
||||
!out/demo-pf-linux.cast
|
||||
!out/screencast/
|
||||
out/screencast/*
|
||||
!out/screencast/art/
|
||||
!out/screencast/art/*
|
||||
!out/screencast/en/
|
||||
out/screencast/en/*
|
||||
!out/screencast/en/*.mp4
|
||||
!out/screencast/pt/
|
||||
out/screencast/pt/*
|
||||
!out/screencast/pt/*.mp4
|
||||
*.o
|
||||
*.ko
|
||||
*.mod
|
||||
@@ -11,7 +21,9 @@ out/
|
||||
modules.order
|
||||
Module.symvers
|
||||
tools/out/
|
||||
!out/demo-pf-linux.cast
|
||||
lab/seeds/
|
||||
*.pcap
|
||||
*.log
|
||||
|
||||
__pycache__/
|
||||
*.pyc
|
||||
@@ -1,5 +1,15 @@
|
||||
# Changelog
|
||||
|
||||
## Unreleased
|
||||
|
||||
- **M5 (first step)**: OpenBSD's `hfsc.c` and `fq_codel.c` now compile and link
|
||||
into `pf.ko` (157 scheduler symbols reachable from `pf_ioctl`). Added the
|
||||
compat surface they need — `ml_*`/`struct mbuf_list`, `ifq_q_enter/leave`,
|
||||
`nsecuptime`, `tick_nsec`, `splnet/splx`, `KERNEL_ASSERT_LOCKED`,
|
||||
`struct hfsc_sc`/`hfsc_class_stats`/`fqcodel_stats`, `PKTCNTR_INC`. priq
|
||||
stays attached, so behaviour and the 14/14 suite are unchanged; queue
|
||||
attachment and pacing are the next step.
|
||||
|
||||
## v0.6.0 — 2026-11-11
|
||||
|
||||
Adds scrub, route pools and `reply-to` verification. Suite: 14 gates, all green.
|
||||
|
||||
@@ -9,7 +9,7 @@ real OpenBSD `pfctl` over the `/dev/pf` ioctl ABI. (A second module for the
|
||||
**Status:** the filter core, stateful filtering, tables, anchors, NAT
|
||||
(`nat-to`/`rdr-to`/`binat-to`), forced routing (`route-to`/`dup-to`), `divert`
|
||||
and `pflog` are **verified in a real libvirt lab** (14/14 gates green). The
|
||||
remaining features (ALTQ/pfq, af-to) are still pending — honest per-feature status:
|
||||
remaining features (queue attachment/pacing, af-to) are still pending — honest per-feature status:
|
||||
[`docs/en/feature-matrix.md`](docs/en/feature-matrix.md).
|
||||
Demo (asciinema): [`docs/en/demo.md`](docs/en/demo.md).
|
||||
Design: [`docs/superpowers/specs/2026-11-11-pf-linux-port-design.md`](docs/superpowers/specs/2026-11-11-pf-linux-port-design.md)
|
||||
@@ -24,7 +24,7 @@ Design: [`docs/superpowers/specs/2026-11-11-pf-linux-port-design.md`](docs/super
|
||||
| M2 | filter core: rules/states/tables/anchors, VM gates 01–04 | **done** — gates 01/02/03/04 green |
|
||||
| M3 | NAT (`nat-to`/`rdr-to`/`binat-to`), scrub | **done** — gates 05/06/07/09 (incl. 2 MB integrity through a redirect, min-ttl/max-mss measured) |
|
||||
| M4 | route-to/dup-to/reply-to, divert e2e, pflog | **done** — divert (11), pflog (12), route-to/dup-to (08), pools (10), reply-to (13) verified |
|
||||
| M5 | queues: HFSC/FQ-CoDel back ends + `pfq` qdisc | pending |
|
||||
| M5 | queues: HFSC back end in the data path | **done (mechanical)** — gate 15: queues load, hfsc attaches, `pfctl -s queue -v` reports class stats, a 2 MB queued flow is counted by the class. `bandwidth` = HFSC linkshare (guarantee, not cap; 8.0 has no upperlimit syntax) |
|
||||
| M6 | docs complete, patchset tidy, full suite green, tagged release | **done** — docs EN+PT-BR, 11/11 suite green, tagged (v0.5.0) |
|
||||
|
||||
## What you get (planned)
|
||||
@@ -35,7 +35,7 @@ Design: [`docs/superpowers/specs/2026-11-11-pf-linux-port-design.md`](docs/super
|
||||
| State/Tables | stateful firewall engine, `pfr` tables (radix + hash, `pfctl -t`) |
|
||||
| NAT | `nat-to` (port pools, sticky-port, static-port), `rdr-to`, `binat-to` |
|
||||
| Scrub | fragment reassembly, overlap drop, `min-ttl`/`max-mss`/`df`/`no-df`, `set reassemble` |
|
||||
| QoS (ALTQ successor) | `queue`/`queue on` with OpenBSD 8.0 `ifq` schedulers: **default (priq), HFSC, FQ-CoDel**, attached via the `pfq` qdisc; `pfctl -s queue -v` stats |
|
||||
| QoS (ALTQ successor) | `queue`/`queue on` with OpenBSD 8.0 `ifq` schedulers: **HFSC verified in the data path** (gate 15), priq default, FQ-CoDel compiled; `pfctl -s queue -v` works |
|
||||
| Divert | `divert-to` / `divert` tag + `pdivert` relay over an in-module `PF_DIVERT` socket family (verified: packet to userspace and back) |
|
||||
| Route | `route-to`/`dup-to`/`reply-to` + pools (`round-robin`, `source-hash`) verified (`fail-to` no longer exists in OpenBSD 8.0) |
|
||||
| Logging | `pflogd` via `/dev/pflog` (DLT_PFLOG pcap, OpenBSD record layout) — verified |
|
||||
@@ -45,6 +45,22 @@ Not in scope: `pfsync`/CARP failover, IPsec/pfkey, `ftp-proxy`, per-netns PF
|
||||
instances, classic-ALTQ-only schedulers (cbq/wfq/red — removed upstream before
|
||||
OpenBSD 8.0).
|
||||
|
||||
## Screencasts
|
||||
|
||||
Seven narrated screencasts per language (English and Portuguese), 1080p30 with
|
||||
burned-in subtitles, the lab's network diagram drawn over the terminal, and a
|
||||
continuous film that crossfades the phases together. Every video is a **real
|
||||
shell session** against the Alpine VMs — commands typed key by key, the kernel
|
||||
answering live — with a MiniMax female voice narrating:
|
||||
|
||||
```sh
|
||||
./scripts/screencast/build-all.sh # -> out/screencast/{en,pt}/*.mp4
|
||||
```
|
||||
|
||||
Details, per-phase breakdown and the YouTube description text:
|
||||
[`docs/en/screencast.md`](docs/en/screencast.md) ·
|
||||
[`docs/pt-BR/screencast.md`](docs/pt-BR/screencast.md)
|
||||
|
||||
## Quick start
|
||||
|
||||
Full recipe: [`docs/en/install.md`](docs/en/install.md). Short version
|
||||
|
||||
@@ -111,6 +111,30 @@ kernel prints `pfk: divert port N: no socket bound`. `log` needs
|
||||
(256 records / 1 MiB) and drops. Neither is a stub any more, but neither does
|
||||
anything until its consumer runs (see [pf-guide.md](pf-guide.md)).
|
||||
|
||||
### 9. Queues: path verified; shaping semantics are HFSC's, not a bug
|
||||
|
||||
`15-queue.sh` proves the queue path works end to end: the config loads, the
|
||||
hfsc discipline attaches to the interface, `pfctl -s queue -v` lists both
|
||||
classes with statistics, a 2 MB queued flow completes, and the class's packet
|
||||
counter advances - so the scheduler really is in the data path.
|
||||
|
||||
What the lab cannot show is a *rate cap*, and that is HFSC semantics rather
|
||||
than a port defect: OpenBSD 8.0's `pf.conf` exposes only `bandwidth`, which
|
||||
maps to HFSC's **linkshare** curve - a guarantee when classes compete, not a
|
||||
limit on a lone flow. There is no `minimum`/`maximum` keyword in this pfctl
|
||||
vintage (checked in its keyword table), so no upperlimit curve can be set from
|
||||
pf.conf. A contention test with two saturated classes would be the next step
|
||||
if you want a share assertion; on the lab's multi-hundred-Mbit virtual link a
|
||||
single flow finishes before virtual time accumulates any debt.
|
||||
|
||||
Bugs found and fixed while getting here (details in `docs/porting-notes.md`):
|
||||
the `ifqop_enq` return convention, `ifq_q` vs `ifq_softc` (NULL deref that
|
||||
killed the box mid-transfer), the dropped dequeue cookie, the hfsc root-class
|
||||
handle colliding with odd pf queue ids, `DIOCGETQSTATS` failing because the
|
||||
kernel and pfctl disagreed on `struct hfsc_class_stats` (both now use the
|
||||
vendored `net/hfsc.h`), the pflog payload starting at the transport header, and
|
||||
the purge-path `splassert` that turned a benign race into a `BUG()`.
|
||||
|
||||
## Compat-layer deviations (from `docs/porting-notes.md`, M1 record)
|
||||
|
||||
These are deliberate, marked-in-code divergences of the compat layer; they
|
||||
|
||||
@@ -60,7 +60,7 @@ As of 2026-11-11; source of per-row evidence: `out/test-logs/`,
|
||||
|---|---|---|---|
|
||||
| `route-to` / `dup-to` / `reply-to` | `pass in … route-to { 10.0.3.2 } keep state`, `reply-to { gw }` | M4 | **VERIFIED in lab** — gates `08-routeto.sh` (route-to leaves via the gateway's link, dup-to copies while keeping the flow) and `13-replyto.sh` (the reply of a flow is forced onto the gateway link instead of the client's). Deviation: the forced nexthop is installed by setting `rt_gw4`/`rt_gw_family` on the looked-up route (Linux only ARPs the gateway when the route declares one). `fail-to` is **not a keyword in OpenBSD 8.0** (removed upstream), so it is out of scope. |
|
||||
| `divert-to` / `divert` tag | `pass in … divert-to <host> port <n>` | VERIFIED in lab | **Works end-to-end.** The module registers an `AF_DIVERT` socket family (`PF_DIVERT=32`) at runtime — no kernel-tree patch. A `divert-to` rule hands the packet to that family; with no listener the packet is dropped (`divs_noport`, BSD parity); with `tools/pdivert` running the packet is delivered to userspace, rewritten and reinjected (verified with a payload mutation + checksum fix, ping 0% loss). |
|
||||
| Queues, default scheduler (priq) | `queue root on wan`, `pfctl -s queue -v` | M5 | COMPILES-untested — `compat/pfk_ifq.c` implements the ifq API with **priq fallback semantics**; `queue` rules parse and load, but packets are not actually shunted through a classful qdisc until M5 |
|
||||
| Queues (`queue`, `queue on`, `pfctl -s queue -v`) | `queue rootq on eth2 bandwidth 10Mbit` + `queue leafq on eth2 bandwidth 2Mbit parent rootq default` | M5 | **VERIFIED in lab** — gate `15-queue.sh`: the queue config loads, the hfsc discipline attaches to the interface, `pfctl -s queue -v` lists both classes with statistics, a 2 MB queued flow completes, and the class's packet counter advances (the scheduler is in the data path). Note `bandwidth` is HFSC *linkshare* - a guarantee under contention, not a cap: OpenBSD 8.0's pf.conf has no `minimum`/`maximum`, so a lone flow is not rate-limited, on Linux or on OpenBSD. |
|
||||
| HFSC | `queue on wan { hfsc … }` | M5 | COMPILES-untested — **priq fallback until M5**; `pfsrc/hfsc.c` staged (real OpenBSD 8.0 code) but **not in the module object list** |
|
||||
| FQ-CoDel | `queue on wan { fq_codel … }` | M5 | COMPILES-untested — same as HFSC: staged, not built |
|
||||
| `pfq` qdisc (`tc qdisc … root pfq`) | n/a (Linux glue) | M5 | PENDING-M5 (does not exist in the tree yet) |
|
||||
|
||||
+1
-1
@@ -92,7 +92,7 @@ rebuilt.
|
||||
| M2 | filter core: rules/states/tables/anchors VM gates | **done** — gates 01–04 green |
|
||||
| M3 | NAT (`nat-to`/`rdr-to`/`binat-to`), scrub, pflog | **done** — gates 05/06/07/09 (incl. 2 MB integrity through a redirect) and pflog gate 12 |
|
||||
| M4 | route-to/dup-to/reply-to, pools, divert e2e | **done** — gates 08/10/13 and divert gate 11 |
|
||||
| M5 | queues: ifq back ends (hfsc/fq_codel) + `pfq` qdisc | pending (scheduler code ported, qdisc missing) |
|
||||
| M5 | queues: ifq back ends (hfsc/fq_codel) | **partial** — back ends compiled into `pf.ko`; queue attachment/pacing pending |
|
||||
| M6 | docs complete, patchset tidy, full suite green, tagged releases | in progress — docs EN+PT-BR complete, 14/14 suite green, tagged |
|
||||
|
||||
## Next reads
|
||||
|
||||
+1
-1
@@ -63,7 +63,7 @@ treat them as load-and-pray until the matrix moves them.
|
||||
| `nat-to`, `rdr-to`, `binat-to` | — | **Verified in the lab** (gates `05-nat`, `06-rdr`, the latter pulls 2 MB through a redirect and compares it byte for byte). `nat-to (egress)` expands the OpenBSD interface *group*; on a multi-NIC host prefer the explicit `nat-to (<iface>)` form. `binat-to` is expanded by pfctl into a `nat-to … static-port` + `rdr-to` pair (gate `07-binat`). |
|
||||
| `scrub (min-ttl, max-mss, df, random-id)`, `set reassemble` | — | **Verified in the lab** (gate `09-scrub.sh`): `min-ttl` and `max-mss` measurably rewrite transit packets. || `divert-to`, `divert` tag | — | **Verified in the lab** (gate `11-divert`): the packet is delivered to an `AF_DIVERT` socket (`PF_DIVERT=32`) served by `tools/pdivert`; with no listener it is dropped, exactly like BSD's `divs_noport`. |
|
||||
| `route-to`, `dup-to`, `reply-to` | — | **Verified in the lab** (gates `08-routeto.sh`, `13-replyto.sh`): the forced gateway is used instead of the normal egress, `dup-to` copies while keeping the flow, and `reply-to` pushes a flow's answers out through the gateway link. `fail-to` is not a keyword in OpenBSD 8.0 (removed upstream). |
|
||||
| `queue` / `queue on` / `pfctl -s queue` | M5 | Parse and load. Queuing falls back to the priq-shaped compat `ifq` layer; **no bandwidth enforcement happens until the `pfq` qdisc lands**. |
|
||||
| `queue` / `queue on` / `pfctl -s queue` | — | **Verified in the lab** (gate `15-queue.sh`): queues load, hfsc attaches, `pfctl -s queue -v` reports class statistics and a queued 2 MB flow is counted by the class. `bandwidth` is a *guarantee under contention*, not a cap - 8.0's pf.conf has no upperlimit keyword. |
|
||||
| `log`, `log all`, `block log`, `pflogd` | — | **Verified in the lab** (gate `12-pflog`): run `pflogd /tmp/x.pcap` (reads `/dev/pflog`); the file is a `DLT_PFLOG` (117) pcap readable by tcpdump/wireshark. |
|
||||
| `route-label …` | never (non-goal) | Rejected at the ioctl layer. |
|
||||
| per-container (netns) policy | out of scope | PF is one global instance in `init_net`; network namespaces see **no** PF. |
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
# Screencasts / Videos da demonstracao
|
||||
|
||||
> PT-BR: [docs/pt-BR/screencast.md](pt-BR/screencast.md) · EN: this page
|
||||
|
||||
Sete videos por idioma, com **narracao com voz feminina** (MiniMax),
|
||||
**legendas queimadas**, **diagrama de rede do lab** e um **filme completo**
|
||||
com transicoes entre as fases. 1920x1080 @ 30fps, H.264/MP4 + AAC — prontos
|
||||
para subir no YouTube.
|
||||
|
||||
Cada video e a gravacao de uma **sessao shell real**: uma conexao SSH de
|
||||
verdade ao firewall Alpine, com os comandos digitados tecla a tecla e a saida
|
||||
do kernel chegando em tempo real. Nada e simulado.
|
||||
|
||||
## As fases
|
||||
|
||||
| # | Video | O que mostra |
|
||||
|---|-------|--------------|
|
||||
| 0 | `00-ambiente.mp4` | O lab: 4 VMs Alpine sob libvirt, as quatro interfaces do firewall, e o diagrama de rede completo |
|
||||
| 1 | `01-implementacao.mp4` | A implementacao no kernel: fontes do OpenBSD 8.0 + camada de compatibilidade = `pf.ko` |
|
||||
| 2 | `02-ativacao.mp4` | `insmod pf.ko`, `/dev/pf`, `pfctl -e`, `pfctl -s info` |
|
||||
| 3 | `03-regras.mp4` | Regras basicas: tabela, anchor, `keep state`, `block drop` (o ping morre) |
|
||||
| 4 | `04-roteamento.mp4` | `route-to` e `reply-to`, com o diagrama mostrando o caminho real do pacote |
|
||||
| 5 | `05-balanceamento.mp4` | Pool `round-robin`, pool `source-hash` e `dup-to` |
|
||||
| 6 | `06-qos.mp4` | Filas `hfsc` do OpenBSD 8.0 e uma transferencia de 2 MB pela classe |
|
||||
|
||||
Mais `pf-on-linux-full.mp4`: o filme completo, com as sete fases emendadas por
|
||||
transicoes (crossfade sobre um cartao de capitulo).
|
||||
|
||||
## Vozes MiniMax
|
||||
|
||||
| Idioma | Voz | Id |
|
||||
|--------|-----|-----|
|
||||
| Ingles (en_US) | Confident Woman | `English_ConfidentWoman` |
|
||||
| Portugues (pt_BR) | Confident Woman | `Portuguese_ConfidentWoman` |
|
||||
|
||||
A audio e gerada pelo MiniMax TTS, **um arquivo por segmento** de fala. Como
|
||||
cada segmento corresponde a uma batida da sessao, a voz comeca exatamente
|
||||
quando a acao correspondente aparece na tela.
|
||||
|
||||
## Como regravar
|
||||
|
||||
```sh
|
||||
# tudo: audio -> sessoes -> videos -> filme completo
|
||||
./scripts/screencast/build-all.sh
|
||||
|
||||
# uma fase so
|
||||
./scripts/screencast/build-all.sh 04-roteamento
|
||||
```
|
||||
|
||||
Etapas individuais:
|
||||
|
||||
```sh
|
||||
python3 scripts/screencast/make_audio.py --lang pt # audio MiniMax
|
||||
python3 scripts/screencast/record.py --plan scripts/screencast/plans/04-roteamento.json \\
|
||||
--out out/screencast/caps/04-roteamento.cast --pace
|
||||
python3 scripts/screencast/assemble.py --phase 04-roteamento --lang pt
|
||||
python3 scripts/screencast/join.py --lang pt
|
||||
```
|
||||
|
||||
### Como o pipeline funciona
|
||||
|
||||
```
|
||||
planos/*.json --record.py--> caps/*.cast --render_video.py--> video mudo
|
||||
(comandos) (pty real) (asciicast) (pyte + PIL) |
|
||||
|
|
||||
narracao/*.json --make_audio.py--> vo/<fase>.<lang>/*.mp3 ----------+
|
||||
(MiniMax TTS) |
|
||||
assemble.py: mix + legendas
|
||||
|
|
||||
join.py: transicoes
|
||||
```
|
||||
|
||||
Pontos que importam:
|
||||
|
||||
* **Sessao real.** `record.py` abre um pty, faz SSH na VM e digita os comandos
|
||||
um caractere por vez (~34 c/s, com jitter). O `bash --noediting` evita que o
|
||||
editor de linha redesenhe o comando e o tty o repita.
|
||||
* **Pacing.** A gravacao segura a tela enquanto a voz fala: cada batida dura
|
||||
no maximo o trecho de audio que a acompanha (o maior entre EN e PT), para
|
||||
uma unica captura servir as duas versoes.
|
||||
* **Legendas.** O texto da narracao e queimado no video, na lingua da trilha.
|
||||
* **Diagrama.** Nas fases 0, 4 e 5 o mapa do lab e desenhado sobre o
|
||||
terminal, e o caminho em discussao fica destacado (eth1/eth2/eth3).
|
||||
|
||||
Requisitos: `ffmpeg`, `python3` com `PIL` e `pyte`, a fonte DejaVu, a chave
|
||||
`MINIMAX_API_KEY` no ambiente, e o lab de VMs no ar.
|
||||
|
||||
## Onde ficam os arquivos
|
||||
|
||||
| Caminho | Conteudo |
|
||||
|---------|----------|
|
||||
| `out/screencast/en/`, `out/screencast/pt/` | os videos finais |
|
||||
| `out/screencast/caps/` | as capturas asciicast das sessoes |
|
||||
| `out/screencast/vo/` | o audio gerado pelo MiniMax |
|
||||
| `scripts/screencast/plans/` | o roteiro de comandos de cada fase |
|
||||
| `scripts/screencast/narration/` | os textos da narracao (EN e PT) |
|
||||
|
||||
## Texto de descricao (YouTube)
|
||||
|
||||
```
|
||||
PF on Linux - o motor do OpenBSD PF 8.0 dentro do kernel Linux
|
||||
|
||||
Port completo do Packet Filter do OpenBSD 8.0 para o kernel Linux 7.3.4,
|
||||
empacotado como modulo (pf.ko) e controlado pelo pfctl original do OpenBSD
|
||||
sobre a ABI de ioctls do /dev/pf.
|
||||
|
||||
O lab e real: 4 VMs Alpine sob libvirt, kernel compilado do fonte com o
|
||||
modulo, trafego real entre cliente, firewall e servidor.
|
||||
|
||||
Fases:
|
||||
0. O ambiente de testes - as 4 VMs e o diagrama de rede
|
||||
1. Implementacao no kernel - das fontes OpenBSD ao pf.ko
|
||||
2. Ativacao - insmod, /dev/pf, pfctl -e, pfctl -s info
|
||||
3. Regras basicas - tabelas, anchors, keep state, block drop
|
||||
4. Roteamento - route-to e reply-to, com o caminho real no diagrama
|
||||
5. Balanceamento - pools round-robin, source-hash e dup-to
|
||||
6. QOS - filas hfsc do OpenBSD 8.0 no caminho de dados
|
||||
|
||||
Suite automatizada: 15 gates, todos verdes.
|
||||
Codigo e documentacao: https://git.labexposed.com/lgcosta/pf-linux-port
|
||||
|
||||
#linux #kernel #openbsd #pf #firewall #netfilter #screencast
|
||||
```
|
||||
@@ -939,3 +939,94 @@ Rationale: a single consumer is the real use case, and Linux gives no bpf hook
|
||||
for a non-`net_device` virtual interface without pulling in a much larger
|
||||
amount of the BSD if layer. The on-disk format is unchanged, so existing
|
||||
tooling works.
|
||||
|
||||
## Scheduler back ends in the build (M5, first step)
|
||||
|
||||
`pfsrc/hfsc.c` and `pfsrc/fq_codel.c` are now compiled into `pf.ko` (the
|
||||
`pf_ioctl` queue manager already references `hfsc_initialize`,
|
||||
`hfsc_pf_addqueue`, `ifq_hfsc_ops`, `fqcodel_pf_addqueue`, `ifq_fqcodel_ops`).
|
||||
The priq shim in `compat/pfk_ifq.c` remains the *attached* scheduler, so
|
||||
behaviour is unchanged until the data path enqueues into a queue.
|
||||
|
||||
Compat surface added for the back ends (all in `module/include/pfk.h` /
|
||||
`pfk_if.h` / `compat/pfk_ifq.c`):
|
||||
|
||||
| OpenBSD | This port |
|
||||
|---|---|
|
||||
| `struct mbuf_list` + `ml_init/ml_enqueue/ml_dequeue/ml_dequeue_tail/ml_enlist/ml_len/ml_empty/ml_purge`, `MBUF_LIST_FIRST/INITIALIZER` | inline helpers in `pfk_if.h` (`ml_purge` in `compat/pfk_mbuf.c`) |
|
||||
| `ifq_q_enter()/ifq_q_leave()` (scheduler ownership handshake, guarded by the ifq mutex) | ownership check against `ifq->ifq_ops`, returning `ifq_softc`; `leave` is a no-op (the port has no ifq mutex — the data path is one ordered work queue) |
|
||||
| `nsecuptime()` | `getnsecuptime()` (`ktime_get_ns`) |
|
||||
| `tick_nsec` (`HFSC_CLK_PER_TICK`) | `1000000000 / HZ` |
|
||||
| `splnet()/splx()` | no-ops, kept as markers of the critical regions |
|
||||
| `KERNEL_ASSERT_LOCKED()` | no-op (structural, not runtime) |
|
||||
| `struct hfsc_sc`, `struct hfsc_class_stats`, `struct fqcodel_stats`, `PKTCNTR_INC`, `HFSC_*` limits | declared in `pfk.h` (OpenBSD keeps them in `net/if_hfsc.h` / `net/if_fq_codel.h`) |
|
||||
|
||||
Two port fixes worth recording:
|
||||
|
||||
- `min(cd->drops - 1, nitems(...) - 1)` → `min_t(unsigned int, ...)`: Linux
|
||||
type-checks `min()` and `nitems()` is `size_t`.
|
||||
- `hfsc_more_slots(u_int current)` → renamed the parameter: **`current` is a
|
||||
kernel macro** (`get_current()`), which turned the definition into a K&R
|
||||
function and produced a baffling "makes pointer from integer" error at the
|
||||
call site.
|
||||
|
||||
## ifq contract details found while wiring the schedulers (M5)
|
||||
|
||||
The staged OpenBSD back ends assume three things the first version of the
|
||||
port's ifq shim got wrong. All three were reproduced in the lab and fixed:
|
||||
|
||||
| Assumption | What the port did | Fix |
|
||||
|---|---|---|
|
||||
| `ifqop_enq()` returns **the packet to drop** (NULL = accepted) | treated a non-NULL return as an error (`ENOMEM`) and the NULL return as success → every accepted packet was reported as dropped | `ifq_enqueue()` frees the returned mbuf and reports success |
|
||||
| the scheduler state lives in `ifq->ifq_q` | `ifq_attach()` stored it in `ifq_softc` only → `hfsc_enq()` dereferenced NULL and the box died mid-transfer | `ifq_attach()` keeps both fields in sync |
|
||||
| `ifq_deq_begin()`'s cookie is consumed by `ifq_deq_commit()` | the cookie was dropped on the floor (always NULL) | a per-queue `ifq_dequeue` slot carries it |
|
||||
|
||||
Also fixed: the hfsc **root-class handle** used bit `0x1`, which is exactly the
|
||||
handle of any odd pf queue id, so `hfsc_pf_addqueue()` saw the root class as a
|
||||
duplicate of the queue being added and returned `EBUSY` ("cannot add queue").
|
||||
Handles now use bit `0x80000000`; pf queue ids are `u_int16_t`, so they can
|
||||
never collide.
|
||||
|
||||
### `splassert` under rw-semas
|
||||
|
||||
`PF_ASSERT_UNLOCKED()` (used by `pf_purge_expired_states()`) is a BSD
|
||||
per-CPU-spl check: "this context did not take the lock". With an rw-semaphore
|
||||
`rw_status()` is global, so the assert fires whenever *another* CPU holds the
|
||||
lock - which is not a bug, and `BUG()` there turned a harmless race into a
|
||||
wedged module (`Unloading`, refcount -1). It now logs once
|
||||
(`pr_warn_ratelimited`) instead of killing the host; the invariant it protects
|
||||
(purge never acquires `pf_lock` itself) is upheld by the call sites, which are
|
||||
only reached from `systqmp` tasks and timer work.
|
||||
|
||||
### What `bandwidth` can and cannot do here
|
||||
|
||||
OpenBSD 8.0's `pfctl` keyword table exposes `burst`, `flows`, `qlimit`,
|
||||
`parent`, `default` - **not** `minimum`/`maximum`. A pf `queue` therefore only
|
||||
sets HFSC's **linkshare** curve (`qspec.linkshare = scspec` in `expand_queue`).
|
||||
Linkshare is a *guarantee under contention*, not a cap: a single active class
|
||||
may run at line rate, and `bandwidth 2Mbit` alone will not make one flow
|
||||
measure 2 Mbit/s. Verifying HFSC in this port therefore requires two competing
|
||||
leaf queues and comparing their shares - which is the remaining M5 item, not a
|
||||
code defect in the queue path (attach, enqueue, drain and transmit are
|
||||
verified: `eth2` carries the hfsc ops, 117 drains observed, transfer completes,
|
||||
box stays alive).
|
||||
|
||||
## Scheduler ABI must be shared, not re-implemented (M5)
|
||||
|
||||
The first version of this note hand-wrote `struct hfsc_class_stats`,
|
||||
`struct fqcodel_stats`, `struct hfsc_sc` and the `HFSC_*` constants in `pfk.h`
|
||||
from memory of the BSD sources. Two consequences showed up in the lab:
|
||||
|
||||
1. `pfctl -s queue -v` failed with `EINVAL` - `hfsc_pf_qstats()` rejects the
|
||||
call when `*nbytes < sizeof(stats)`, and pfctl's struct (from the vendored
|
||||
`net/hfsc.h`) was smaller than the kernel's invented one.
|
||||
2. `HFSC_ROOT_CLASS` was guessed as `0x1`, which is exactly the handle of every
|
||||
odd pf queue id, so `hfsc_pf_addqueue()` refused every root queue with
|
||||
`EBUSY`. The real value is `0x10000`, safely above the `u_int16_t` qid
|
||||
range.
|
||||
|
||||
The port now vendors OpenBSD's `net/hfsc.h` and `net/fq_codel.h` into
|
||||
`module/include/net/` and includes them from `pfk.h`, so kernel and pfctl
|
||||
compile against one definition. Rule for anything else in this area: if a
|
||||
struct crosses the ioctl boundary, take the header from OpenBSD's tree - do not
|
||||
reconstruct it.
|
||||
@@ -111,6 +111,32 @@ lendo `/dev/pflog`; sem ele a fila atinge o limite (256 registros / 1 MiB) e
|
||||
descarta. Nenhum dos dois é mais stub, mas nenhum faz algo sem o consumidor
|
||||
rodar (ver [pf-guide.md](pf-guide.md)).
|
||||
|
||||
### 9. Filas: caminho verificado; a semântica de modelagem é do HFSC, não um bug
|
||||
|
||||
`15-queue.sh` prova que o caminho de filas funciona de ponta a ponta: a
|
||||
configuração carrega, a disciplina hfsc é anexada à interface, `pfctl -s
|
||||
queue -v` lista as duas classes com estatísticas, um fluxo de 2 MB pela fila
|
||||
completa, e o contador de pacotes da classe avança - o escalonador está mesmo
|
||||
no caminho de dados.
|
||||
|
||||
O que o lab não mostra é um *teto de taxa*, e isso é semântica do HFSC, não
|
||||
defeito do port: o `pf.conf` do OpenBSD 8.0 expõe apenas `bandwidth`, que mapeia
|
||||
para a curva **linkshare** do HFSC - garantia quando classes competem, não limite
|
||||
de um fluxo isolado. Não existe palavra-chave `minimum`/`maximum` nesta versão do
|
||||
pfctl (verificado na sua tabela de palavras-chave), logo não há como definir
|
||||
upperlimit pelo pf.conf. Um teste de contenção com duas classes saturadas seria o
|
||||
próximo passo se você quiser asserir sobre share; no enlace virtual de centenas de
|
||||
Mbit do lab, um único fluxo termina antes de acumular dívida de tempo virtual.
|
||||
|
||||
Bugs encontrados e corrigidos no caminho (detalhes em `docs/porting-notes.md`): a
|
||||
convenção de retorno de `ifqop_enq`, `ifq_q` vs `ifq_softc` (deref de NULL que
|
||||
matava a máquina no meio da transferência), o cookie de dequeue perdido, o
|
||||
handle da classe raiz do hfsc colidindo com qids ímpares do pf, o
|
||||
`DIOCGETQSTATS` falhando porque kernel e pfctl discordavam sobre
|
||||
`struct hfsc_class_stats` (ambos agora usam o `net/hfsc.h` vendorizado), o
|
||||
payload do pflog começando no cabeçalho de transporte, e o `splassert` do caminho
|
||||
de purge que transformava uma corrida benigna em `BUG()`.
|
||||
|
||||
## Desvios da camada de compat (de `docs/porting-notes.md`, registro do M1)
|
||||
|
||||
Estes são desvios deliberados e marcados no código; importam ao raciocinar
|
||||
|
||||
@@ -60,7 +60,7 @@ Estado em 2026-11-11; evidência por linha: `out/test-logs/`,
|
||||
|---|---|---|---|
|
||||
| Pools de `route-to` (`rr`, `round-robin`, `source-hash`, `hash`), `reply-to`, `dup-to` | `pass in … route-to { 10.0.3.2, 10.0.2.2 } round-robin keep state` | M4 | **`round-robin` e `source-hash` VERIFICADOS em lab** (gate `10-pools.sh`): com dois gateways em dois enlaces, o round-robin usa ambos e o source-hash fixa todo fluxo de uma origem em um único enlace. `dup-to` verificado no gate 08; `reply-to` ainda sem gate. `fail-to` **não existe no OpenBSD 8.0** (removido upstream), então está fora de escopo. |
|
||||
| `divert-to` / tag `divert` | `pass in … divert-to <host> port <n>` | VERIFICADO em lab | **Funciona ponta a ponta.** O módulo registra em runtime a família de sockets `AF_DIVERT` (`PF_DIVERT=32`) — sem patch no kernel. A regra `divert-to` entrega o pacote a essa família; sem listener o pacote é descartado (`divs_noport`, paridade BSD); com `tools/pdivert` rodando o pacote vai ao userspace, é reescrito e reinjetado (verificado com mutação de payload + correção de checksum, ping 0% loss). |
|
||||
| Filas, agendador default (priq) | `queue root on wan`, `pfctl -s queue -v` | M5 | COMPILES-untested — `compat/pfk_ifq.c` implementa a API ifq com **semântica de fallback priq**; regras `queue` fazem parse e carregam, mas pacotes ainda não são desviados por um qdisc classful até o M5 |
|
||||
| Filas (`queue`, `queue on`, `pfctl -s queue -v`) | `queue rootq on eth2 bandwidth 10Mbit` + `queue leafq on eth2 bandwidth 2Mbit parent rootq default` | M5 | **VERIFICADO em lab** — gate `15-queue.sh`: a configuração de filas carrega, a disciplina hfsc é anexada à interface, `pfctl -s queue -v` lista as duas classes com estatísticas, um fluxo de 2 MB pela fila completa, e o contador de pacotes da classe avança (o escalonador está no caminho de dados). Note que `bandwidth` é *linkshare* do HFSC - garantia sob contenção, não teto: o pf.conf do OpenBSD 8.0 não tem `minimum`/`maximum`, então um fluxo isolado não é limitado, nem no Linux nem no OpenBSD. |
|
||||
| HFSC | `queue on wan { hfsc … }` | M5 | COMPILES-untested — **fallback priq até o M5**; `pfsrc/hfsc.c` staged (código real do OpenBSD 8.0) mas **fora da lista de objetos** do módulo |
|
||||
| FQ-CoDel | `queue on wan { fq_codel … }` | M5 | COMPILES-untested — igual ao HFSC: staged, não compilado |
|
||||
| Qdisc `pfq` (`tc qdisc … root pfq`) | n/a (glue Linux) | M5 | PENDING-M5 (ainda não existe na árvore) |
|
||||
|
||||
+1
-1
@@ -94,7 +94,7 @@ tamanho embutido na codificação `_IOR/_IOW` OpenBSD e despacha para o
|
||||
| M2 | núcleo do filtro: regras/estados/tabelas/âncoras, gates VM | **pronto** — gates 01–04 verdes |
|
||||
| M3 | NAT (`nat-to`/`rdr-to`/`binat-to`), scrub, pflog | **pronto** — gates 05/06/07/09 (incl. integridade de 2 MB via redirect) e pflog gate 12 |
|
||||
| M4 | route-to/dup-to/reply-to, pools, divert e2e | **pronto** — gates 08/10/13 e divert gate 11 |
|
||||
| M5 | filas: back ends ifq (hfsc/fq_codel) + qdisc `pfq` | pendente (código dos escalonadores portado, qdisc faltando) |
|
||||
| M5 | filas: back ends ifq (hfsc/fq_codel) | **parcial** — back ends compilados no `pf.ko`; anexação/ritmo pendentes |
|
||||
| M6 | docs completas, patchset limpo, suíte verde, releases com tag | em andamento — docs EN+PT-BR completas, suíte 14/14 verde, com tag |
|
||||
|
||||
## Próximas leituras
|
||||
|
||||
@@ -66,7 +66,7 @@ matriz mudar.
|
||||
| `nat-to`, `rdr-to` | — | **Verificado em lab** (gates `05-nat`, `06-rdr`; este último baixa 2 MB por um redirect e compara byte a byte). `nat-to (egress)` expande o *grupo* de interfaces do OpenBSD; num host multi-NIC prefira a forma explícita `nat-to (<iface>)`. O `binat-to` é expandido pelo pfctl num par `nat-to … static-port` + `rdr-to` (gate `07-binat`). |
|
||||
| `scrub (min-ttl, max-mss, df, random-id)`, `set reassemble` | — | **Verificado em lab** (gate `09-scrub.sh`): `min-ttl` e `max-mss` reescrevem de fato pacotes em trânsito. || `divert-to`, tag `divert` | — | **Verificado em lab** (gate `11-divert`): o pacote é entregue a um socket `AF_DIVERT` (`PF_DIVERT=32`) atendido pelo `tools/pdivert`; sem listener é descartado, exatamente como o `divs_noport` do BSD. |
|
||||
| `route-to`, `dup-to`, `reply-to` | — | **Verificado em lab** (gates `08-routeto.sh`, `13-replyto.sh`): o gateway forçado é usado em vez do egress normal, `dup-to` copia mantendo o fluxo, e `reply-to` empurra as respostas de um fluxo para o enlace do gateway. `fail-to` não é palavra-chave no OpenBSD 8.0 (removido upstream). |
|
||||
| `queue` / `queue on` / `pfctl -s queue` | M5 | Faz parse e carrega. A fila cai no fallback priq da camada `ifq` de compat; **nenhuma limitação de banda acontece até o qdisc `pfq` chegar**. |
|
||||
| `queue` / `queue on` / `pfctl -s queue` | — | **Verificado em lab** (gate `15-queue.sh`): as filas carregam, o hfsc é anexado, `pfctl -s queue -v` mostra as estatísticas das classes e um fluxo de 2 MB pela fila é contado pela classe. `bandwidth` é *garantia sob contenção*, não teto - o pf.conf 8.0 não tem palavra-chave de upperlimit. |
|
||||
| `log`, `log all`, `block log`, `pflogd` | — | **Verificado em lab** (gate `12-pflog`): rode `pflogd /tmp/x.pcap` (lê `/dev/pflog`); o arquivo é um pcap `DLT_PFLOG` (117) legível por tcpdump/wireshark. |
|
||||
| `route-label …` | nunca (não-objetivo) | Rejeitado na camada de ioctl. |
|
||||
| política por container (netns) | fora de escopo | O PF é uma instância global em `init_net`; network namespaces **não** enxergam o PF. |
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
# Screencasts / Videos da demonstracao
|
||||
|
||||
> PT-BR: esta pagina · EN: [docs/en/screencast.md](../en/screencast.md)
|
||||
|
||||
Sete videos por idioma, com **narracao de voz feminina** (MiniMax), **legendas
|
||||
queimadas**, **diagrama de rede do lab** e um **filme completo** com transicoes
|
||||
entre as fases. 1920x1080 @ 30fps, H.264/MP4 + AAC — prontos para subir no
|
||||
YouTube.
|
||||
|
||||
Cada video e a gravacao de uma **sessao shell real**: uma conexao SSH de verdade
|
||||
ao firewall Alpine, com os comandos digitados tecla a tecla e a saida do kernel
|
||||
chegando em tempo real. Nada e simulado.
|
||||
|
||||
## As fases
|
||||
|
||||
| # | Video | O que mostra |
|
||||
|---|-------|--------------|
|
||||
| 0 | `00-ambiente.mp4` | O lab: 4 VMs Alpine sob libvirt, as quatro interfaces do firewall e o diagrama de rede completo |
|
||||
| 1 | `01-implementacao.mp4` | A implementacao no kernel: fontes do OpenBSD 8.0 + camada de compatibilidade = `pf.ko` |
|
||||
| 2 | `02-ativacao.mp4` | `insmod pf.ko`, `/dev/pf`, `pfctl -e`, `pfctl -s info` |
|
||||
| 3 | `03-regras.mp4` | Regras basicas: tabela, anchor, `keep state`, `block drop` (o ping morre) |
|
||||
| 4 | `04-roteamento.mp4` | `route-to` e `reply-to`, com o diagrama mostrando o caminho real do pacote |
|
||||
| 5 | `05-balanceamento.mp4` | Pool `round-robin`, pool `source-hash` e `dup-to` |
|
||||
| 6 | `06-qos.mp4` | Filas `hfsc` do OpenBSD 8.0 e uma transferencia de 2 MB pela classe |
|
||||
|
||||
Mais `pf-on-linux-full.mp4`: o filme completo, com as sete fases emendadas por
|
||||
transicoes (crossfade sobre um cartao de capitulo).
|
||||
|
||||
## Vozes MiniMax
|
||||
|
||||
| Idioma | Voz | Id |
|
||||
|--------|-----|-----|
|
||||
| Ingles (en_US) | Confident Woman | `English_ConfidentWoman` |
|
||||
| Portugues (pt_BR) | Confident Woman | `Portuguese_ConfidentWoman` |
|
||||
|
||||
O audio e gerado pelo MiniMax TTS, **um arquivo por segmento** de fala. Como cada
|
||||
segmento corresponde a uma batida da sessao, a voz comeca exatamente quando a
|
||||
acao correspondente aparece na tela.
|
||||
|
||||
## Como regravar
|
||||
|
||||
```sh
|
||||
# tudo: audio -> sessoes -> videos -> filme completo
|
||||
./scripts/screencast/build-all.sh
|
||||
|
||||
# uma fase so
|
||||
./scripts/screencast/build-all.sh 04-roteamento
|
||||
```
|
||||
|
||||
Etapas individuais:
|
||||
|
||||
```sh
|
||||
python3 scripts/screencast/make_audio.py --lang pt # audio MiniMax
|
||||
python3 scripts/screencast/record.py --plan scripts/screencast/plans/04-roteamento.json \\
|
||||
--out out/screencast/caps/04-roteamento.cast --pace
|
||||
python3 scripts/screencast/assemble.py --phase 04-roteamento --lang pt
|
||||
python3 scripts/screencast/join.py --lang pt
|
||||
```
|
||||
|
||||
### Como o pipeline funciona
|
||||
|
||||
```
|
||||
planos/*.json --record.py--> caps/*.cast --render_video.py--> video mudo
|
||||
(comandos) (pty real) (asciicast) (pyte + PIL) |
|
||||
|
|
||||
narracao/*.json --make_audio.py--> vo/<fase>.<lang>/*.mp3 ----------+
|
||||
(MiniMax TTS) |
|
||||
assemble.py: mix + legendas
|
||||
|
|
||||
join.py: transicoes
|
||||
```
|
||||
|
||||
Pontos que importam:
|
||||
|
||||
* **Sessao real.** `record.py` abre um pty, faz SSH na VM e digita os comandos um
|
||||
caractere por vez (~34 c/s, com jitter). O `bash --noediting` evita que o
|
||||
editor de linha redesenhe o comando e o tty o repita.
|
||||
* **Pacing.** A gravacao segura a tela enquanto a voz fala: cada batida dura no
|
||||
maximo o trecho de audio que a acompanha (o maior entre EN e PT), para uma
|
||||
unica captura servir as duas versoes.
|
||||
* **Legendas.** O texto da narracao e queimado no video, na lingua da trilha.
|
||||
* **Diagrama.** Nas fases 0, 4 e 5 o mapa do lab e desenhado sobre o terminal, e o
|
||||
caminho em discussao fica destacado (eth1/eth2/eth3).
|
||||
|
||||
Requisitos: `ffmpeg`, `python3` com `PIL` e `pyte`, a fonte DejaVu, a chave
|
||||
`MINIMAX_API_KEY` no ambiente, e o lab de VMs no ar.
|
||||
|
||||
## Onde ficam os arquivos
|
||||
|
||||
| Caminho | Conteudo |
|
||||
|---------|----------|
|
||||
| `out/screencast/en/`, `out/screencast/pt/` | os videos finais |
|
||||
| `out/screencast/caps/` | as capturas asciicast das sessoes |
|
||||
| `out/screencast/vo/` | o audio gerado pelo MiniMax |
|
||||
| `scripts/screencast/plans/` | o roteiro de comandos de cada fase |
|
||||
| `scripts/screencast/narration/` | os textos da narracao (EN e PT) |
|
||||
|
||||
## Texto de descricao (YouTube)
|
||||
|
||||
```
|
||||
PF on Linux - o motor do OpenBSD PF 8.0 dentro do kernel Linux
|
||||
|
||||
Port completo do Packet Filter do OpenBSD 8.0 para o kernel Linux 7.3.4,
|
||||
empacotado como modulo (pf.ko) e controlado pelo pfctl original do OpenBSD sobre
|
||||
a ABI de ioctls do /dev/pf.
|
||||
|
||||
O lab e real: 4 VMs Alpine sob libvirt, kernel compilado do fonte com o modulo,
|
||||
trafego real entre cliente, firewall e servidor.
|
||||
|
||||
Fases:
|
||||
0. O ambiente de testes - as 4 VMs e o diagrama de rede
|
||||
1. Implementacao no kernel - das fontes OpenBSD ao pf.ko
|
||||
2. Ativacao - insmod, /dev/pf, pfctl -e, pfctl -s info
|
||||
3. Regras basicas - tabelas, anchors, keep state, block drop
|
||||
4. Roteamento - route-to e reply-to, com o caminho real no diagrama
|
||||
5. Balanceamento - pools round-robin, source-hash e dup-to
|
||||
6. QOS - filas hfsc do OpenBSD 8.0 no caminho de dados
|
||||
|
||||
Suite automatizada: 15 gates, todos verdes.
|
||||
Codigo e documentacao: https://git.labexposed.com/lgcosta/pf-linux-port
|
||||
|
||||
#linux #kernel #openbsd #pf #firewall #netfilter #screencast
|
||||
```
|
||||
+1
-1
@@ -1 +1 @@
|
||||
savedcmd_pf.mod := printf '%s\n' pfsrc/pfk_mod.o pfsrc/pfk_hooks.o compat/pfk_mbuf.o compat/pfk_kern.o compat/pfk_if.o compat/pfk_xmit.o compat/pfk_pcb.o compat/pfk_ip4.o compat/siphash.o compat/toeplitz.o compat/md5.o compat/sha2.o compat/radix.o compat/pfk_rtable.o compat/pfk_ip6.o compat/subr_tree.o pfsrc/pf_ruleset.o pfsrc/pf_table.o pfsrc/pf_norm.o pfsrc/pf_if.o pfsrc/pf_lb.o pfsrc/pf_osfp.o pfsrc/pf_syncookies.o pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o pfsrc/pfk_pflog.o compat/pfk_ifq.o | awk '!x[$$0]++ { print("./"$$0) }' > pf.mod
|
||||
savedcmd_pf.mod := printf '%s\n' pfsrc/pfk_mod.o pfsrc/pfk_hooks.o compat/pfk_mbuf.o compat/pfk_kern.o compat/pfk_if.o compat/pfk_xmit.o compat/pfk_pcb.o compat/pfk_ip4.o compat/siphash.o compat/toeplitz.o compat/md5.o compat/sha2.o compat/radix.o compat/pfk_rtable.o compat/pfk_ip6.o compat/subr_tree.o pfsrc/pf_ruleset.o pfsrc/pf_table.o pfsrc/pf_norm.o pfsrc/pf_if.o pfsrc/pf_lb.o pfsrc/pf_osfp.o pfsrc/pf_syncookies.o pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o pfsrc/pfk_pflog.o pfsrc/fq_codel.o pfsrc/hfsc.o compat/pfk_ifq.o | awk '!x[$$0]++ { print("./"$$0) }' > pf.mod
|
||||
+1
-1
@@ -20,7 +20,7 @@ pf-objs := pfsrc/pfk_mod.o pfsrc/pfk_hooks.o \
|
||||
# ported OpenBSD 8.0 engine
|
||||
pf-objs += pfsrc/pf_ruleset.o pfsrc/pf_table.o pfsrc/pf_norm.o pfsrc/pf_if.o \
|
||||
pfsrc/pf_lb.o pfsrc/pf_osfp.o pfsrc/pf_syncookies.o \
|
||||
pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o pfsrc/pfk_pflog.o
|
||||
pfsrc/pf_ioctl.o pfsrc/pf.o pfsrc/divert_sock.o pfsrc/pfk_pflog.o pfsrc/fq_codel.o pfsrc/hfsc.o
|
||||
|
||||
# queueing framework (compat/pfk_ifq.c implements the ifq API with priq
|
||||
# semantics; the OpenBSD hfsc/fq_codel back ends land in milestone M5)
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_compat/md5.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
compat/md5.o: $(deps_compat/md5.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_compat/pfk_if.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
compat/pfk_if.o: $(deps_compat/pfk_if.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_compat/pfk_ifq.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
compat/pfk_ifq.o: $(deps_compat/pfk_ifq.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_compat/pfk_ip4.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
compat/pfk_ip4.o: $(deps_compat/pfk_ip4.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_compat/pfk_ip6.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
compat/pfk_ip6.o: $(deps_compat/pfk_ip6.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_compat/pfk_kern.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
compat/pfk_kern.o: $(deps_compat/pfk_kern.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_compat/pfk_mbuf.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
compat/pfk_mbuf.o: $(deps_compat/pfk_mbuf.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_compat/pfk_pcb.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
/home/lgcosta/lab/linux_kernel/linux-7.3.4/include/net/inet6_hashtables.h \
|
||||
|
||||
compat/pfk_pcb.o: $(deps_compat/pfk_pcb.o)
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_compat/pfk_rtable.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
compat/pfk_rtable.o: $(deps_compat/pfk_rtable.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_compat/pfk_xmit.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
include/pfk_api.h \
|
||||
|
||||
compat/pfk_xmit.o: $(deps_compat/pfk_xmit.o)
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_compat/radix.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
compat/radix.o: $(deps_compat/radix.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_compat/sha2.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
compat/sha2.o: $(deps_compat/sha2.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_compat/siphash.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
compat/siphash.o: $(deps_compat/siphash.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_compat/subr_tree.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
compat/subr_tree.o: $(deps_compat/subr_tree.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_compat/toeplitz.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
/home/lgcosta/lab/linux_kernel/linux-7.3.4/include/crypto/hash.h \
|
||||
/home/lgcosta/lab/linux_kernel/linux-7.3.4/include/linux/crypto.h \
|
||||
|
||||
|
||||
+103
-27
@@ -217,10 +217,14 @@ pfk_pfq_enqueue(void *arg, struct mbuf *m)
|
||||
struct pfk_pfq *pfq = arg;
|
||||
|
||||
if (!pfq)
|
||||
return NULL;
|
||||
if (ifq_enqueue(&pfq->ifq, m) != 0)
|
||||
return NULL;
|
||||
return m;
|
||||
return m; /* caller keeps ownership */
|
||||
/*
|
||||
* ifq_enqueue takes ownership of m (the scheduler may drop it); we then
|
||||
* drain immediately, so nothing is returned for the caller to send.
|
||||
*/
|
||||
ifq_enqueue(&pfq->ifq, m);
|
||||
ifq_start(&pfq->ifq);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
static struct mbuf *
|
||||
@@ -273,10 +277,11 @@ static const struct pfq_ops pfk_priq_pfq_ops = {
|
||||
* packets are still queued, but the traffic shaping curves are not yet
|
||||
* applied — recorded in the feature matrix as pending M5).
|
||||
*/
|
||||
const struct ifq_ops * const ifq_hfsc_ops = &pfk_priq_ifq_ops;
|
||||
const struct ifq_ops * const ifq_fqcodel_ops = &pfk_priq_ifq_ops;
|
||||
const struct pfq_ops * const pfq_hfsc_ops = &pfk_priq_pfq_ops;
|
||||
const struct pfq_ops * const pfq_fqcodel_ops = &pfk_priq_pfq_ops;
|
||||
/*
|
||||
* M5: the real scheduler back ends (pfsrc/hfsc.c, pfsrc/fq_codel.c) define
|
||||
* ifq_hfsc_ops/pfq_hfsc_ops/ifq_fqcodel_ops/pfq_fqcodel_ops themselves; the
|
||||
* priq shim above stays as the default for interfaces without a queue rule.
|
||||
*/
|
||||
|
||||
/* ---------------- framework ---------------- */
|
||||
void
|
||||
@@ -302,7 +307,36 @@ ifq_attach(struct ifqueue *ifq, const struct ifq_ops *ops, void *opsarg)
|
||||
if (!opsarg && ops->ifqop_alloc)
|
||||
opsarg = ops->ifqop_alloc(0, NULL);
|
||||
ifq->ifq_ops = ops;
|
||||
/*
|
||||
* The port's own helpers use ifq_softc; the staged OpenBSD schedulers
|
||||
* (hfsc/fq_codel) read ifq->ifq_q. Keep both in sync or the back ends
|
||||
* dereference a NULL scheduler state.
|
||||
*/
|
||||
ifq->ifq_softc = opsarg;
|
||||
ifq->ifq_q = opsarg;
|
||||
}
|
||||
|
||||
/*
|
||||
* Scheduler hand-off used by the ported back ends (hfsc/fq_codel) when they
|
||||
* need their per-interface state. PFLINUX: the port has no ifq mutex (the
|
||||
* data path is a single ordered work queue), so "enter" is a plain ownership
|
||||
* check and "leave" is a no-op - the pair exists to mirror OpenBSD's API.
|
||||
*/
|
||||
void *
|
||||
ifq_q_enter(struct ifqueue *ifq, const struct ifq_ops *ops)
|
||||
{
|
||||
if (ifq == NULL || ops == NULL || ifq->ifq_ops != ops)
|
||||
return NULL; /* another scheduler owns it */
|
||||
if (ifq->ifq_softc == NULL && ops->ifqop_alloc)
|
||||
ifq->ifq_softc = ops->ifqop_alloc(0, NULL);
|
||||
return ifq->ifq_softc;
|
||||
}
|
||||
|
||||
void
|
||||
ifq_q_leave(struct ifqueue *ifq, void *q)
|
||||
{
|
||||
(void)ifq;
|
||||
(void)q;
|
||||
}
|
||||
|
||||
void
|
||||
@@ -312,19 +346,35 @@ ifq_destroy(struct ifqueue *ifq)
|
||||
ifq->ifq_ops->ifqop_free(0, ifq->ifq_softc);
|
||||
ifq->ifq_ops = NULL;
|
||||
ifq->ifq_softc = NULL;
|
||||
ifq->ifq_q = NULL;
|
||||
}
|
||||
|
||||
/*
|
||||
* ifqop_enq returns the packet the scheduler decided to DROP (possibly the one
|
||||
* just offered), and NULL when the packet was accepted - it is NOT an error
|
||||
* code. Freeing the returned mbuf is this layer's job.
|
||||
*/
|
||||
int
|
||||
ifq_enqueue(struct ifqueue *ifq, struct mbuf *m)
|
||||
{
|
||||
if (!ifq->ifq_ops || !ifq->ifq_ops->ifqop_enq)
|
||||
struct mbuf *dm;
|
||||
|
||||
if (!ifq->ifq_ops || !ifq->ifq_ops->ifqop_enq) {
|
||||
m_freem(m);
|
||||
return EINVAL;
|
||||
}
|
||||
if (ifq->ifq_maxlen && ifq->ifq_len >= ifq->ifq_maxlen) {
|
||||
ifq->ifq_errors++;
|
||||
ifq->ifq_qdrops++;
|
||||
m_freem(m);
|
||||
return ENOBUFS;
|
||||
}
|
||||
return ifq->ifq_ops->ifqop_enq(ifq, m) ? 0 : ENOMEM;
|
||||
dm = ifq->ifq_ops->ifqop_enq(ifq, m);
|
||||
if (dm) {
|
||||
ifq->ifq_qdrops++;
|
||||
m_freem(dm);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
struct mbuf *
|
||||
@@ -336,14 +386,16 @@ ifq_deq_begin(struct ifqueue *ifq)
|
||||
if (!ifq->ifq_ops || !ifq->ifq_ops->ifqop_deq_begin)
|
||||
return NULL;
|
||||
m = ifq->ifq_ops->ifqop_deq_begin(ifq, &cookie);
|
||||
ifq->ifq_dequeue = cookie;
|
||||
return m;
|
||||
}
|
||||
|
||||
void
|
||||
ifq_deq_commit(struct ifqueue *ifq, struct mbuf *m)
|
||||
{
|
||||
void *cookie = NULL;
|
||||
void *cookie = ifq->ifq_dequeue;
|
||||
|
||||
ifq->ifq_dequeue = NULL;
|
||||
if (ifq->ifq_ops && ifq->ifq_ops->ifqop_deq_commit)
|
||||
ifq->ifq_ops->ifqop_deq_commit(ifq, m, cookie);
|
||||
}
|
||||
@@ -365,12 +417,52 @@ ifq_dequeue(struct ifqueue *ifq)
|
||||
return m;
|
||||
}
|
||||
|
||||
/*
|
||||
* Drain the queue as far as the scheduler allows and hand each packet to the
|
||||
* stack's output path. OpenBSD calls if_start from softint context; here it
|
||||
* runs either from the pf work queue (enqueue side) or from the scheduler's
|
||||
* deferred timeout (pacing side), which is what makes hfsc's bandwidth
|
||||
* limits take effect.
|
||||
*/
|
||||
void
|
||||
ifq_start(struct ifqueue *ifq)
|
||||
{
|
||||
struct mbuf *m;
|
||||
int n;
|
||||
|
||||
n = 0;
|
||||
ifq->ifq_oactive = 1;
|
||||
while ((m = ifq_dequeue(ifq)) != NULL) {
|
||||
n++;
|
||||
if (pfk_ifq_transmit(ifq, m))
|
||||
break; /* device busy: packet kept */
|
||||
}
|
||||
ifq->ifq_oactive = 0;
|
||||
}
|
||||
|
||||
/*
|
||||
* pf data-path entry: hand a packet to the interface's scheduler when one is
|
||||
* attached (a pf `queue' rule is configured). Returns 1 when the queue took
|
||||
* ownership of m, 0 when the caller must keep going (priq/no scheduler).
|
||||
*/
|
||||
int
|
||||
pf_kifq_input(struct pfk_ifnet *ifp, struct mbuf *m)
|
||||
{
|
||||
struct ifqueue *ifq;
|
||||
|
||||
if (ifp == NULL || m == NULL)
|
||||
return 0;
|
||||
if (ifp == NULL)
|
||||
return 0;
|
||||
ifq = &ifp->if_snd;
|
||||
if (ifq->ifq_ops == NULL || ifq->ifq_ops == ifq_priq_ops)
|
||||
return 0; /* no scheduler: fast path */
|
||||
if (ifq_enqueue(ifq, m) != 0)
|
||||
return 1; /* dropped by the queue: owned */
|
||||
ifq_start(ifq);
|
||||
return 1;
|
||||
}
|
||||
|
||||
unsigned int
|
||||
ifq_purge(struct ifqueue *ifq)
|
||||
{
|
||||
@@ -423,19 +515,3 @@ ifq_hdatalen(struct ifqueue *ifq)
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* ---------------- scheduler back ends (milestone M5) ---------------- */
|
||||
/* The staged OpenBSD hfsc.c/fq_codel.c are not in the build yet; the priq
|
||||
* ops above are the active scheduler. These wrappers keep pf_ioctl's
|
||||
* initialisation calls valid until M5 replaces them with the ported
|
||||
* schedulers (docs/porting-notes.md). */
|
||||
void
|
||||
hfsc_initialize(void)
|
||||
{
|
||||
pr_info("pfk: hfsc back end pending M5 (priq active)\n");
|
||||
}
|
||||
|
||||
void
|
||||
fq_codel_initialize(void)
|
||||
{
|
||||
pr_info("pfk: fq_codel back end pending M5 (priq active)\n");
|
||||
}
|
||||
@@ -47,8 +47,21 @@ pfk_printk_compat(const char *fmt, ...)
|
||||
void
|
||||
splassert_fail(int expected, int actual, const char *func)
|
||||
{
|
||||
pr_err("splassert: %s: expected %d, got %d\n", func, expected, actual);
|
||||
BUG();
|
||||
/*
|
||||
* PFLINUX: OpenBSD's splassert() checks a per-CPU spl level, so
|
||||
* "assert unlocked" really means "this context did not take the lock".
|
||||
* The port replaced spl with rw-semas plus work queues, where
|
||||
* rw_status() is a global counter: an assert-unlocked in a work-queue
|
||||
* task fires whenever *another* CPU happens to hold the lock, which is
|
||||
* not a bug (the task never acquires pf_lock itself) and was turning a
|
||||
* harmless race into a kernel BUG() that wedged the module in
|
||||
* "Unloading". Report it once instead of killing the host; the
|
||||
* structural invariant the BSD assert protects - purge never runs while
|
||||
* holding pf_lock - is upheld by the call sites (pf_purge_* are only
|
||||
* reached from systqmp tasks and timer work).
|
||||
*/
|
||||
pr_warn_ratelimited("pfk: splassert: %s: expected %d, got %d\n",
|
||||
func, expected, actual);
|
||||
}
|
||||
|
||||
/* ---------------- addlog (OpenBSD kern.log) ---------------- */
|
||||
|
||||
@@ -607,3 +607,14 @@ pfk_mbuf_exit(void)
|
||||
{
|
||||
kmem_cache_destroy(pfk_mbuf_cache);
|
||||
}
|
||||
|
||||
void
|
||||
ml_purge(struct mbuf_list *ml)
|
||||
{
|
||||
struct mbuf *m;
|
||||
|
||||
while ((m = ml_dequeue(ml)) != NULL)
|
||||
m_freem(m);
|
||||
ml->ml_first = ml->ml_last = NULL;
|
||||
ml->ml_len = 0;
|
||||
}
|
||||
@@ -270,3 +270,40 @@ pfk_if_output(struct pfk_ifnet *ifp, struct mbuf *m, void *dst,
|
||||
m_freem(mp);
|
||||
return err;
|
||||
}
|
||||
|
||||
/*
|
||||
* Transmit one queued packet. Called from ifq_start() - i.e. from the pf
|
||||
* work queue or from a scheduler's deferred timeout - and always consumes the
|
||||
* mbuf. Returns 0 (keep draining); a scheduler that wants to defer does so by
|
||||
* simply not returning a packet from ifqop_deq_begin().
|
||||
*/
|
||||
int
|
||||
pfk_ifq_transmit(struct ifqueue *ifq, struct mbuf *m)
|
||||
{
|
||||
struct sk_buff *skb = m_to_skb(m);
|
||||
struct dst_entry *d;
|
||||
|
||||
(void)ifq;
|
||||
if (skb == NULL) {
|
||||
m_freem(m);
|
||||
return 0;
|
||||
}
|
||||
d = skb_dst(skb);
|
||||
if (d == NULL || IS_ERR(d) || d->dev == NULL || !netif_running(d->dev)) {
|
||||
m_freem(m);
|
||||
return 0;
|
||||
}
|
||||
skb->dev = d->dev;
|
||||
skb_reset_network_header(skb);
|
||||
if (skb->protocol == 0)
|
||||
skb->protocol = htons(ip_hdr(skb)->protocol == IPPROTO_IPV6 ?
|
||||
ETH_P_IPV6 : ETH_P_IP);
|
||||
/* the stack owns skb->cb (IPCB) from here; pf already tested it */
|
||||
memset(skb->cb, 0, sizeof(skb->cb));
|
||||
skb->mark |= PFK_DONE_OUT;
|
||||
skb_get(skb); /* the output path consumes one reference */
|
||||
m->skb = NULL; /* the mbuf's reference moves to the stack */
|
||||
m_freem(m);
|
||||
d->output(&init_net, NULL, skb);
|
||||
return 0;
|
||||
}
|
||||
@@ -27,6 +27,7 @@ struct ifqueue {
|
||||
struct mbuf_list *ifq_free; /* free list (unused in the port) */
|
||||
unsigned int ifq_len;
|
||||
unsigned int ifq_oactive;
|
||||
void *ifq_dequeue; /* cookie between deq_begin/commit */
|
||||
|
||||
/* statistics */
|
||||
u_int64_t ifq_packets;
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
/* $OpenBSD: fq_codel.h,v 1.4 2020/06/18 23:29:59 dlg Exp $ */
|
||||
|
||||
/*
|
||||
* Copyright (c) 2017 Mike Belopuhov
|
||||
*
|
||||
* Permission to use, copy, modify, and distribute this software for any
|
||||
* purpose with or without fee is hereby granted, provided that the above
|
||||
* copyright notice and this permission notice appear in all copies.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
|
||||
* WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
|
||||
* MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
|
||||
* ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
|
||||
* WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
|
||||
* ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
|
||||
* OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
|
||||
*/
|
||||
|
||||
#ifndef _NET_FQ_CODEL_H_
|
||||
#define _NET_FQ_CODEL_H_
|
||||
|
||||
/* compatible with hfsc_pktcntr */
|
||||
struct fqcodel_pktcntr {
|
||||
uint64_t packets;
|
||||
uint64_t bytes;
|
||||
};
|
||||
|
||||
struct fqcodel_stats {
|
||||
struct fqcodel_pktcntr xmit_cnt;
|
||||
struct fqcodel_pktcntr drop_cnt;
|
||||
|
||||
uint32_t qlength;
|
||||
uint32_t qlimit;
|
||||
|
||||
uint32_t flows;
|
||||
uint32_t _unused; /* padding */
|
||||
|
||||
uint32_t target;
|
||||
uint32_t interval;
|
||||
|
||||
/* the values below are used to calculate standard deviation */
|
||||
uint64_t delaysum; /* sum of delays, us */
|
||||
uint64_t delaysumsq;/* sum of squared delays, us */
|
||||
};
|
||||
|
||||
#ifdef _KERNEL
|
||||
extern const struct ifq_ops * const ifq_fqcodel_ops;
|
||||
extern const struct pfq_ops * const pfq_fqcodel_ops;
|
||||
#endif /* _KERNEL */
|
||||
|
||||
#endif /* _NET_FQ_CODEL_H_ */
|
||||
@@ -0,0 +1,122 @@
|
||||
/* $OpenBSD: hfsc.h,v 1.15 2026/03/19 14:59:05 sthen Exp $ */
|
||||
|
||||
/*
|
||||
* Copyright (c) 2012-2013 Henning Brauer <henning@openbsd.org>
|
||||
* Copyright (c) 1997-1999 Carnegie Mellon University. All Rights Reserved.
|
||||
*
|
||||
* Permission to use, copy, modify, and distribute this software and
|
||||
* its documentation is hereby granted (including for commercial or
|
||||
* for-profit use), provided that both the copyright notice and this
|
||||
* permission notice appear in all copies of the software, derivative
|
||||
* works, or modified versions, and any portions thereof.
|
||||
*
|
||||
* THIS SOFTWARE IS EXPERIMENTAL AND IS KNOWN TO HAVE BUGS, SOME OF
|
||||
* WHICH MAY HAVE SERIOUS CONSEQUENCES. CARNEGIE MELLON PROVIDES THIS
|
||||
* SOFTWARE IN ITS ``AS IS'' CONDITION, AND ANY EXPRESS OR IMPLIED
|
||||
* WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
|
||||
* OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
||||
* DISCLAIMED. IN NO EVENT SHALL CARNEGIE MELLON UNIVERSITY BE LIABLE
|
||||
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
|
||||
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
|
||||
* OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
|
||||
* BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
|
||||
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
|
||||
* USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
|
||||
* DAMAGE.
|
||||
*
|
||||
* Carnegie Mellon encourages (but does not require) users of this
|
||||
* software to return any improvements or extensions that they make,
|
||||
* and to grant Carnegie Mellon the rights to redistribute these
|
||||
* changes without encumbrance.
|
||||
*/
|
||||
#ifndef _HFSC_H_
|
||||
#define _HFSC_H_
|
||||
|
||||
/* hfsc class flags */
|
||||
#define HFSC_DEFAULTCLASS 0x1000 /* default class */
|
||||
|
||||
struct hfsc_pktcntr {
|
||||
u_int64_t packets;
|
||||
u_int64_t bytes;
|
||||
};
|
||||
|
||||
#define PKTCNTR_INC(cntr, len) \
|
||||
do { (cntr)->packets++; (cntr)->bytes += len; } while (0)
|
||||
|
||||
struct hfsc_sc {
|
||||
u_int64_t m1; /* slope of the first segment in bits/sec */
|
||||
u_int64_t m2; /* slope of the second segment in bits/sec */
|
||||
u_int d; /* the x-projection of the first segment in msec */
|
||||
};
|
||||
|
||||
/* special class handles */
|
||||
#define HFSC_ROOT_CLASS 0x10000
|
||||
#define HFSC_DEFAULT_CLASSES 64
|
||||
#define HFSC_MAX_CLASSES 65535
|
||||
|
||||
/* service curve types */
|
||||
#define HFSC_REALTIMESC 1
|
||||
#define HFSC_LINKSHARINGSC 2
|
||||
#define HFSC_UPPERLIMITSC 4
|
||||
#define HFSC_DEFAULTSC (HFSC_REALTIMESC|HFSC_LINKSHARINGSC)
|
||||
|
||||
struct hfsc_class_stats {
|
||||
struct hfsc_pktcntr xmit_cnt;
|
||||
struct hfsc_pktcntr drop_cnt;
|
||||
|
||||
u_int qlength;
|
||||
u_int qlimit;
|
||||
u_int period;
|
||||
|
||||
u_int class_id;
|
||||
u_int32_t class_handle;
|
||||
struct hfsc_sc rsc;
|
||||
struct hfsc_sc fsc;
|
||||
struct hfsc_sc usc; /* upper limit service curve */
|
||||
|
||||
u_int64_t total; /* total work in bytes */
|
||||
u_int64_t cumul; /* cumulative work in bytes
|
||||
done by real-time criteria */
|
||||
u_int64_t d; /* deadline */
|
||||
u_int64_t e; /* eligible time */
|
||||
u_int64_t vt; /* virtual time */
|
||||
u_int64_t f; /* fit time for upper-limit */
|
||||
|
||||
/* info helpful for debugging */
|
||||
u_int64_t initvt; /* init virtual time */
|
||||
u_int64_t vtoff; /* cl_vt_ipoff */
|
||||
u_int64_t cvtmax; /* cl_maxvt */
|
||||
u_int64_t myf; /* cl_myf */
|
||||
u_int64_t cfmin; /* cl_mincf */
|
||||
u_int64_t cvtmin; /* cl_mincvt */
|
||||
u_int64_t myfadj; /* cl_myfadj */
|
||||
u_int64_t vtadj; /* cl_vtadj */
|
||||
u_int64_t cur_time;
|
||||
u_int32_t machclk_freq;
|
||||
|
||||
u_int vtperiod; /* vt period sequence no */
|
||||
u_int parentperiod; /* parent's vt period seqno */
|
||||
int nactive; /* number of active children */
|
||||
|
||||
/* red and rio related info */
|
||||
int qtype;
|
||||
/* struct redstats red[3]; */
|
||||
};
|
||||
|
||||
#ifdef _KERNEL
|
||||
struct ifnet;
|
||||
struct ifqueue;
|
||||
struct pf_queuespec;
|
||||
struct hfsc_if;
|
||||
|
||||
extern const struct ifq_ops * const ifq_hfsc_ops;
|
||||
extern const struct pfq_ops * const pfq_hfsc_ops;
|
||||
|
||||
#define HFSC_ENABLED(ifq) ((ifq)->ifq_ops == ifq_hfsc_ops)
|
||||
#define HFSC_DEFAULT_QLIMIT 50
|
||||
|
||||
void hfsc_initialize(void);
|
||||
|
||||
#endif /* _KERNEL */
|
||||
#endif /* _HFSC_H_ */
|
||||
@@ -1212,5 +1212,51 @@ struct pfk_pflog_stat {
|
||||
};
|
||||
#define PFKIOCGETPFLOGSTAT _IOR('L', 1, struct pfk_pflog_stat)
|
||||
|
||||
|
||||
/*
|
||||
* Scheduler back-end ABI: the very headers OpenBSD ships (net/hfsc.h,
|
||||
* net/fq_codel.h), vendored into the port so that pfctl and pf.ko agree on the
|
||||
* stats layouts and on the class-handle/service-curve constants. (Hand-rolled
|
||||
* approximations of these structs made DIOCGETQSTATS fail with EINVAL, because
|
||||
* pfctl's sizeof(struct hfsc_class_stats) did not match the kernel's.)
|
||||
*/
|
||||
#ifndef nsecuptime
|
||||
#define nsecuptime() getnsecuptime()
|
||||
#endif
|
||||
|
||||
#include <net/hfsc.h>
|
||||
#include <net/fq_codel.h>
|
||||
|
||||
/* OpenBSD's global nanoseconds-per-clocktick (HFSC_CLK_PER_TICK). The port
|
||||
* has no clock_rate machinery, so derive it from the kernel's HZ. */
|
||||
#ifndef tick_nsec
|
||||
#define tick_nsec (1000000000LL / HZ)
|
||||
#endif
|
||||
|
||||
/* ifq scheduler hand-off (see pfsrc/ifq.c): enter returns the scheduler's
|
||||
* queue state when it owns the interface queue, NULL otherwise. */
|
||||
void *ifq_q_enter(struct ifqueue *, const struct ifq_ops *);
|
||||
void ifq_q_leave(struct ifqueue *, void *);
|
||||
int pfk_ifq_transmit(struct ifqueue *, struct mbuf *);
|
||||
int pf_kifq_input(struct pfk_ifnet *, struct mbuf *);
|
||||
|
||||
|
||||
/*
|
||||
* PFLINUX notes on the scheduler back ends:
|
||||
*
|
||||
* - splnet()/splx(): OpenBSD raises the software interrupt level around
|
||||
* class-table surgery. This port runs the whole data path in one ordered
|
||||
* work queue and guards the queue with ifq_mtx, so these are no-ops; they
|
||||
* are kept as markers of the critical regions.
|
||||
* - KERNEL_ASSERT_LOCKED(): the schedulers are only reached from ifq/pf paths
|
||||
* that already hold the queue mutex; asserted structurally, not by spl.
|
||||
* - HFSC_ENABLED(), PKTCNTR_INC(), the class-handle constants and the stats
|
||||
* layouts all come from the vendored net/hfsc.h / net/fq_codel.h above, so
|
||||
* kernel and pfctl cannot drift apart.
|
||||
*/
|
||||
#define splnet() 0
|
||||
#define splx(s) ((void)(s))
|
||||
#define KERNEL_ASSERT_LOCKED() do { } while (0)
|
||||
|
||||
#endif /* _PFK_H */
|
||||
|
||||
@@ -74,6 +74,79 @@ ml_dequeue(struct mbuf_list *ml)
|
||||
return m;
|
||||
}
|
||||
|
||||
|
||||
/* BSD <net/if_var.h> mbuf_list helpers (net/if_var.h ml_*) */
|
||||
static inline void
|
||||
ml_init(struct mbuf_list *ml)
|
||||
{
|
||||
ml->ml_first = ml->ml_last = NULL;
|
||||
ml->ml_len = 0;
|
||||
}
|
||||
|
||||
static inline void
|
||||
ml_enqueue(struct mbuf_list *ml, struct mbuf *m)
|
||||
{
|
||||
m->m_next = NULL;
|
||||
if (ml->ml_last == NULL)
|
||||
ml->ml_first = m;
|
||||
else
|
||||
ml->ml_last->m_next = m;
|
||||
ml->ml_last = m;
|
||||
ml->ml_len++;
|
||||
}
|
||||
|
||||
static inline struct mbuf *
|
||||
ml_dequeue_tail(struct mbuf_list *ml)
|
||||
{
|
||||
struct mbuf *m, *prev = NULL;
|
||||
|
||||
if (ml->ml_first == NULL)
|
||||
return NULL;
|
||||
for (m = ml->ml_first; m->m_next; m = m->m_next)
|
||||
prev = m;
|
||||
if (prev == NULL) /* single element */
|
||||
ml->ml_first = NULL;
|
||||
else
|
||||
prev->m_next = NULL;
|
||||
ml->ml_last = prev;
|
||||
ml->ml_len--;
|
||||
m->m_next = NULL;
|
||||
return m;
|
||||
}
|
||||
|
||||
/* append every packet in m2 to m1 (m2 becomes empty) */
|
||||
static inline void
|
||||
ml_enlist(struct mbuf_list *m1, struct mbuf_list *m2)
|
||||
{
|
||||
if (m2->ml_first == NULL)
|
||||
return;
|
||||
if (m1->ml_last == NULL)
|
||||
m1->ml_first = m2->ml_first;
|
||||
else
|
||||
m1->ml_last->m_next = m2->ml_first;
|
||||
m1->ml_last = m2->ml_last;
|
||||
m1->ml_len += m2->ml_len;
|
||||
m2->ml_first = m2->ml_last = NULL;
|
||||
m2->ml_len = 0;
|
||||
}
|
||||
|
||||
static inline unsigned int
|
||||
ml_len(const struct mbuf_list *ml)
|
||||
{
|
||||
return (unsigned int)ml->ml_len;
|
||||
}
|
||||
|
||||
static inline int
|
||||
ml_empty(const struct mbuf_list *ml)
|
||||
{
|
||||
return ml->ml_first == NULL;
|
||||
}
|
||||
|
||||
void ml_purge(struct mbuf_list *ml); /* compat/pfk_mbuf.c (needs m_freem) */
|
||||
|
||||
#define MBUF_LIST_FIRST(_ml) ((_ml)->ml_first)
|
||||
#define MBUF_LIST_INITIALIZER() { NULL, NULL, 0 }
|
||||
|
||||
/* egress interface (default-route dev), maintained on fib/net events */
|
||||
struct pfk_ifnet *pfk_egress_if(void);
|
||||
void pfk_egress_refresh(void);
|
||||
|
||||
@@ -73,7 +73,7 @@ struct pkthdr {
|
||||
u_int32_t ph_flowid;
|
||||
u_int ph_ifidx;
|
||||
u_int ph_rtableid;
|
||||
struct pfk_timeval ph_timestamp;
|
||||
int64_t ph_timestamp; /* OpenBSD 8.0: nanoseconds */
|
||||
struct pfk_ifnet *ifp;
|
||||
struct pfk_ifnet *rcvif;
|
||||
u_short ph; /* tso segs: unused, kept as pad */
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_pfsrc/divert_sock.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
include/pfk_api.h \
|
||||
|
||||
pfsrc/divert_sock.o: $(deps_pfsrc/divert_sock.o)
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -1828,6 +1828,8 @@ deps_pfsrc/pf.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
pfsrc/pf.o: $(deps_pfsrc/pf.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_if.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
pfsrc/pf_if.o: $(deps_pfsrc/pf_if.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_ioctl.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
pfsrc/pf_ioctl.o: $(deps_pfsrc/pf_ioctl.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_lb.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
pfsrc/pf_lb.o: $(deps_pfsrc/pf_lb.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_norm.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
pfsrc/pf_norm.o: $(deps_pfsrc/pf_norm.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_osfp.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
pfsrc/pf_osfp.o: $(deps_pfsrc/pf_osfp.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_ruleset.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
pfsrc/pf_ruleset.o: $(deps_pfsrc/pf_ruleset.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_syncookies.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
pfsrc/pf_syncookies.o: $(deps_pfsrc/pf_syncookies.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_pfsrc/pf_table.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
|
||||
pfsrc/pf_table.o: $(deps_pfsrc/pf_table.o)
|
||||
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_pfsrc/pfk_hooks.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
include/pfk_api.h \
|
||||
|
||||
pfsrc/pfk_hooks.o: $(deps_pfsrc/pfk_hooks.o)
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_pfsrc/pfk_mod.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
include/pfk_api.h \
|
||||
|
||||
pfsrc/pfk_mod.o: $(deps_pfsrc/pfk_mod.o)
|
||||
|
||||
@@ -1828,6 +1828,8 @@ deps_pfsrc/pfk_pflog.o := \
|
||||
include/compat/sys/rwlock.h \
|
||||
include/compat/sys/mutex.h \
|
||||
include/compat/sys/pclock.h \
|
||||
include/net/hfsc.h \
|
||||
include/net/fq_codel.h \
|
||||
include/pfk_api.h \
|
||||
|
||||
pfsrc/pfk_pflog.o: $(deps_pfsrc/pfk_pflog.o)
|
||||
|
||||
@@ -332,7 +332,9 @@ control_law(struct codel *cd, struct codel_params *cp, int64_t rts)
|
||||
unsigned int idx;
|
||||
|
||||
KASSERT(cd->drops > 0);
|
||||
idx = min(cd->drops - 1, nitems(codel_intervals) - 1);
|
||||
/* PFLINUX: Linux' min() type-checks its arguments; nitems() is size_t */
|
||||
idx = min_t(unsigned int, cd->drops - 1,
|
||||
(unsigned int)(nitems(codel_intervals) - 1));
|
||||
cd->next = rts + cp->intervals[idx];
|
||||
}
|
||||
|
||||
|
||||
+12
-3
@@ -335,9 +335,9 @@ hfsc_class_purge(struct hfsc_class *cl, struct mbuf_list *ml)
|
||||
}
|
||||
|
||||
static inline u_int
|
||||
hfsc_more_slots(u_int current)
|
||||
hfsc_more_slots(u_int cur) /* PFLINUX: 'current' is a kernel macro */
|
||||
{
|
||||
u_int want = current * 2;
|
||||
u_int want = cur * 2;
|
||||
|
||||
return (want > HFSC_MAX_CLASSES ? HFSC_MAX_CLASSES : want);
|
||||
}
|
||||
@@ -481,7 +481,16 @@ hfsc_pf_qstats(struct pf_queuespec *q, void *ubuf, int *nbytes)
|
||||
if (hif == NULL)
|
||||
return (EBADF);
|
||||
|
||||
if ((cl = hfsc_clh2cph(hif, q->qid)) == NULL) {
|
||||
/*
|
||||
* A root queue's class is created with handle HFSC_ROOT_CLASS | qid
|
||||
* (see hfsc_pf_addqueue), so looking the queue up by its plain pf qid
|
||||
* only finds non-root classes: fall back to the root handle, otherwise
|
||||
* "pfctl -s queue" fails with EINVAL on the first (root) queue.
|
||||
*/
|
||||
if ((cl = hfsc_clh2cph(hif, q->qid)) == NULL &&
|
||||
(q->flags & PFQS_ROOTCLASS))
|
||||
cl = hfsc_clh2cph(hif, HFSC_ROOT_CLASS | q->qid);
|
||||
if (cl == NULL) {
|
||||
ifq_q_leave(&ifp->if_snd, hif);
|
||||
return (EINVAL);
|
||||
}
|
||||
|
||||
@@ -662,6 +662,11 @@ pf_create_queues(void)
|
||||
}
|
||||
|
||||
qif->disc = qif->pfqops->pfq_alloc(ifp);
|
||||
if (qif->disc == NULL) {
|
||||
error = ENOBUFS;
|
||||
free(qif, M_PF, sizeof(*qif));
|
||||
goto error;
|
||||
}
|
||||
|
||||
qif->next = list;
|
||||
list = qif;
|
||||
|
||||
@@ -204,6 +204,22 @@ pfk_pkt_work(struct work_struct *w)
|
||||
|
||||
|
||||
if (m == m0) {
|
||||
/*
|
||||
* Queueing: when a pf `queue' rule attached a scheduler to the
|
||||
* egress interface, hand the packet to it instead of sending it
|
||||
* straight out - this is what makes bandwidth limits real.
|
||||
*/
|
||||
if (action == PF_PASS && pk->dir == PF_OUT &&
|
||||
pskb_may_pull(pk->skb, sizeof(struct iphdr)) &&
|
||||
skb_dst(pk->skb) && !IS_ERR(skb_dst(pk->skb)) &&
|
||||
pf_kifq_input(skb_dst(pk->skb)->dev ?
|
||||
pfk_ifp_bydev(skb_dst(pk->skb)->dev) : NULL, m)) {
|
||||
orig = pk->skb;
|
||||
pk->skb = NULL; /* the queue owns the packet */
|
||||
kfree_skb(orig);
|
||||
kfree(pk);
|
||||
return;
|
||||
}
|
||||
/* same buffer, possibly rewritten in place: our wrapper
|
||||
* held one extra skb reference — release it */
|
||||
m_freem(m);
|
||||
|
||||
@@ -115,11 +115,22 @@ pfk_pflog_pull(struct mbuf *m, char *buf, size_t want)
|
||||
{
|
||||
struct sk_buff *skb = m ? m_to_skb(m) : NULL;
|
||||
size_t n;
|
||||
int off;
|
||||
|
||||
if (skb == NULL)
|
||||
return 0;
|
||||
n = min_t(size_t, want, skb->len);
|
||||
if (skb_copy_bits(skb, 0, buf, n))
|
||||
/*
|
||||
* BSD logs from the network header; pf may already have advanced the
|
||||
* mbuf/skb data pointer to the transport header while parsing, so
|
||||
* anchor explicitly at the IP header instead of at skb->data.
|
||||
*/
|
||||
off = skb_network_offset(skb);
|
||||
if (off < 0)
|
||||
off = 0;
|
||||
if (off > skb->len)
|
||||
return 0;
|
||||
n = min_t(size_t, want, skb->len - off);
|
||||
if (skb_copy_bits(skb, off, buf, n))
|
||||
return 0;
|
||||
return n;
|
||||
}
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 2.8 MiB |
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Executable
+149
@@ -0,0 +1,149 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Assemble the final MP4s: silent video track + MiniMax narration + subtitles.
|
||||
|
||||
Each narration segment was recorded as its own audio file, generated by
|
||||
MiniMax with a female voice for the language of the track. The segments are
|
||||
placed at the moment their beat starts on screen, so the voice, the caption
|
||||
and the terminal action always agree.
|
||||
|
||||
./scripts/screencast/assemble.py --phase 04-roteamento --lang en
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
REPO = os.path.dirname(os.path.dirname(HERE))
|
||||
CAPS = os.path.join(REPO, "out", "screencast", "caps")
|
||||
VO = os.path.join(REPO, "out", "screencast", "vo")
|
||||
OUT = os.path.join(REPO, "out", "screencast")
|
||||
ART = os.path.join(OUT, "art")
|
||||
|
||||
# which phases draw the lab diagram over the terminal
|
||||
DIAGRAM_PHASES = {"00-ambiente", "04-roteamento", "05-balanceamento"}
|
||||
# phases that show the module/host instead of the firewall VM
|
||||
HOST_PHASES = {"01-implementacao"}
|
||||
|
||||
PHASE_ORDER = ["00-ambiente", "01-implementacao", "02-ativacao", "03-regras",
|
||||
"04-roteamento", "05-balanceamento", "06-qos"]
|
||||
|
||||
|
||||
def run(cmd):
|
||||
subprocess.run(cmd, check=True)
|
||||
|
||||
|
||||
def probe(path):
|
||||
out = subprocess.run(["ffprobe", "-v", "error", "-show_entries",
|
||||
"format=duration", "-of", "csv=p=0", path],
|
||||
capture_output=True, text=True).stdout.strip()
|
||||
return float(out or 0)
|
||||
|
||||
|
||||
def audio_duration(path):
|
||||
return probe(path)
|
||||
|
||||
|
||||
def make_video(phase, lang):
|
||||
"""Render the silent video (terminal + diagram + burned-in subtitles)."""
|
||||
cast = os.path.join(CAPS, f"{phase}.cast")
|
||||
nar = os.path.join(HERE, "narration", f"{phase}.{lang}.json")
|
||||
if not os.path.exists(cast):
|
||||
raise SystemExit(f"[assemble] missing capture: {cast}")
|
||||
art = None
|
||||
if os.path.isdir(ART):
|
||||
imgs = sorted(f for f in os.listdir(ART) if f.endswith((".jpg", ".png")))
|
||||
if imgs:
|
||||
art = os.path.join(ART, imgs[0])
|
||||
outdir = os.path.join(OUT, lang)
|
||||
os.makedirs(outdir, exist_ok=True)
|
||||
silent = os.path.join(outdir, f"{phase}.silent.mp4")
|
||||
index = PHASE_ORDER.index(phase) + 1
|
||||
cmd = [sys.executable, os.path.join(HERE, "render_video.py"),
|
||||
"--cast", cast, "--narration", nar, "--out", silent,
|
||||
"--lang", lang, "--phase-no", str(index),
|
||||
"--phase-total", str(len(PHASE_ORDER))]
|
||||
if phase in DIAGRAM_PHASES:
|
||||
cmd.append("--diagram")
|
||||
if art:
|
||||
cmd += ["--art", art]
|
||||
run(cmd)
|
||||
return silent
|
||||
|
||||
|
||||
def mux_audio(phase, lang, silent):
|
||||
"""Lay every narration segment on the timeline and add it to the video."""
|
||||
meta = json.load(open(os.path.join(CAPS, f"{phase}.json")))
|
||||
beats = {b["id"]: b for b in meta["beats"]}
|
||||
nar = json.load(open(os.path.join(HERE, "narration", f"{phase}.{lang}.json")))
|
||||
vodir = os.path.join(VO, f"{phase}.{lang}")
|
||||
|
||||
segments = []
|
||||
for i, seg in enumerate(nar["segments"]):
|
||||
beat = beats.get(seg["beat"])
|
||||
audio = os.path.join(vodir, f"{i:02d}-{seg['beat']}.mp3")
|
||||
if beat is None or not os.path.exists(audio):
|
||||
sys.stderr.write(f"[assemble] missing audio: {audio}\n")
|
||||
continue
|
||||
segments.append((beat["start"], audio))
|
||||
|
||||
tmp = os.path.join(OUT, "_tmp")
|
||||
os.makedirs(tmp, exist_ok=True)
|
||||
if not segments:
|
||||
out = os.path.join(tmp, f"{phase}.{lang}.m4v")
|
||||
run(["ffmpeg", "-y", "-loglevel", "error", "-i", silent,
|
||||
"-c:v", "copy", "-an", out])
|
||||
return out
|
||||
|
||||
vdur = probe(silent)
|
||||
inputs = []
|
||||
filters = []
|
||||
# input 0 is the video; the narration files are inputs 1..N
|
||||
for k, (start, audio) in enumerate(segments):
|
||||
inputs += ["-i", audio]
|
||||
delay = int(start * 1000)
|
||||
# pad so every segment reaches the end of the video, then the mix is
|
||||
# trimmed to the video length
|
||||
filters.append(f"[{k + 1}:a]adelay={delay}|{delay},apad[aud{k}]")
|
||||
mix_in = "".join(f"[aud{k}]" for k in range(len(segments)))
|
||||
filters.append(f"{mix_in}amix=inputs={len(segments)}:normalize=0,"
|
||||
f"atrim=0:{vdur:.3f},asetpts=N/SR/TB[mixed]")
|
||||
cmd = ["ffmpeg", "-y", "-loglevel", "error", "-i", silent] + inputs + \
|
||||
["-filter_complex", ";".join(filters),
|
||||
"-map", "0:v", "-map", "[mixed]", "-c:v", "copy", "-c:a", "aac",
|
||||
"-b:a", "160k", "-ar", "44100", "-ac", "2",
|
||||
os.path.join(tmp, f"{phase}.{lang}.m4v")]
|
||||
run(cmd)
|
||||
return os.path.join(tmp, f"{phase}.{lang}.m4v")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--phase", required=True)
|
||||
ap.add_argument("--lang", required=True, choices=["en", "pt"])
|
||||
ap.add_argument("--keep-silent", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
phase, lang = args.phase, args.lang
|
||||
silent = make_video(phase, lang)
|
||||
muxed = mux_audio(phase, lang, silent)
|
||||
|
||||
outdir = os.path.join(OUT, lang)
|
||||
os.makedirs(outdir, exist_ok=True)
|
||||
final = os.path.join(outdir, f"{phase}.mp4")
|
||||
run(["ffmpeg", "-y", "-loglevel", "error", "-i", muxed,
|
||||
"-c", "copy", "-movflags", "+faststart", final])
|
||||
if not args.keep_silent:
|
||||
os.remove(silent)
|
||||
os.remove(muxed)
|
||||
|
||||
dur = probe(final)
|
||||
size = os.path.getsize(final) / 1e6
|
||||
print(f"[assemble] {final} {dur:.1f}s {size:.1f}MB")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+48
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build every screencast: capture -> narration -> video -> full movie.
|
||||
#
|
||||
# ./scripts/screencast/build-all.sh # everything
|
||||
# ./scripts/screencast/build-all.sh 04-roteamento
|
||||
set -uo pipefail
|
||||
cd "$(dirname "$0")/../.."
|
||||
REPO=$(pwd)
|
||||
|
||||
PHASES=("$@")
|
||||
[ ${#PHASES[@]} -eq 0 ] && PHASES=(00-ambiente 01-implementacao 02-ativacao 03-regras \
|
||||
04-roteamento 05-balanceamento 06-qos)
|
||||
|
||||
echo "== narration audio =="
|
||||
for lang in en pt; do
|
||||
python3 scripts/screencast/make_audio.py --lang "$lang" | tail -1
|
||||
done
|
||||
|
||||
echo "== sessions =="
|
||||
for p in "${PHASES[@]}"; do
|
||||
# the activation phase shows the module being loaded, so it is the one
|
||||
# that must not have it preloaded
|
||||
extra=""
|
||||
[ "$p" = "02-ativacao" ] || extra="--ensure-pf"
|
||||
python3 scripts/screencast/record.py --plan "scripts/screencast/plans/$p.json" \
|
||||
--out "out/screencast/caps/$p.cast" --pace $extra
|
||||
done
|
||||
|
||||
echo "== videos =="
|
||||
for lang in en pt; do
|
||||
for p in "${PHASES[@]}"; do
|
||||
python3 scripts/screencast/assemble.py --phase "$p" --lang "$lang" || echo "FALHOU: $p $lang"
|
||||
done
|
||||
done
|
||||
|
||||
echo "== full movie =="
|
||||
for lang in en pt; do
|
||||
python3 scripts/screencast/join.py --lang "$lang" || echo "FALHOU: join $lang"
|
||||
done
|
||||
|
||||
echo
|
||||
echo "== resultado =="
|
||||
for f in out/screencast/en/*.mp4 out/screencast/pt/*.mp4; do
|
||||
[ -e "$f" ] || continue
|
||||
d=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$f")
|
||||
a=$(ffprobe -v error -select_streams a -show_entries stream=codec_name -of csv=p=0 "$f")
|
||||
printf '%-52s %6.1fs audio=%s\n' "$f" "$d" "${a:-none}"
|
||||
done
|
||||
Executable
+189
@@ -0,0 +1,189 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Join the phases into one continuous video per language, with transitions.
|
||||
|
||||
Each phase fades into a short "chapter" card that names it, and the card
|
||||
fades into the next phase - one continuous stream, no hard cuts, so the
|
||||
viewer always knows where they are.
|
||||
|
||||
./scripts/screencast/join.py --lang en
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
REPO = os.path.dirname(os.path.dirname(HERE))
|
||||
OUT = os.path.join(REPO, "out", "screencast")
|
||||
ART = os.path.join(OUT, "art")
|
||||
|
||||
PHASE_ORDER = ["00-ambiente", "01-implementacao", "02-ativacao", "03-regras",
|
||||
"04-roteamento", "05-balanceamento", "06-qos"]
|
||||
|
||||
TITLES = {
|
||||
"00-ambiente": ("The lab environment", "O ambiente de testes",
|
||||
"Four Alpine VMs under libvirt", "Quatro VMs Alpine sob libvirt"),
|
||||
"01-implementacao": ("Implementation in the kernel", "Implementacao no kernel",
|
||||
"OpenBSD 8.0 sources + a Linux compatibility layer",
|
||||
"Fontes do OpenBSD 8.0 + camada de compatibilidade Linux"),
|
||||
"02-ativacao": ("Activation", "Ativacao",
|
||||
"insmod pf.ko, then pfctl over the OpenBSD ioctl ABI",
|
||||
"insmod pf.ko e o pfctl pela ABI de ioctls do OpenBSD"),
|
||||
"03-regras": ("Basic rules", "Regras basicas",
|
||||
"Tables, anchors, keep state, block drop",
|
||||
"Tabelas, anchors, keep state, block drop"),
|
||||
"04-roteamento": ("Routing", "Roteamento",
|
||||
"route-to and reply-to", "route-to e reply-to"),
|
||||
"05-balanceamento": ("Load balancing", "Balanceamento",
|
||||
"Pools round-robin, source-hash, dup-to",
|
||||
"Pools round-robin, source-hash, dup-to"),
|
||||
"06-qos": ("Quality of service", "Qualidade de servico",
|
||||
"hfsc queues in the data path", "Filas hfsc no caminho de dados"),
|
||||
}
|
||||
|
||||
|
||||
def run(cmd):
|
||||
subprocess.run(cmd, check=True)
|
||||
|
||||
|
||||
def probe(path):
|
||||
out = subprocess.run(["ffprobe", "-v", "error", "-show_entries",
|
||||
"format=duration", "-of", "csv=p=0", path],
|
||||
capture_output=True, text=True).stdout.strip()
|
||||
return float(out or 0)
|
||||
|
||||
|
||||
def chapter_card(phase, lang, out_path, seconds=1.8):
|
||||
"""A short card naming the phase; it is the pivot of the crossfade."""
|
||||
title = TITLES[phase][0 if lang == "en" else 1]
|
||||
sub = TITLES[phase][2 if lang == "en" else 3]
|
||||
num = PHASE_ORDER.index(phase) + 1
|
||||
art = None
|
||||
if os.path.isdir(ART):
|
||||
imgs = sorted(f for f in os.listdir(ART) if f.endswith((".jpg", ".png")))
|
||||
if imgs:
|
||||
art = os.path.join(ART, imgs[0])
|
||||
|
||||
script = os.path.join(OUT, "_chapter.py")
|
||||
with open(script, "w") as f:
|
||||
f.write(f"""
|
||||
import os, sys
|
||||
from PIL import Image, ImageDraw, ImageFont
|
||||
W, H = 1920, 1080
|
||||
BG = (12, 16, 22); ACCENT = (88, 168, 255); FG = (240, 246, 252); MUTED = (140, 150, 165)
|
||||
mono = "/usr/share/fonts/dejavu-sans-mono-fonts/DejaVuSansMono.ttf"
|
||||
sans = "/usr/share/fonts/dejavu-sans-fonts/DejaVuSans.ttf"
|
||||
sansb = "/usr/share/fonts/dejavu-sans-fonts/DejaVuSans-Bold.ttf"
|
||||
art = {art!r}
|
||||
img = Image.new("RGB", (W, H), BG)
|
||||
if art and os.path.exists(art):
|
||||
img = Image.blend(img, Image.open(art).resize((W, H)).convert("RGB"), 0.5)
|
||||
d = ImageDraw.Draw(img, "RGBA")
|
||||
d.rectangle([0, 0, W, 8], fill=ACCENT)
|
||||
d.text((140, 430), {title!r}, font=ImageFont.truetype(sansb, 62), fill=FG)
|
||||
d.text((140, 520), {sub!r}, font=ImageFont.truetype(sans, 30), fill=MUTED)
|
||||
d.text((140, 586), "PHASE %d OF %d" % ({num}, {len(PHASE_ORDER)}),
|
||||
font=ImageFont.truetype(mono, 24), fill=ACCENT)
|
||||
d.text((140, 640), "PF on Linux - OpenBSD PF 8.0 no kernel Linux",
|
||||
font=ImageFont.truetype(mono, 24), fill=MUTED)
|
||||
img.save(sys.argv[1])
|
||||
""")
|
||||
png = out_path + ".png"
|
||||
run([sys.executable, script, png])
|
||||
have = subprocess.run(["ffmpeg", "-hide_banner", "-encoders"],
|
||||
capture_output=True, text=True).stdout
|
||||
venc = (["-c:v", "libx264", "-preset", "veryfast", "-crf", "20", "-profile:v", "high"]
|
||||
if "libx264" in have else
|
||||
["-c:v", "libopenh264", "-b:v", "6M", "-profile:v", "high"])
|
||||
run(["ffmpeg", "-y", "-loglevel", "error", "-loop", "1", "-i", png,
|
||||
"-t", str(seconds), "-r", "30"] + venc + ["-pix_fmt", "yuv420p", out_path])
|
||||
os.remove(png)
|
||||
os.remove(script)
|
||||
return out_path
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--lang", required=True, choices=["en", "pt"])
|
||||
ap.add_argument("--out")
|
||||
ap.add_argument("--fade", type=float, default=0.7)
|
||||
args = ap.parse_args()
|
||||
|
||||
lang = args.lang
|
||||
tmp = os.path.join(OUT, "_tmp")
|
||||
os.makedirs(tmp, exist_ok=True)
|
||||
|
||||
segs = [] # (path, duration)
|
||||
for phase in PHASE_ORDER:
|
||||
src = os.path.join(OUT, lang, f"{phase}.mp4")
|
||||
if not os.path.exists(src):
|
||||
sys.stderr.write(f"[join] missing {src}, skipping\n")
|
||||
continue
|
||||
segs.append((src, probe(src)))
|
||||
card = chapter_card(phase, lang, os.path.join(tmp, f"card-{phase}.mp4"))
|
||||
segs.append((card, probe(card)))
|
||||
|
||||
if not segs:
|
||||
raise SystemExit("[join] nothing to join")
|
||||
|
||||
inputs = []
|
||||
for path, _ in segs:
|
||||
inputs += ["-i", path]
|
||||
|
||||
# sequential xfade: each segment fades into the next
|
||||
filters = []
|
||||
prev = "0:v"
|
||||
offset = segs[0][1] - args.fade
|
||||
for i in range(1, len(segs)):
|
||||
filters.append(f"[{prev}][{i}:v]xfade=transition=fade:"
|
||||
f"duration={args.fade}:offset={offset:.3f}[v{i}]")
|
||||
prev = f"v{i}"
|
||||
offset += segs[i][1] - args.fade
|
||||
|
||||
# The audio has to follow the same timeline: with xfade the segments
|
||||
# overlap by the fade duration, so each one starts `fade` earlier than
|
||||
# the plain sum would suggest.
|
||||
audio_at = []
|
||||
acc = 0.0
|
||||
for i, (_, dur) in enumerate(segs):
|
||||
audio_at.append(max(0.0, acc - args.fade * i))
|
||||
acc += dur
|
||||
|
||||
# Only the phase videos carry narration; the chapter cards are silent.
|
||||
# Their input indices are the even ones in `inputs`.
|
||||
phase_idx = [i for i in range(len(segs)) if i % 2 == 0]
|
||||
afilters = []
|
||||
labels = []
|
||||
for k, i in enumerate(phase_idx):
|
||||
ms = int(audio_at[i] * 1000)
|
||||
afilters.append(f"[{i}:a]adelay={ms}|{ms}[a{k}]")
|
||||
labels.append(f"[a{k}]")
|
||||
afilters.append("".join(labels) +
|
||||
f"amix=inputs={len(phase_idx)}:normalize=0,"
|
||||
f"atrim=0:{audio_at[-1] + segs[-1][1]:.3f},"
|
||||
f"asetpts=N/SR/TB[amix]")
|
||||
filters += afilters
|
||||
amap = "[amix]"
|
||||
|
||||
out = args.out or os.path.join(OUT, lang, "pf-on-linux-full.mp4")
|
||||
have = subprocess.run(["ffmpeg", "-hide_banner", "-encoders"],
|
||||
capture_output=True, text=True).stdout
|
||||
if "libx264" in have:
|
||||
venc = ["-c:v", "libx264", "-preset", "veryfast", "-crf", "20",
|
||||
"-profile:v", "high", "-level", "4.0"]
|
||||
elif "libopenh264" in have:
|
||||
venc = ["-c:v", "libopenh264", "-b:v", "6M", "-profile:v", "high"]
|
||||
else:
|
||||
venc = ["-c:v", "libvpx-vp9", "-b:v", "0", "-crf", "30", "-row-mt", "1",
|
||||
"-cpu-used", "4"]
|
||||
# the phases already carry their narration; the cards are silent
|
||||
run(["ffmpeg", "-y", "-loglevel", "error"] + inputs +
|
||||
["-filter_complex", ";".join(filters), "-map", f"[{prev}]",
|
||||
"-map", amap] + venc +
|
||||
["-pix_fmt", "yuv420p", "-movflags", "+faststart", out])
|
||||
print(f"[join] {out} {probe(out):.1f}s")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+121
@@ -0,0 +1,121 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Generate the narration audio for every screencast phase with MiniMax TTS.
|
||||
|
||||
Female voices, one per language:
|
||||
en -> English_ConfidentWoman
|
||||
pt -> Portuguese_ConfidentWoman
|
||||
|
||||
Each *segment* of a narration script becomes its own audio file, so the
|
||||
renderer can start the voice exactly when the matching beat appears on
|
||||
screen and burn the same sentence as a subtitle.
|
||||
|
||||
./scripts/screencast/make_audio.py --lang en
|
||||
./scripts/screencast/make_audio.py --lang pt --phase 04-roteamento
|
||||
"""
|
||||
import argparse
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
NARR = os.path.join(HERE, "narration")
|
||||
VO_DIR = os.path.join(HERE, "..", "..", "out", "screencast", "vo")
|
||||
|
||||
VOICES = {
|
||||
"en": {"voice_id": "English_ConfidentWoman", "language_boost": "English"},
|
||||
"pt": {"voice_id": "Portuguese_ConfidentWoman", "language_boost": "Portuguese"},
|
||||
}
|
||||
|
||||
# MiniMax occasionally stumbles on the punctuation of long sentences; keeping
|
||||
# the text plain (no dashes, no slashes) makes the prosody predictable.
|
||||
MODEL = "speech-02-hd"
|
||||
API = "https://api.minimax.io/v1/t2a_v2"
|
||||
|
||||
|
||||
def decode_audio(payload):
|
||||
"""MiniMax returns the audio as a hex string (it starts with the ASCII
|
||||
bytes of the ID3 header, 49443304). Older/other endpoints use base64,
|
||||
so both are handled."""
|
||||
if len(payload) > 8 and all(c in "0123456789abcdefABCDEF" for c in payload[:64]):
|
||||
return bytes.fromhex(payload)
|
||||
data = payload.replace("-", "+").replace("_", "/")
|
||||
data += "=" * (-len(data) % 4)
|
||||
return base64.b64decode(data, validate=False)
|
||||
|
||||
|
||||
def tts(text, voice, lang, out_path, retries=3):
|
||||
body = {
|
||||
"model": MODEL,
|
||||
"text": text,
|
||||
"stream": False,
|
||||
"voice_setting": {
|
||||
"voice_id": voice["voice_id"],
|
||||
"speed": 0.97, "vol": 1.0, "pitch": 0, "emotion": "neutral",
|
||||
},
|
||||
"audio_setting": {"sample_rate": 44100, "bitrate": 128000,
|
||||
"format": "mp3", "channel": 1},
|
||||
"language_boost": voice["language_boost"],
|
||||
}
|
||||
key = os.environ.get("MINIMAX_API_KEY")
|
||||
if not key:
|
||||
raise SystemExit("MINIMAX_API_KEY is not set")
|
||||
for attempt in range(retries):
|
||||
req = urllib.request.Request(
|
||||
API, data=json.dumps(body).encode(),
|
||||
headers={"Authorization": f"Bearer {key}", "Content-Type": "application/json"})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=120) as r:
|
||||
data = json.loads(r.read())
|
||||
if data.get("base_resp", {}).get("status_code") != 0:
|
||||
raise RuntimeError(str(data.get("base_resp")))
|
||||
audio = data.get("data", {}).get("audio")
|
||||
if not audio:
|
||||
raise RuntimeError("no audio in response")
|
||||
with open(out_path, "wb") as f:
|
||||
f.write(decode_audio(audio))
|
||||
return True
|
||||
except Exception as exc: # noqa: BLE001
|
||||
sys.stderr.write(f"[tts] attempt {attempt + 1} failed: {exc}\n")
|
||||
time.sleep(2 + attempt * 3)
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--lang", required=True, choices=["en", "pt"])
|
||||
ap.add_argument("--phase", help="only this phase")
|
||||
ap.add_argument("--force", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
voice = VOICES[args.lang]
|
||||
files = sorted(f for f in os.listdir(NARR) if f.endswith(f".{args.lang}.json"))
|
||||
if args.phase:
|
||||
files = [f for f in files if f.startswith(args.phase + ".")]
|
||||
|
||||
total = 0
|
||||
for fn in files:
|
||||
phase = fn.split(".")[0]
|
||||
script = json.load(open(os.path.join(NARR, fn)))
|
||||
outdir = os.path.abspath(os.path.join(VO_DIR, f"{phase}.{args.lang}"))
|
||||
os.makedirs(outdir, exist_ok=True)
|
||||
for i, seg in enumerate(script["segments"]):
|
||||
out = os.path.join(outdir, f"{i:02d}-{seg['beat']}.mp3")
|
||||
if os.path.exists(out) and not args.force:
|
||||
print(f"skip {phase}.{args.lang} seg{i} ({seg['beat']})")
|
||||
total += 1
|
||||
continue
|
||||
ok = tts(seg["text"], voice, args.lang, out)
|
||||
size = os.path.getsize(out) if os.path.exists(out) else 0
|
||||
print(f"{'ok ' if ok else 'FAIL'} {phase}.{args.lang} seg{i} "
|
||||
f"({seg['beat']}) {size} bytes")
|
||||
total += 1
|
||||
print(f"[audio] {total} segments processed for lang={args.lang}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+227
@@ -0,0 +1,227 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Write the narration scripts for every screencast phase, in English and in
|
||||
Brazilian Portuguese.
|
||||
|
||||
Each segment belongs to one *beat* of the recorded session (see
|
||||
scripts/screencast/plans/*.json). The renderer places the segment's audio at
|
||||
the moment that beat starts on screen, and burns the same text as a subtitle,
|
||||
so the voice, the caption and the terminal action always agree.
|
||||
|
||||
A segment may also carry:
|
||||
highlight - nodes of the lab diagram to light up while it is spoken
|
||||
diagram_caption - one-line caption shown inside the diagram panel
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
|
||||
OUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "narration")
|
||||
os.makedirs(OUT, exist_ok=True)
|
||||
|
||||
EN_VOICE = "English_ConfidentWoman"
|
||||
PT_VOICE = "Portuguese_ConfidentWoman"
|
||||
|
||||
PHASES = {}
|
||||
|
||||
# ------------------------------------------------------------------ 00 ----
|
||||
PHASES["00-ambiente"] = {
|
||||
"en": {
|
||||
"title": "The lab environment",
|
||||
"subtitle": "Four Alpine VMs under libvirt, running a kernel built from source",
|
||||
"segments": [
|
||||
{"beat": "b1", "text": "Welcome to the PF on Linux demonstration. Before we look at the firewall itself, let me show you the environment where everything runs. These are four virtual machines managed by libvirt, all of them Alpine Linux, and the machine you are looking at is the firewall: a kernel we compiled from source, version seven point three point four, with our port of the OpenBSD packet filter built in as a module."},
|
||||
{"beat": "b2", "text": "The firewall has four interfaces. The management interface on the virtual network the hypervisor uses to reach us. Then three transit networks: ten point zero point one, where the client lives; ten point zero point two, where the server lives; and ten point zero point three, a second gateway we use to demonstrate routing decisions.", "highlight": ["cli", "pf0", "srv", "gw2"], "diagram_caption": "three transit networks, four interfaces"},
|
||||
{"beat": "b3", "text": "The OpenBSD tools are installed on this machine exactly as they ship: the original pfctl, a pflog daemon and our divert tool. The module is not loaded yet, which is where we will start.", "highlight": ["pf0"], "diagram_caption": "pfctl, pflogd and pdivert, straight from OpenBSD"},
|
||||
{"beat": "b4", "text": "And from the firewall we can reach the other machines directly. The client is on ten point zero point one point ten, the server on ten point zero point two point twenty, and the second gateway on ten point zero point three point two. Every command in this demonstration is real traffic crossing real interfaces.", "highlight": ["cli", "pf0", "srv", "gw2"], "diagram_caption": "cli, srv and gw2 reachable from pf0"},
|
||||
],
|
||||
},
|
||||
"pt": {
|
||||
"title": "O ambiente de testes",
|
||||
"subtitle": "Quatro VMs Alpine sob libvirt, rodando um kernel compilado do fonte",
|
||||
"segments": [
|
||||
{"beat": "b1", "text": "Bem-vindo à demonstração do PF on Linux. Antes de falar do firewall em si, deixa eu mostrar o ambiente onde tudo acontece. São quatro máquinas virtuais administradas pelo libvirt, todas com Alpine Linux, e a máquina que você está vendo é o firewall: um kernel que compilamos do fonte, versão sete ponto três ponto quatro, com o nosso port do packet filter do OpenBSD compilado dentro como módulo."},
|
||||
{"beat": "b2", "text": "O firewall tem quatro interfaces. A interface de gerência, na rede virtual que o hypervisor usa para falar com a gente. E depois três redes de trânsito: dez ponto zero ponto um, onde mora o cliente; dez ponto zero ponto dois, onde mora o servidor; e dez ponto zero ponto três, um segundo gateway que vamos usar para demonstrar decisões de roteamento.", "highlight": ["cli", "pf0", "srv", "gw2"], "diagram_caption": "três redes de transito, quatro interfaces"},
|
||||
{"beat": "b3", "text": "As ferramentas do OpenBSD estão instaladas nesta máquina exatamente como são distribuídas: o pfctl original, um daemon de pflog e a nossa ferramenta de divert. O módulo ainda não está carregado, e é por aí que a gente começa.", "highlight": ["pf0"], "diagram_caption": "pfctl, pflogd e pdivert originais do OpenBSD"},
|
||||
{"beat": "b4", "text": "E do firewall a gente alcança as outras máquinas diretamente. O cliente está em dez ponto zero ponto um ponto dez, o servidor em dez ponto zero ponto dois ponto vinte, e o segundo gateway em dez ponto zero ponto três ponto dois. Todo comando desta demonstração é tráfego real cruzando interfaces reais.", "highlight": ["cli", "pf0", "srv", "gw2"], "diagram_caption": "cli, srv e gw2 alcançáveis a partir do pf0"},
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------ 01 ----
|
||||
PHASES["01-implementacao"] = {
|
||||
"en": {
|
||||
"title": "Implementation in the kernel",
|
||||
"subtitle": "OpenBSD 8.0 sources plus a Linux compatibility layer equals pf.ko",
|
||||
"segments": [
|
||||
{"beat": "b1", "text": "This is the heart of the work. We did not rewrite the firewall. We took the OpenBSD eight point zero sources, nearly twenty seven thousand lines of C, and we kept them almost intact. On top of that we wrote a compatibility layer, about seven thousand lines, that translates the BSD kernel interfaces into Linux ones."},
|
||||
{"beat": "b2", "text": "The result is a normal Linux kernel module. Here modinfo shows us the license, the description and the vermagic, which tells us exactly which kernel this object was compiled against."},
|
||||
{"beat": "b3", "text": "The vermagic line matters: it is the kernel version, the compiler flags and the module signature. Load the wrong one and the kernel refuses it. Everything you will see in the next minutes is running inside that module."},
|
||||
{"beat": "b4", "text": "This file is what we load into the firewall. A single artifact, built out of tree, carrying the entire OpenBSD packet filter."},
|
||||
],
|
||||
},
|
||||
"pt": {
|
||||
"title": "Implementação no kernel",
|
||||
"subtitle": "Fontes do OpenBSD 8.0 mais uma camada de compatibilidade Linux resultam em pf.ko",
|
||||
"segments": [
|
||||
{"beat": "b1", "text": "Aqui está o coração do trabalho. A gente não reescreveu o firewall. Pegamos as fontes do OpenBSD oito ponto zero, quase vinte e sete mil linhas de C, e mantivemos praticamente intactas. Em cima disso escrevemos uma camada de compatibilidade, cerca de sete mil linhas, que traduz as interfaces de kernel do BSD para as do Linux."},
|
||||
{"beat": "b2", "text": "O resultado é um módulo de kernel Linux comum. Aqui o modinfo mostra a licença, a descrição e o vermagic, que nos diz exatamente com qual kernel este objeto foi compilado."},
|
||||
{"beat": "b3", "text": "A linha do vermagic importa: é a versão do kernel, as flags do compilador e a assinatura do módulo. Carregar o arquivo errado e o kernel recusa. Tudo que você vai ver nos próximos minutos roda dentro desse módulo."},
|
||||
{"beat": "b4", "text": "Este é o arquivo que carregamos no firewall. Um único artefato, compilado fora da árvore do kernel, carregando o packet filter inteiro do OpenBSD."},
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------ 02 ----
|
||||
PHASES["02-ativacao"] = {
|
||||
"en": {
|
||||
"title": "Activation",
|
||||
"subtitle": "insmod pf.ko, then pfctl talks to it over the OpenBSD ioctl ABI",
|
||||
"segments": [
|
||||
{"beat": "b1", "text": "First we make sure nothing is loaded, so the activation is honest. No module, no firewall."},
|
||||
{"beat": "b2", "text": "Now we insert it. The kernel accepts the module and the log shows our initialization message. This is the OpenBSD engine running inside Linux."},
|
||||
{"beat": "b3", "text": "The device appears, owned by root with mode six hundred, exactly like the BSD original, and the module shows up in the list of loaded modules."},
|
||||
{"beat": "b4", "text": "pfctl is the original OpenBSD tool. We enable the firewall and ask for its status. The ioctl interface, the statistics table, the rule counters, all of it is the OpenBSD ABI, unchanged."},
|
||||
{"beat": "b5", "text": "The ruleset is still empty, and that is exactly right: with no rules, pf blocks everything and returns. The engine is armed and waiting for instructions."},
|
||||
],
|
||||
},
|
||||
"pt": {
|
||||
"title": "Ativação",
|
||||
"subtitle": "insmod pf.ko e o pfctl conversa com ele pela ABI de ioctls do OpenBSD",
|
||||
"segments": [
|
||||
{"beat": "b1", "text": "Primeiro a gente garante que não tem nada carregado, para a ativação ser honesta. Sem módulo, sem firewall."},
|
||||
{"beat": "b2", "text": "Agora inserimos. O kernel aceita o módulo e o log mostra a nossa mensagem de inicialização. Este é o motor do OpenBSD rodando dentro do Linux."},
|
||||
{"beat": "b3", "text": "O dispositivo aparece, do root, com modo seiscentos, exatamente como no BSD original, e o módulo aparece na lista de módulos carregados."},
|
||||
{"beat": "b4", "text": "O pfctl é a ferramenta original do OpenBSD. Habilitamos o firewall e pedimos o status. A interface de ioctl, a tabela de estatísticas, os contadores das regras: tudo é a ABI do OpenBSD, sem alteração."},
|
||||
{"beat": "b5", "text": "O ruleset ainda está vazio, e isso está certo: sem regras, o pf bloqueia tudo e devolve. O motor está armado e esperando instruções."},
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------ 03 ----
|
||||
PHASES["03-regras"] = {
|
||||
"en": {
|
||||
"title": "Basic rules",
|
||||
"subtitle": "Tables, anchors, keep state and block drop",
|
||||
"segments": [
|
||||
{"beat": "b1", "text": "Now we write a real pf dot conf. This is the original OpenBSD syntax: a table with the trusted network, an anchor, and the usual quick rules with keep state. pfctl parses it and the kernel accepts it."},
|
||||
{"beat": "b2", "text": "pfctl shows us what the kernel understood: the rules as they were compiled, the list of tables, and the contents of the trusted table."},
|
||||
{"beat": "b3", "text": "Let the traffic through. From the client we ping the server, and the firewall creates a state. You can see it in the state table: the five tuple, the timeout, the counters."},
|
||||
{"beat": "b4", "text": "Now the same thing with a drop. We replace the pass rule with a block drop rule for the client network, and reload."},
|
||||
{"beat": "b5", "text": "The ping is dead. One hundred percent packet loss. And the rule counters show the packets that were evaluated and the ones that were blocked. This is a stateful firewall behaving exactly like the original."},
|
||||
{"beat": "b6", "text": "Finally, anchors. We load a ruleset into the anchor called demo, and pfctl lists the rules that live inside it, separately from the main ruleset."},
|
||||
],
|
||||
},
|
||||
"pt": {
|
||||
"title": "Regras básicas",
|
||||
"subtitle": "Tabelas, anchors, keep state e block drop",
|
||||
"segments": [
|
||||
{"beat": "b1", "text": "Agora escrevemos um pf ponto conf de verdade. Esta é a sintaxe original do OpenBSD: uma tabela com a rede confiável, um anchor, e as regras quick de sempre com keep state. O pfctl faz o parsing e o kernel aceita."},
|
||||
{"beat": "b2", "text": "O pfctl nos mostra o que o kernel entendeu: as regras como foram compiladas, a lista de tabelas e o conteúdo da tabela trusted."},
|
||||
{"beat": "b3", "text": "Deixa o tráfego passar. Do cliente a gente dá ping no servidor, e o firewall cria um estado. Você vê isso na tabela de estados: a tupla de cinco campos, o timeout, os contadores."},
|
||||
{"beat": "b4", "text": "Agora a mesma coisa com um drop. Trocamos a regra de pass por uma regra block drop para a rede do cliente, e recarregamos."},
|
||||
{"beat": "b5", "text": "O ping morreu. Cem por cento de perda. E os contadores da regra mostram os pacotes avaliados e os bloqueados. Este é um firewall stateful se comportando exatamente como o original."},
|
||||
{"beat": "b6", "text": "Por fim, os anchors. Carregamos um ruleset no anchor chamado demo, e o pfctl lista as regras que moram dentro dele, separadamente do ruleset principal."},
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------ 04 ----
|
||||
PHASES["04-roteamento"] = {
|
||||
"en": {
|
||||
"title": "Routing",
|
||||
"subtitle": "route-to forces the next hop, reply-to forces the way back",
|
||||
"segments": [
|
||||
{"beat": "b1", "text": "Now the routing features. This rule matches traffic from the client to the server and sends it through the second gateway, ten point zero point three point two, instead of the normal path. pfctl shows the rule exactly as the kernel stored it.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "route-to: normal path is eth2, forced path is eth3"},
|
||||
{"beat": "b2", "text": "To prove where the packet actually went, we capture on both interfaces at the same time: eth three, towards the gateway, and eth two, towards the server."},
|
||||
{"beat": "b3", "text": "And now the client pings the server.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "the request leaves through eth3, to the gateway"},
|
||||
{"beat": "b4", "text": "The evidence is in the captures. The request appears on eth three, on the way to the gateway, and it never appears on eth two. The firewall decided the next hop, not the routing table.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "1 request on eth3, 0 on eth2"},
|
||||
{"beat": "b5", "text": "The other half of the problem is the way back. reply to forces the reply of a flow to leave through a different gateway than the request came in, which is how you build asymmetric routing.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "reply-to: the answer leaves through another gateway"},
|
||||
{"beat": "b6", "text": "We capture eth three again and also eth one, the link where the client is."},
|
||||
{"beat": "b7", "text": "The replies show up on eth three, on the gateway link, and not on the client link. The request and the reply now take different paths, exactly as commanded.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "reply on eth3, nothing on the client link"},
|
||||
],
|
||||
},
|
||||
"pt": {
|
||||
"title": "Roteamento",
|
||||
"subtitle": "route-to força o próximo salto, reply-to força o caminho de volta",
|
||||
"segments": [
|
||||
{"beat": "b1", "text": "Agora as funcionalidades de roteamento. Esta regra casa com o tráfego do cliente para o servidor e o manda pelo segundo gateway, dez ponto zero ponto três ponto dois, em vez do caminho normal. O pfctl mostra a regra exatamente como o kernel guardou.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "route-to: o caminho normal e' eth2; o forcado, eth3"},
|
||||
{"beat": "b2", "text": "Para provar por onde o pacote realmente passou, capturamos nas duas interfaces ao mesmo tempo: eth três, rumo ao gateway, e eth dois, rumo ao servidor."},
|
||||
{"beat": "b3", "text": "E agora o cliente dá ping no servidor.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "o pedido sai por eth3, rumo ao gateway"},
|
||||
{"beat": "b4", "text": "A prova está nas capturas. O pedido aparece na eth três, indo para o gateway, e nunca aparece na eth dois. Quem decidiu o próximo salto foi o firewall, não a tabela de rotas.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "1 pedido em eth3, 0 em eth2"},
|
||||
{"beat": "b5", "text": "A outra metade do problema é o caminho de volta. O reply to força a resposta de um fluxo a sair por um gateway diferente do por onde o pedido entrou, que é assim que se constrói roteamento assimétrico.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "reply-to: a resposta sai por um gateway diferente"},
|
||||
{"beat": "b6", "text": "Capturamos a eth três de novo e também a eth um, o link onde está o cliente."},
|
||||
{"beat": "b7", "text": "As respostas aparecem na eth três, no link do gateway, e não no link do cliente. O pedido e a resposta agora seguem caminhos diferentes, exatamente como foi pedido.", "highlight": ["cli", "pf0", "gw2"], "diagram_caption": "resposta em eth3, nada no link do cliente"},
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------ 05 ----
|
||||
PHASES["05-balanceamento"] = {
|
||||
"en": {
|
||||
"title": "Load balancing",
|
||||
"subtitle": "NAT pools in round-robin and source-hash, plus dup-to",
|
||||
"segments": [
|
||||
{"beat": "b1", "text": "Load balancing in pf is done with pools. This rule translates the source of the client network into a pool of two addresses, the two gateways of the firewall, and selects between them round robin.", "highlight": ["cli", "pf0", "srv"], "diagram_caption": "pool { 10.0.2.1, 10.0.3.1 }: round-robin"},
|
||||
{"beat": "b2", "text": "We capture on the interface towards the server to see which source address each packet carries."},
|
||||
{"beat": "b3", "text": "Six packets from the client."},
|
||||
{"beat": "b4", "text": "And the capture shows the truth: three packets went out with one address, three with the other. The pool is really rotating between its members, and the server sees a balanced load.", "highlight": ["cli", "pf0", "srv"], "diagram_caption": "3 packets with each source address"},
|
||||
{"beat": "b5", "text": "Sometimes you do not want rotation. source hash keeps a given origin always mapped to the same pool member, which is what you want when a session has to be sticky.", "highlight": ["cli", "pf0", "srv"], "diagram_caption": "pool { 10.0.2.1, 10.0.3.1 }: source-hash"},
|
||||
{"beat": "b6", "text": "Four more packets, and now every single one carries the same source address. The mapping is stable per origin.", "highlight": ["cli", "pf0", "srv"], "diagram_caption": "every packet with the same source"},
|
||||
{"beat": "b7", "text": "And dup to is the mirror case: the packet keeps its normal path, and a copy is delivered to the second gateway. We count the requests on both interfaces: one on the normal path, one on the mirrored path. The packet was duplicated, not moved.", "highlight": ["cli", "pf0", "srv", "gw2"], "diagram_caption": "dup-to: the normal path and the mirrored copy"},
|
||||
],
|
||||
},
|
||||
"pt": {
|
||||
"title": "Balanceamento",
|
||||
"subtitle": "Pools de NAT em round-robin e source-hash, mais o dup-to",
|
||||
"segments": [
|
||||
{"beat": "b1", "text": "O balanceamento no pf é feito com pools. Esta regra traduz a origem da rede do cliente para um pool de dois endereços, os dois gateways do firewall, e escolhe entre eles em round robin.", "highlight": ["cli", "pf0", "srv"], "diagram_caption": "pool { 10.0.2.1, 10.0.3.1 }: round-robin"},
|
||||
{"beat": "b2", "text": "Capturamos na interface que vai para o servidor para ver qual endereço de origem cada pacote carrega."},
|
||||
{"beat": "b3", "text": "Seis pacotes vindo do cliente."},
|
||||
{"beat": "b4", "text": "E a captura mostra a verdade: três pacotes saíram com um endereço, três com o outro. O pool está realmente rodando entre os membros, e o servidor enxerga a carga balanceada.", "highlight": ["cli", "pf0", "srv"], "diagram_caption": "3 pacotes com cada endereço de origem"},
|
||||
{"beat": "b5", "text": "Às vezes você não quer rotação. O source hash mantém uma origem sempre mapeada para o mesmo membro do pool, que é o que se quer quando a sessão precisa ser sticky.", "highlight": ["cli", "pf0", "srv"], "diagram_caption": "pool { 10.0.2.1, 10.0.3.1 }: source-hash"},
|
||||
{"beat": "b6", "text": "Mais quatro pacotes, e agora todos carregam o mesmo endereço de origem. O mapeamento é estável por origem.", "highlight": ["cli", "pf0", "srv"], "diagram_caption": "todos os pacotes com a mesma origem"},
|
||||
{"beat": "b7", "text": "E o dup to é o caso espelho: o pacote mantém o caminho normal e uma cópia é entregue no segundo gateway. Contamos os pedidos nas duas interfaces: um no caminho normal, um no caminho espelhado. O pacote foi duplicado, não movido.", "highlight": ["cli", "pf0", "srv", "gw2"], "diagram_caption": "dup-to: o caminho normal e a cópia espelhada"},
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------ 06 ----
|
||||
PHASES["06-qos"] = {
|
||||
"en": {
|
||||
"title": "Quality of service",
|
||||
"subtitle": "OpenBSD 8.0 queues: hfsc classes attached to the interface",
|
||||
"segments": [
|
||||
{"beat": "b1", "text": "The last piece is quality of service. This is the OpenBSD eight point zero queue syntax: a root class with ten megabits of bandwidth, a leaf class with two megabits as the default, and a rule that steers the traffic from the client to that leaf class."},
|
||||
{"beat": "b2", "text": "pfctl shows the queues with their statistics, and the rule set shows the queue attached to the rule."},
|
||||
{"beat": "b3", "text": "Now we move real traffic through the queue. On the server we prepare two megabytes and listen on port ninety ninety five."},
|
||||
{"beat": "b4", "text": "And the client downloads it, through the firewall, through the class."},
|
||||
{"beat": "b5", "text": "Two million bytes transferred, and look at the queue counters: the hfsc class counted the packets and the bytes. The scheduler is in the data path, not just in the configuration."},
|
||||
],
|
||||
},
|
||||
"pt": {
|
||||
"title": "Qualidade de serviço",
|
||||
"subtitle": "Filas do OpenBSD 8.0: classes hfsc anexadas à interface",
|
||||
"segments": [
|
||||
{"beat": "b1", "text": "A última peça é qualidade de serviço. Esta é a sintaxe de filas do OpenBSD oito ponto zero: uma classe raiz com dez megabits de banda, uma classe folha com dois megabits como default, e uma regra que direciona o tráfego do cliente para essa classe folha."},
|
||||
{"beat": "b2", "text": "O pfctl mostra as filas com as estatísticas, e o ruleset mostra a fila anexada à regra."},
|
||||
{"beat": "b3", "text": "Agora movemos tráfego de verdade pela fila. No servidor preparamos dois megabytes e escutamos na porta noventa e noventa e cinco."},
|
||||
{"beat": "b4", "text": "E o cliente baixa isso, passando pelo firewall, passando pela classe."},
|
||||
{"beat": "b5", "text": "Dois milhões de bytes transferidos, e olha os contadores das filas: a classe hfsc contou os pacotes e os bytes. O escalonador está no caminho de dados, não só na configuração."},
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
# the diagram is drawn over these phases
|
||||
DIAGRAM_PHASES = {"00-ambiente", "04-roteamento", "05-balanceamento"}
|
||||
|
||||
for phase, langs in PHASES.items():
|
||||
for lang, data in langs.items():
|
||||
out = {
|
||||
"phase": phase, "lang": lang,
|
||||
"voice": EN_VOICE if lang == "en" else PT_VOICE,
|
||||
"title": data["title"], "subtitle": data["subtitle"],
|
||||
"segments": data["segments"],
|
||||
}
|
||||
with open(os.path.join(OUT, f"{phase}.{lang}.json"), "w") as f:
|
||||
json.dump(out, f, ensure_ascii=False, indent=2)
|
||||
print(f"wrote {sum(len(v) for v in PHASES.values())} narration scripts to {OUT}")
|
||||
@@ -0,0 +1,43 @@
|
||||
{
|
||||
"phase": "00-ambiente",
|
||||
"lang": "en",
|
||||
"voice": "English_ConfidentWoman",
|
||||
"title": "The lab environment",
|
||||
"subtitle": "Four Alpine VMs under libvirt, running a kernel built from source",
|
||||
"segments": [
|
||||
{
|
||||
"beat": "b1",
|
||||
"text": "Welcome to the PF on Linux demonstration. Before we look at the firewall itself, let me show you the environment where everything runs. These are four virtual machines managed by libvirt, all of them Alpine Linux, and the machine you are looking at is the firewall: a kernel we compiled from source, version seven point three point four, with our port of the OpenBSD packet filter built in as a module."
|
||||
},
|
||||
{
|
||||
"beat": "b2",
|
||||
"text": "The firewall has four interfaces. The management interface on the virtual network the hypervisor uses to reach us. Then three transit networks: ten point zero point one, where the client lives; ten point zero point two, where the server lives; and ten point zero point three, a second gateway we use to demonstrate routing decisions.",
|
||||
"highlight": [
|
||||
"cli",
|
||||
"pf0",
|
||||
"srv",
|
||||
"gw2"
|
||||
],
|
||||
"diagram_caption": "three transit networks, four interfaces"
|
||||
},
|
||||
{
|
||||
"beat": "b3",
|
||||
"text": "The OpenBSD tools are installed on this machine exactly as they ship: the original pfctl, a pflog daemon and our divert tool. The module is not loaded yet, which is where we will start.",
|
||||
"highlight": [
|
||||
"pf0"
|
||||
],
|
||||
"diagram_caption": "pfctl, pflogd and pdivert, straight from OpenBSD"
|
||||
},
|
||||
{
|
||||
"beat": "b4",
|
||||
"text": "And from the firewall we can reach the other machines directly. The client is on ten point zero point one point ten, the server on ten point zero point two point twenty, and the second gateway on ten point zero point three point two. Every command in this demonstration is real traffic crossing real interfaces.",
|
||||
"highlight": [
|
||||
"cli",
|
||||
"pf0",
|
||||
"srv",
|
||||
"gw2"
|
||||
],
|
||||
"diagram_caption": "cli, srv and gw2 reachable from pf0"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
{
|
||||
"phase": "00-ambiente",
|
||||
"lang": "pt",
|
||||
"voice": "Portuguese_ConfidentWoman",
|
||||
"title": "O ambiente de testes",
|
||||
"subtitle": "Quatro VMs Alpine sob libvirt, rodando um kernel compilado do fonte",
|
||||
"segments": [
|
||||
{
|
||||
"beat": "b1",
|
||||
"text": "Bem-vindo à demonstração do PF on Linux. Antes de falar do firewall em si, deixa eu mostrar o ambiente onde tudo acontece. São quatro máquinas virtuais administradas pelo libvirt, todas com Alpine Linux, e a máquina que você está vendo é o firewall: um kernel que compilamos do fonte, versão sete ponto três ponto quatro, com o nosso port do packet filter do OpenBSD compilado dentro como módulo."
|
||||
},
|
||||
{
|
||||
"beat": "b2",
|
||||
"text": "O firewall tem quatro interfaces. A interface de gerência, na rede virtual que o hypervisor usa para falar com a gente. E depois três redes de trânsito: dez ponto zero ponto um, onde mora o cliente; dez ponto zero ponto dois, onde mora o servidor; e dez ponto zero ponto três, um segundo gateway que vamos usar para demonstrar decisões de roteamento.",
|
||||
"highlight": [
|
||||
"cli",
|
||||
"pf0",
|
||||
"srv",
|
||||
"gw2"
|
||||
],
|
||||
"diagram_caption": "três redes de transito, quatro interfaces"
|
||||
},
|
||||
{
|
||||
"beat": "b3",
|
||||
"text": "As ferramentas do OpenBSD estão instaladas nesta máquina exatamente como são distribuídas: o pfctl original, um daemon de pflog e a nossa ferramenta de divert. O módulo ainda não está carregado, e é por aí que a gente começa.",
|
||||
"highlight": [
|
||||
"pf0"
|
||||
],
|
||||
"diagram_caption": "pfctl, pflogd e pdivert originais do OpenBSD"
|
||||
},
|
||||
{
|
||||
"beat": "b4",
|
||||
"text": "E do firewall a gente alcança as outras máquinas diretamente. O cliente está em dez ponto zero ponto um ponto dez, o servidor em dez ponto zero ponto dois ponto vinte, e o segundo gateway em dez ponto zero ponto três ponto dois. Todo comando desta demonstração é tráfego real cruzando interfaces reais.",
|
||||
"highlight": [
|
||||
"cli",
|
||||
"pf0",
|
||||
"srv",
|
||||
"gw2"
|
||||
],
|
||||
"diagram_caption": "cli, srv e gw2 alcançáveis a partir do pf0"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"phase": "01-implementacao",
|
||||
"lang": "en",
|
||||
"voice": "English_ConfidentWoman",
|
||||
"title": "Implementation in the kernel",
|
||||
"subtitle": "OpenBSD 8.0 sources plus a Linux compatibility layer equals pf.ko",
|
||||
"segments": [
|
||||
{
|
||||
"beat": "b1",
|
||||
"text": "This is the heart of the work. We did not rewrite the firewall. We took the OpenBSD eight point zero sources, nearly twenty seven thousand lines of C, and we kept them almost intact. On top of that we wrote a compatibility layer, about seven thousand lines, that translates the BSD kernel interfaces into Linux ones."
|
||||
},
|
||||
{
|
||||
"beat": "b2",
|
||||
"text": "The result is a normal Linux kernel module. Here modinfo shows us the license, the description and the vermagic, which tells us exactly which kernel this object was compiled against."
|
||||
},
|
||||
{
|
||||
"beat": "b3",
|
||||
"text": "The vermagic line matters: it is the kernel version, the compiler flags and the module signature. Load the wrong one and the kernel refuses it. Everything you will see in the next minutes is running inside that module."
|
||||
},
|
||||
{
|
||||
"beat": "b4",
|
||||
"text": "This file is what we load into the firewall. A single artifact, built out of tree, carrying the entire OpenBSD packet filter."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"phase": "01-implementacao",
|
||||
"lang": "pt",
|
||||
"voice": "Portuguese_ConfidentWoman",
|
||||
"title": "Implementação no kernel",
|
||||
"subtitle": "Fontes do OpenBSD 8.0 mais uma camada de compatibilidade Linux resultam em pf.ko",
|
||||
"segments": [
|
||||
{
|
||||
"beat": "b1",
|
||||
"text": "Aqui está o coração do trabalho. A gente não reescreveu o firewall. Pegamos as fontes do OpenBSD oito ponto zero, quase vinte e sete mil linhas de C, e mantivemos praticamente intactas. Em cima disso escrevemos uma camada de compatibilidade, cerca de sete mil linhas, que traduz as interfaces de kernel do BSD para as do Linux."
|
||||
},
|
||||
{
|
||||
"beat": "b2",
|
||||
"text": "O resultado é um módulo de kernel Linux comum. Aqui o modinfo mostra a licença, a descrição e o vermagic, que nos diz exatamente com qual kernel este objeto foi compilado."
|
||||
},
|
||||
{
|
||||
"beat": "b3",
|
||||
"text": "A linha do vermagic importa: é a versão do kernel, as flags do compilador e a assinatura do módulo. Carregar o arquivo errado e o kernel recusa. Tudo que você vai ver nos próximos minutos roda dentro desse módulo."
|
||||
},
|
||||
{
|
||||
"beat": "b4",
|
||||
"text": "Este é o arquivo que carregamos no firewall. Um único artefato, compilado fora da árvore do kernel, carregando o packet filter inteiro do OpenBSD."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"phase": "02-ativacao",
|
||||
"lang": "en",
|
||||
"voice": "English_ConfidentWoman",
|
||||
"title": "Activation",
|
||||
"subtitle": "insmod pf.ko, then pfctl talks to it over the OpenBSD ioctl ABI",
|
||||
"segments": [
|
||||
{
|
||||
"beat": "b1",
|
||||
"text": "First we make sure nothing is loaded, so the activation is honest. No module, no firewall."
|
||||
},
|
||||
{
|
||||
"beat": "b2",
|
||||
"text": "Now we insert it. The kernel accepts the module and the log shows our initialization message. This is the OpenBSD engine running inside Linux."
|
||||
},
|
||||
{
|
||||
"beat": "b3",
|
||||
"text": "The device appears, owned by root with mode six hundred, exactly like the BSD original, and the module shows up in the list of loaded modules."
|
||||
},
|
||||
{
|
||||
"beat": "b4",
|
||||
"text": "pfctl is the original OpenBSD tool. We enable the firewall and ask for its status. The ioctl interface, the statistics table, the rule counters, all of it is the OpenBSD ABI, unchanged."
|
||||
},
|
||||
{
|
||||
"beat": "b5",
|
||||
"text": "The ruleset is still empty, and that is exactly right: with no rules, pf blocks everything and returns. The engine is armed and waiting for instructions."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"phase": "02-ativacao",
|
||||
"lang": "pt",
|
||||
"voice": "Portuguese_ConfidentWoman",
|
||||
"title": "Ativação",
|
||||
"subtitle": "insmod pf.ko e o pfctl conversa com ele pela ABI de ioctls do OpenBSD",
|
||||
"segments": [
|
||||
{
|
||||
"beat": "b1",
|
||||
"text": "Primeiro a gente garante que não tem nada carregado, para a ativação ser honesta. Sem módulo, sem firewall."
|
||||
},
|
||||
{
|
||||
"beat": "b2",
|
||||
"text": "Agora inserimos. O kernel aceita o módulo e o log mostra a nossa mensagem de inicialização. Este é o motor do OpenBSD rodando dentro do Linux."
|
||||
},
|
||||
{
|
||||
"beat": "b3",
|
||||
"text": "O dispositivo aparece, do root, com modo seiscentos, exatamente como no BSD original, e o módulo aparece na lista de módulos carregados."
|
||||
},
|
||||
{
|
||||
"beat": "b4",
|
||||
"text": "O pfctl é a ferramenta original do OpenBSD. Habilitamos o firewall e pedimos o status. A interface de ioctl, a tabela de estatísticas, os contadores das regras: tudo é a ABI do OpenBSD, sem alteração."
|
||||
},
|
||||
{
|
||||
"beat": "b5",
|
||||
"text": "O ruleset ainda está vazio, e isso está certo: sem regras, o pf bloqueia tudo e devolve. O motor está armado e esperando instruções."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"phase": "03-regras",
|
||||
"lang": "en",
|
||||
"voice": "English_ConfidentWoman",
|
||||
"title": "Basic rules",
|
||||
"subtitle": "Tables, anchors, keep state and block drop",
|
||||
"segments": [
|
||||
{
|
||||
"beat": "b1",
|
||||
"text": "Now we write a real pf dot conf. This is the original OpenBSD syntax: a table with the trusted network, an anchor, and the usual quick rules with keep state. pfctl parses it and the kernel accepts it."
|
||||
},
|
||||
{
|
||||
"beat": "b2",
|
||||
"text": "pfctl shows us what the kernel understood: the rules as they were compiled, the list of tables, and the contents of the trusted table."
|
||||
},
|
||||
{
|
||||
"beat": "b3",
|
||||
"text": "Let the traffic through. From the client we ping the server, and the firewall creates a state. You can see it in the state table: the five tuple, the timeout, the counters."
|
||||
},
|
||||
{
|
||||
"beat": "b4",
|
||||
"text": "Now the same thing with a drop. We replace the pass rule with a block drop rule for the client network, and reload."
|
||||
},
|
||||
{
|
||||
"beat": "b5",
|
||||
"text": "The ping is dead. One hundred percent packet loss. And the rule counters show the packets that were evaluated and the ones that were blocked. This is a stateful firewall behaving exactly like the original."
|
||||
},
|
||||
{
|
||||
"beat": "b6",
|
||||
"text": "Finally, anchors. We load a ruleset into the anchor called demo, and pfctl lists the rules that live inside it, separately from the main ruleset."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"phase": "03-regras",
|
||||
"lang": "pt",
|
||||
"voice": "Portuguese_ConfidentWoman",
|
||||
"title": "Regras básicas",
|
||||
"subtitle": "Tabelas, anchors, keep state e block drop",
|
||||
"segments": [
|
||||
{
|
||||
"beat": "b1",
|
||||
"text": "Agora escrevemos um pf ponto conf de verdade. Esta é a sintaxe original do OpenBSD: uma tabela com a rede confiável, um anchor, e as regras quick de sempre com keep state. O pfctl faz o parsing e o kernel aceita."
|
||||
},
|
||||
{
|
||||
"beat": "b2",
|
||||
"text": "O pfctl nos mostra o que o kernel entendeu: as regras como foram compiladas, a lista de tabelas e o conteúdo da tabela trusted."
|
||||
},
|
||||
{
|
||||
"beat": "b3",
|
||||
"text": "Deixa o tráfego passar. Do cliente a gente dá ping no servidor, e o firewall cria um estado. Você vê isso na tabela de estados: a tupla de cinco campos, o timeout, os contadores."
|
||||
},
|
||||
{
|
||||
"beat": "b4",
|
||||
"text": "Agora a mesma coisa com um drop. Trocamos a regra de pass por uma regra block drop para a rede do cliente, e recarregamos."
|
||||
},
|
||||
{
|
||||
"beat": "b5",
|
||||
"text": "O ping morreu. Cem por cento de perda. E os contadores da regra mostram os pacotes avaliados e os bloqueados. Este é um firewall stateful se comportando exatamente como o original."
|
||||
},
|
||||
{
|
||||
"beat": "b6",
|
||||
"text": "Por fim, os anchors. Carregamos um ruleset no anchor chamado demo, e o pfctl lista as regras que moram dentro dele, separadamente do ruleset principal."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
{
|
||||
"phase": "04-roteamento",
|
||||
"lang": "en",
|
||||
"voice": "English_ConfidentWoman",
|
||||
"title": "Routing",
|
||||
"subtitle": "route-to forces the next hop, reply-to forces the way back",
|
||||
"segments": [
|
||||
{
|
||||
"beat": "b1",
|
||||
"text": "Now the routing features. This rule matches traffic from the client to the server and sends it through the second gateway, ten point zero point three point two, instead of the normal path. pfctl shows the rule exactly as the kernel stored it.",
|
||||
"highlight": [
|
||||
"cli",
|
||||
"pf0",
|
||||
"gw2"
|
||||
],
|
||||
"diagram_caption": "route-to: normal path is eth2, forced path is eth3"
|
||||
},
|
||||
{
|
||||
"beat": "b2",
|
||||
"text": "To prove where the packet actually went, we capture on both interfaces at the same time: eth three, towards the gateway, and eth two, towards the server."
|
||||
},
|
||||
{
|
||||
"beat": "b3",
|
||||
"text": "And now the client pings the server.",
|
||||
"highlight": [
|
||||
"cli",
|
||||
"pf0",
|
||||
"gw2"
|
||||
],
|
||||
"diagram_caption": "the request leaves through eth3, to the gateway"
|
||||
},
|
||||
{
|
||||
"beat": "b4",
|
||||
"text": "The evidence is in the captures. The request appears on eth three, on the way to the gateway, and it never appears on eth two. The firewall decided the next hop, not the routing table.",
|
||||
"highlight": [
|
||||
"cli",
|
||||
"pf0",
|
||||
"gw2"
|
||||
],
|
||||
"diagram_caption": "1 request on eth3, 0 on eth2"
|
||||
},
|
||||
{
|
||||
"beat": "b5",
|
||||
"text": "The other half of the problem is the way back. reply to forces the reply of a flow to leave through a different gateway than the request came in, which is how you build asymmetric routing.",
|
||||
"highlight": [
|
||||
"cli",
|
||||
"pf0",
|
||||
"gw2"
|
||||
],
|
||||
"diagram_caption": "reply-to: the answer leaves through another gateway"
|
||||
},
|
||||
{
|
||||
"beat": "b6",
|
||||
"text": "We capture eth three again and also eth one, the link where the client is."
|
||||
},
|
||||
{
|
||||
"beat": "b7",
|
||||
"text": "The replies show up on eth three, on the gateway link, and not on the client link. The request and the reply now take different paths, exactly as commanded.",
|
||||
"highlight": [
|
||||
"cli",
|
||||
"pf0",
|
||||
"gw2"
|
||||
],
|
||||
"diagram_caption": "reply on eth3, nothing on the client link"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
{
|
||||
"phase": "04-roteamento",
|
||||
"lang": "pt",
|
||||
"voice": "Portuguese_ConfidentWoman",
|
||||
"title": "Roteamento",
|
||||
"subtitle": "route-to força o próximo salto, reply-to força o caminho de volta",
|
||||
"segments": [
|
||||
{
|
||||
"beat": "b1",
|
||||
"text": "Agora as funcionalidades de roteamento. Esta regra casa com o tráfego do cliente para o servidor e o manda pelo segundo gateway, dez ponto zero ponto três ponto dois, em vez do caminho normal. O pfctl mostra a regra exatamente como o kernel guardou.",
|
||||
"highlight": [
|
||||
"cli",
|
||||
"pf0",
|
||||
"gw2"
|
||||
],
|
||||
"diagram_caption": "route-to: o caminho normal e' eth2; o forcado, eth3"
|
||||
},
|
||||
{
|
||||
"beat": "b2",
|
||||
"text": "Para provar por onde o pacote realmente passou, capturamos nas duas interfaces ao mesmo tempo: eth três, rumo ao gateway, e eth dois, rumo ao servidor."
|
||||
},
|
||||
{
|
||||
"beat": "b3",
|
||||
"text": "E agora o cliente dá ping no servidor.",
|
||||
"highlight": [
|
||||
"cli",
|
||||
"pf0",
|
||||
"gw2"
|
||||
],
|
||||
"diagram_caption": "o pedido sai por eth3, rumo ao gateway"
|
||||
},
|
||||
{
|
||||
"beat": "b4",
|
||||
"text": "A prova está nas capturas. O pedido aparece na eth três, indo para o gateway, e nunca aparece na eth dois. Quem decidiu o próximo salto foi o firewall, não a tabela de rotas.",
|
||||
"highlight": [
|
||||
"cli",
|
||||
"pf0",
|
||||
"gw2"
|
||||
],
|
||||
"diagram_caption": "1 pedido em eth3, 0 em eth2"
|
||||
},
|
||||
{
|
||||
"beat": "b5",
|
||||
"text": "A outra metade do problema é o caminho de volta. O reply to força a resposta de um fluxo a sair por um gateway diferente do por onde o pedido entrou, que é assim que se constrói roteamento assimétrico.",
|
||||
"highlight": [
|
||||
"cli",
|
||||
"pf0",
|
||||
"gw2"
|
||||
],
|
||||
"diagram_caption": "reply-to: a resposta sai por um gateway diferente"
|
||||
},
|
||||
{
|
||||
"beat": "b6",
|
||||
"text": "Capturamos a eth três de novo e também a eth um, o link onde está o cliente."
|
||||
},
|
||||
{
|
||||
"beat": "b7",
|
||||
"text": "As respostas aparecem na eth três, no link do gateway, e não no link do cliente. O pedido e a resposta agora seguem caminhos diferentes, exatamente como foi pedido.",
|
||||
"highlight": [
|
||||
"cli",
|
||||
"pf0",
|
||||
"gw2"
|
||||
],
|
||||
"diagram_caption": "resposta em eth3, nada no link do cliente"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
{
|
||||
"phase": "05-balanceamento",
|
||||
"lang": "en",
|
||||
"voice": "English_ConfidentWoman",
|
||||
"title": "Load balancing",
|
||||
"subtitle": "NAT pools in round-robin and source-hash, plus dup-to",
|
||||
"segments": [
|
||||
{
|
||||
"beat": "b1",
|
||||
"text": "Load balancing in pf is done with pools. This rule translates the source of the client network into a pool of two addresses, the two gateways of the firewall, and selects between them round robin.",
|
||||
"highlight": [
|
||||
"cli",
|
||||
"pf0",
|
||||
"srv"
|
||||
],
|
||||
"diagram_caption": "pool { 10.0.2.1, 10.0.3.1 }: round-robin"
|
||||
},
|
||||
{
|
||||
"beat": "b2",
|
||||
"text": "We capture on the interface towards the server to see which source address each packet carries."
|
||||
},
|
||||
{
|
||||
"beat": "b3",
|
||||
"text": "Six packets from the client."
|
||||
},
|
||||
{
|
||||
"beat": "b4",
|
||||
"text": "And the capture shows the truth: three packets went out with one address, three with the other. The pool is really rotating between its members, and the server sees a balanced load.",
|
||||
"highlight": [
|
||||
"cli",
|
||||
"pf0",
|
||||
"srv"
|
||||
],
|
||||
"diagram_caption": "3 packets with each source address"
|
||||
},
|
||||
{
|
||||
"beat": "b5",
|
||||
"text": "Sometimes you do not want rotation. source hash keeps a given origin always mapped to the same pool member, which is what you want when a session has to be sticky.",
|
||||
"highlight": [
|
||||
"cli",
|
||||
"pf0",
|
||||
"srv"
|
||||
],
|
||||
"diagram_caption": "pool { 10.0.2.1, 10.0.3.1 }: source-hash"
|
||||
},
|
||||
{
|
||||
"beat": "b6",
|
||||
"text": "Four more packets, and now every single one carries the same source address. The mapping is stable per origin.",
|
||||
"highlight": [
|
||||
"cli",
|
||||
"pf0",
|
||||
"srv"
|
||||
],
|
||||
"diagram_caption": "every packet with the same source"
|
||||
},
|
||||
{
|
||||
"beat": "b7",
|
||||
"text": "And dup to is the mirror case: the packet keeps its normal path, and a copy is delivered to the second gateway. We count the requests on both interfaces: one on the normal path, one on the mirrored path. The packet was duplicated, not moved.",
|
||||
"highlight": [
|
||||
"cli",
|
||||
"pf0",
|
||||
"srv",
|
||||
"gw2"
|
||||
],
|
||||
"diagram_caption": "dup-to: the normal path and the mirrored copy"
|
||||
}
|
||||
]
|
||||
}
|
||||
Loaded 100 of 125 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user