nperft · netperftool
Find the signal in the noise.
nperft results | live run | finished in 235 s; stored 1 run (178 values) nperft suite --peer 203.0.113.20 --events --state-dir ~/.local/state/nperft/runs/1795189963/this-machine --tests all ┌ tests ─────────────────┐┌ sustained: Gb/s | 20 measurement(s), latest 0.0339 | 208 segments retransmitted ──────────────────────────────────────────────────┐ │ ✓ pair ││ 0.0705 │ │ ✓ paths ││ █ │ │ ✓ tcppaths ││ █▃ │ │ ✓ transfer ││ ██ │ │ ✓ connrate ││ ██▆ │ │ ✓ scaling ││ ███▁ │ │ ✓ pktsize ││ ████ ▁▃▄▅ ▂▆▄ ▁▃▂▂▁▁▁ │ │ ✓ sustained ││ ████▄ ▂▅▂ ▂▄▆▇█████▄ ▁ ▁▃▄▅▆▇█▄ ▁▃▁ ▂▆███▄ ▂███████▇▂ │ │ ✓ idle ││ █████ ▅███▇▄▁ ▂▅▇███████████▄ ▁▃▄▆▇█▆▄▂ ▃▅▇█████████▄ ▂▅███▇▅▃▁ ▂▆██████▄ ▂██████████▅ │ │ – public ││ █████▇ ▃████████▅▂▄▇███████████████▇▃▃▃▄▅▆▇██████████▇▅▃▄▆██████████████▇▃ ▃▆█████████▇█████████▃ ▃█████████████▃ │ │ – neighbour ││ 0.0352 ██████▂▁▆███████████████████████████████████████████████████████████████████▇▇██████████████████████▃ ▃███████████████▆▁│ │ ││ █████████████████████████████████████████████████████████████████████████████████████████████████████▃ ▄██████████████████│ │ ││ ██████████████████████████████████████████████████████████████████████████████████████████████████████▄███████████████████│ │ ││ ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ │ ││ ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ │ ││ ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ │ ││ ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ │ ││ ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ │ ││ ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ │ ││ ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ │ ││ 0 ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ │ ││ 1.00 seconds (height: Gb/s) 20.00│ │ │└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ │┌ report ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ││ most that made the round trip: 41697 packets/s │ │ ││sustained 0.04 Gb/s over 20 s on 8 flows; first 1 s 0.03, last 1 s 0.03, lowest second 0.03 │ │ ││ no step down: the rate at the end matches the start │ │ ││public skipped: needs --peer-public ADDR, the peer's public address │ │ ││neighbour skipped: needs --neighbour ADDR (a third machine) and --neighbour-target IPV4 (where it sends) │ │ ││idle a connection left silent, then used again │ │ ││ 5 s still worked (answer in 82848 us) │ │ ││ 20 s still worked (answer in 104322 us) │ │ ││ nothing dropped a silent connection in this time │ └────────────────────────┘└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ Up/Down look at a test | b browse stored results | r run again | q quit | colour: green clean, to red errors
Terminal views
Identify problems at a glance
Incidents, each with a likely cause.
1 Overview │ 2 Incidents │ 3 Destinations │ 4 Tests │ 5 Fleet ┌ 9 record(s) in the last 24h, cause: all ─────────────────────────────────────────────────────────┐ │when state cause incident detail │ │25s ago ended, 14 s network endpoint_slow:10.99.0.3:7420 cleared; peak 5405.000 │ │26s ago ended, 15 s network retransmits cleared; peak 0.573 │ │27s ago ended, 11 s network peer_rtt_slow:10.99.0.3:7420 cleared; peak 4251.000 │ │27s ago 924 ms local stall 924 ms: only the monitor was delay│ │35s ago ended, 25 s local cpu_pressure cleared; peak 50.200 │ │38s ago started network peer_rtt_slow:10.99.0.3:7420 2843 us, usually 76 us │ │39s ago started network endpoint_slow:10.99.0.3:7420 4986 us, usually 60 us │ │41s ago started network retransmits 0.57% of 251527 segments retransmi│ │1m ago started local cpu_pressure tasks waited for a CPU 34% of the │ └──────────────────────────────────────────────────────────────────────────────────────────────────┘
- networkstalls, retransmits, connect retries, slow endpoints
- hostCPU steal, loss of the accelerated NIC, NIC drops
- localCPU, memory and disk pressure, swapping
- configlisten-queue and input-queue drops, a full connection table
Paths, receive queues and offloads
Solving spooky issues
paths 256 UDP flows to 10.99.0.2, differing only in port numbers; 240 probes of 64 bytes each
typical round trip 82 us (flow medians 58 us to 110 us), 0.00% lost
groups by fastest round trip the flows fall into 2 separate groups, so the
route between these two machines depends on the port numbers:
75% of flows 18 us to 35 us
25% of flows 3020 us to 3036 us, 4.87% lost
the group follows the source port modulo 4 (100% of flows agree):
remainder 1-3 -> group 1
remainder 0 -> group 2
the slowest group takes 86.3x as long as the fastest at best
result 64 flows (25%) are consistently worse: round trip 3075 us, 4.87% lost
they are the flows whose source port modulo 4 is 0 (100% of flows agree)
port 40012 -> 7420: round trip 3074 us, 21 of 240 lost
port 40140 -> 7420: round trip 3086 us, 20 of 240 lost
... and 59 more (see --json)
Some problems only hit some connections. One transfer in four crawls. A service is slow from one client port and fine from the next. They follow how packets are spread by port number, over parallel links in the network and over a NIC’s receive queues (RSS).
nperft paths sends small UDP probes over 256 flows that differ only in their port numbers, all at once, and compares the flows with each other. A flow that is consistently slower or lossier than the rest is being treated differently.
When the flows split into groups, it looks for what decides the group. It reads each NIC’s RSS key and table and works out the receive queue every flow lands on, and it tries the source port modulo every number up to 64. The report names the pattern and how many flows agree with it.
Offloads are suspects too. sudo nperft sweep turns TSO and GRO off, changes ring sizes and tries other settings one at a time against a baseline, then puts each one back.
nperft transfer runs many fixed-size TCP transfers at once, each on its own ports. It runs the set twice and reports a band of slow flows only when the same ports stand apart both times, so flows competing for the link are not mistaken for a pattern.
Sizes, regions and physical hosts
Auto-check your assumptions
Two VMs of the same size are not always the same hardware, and two nodes in one region are not always the same distance apart. Every result records what the machine actually is: cloud, VM size, region, zone, CPU, NIC driver and, where the hypervisor reports it, the physical host.
Each test says what lies between the two machines: the same physical host, the same zone on different hosts, or further apart. When both ends share a host the report says so, because that traffic never leaves it.
Group stored results by VM size, host or path to see whether a number belongs to the SKU or to where the VM happened to land. --host matches part of a host name, so hosts with similar names can be looked at together.
nperft watch notices when the machine changes underneath you: a different size, vCPU count, NIC driver or kernel.
pair.bulk.gbps by peer no filters group runs lowest median highest latest 10.99.0.2:7420 2 151.1 153.4 153.4 151.1 10.99.0.3:7420 1 3.56 3.56 3.56 3.56 3m ago 10.99.0.2:7420 153.4 ████████████████████████████████████████ 3m ago 10.99.0.3:7420 3.56 ▉ 87s ago 10.99.0.2:7420 151.1 ███████████████████████████████████████▍
The results database
Travel through time
nperft results | pair.bulk.gbps | by peer | window all | no filters ┌ metrics (258) ───────────────────────────┐┌ 3 runs, oldest on the left (4m ago to 2m ago) ───────┐ │idle.connections[idle_secs=5].alive 1 ││153.4│⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤│ │idle.connections[idle_secs=5].idle_s 1 ││ │ │ │idle.connections[idle_secs=5].respon 1 ││ │ │ │neighbour.loaded.gbps 1 ││ │ │ │neighbour.loaded.p50_us 1 ││ │ │ │neighbour.loaded.p99_us 1 ││76.68│ │ │neighbour.loaded.rps 1 ││ │ │ │neighbour.quiet.gbps 1 ││ │ │ │neighbour.quiet.p50_us 1 ││ │ │ │neighbour.quiet.p99_us 1 ││0 │ ⠄ │ │neighbour.quiet.rps 1 │└──────────────────────────────────────────────────────┘ │pair.bulk.bytes 3 │┌──────────────────────────────────────────────────────┐ │pair.bulk.errors 3 ││group runs lowes media highe lates│ │pair.bulk.flows 3 ││10.99.0.2:7420 2 151.1 153.4 153.4 151.1│ │pair.bulk.gbps 3 ││10.99.0.3:7420 1 3.56 3.56 3.56 3.56 │ └──────────────────────────────────────────┘└──────────────────────────────────────────────────────┘ Up/Down metric | / search | g group by | w window | r run tests | q quit
nperft-results import /var/lib/nperft # or an orchestrator's store, or files nperft-results runs --since 7d nperft-results show --metric pair.bulk.gbps --by path nperft-results tui --since 24h --size D4ds # / search, g group by, w time window
Fleets
Production monitor with quiet reporting
By default the agent stays out of the way of the machine’s real work. It samples the kernel’s own counters at each interval and reports them: retransmits, queue drops, pressure and round-trip times on the connections the machine already has.
After a reboot it runs one short tuning burst to check the state of the environment and set its thresholds, then goes quiet.
Machines can be configured to allow periodic network verification bursts while they are idle and carry no other workload. Persistent network degradation is then found during downtime, before it costs you when it matters.
A lightweight orchestration node keeps track of the fleet. It stores each node’s records and reports when a node stops reporting in. Owners check fleet status over SSH with the reporting terminal view, nperft-report.
Example usage
From two machines to a fleet
Build it
cargo build --release cargo build --release -p nperft-results # results tool
The node binary is small and builds statically with musl. The results tool is a separate package so nodes need no SQLite.
Test one machine against another
nperft serve --open # on B: the receiver nperft --peer B --fast # on A: about 12 s nperft --peer B # about 33 s nperft --peer B --reverse # B sends, A receives
With --reverse connections are still opened from A, so it works from behind a router that blocks incoming connections.
Look for a bad route
nperft paths --peer B nperft paths --peer B --tcp nperft transfer --peer B # 32 transfers of 200 MB
If the last paths run found a port modulus N, transfer runs 4 × N flows so every route carries some.
Try tuning, safely
nperft tune # audit only sudo nperft sweep --peer B
The sweep measures a baseline, then each tuning variant one at a time, putting each change back, then the baseline again.
Run everything and keep it
nperft suite --peer B --quick nperft-results run --peer B # run and store it nperft-results show --metric pair.bulk.gbps --by path nperft-results tui
Results go into a SQLite database under names such as connrate.ceiling_cps. Filter by peer, host, VM size, path or age, and group by any of them.
Keep watching
nperft watch --endpoint db.internal:5432 --dns example.com nperft-report # live terminal report nperft-report --once # one frame as text
Systemd units for the receiver, monitor, agent and orchestrator are in dist/, with an rc.d script for FreeBSD.
Join nodes into a fleet
nperft orchestrate # on a small machine
nperft token --advertise HOST > join # there too
nperft agent # on every node, with the
# join in /etc/nperft/joinThe agent unit does not start without the join file, so it can be enabled in an image and the file supplied at deployment.
Try it on one machine first
tools/lab-netns.sh --nodes 3 # namespaces, no root tools/lab-netns.sh --run paths # 1 flow in 4 on a bad route tools/lab-netns.sh --run fleet # orchestrator + agents tools/lab-qemu.sh # KVM guests, no root
The labs check that things work and that degradation is reported. They do not measure a real network.
Where it runs
Linux, FreeBSD, cloud and bare metal
Linux
Everything. The monitor reads hypervisor steal time, pressure figures, per-connection round-trip times and NIC ring sizes.
FreeBSD
The same program, with the operating-system layer chosen at compile time. Tests, path probes, receiver, agent, orchestrator and report work the same. Checks that need counters FreeBSD lacks stay quiet.
Azure lab
azure/lab.json is an ARM template that deploys a set of nodes with agents, optionally across zones. Results record the physical host each node landed on.