nperft · netperftool

Find the signal in the noise.

$ nperft-results tui  # then r to run the suite
 nperft results | live run | finished in 235 s; stored 1 run (178 values)
 nperft suite --peer 203.0.113.20 --events --state-dir ~/.local/state/nperft/runs/1795189963/this-machine --tests all
┌ tests ─────────────────┐┌ sustained: Gb/s | 20 measurement(s), latest 0.0339 | 208 segments retransmitted ──────────────────────────────────────────────────┐
│ ✓ pair                 ││  0.0705                                                                                                                           │
│ ✓ paths                ││         █                                                                                                                         │
│ ✓ tcppaths             ││         █▃                                                                                                                        │
│ ✓ transfer             ││         ██                                                                                                                        │
│ ✓ connrate             ││         ██▆                                                                                                                       │
│ ✓ scaling              ││         ███▁                                                                                                                      │
│ ✓ pktsize              ││         ████                         ▁▃▄▅                                                             ▂▆▄           ▁▃▂▂▁▁▁       │
│ ✓ sustained            ││         ████▄       ▂▅▂          ▂▄▆▇█████▄                ▁            ▁▃▄▅▆▇█▄          ▁▃▁       ▂▆███▄         ▂███████▇▂     │
│ ✓ idle                 ││         █████      ▅███▇▄▁    ▂▅▇███████████▄         ▁▃▄▆▇█▆▄▂      ▃▅▇█████████▄      ▂▅███▇▅▃▁ ▂▆██████▄       ▂██████████▅    │
│ – public               ││         █████▇   ▃████████▅▂▄▇███████████████▇▃▃▃▄▅▆▇██████████▇▅▃▄▆██████████████▇▃  ▃▆█████████▇█████████▃     ▃█████████████▃  │
│ – neighbour            ││  0.0352 ██████▂▁▆███████████████████████████████████████████████████████████████████▇▇██████████████████████▃   ▃███████████████▆▁│
│                        ││         █████████████████████████████████████████████████████████████████████████████████████████████████████▃ ▄██████████████████│
│                        ││         ██████████████████████████████████████████████████████████████████████████████████████████████████████▄███████████████████│
│                        ││         ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│
│                        ││         ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│
│                        ││         ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│
│                        ││         ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│
│                        ││         ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│
│                        ││         ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│
│                        ││         ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│
│                        ││       0 ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│
│                        ││         1.00                                             seconds  (height: Gb/s)                                             20.00│
│                        │└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
│                        │┌ report ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                        ││          most that made the round trip: 41697 packets/s                                                                           │
│                        ││sustained 0.04 Gb/s over 20 s on 8 flows; first 1 s 0.03, last 1 s 0.03, lowest second 0.03                                        │
│                        ││          no step down: the rate at the end matches the start                                                                      │
│                        ││public    skipped: needs --peer-public ADDR, the peer's public address                                                             │
│                        ││neighbour skipped: needs --neighbour ADDR (a third machine) and --neighbour-target IPV4 (where it sends)                           │
│                        ││idle      a connection left silent, then used again                                                                                │
│                        ││              5 s  still worked (answer in 82848 us)                                                                               │
│                        ││             20 s  still worked (answer in 104322 us)                                                                              │
│                        ││          nothing dropped a silent connection in this time                                                                         │
└────────────────────────┘└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
 Up/Down look at a test | b browse stored results | r run again | q quit | colour: green clean, to red errors

Terminal views

Identify problems at a glance

Incidents, each with a likely cause.

node-1 $ nperft-report --view incidents
 1 Overview │ 2 Incidents │ 3 Destinations │ 4 Tests │ 5 Fleet
┌ 9 record(s) in the last 24h, cause: all ─────────────────────────────────────────────────────────┐
│when     state          cause    incident                       detail                            │
│25s ago  ended, 14 s    network  endpoint_slow:10.99.0.3:7420   cleared; peak 5405.000            │
│26s ago  ended, 15 s    network  retransmits                    cleared; peak 0.573               │
│27s ago  ended, 11 s    network  peer_rtt_slow:10.99.0.3:7420   cleared; peak 4251.000            │
│27s ago  924 ms         local    stall                          924 ms: only the monitor was delay│
│35s ago  ended, 25 s    local    cpu_pressure                   cleared; peak 50.200              │
│38s ago  started        network  peer_rtt_slow:10.99.0.3:7420   2843 us, usually 76 us            │
│39s ago  started        network  endpoint_slow:10.99.0.3:7420   4986 us, usually 60 us            │
│41s ago  started        network  retransmits                    0.57% of 251527 segments retransmi│
│1m ago   started        local    cpu_pressure                   tasks waited for a CPU 34% of the │
└──────────────────────────────────────────────────────────────────────────────────────────────────┘
The report after the lab degraded one node for about 15 seconds and then froze the monitor for one. The slow node is blamed on the network; the freeze is not.
  • networkstalls, retransmits, connect retries, slow endpoints
  • hostCPU steal, loss of the accelerated NIC, NIC drops
  • localCPU, memory and disk pressure, swapping
  • configlisten-queue and input-queue drops, a full connection table

Paths, receive queues and offloads

Solving spooky issues

node-a $ nperft paths --peer 10.99.0.2
paths     256 UDP flows to 10.99.0.2, differing only in port numbers; 240 probes of 64 bytes each
typical   round trip 82 us (flow medians 58 us to 110 us), 0.00% lost
groups    by fastest round trip the flows fall into 2 separate groups, so the
          route between these two machines depends on the port numbers:
           75% of flows  18 us to 35 us
           25% of flows  3020 us to 3036 us, 4.87% lost
          the group follows the source port modulo 4 (100% of flows agree):
            remainder 1-3 -> group 1
            remainder 0 -> group 2
          the slowest group takes 86.3x as long as the fastest at best
result    64 flows (25%) are consistently worse: round trip 3075 us, 4.87% lost
          they are the flows whose source port modulo 4 is 0 (100% of flows agree)
          port 40012 -> 7420: round trip 3074 us, 21 of 240 lost
          port 40140 -> 7420: round trip 3086 us, 20 of 240 lost
          ... and 59 more (see --json)
From the bundled namespace lab, where one flow in four is sent over a link with 3 ms of delay and 5% loss. A few lines are left out for length.
Flows spread over parallel links by port number Machine A sends 256 flows to machine B. The network spreads them over four parallel links by their port numbers. Three links are healthy. The link carrying flows whose source port modulo 4 is 0 adds 3 milliseconds and loses 5 percent of packets. the network between them port mod 4 = 0 · +3 ms, 5% lost port mod 4 = 1 port mod 4 = 2 port mod 4 = 3 machine A 256 flows machine B nperft serve same two machines, four routes, chosen by port
The 256 flows arranged by source port modulo 4 Four rows of 64 cells, one cell per flow, ports 40000 to 40255. Every cell in the row for remainder 0 is marked slow and lossy; the other three rows are healthy. remainder source ports 40000 to 40255, one cell per flow 0 1 2 3
What that run found: every flow whose source port is a multiple of 4 is on the bad link, and no others.

Some problems only hit some connections. One transfer in four crawls. A service is slow from one client port and fine from the next. They follow how packets are spread by port number, over parallel links in the network and over a NIC’s receive queues (RSS).

nperft paths sends small UDP probes over 256 flows that differ only in their port numbers, all at once, and compares the flows with each other. A flow that is consistently slower or lossier than the rest is being treated differently.

When the flows split into groups, it looks for what decides the group. It reads each NIC’s RSS key and table and works out the receive queue every flow lands on, and it tries the source port modulo every number up to 64. The report names the pattern and how many flows agree with it.

Offloads are suspects too. sudo nperft sweep turns TSO and GRO off, changes ring sizes and tries other settings one at a time against a baseline, then puts each one back.

nperft transfer runs many fixed-size TCP transfers at once, each on its own ports. It runs the set twice and reports a band of slow flows only when the same ports stand apart both times, so flows competing for the link are not mistaken for a pattern.

Sizes, regions and physical hosts

Auto-check your assumptions

Two VMs of the same size are not always the same hardware, and two nodes in one region are not always the same distance apart. Every result records what the machine actually is: cloud, VM size, region, zone, CPU, NIC driver and, where the hypervisor reports it, the physical host.

Each test says what lies between the two machines: the same physical host, the same zone on different hosts, or further apart. When both ends share a host the report says so, because that traffic never leaves it.

Group stored results by VM size, host or path to see whether a number belongs to the SKU or to where the VM happened to land. --host matches part of a host name, so hosts with similar names can be looked at together.

nperft watch notices when the machine changes underneath you: a different size, vCPU count, NIC driver or kernel.

What lies between two machines Three VMs of the same size. VM 1 and VM 2 run on the same physical host, so a test between them never leaves it. VM 3 runs on a different host in the same zone, so a test from VM 2 to VM 3 crosses the network. physical host A physical host B VM 1 D4ds_v6 VM 2 D4ds_v6 VM 3 D4ds_v6 same physical host same zone, different hosts same size, same zone, two different answers
Each result is labelled with what lay between the two machines, so results are compared like with like.
node-1 $ nperft-results show --metric pair.bulk.gbps --by peer
pair.bulk.gbps   by peer   no filters

group              runs    lowest    median   highest    latest
10.99.0.2:7420        2     151.1     153.4     153.4     151.1
10.99.0.3:7420        1      3.56      3.56      3.56      3.56

3m ago    10.99.0.2:7420    153.4 ████████████████████████████████████████
3m ago    10.99.0.3:7420     3.56 ▉
87s ago   10.99.0.2:7420    151.1 ███████████████████████████████████████▍
Bulk rate from stored runs, grouped by peer. Group by VM size, host or path the same way. Column spacing is tightened to fit.

The results database

Travel through time

node-1 $ nperft-results tui
 nperft results | pair.bulk.gbps | by peer | window all | no filters
┌ metrics (258) ───────────────────────────┐┌ 3 runs, oldest on the left (4m ago to 2m ago) ───────┐
│idle.connections[idle_secs=5].alive     1 ││153.4│⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤│
│idle.connections[idle_secs=5].idle_s    1 ││     │                                                │
│idle.connections[idle_secs=5].respon    1 ││     │                                                │
│neighbour.loaded.gbps                   1 ││     │                                                │
│neighbour.loaded.p50_us                 1 ││     │                                                │
│neighbour.loaded.p99_us                 1 ││76.68│                                                │
│neighbour.loaded.rps                    1 ││     │                                                │
│neighbour.quiet.gbps                    1 ││     │                                                │
│neighbour.quiet.p50_us                  1 ││     │                                                │
│neighbour.quiet.p99_us                  1 ││0    │                        ⠄                       │
│neighbour.quiet.rps                     1 │└──────────────────────────────────────────────────────┘
│pair.bulk.bytes                         3 │┌──────────────────────────────────────────────────────┐
│pair.bulk.errors                        3 ││group                    runs  lowes media highe lates│
│pair.bulk.flows                         3 ││10.99.0.2:7420           2     151.1 153.4 153.4 151.1│
│pair.bulk.gbps                          3 ││10.99.0.3:7420           1     3.56  3.56  3.56  3.56 │
└──────────────────────────────────────────┘└──────────────────────────────────────────────────────┘
 Up/Down metric | / search | g group by | w window | r run tests | q quit
Every run goes into a SQLite database you build as you go. Scroll the metrics on the left; the chosen one is charted across runs, with a summary for each group. Filter by kind, profile, source, VM size, host, peer, path or age.
nperft-results import /var/lib/nperft          # or an orchestrator's store, or files
nperft-results runs --since 7d
nperft-results show --metric pair.bulk.gbps --by path
nperft-results tui --since 24h --size D4ds     # / search, g group by, w time window
Agents reporting to an orchestration node Four production nodes each run an agent that samples quietly and reports to one orchestration node over TLS 1.3. Node 2 is idle and runs a verification burst. Node 3 has stopped reporting, and the orchestration node raises an alert. The owner checks fleet status over SSH with the reporting terminal view. orchestration node fleet records and status alert node 3 is silent owner nperft-report ssh TLS 1.3, keyed node 1 agent node 2 agent, idle: burst node 3 not reporting node 4 agent
Dashed lines are agents reporting in. Verification bursts run only on machines set to allow them, and only while idle.

Fleets

Production monitor with quiet reporting

By default the agent stays out of the way of the machine’s real work. It samples the kernel’s own counters at each interval and reports them: retransmits, queue drops, pressure and round-trip times on the connections the machine already has.

After a reboot it runs one short tuning burst to check the state of the environment and set its thresholds, then goes quiet.

Machines can be configured to allow periodic network verification bursts while they are idle and carry no other workload. Persistent network degradation is then found during downtime, before it costs you when it matters.

A lightweight orchestration node keeps track of the fleet. It stores each node’s records and reports when a node stops reporting in. Owners check fleet status over SSH with the reporting terminal view, nperft-report.

Example usage

From two machines to a fleet

Build it

cargo build --release
cargo build --release -p nperft-results   # results tool

The node binary is small and builds statically with musl. The results tool is a separate package so nodes need no SQLite.

Test one machine against another

nperft serve --open          # on B: the receiver
nperft --peer B --fast       # on A: about 12 s
nperft --peer B              # about 33 s
nperft --peer B --reverse    # B sends, A receives

With --reverse connections are still opened from A, so it works from behind a router that blocks incoming connections.

Look for a bad route

nperft paths --peer B
nperft paths --peer B --tcp
nperft transfer --peer B     # 32 transfers of 200 MB

If the last paths run found a port modulus N, transfer runs 4 × N flows so every route carries some.

Try tuning, safely

nperft tune                  # audit only
sudo nperft sweep --peer B

The sweep measures a baseline, then each tuning variant one at a time, putting each change back, then the baseline again.

Run everything and keep it

nperft suite --peer B --quick
nperft-results run --peer B  # run and store it
nperft-results show --metric pair.bulk.gbps --by path
nperft-results tui

Results go into a SQLite database under names such as connrate.ceiling_cps. Filter by peer, host, VM size, path or age, and group by any of them.

Keep watching

nperft watch --endpoint db.internal:5432 --dns example.com
nperft-report                # live terminal report
nperft-report --once         # one frame as text

Systemd units for the receiver, monitor, agent and orchestrator are in dist/, with an rc.d script for FreeBSD.

Join nodes into a fleet

nperft orchestrate                    # on a small machine
nperft token --advertise HOST > join  # there too
nperft agent                          # on every node, with the
                                      # join in /etc/nperft/join

The agent unit does not start without the join file, so it can be enabled in an image and the file supplied at deployment.

Try it on one machine first

tools/lab-netns.sh --nodes 3     # namespaces, no root
tools/lab-netns.sh --run paths   # 1 flow in 4 on a bad route
tools/lab-netns.sh --run fleet   # orchestrator + agents
tools/lab-qemu.sh                # KVM guests, no root

The labs check that things work and that degradation is reported. They do not measure a real network.

Where it runs

Linux, FreeBSD, cloud and bare metal

Linux

Everything. The monitor reads hypervisor steal time, pressure figures, per-connection round-trip times and NIC ring sizes.

FreeBSD

The same program, with the operating-system layer chosen at compile time. Tests, path probes, receiver, agent, orchestrator and report work the same. Checks that need counters FreeBSD lacks stay quiet.

Azure lab

azure/lab.json is an ARM template that deploys a set of nodes with agents, optionally across zones. Results record the physical host each node landed on.