xin

Measurement, and how it goes wrong

The numbers, including the ones we lost.

Every proxy's landing page has a bar chart where its bar is longer. Here is the table, the method, and the four times in one session that this project caught itself measuring the wrong thing.

Standing

Against nginx 1.26.3

Same box, same client, paired runs. The method that produced the first row is the interesting part, and it is below.

Workload xin nginx ratio what it actually measures
Static, 3-byte file 341,030 rps 443,483 rps 77% Header generation and syscall count. File I/O is noise at this size.
Static, 1 MB, sendfile off 6,475 rps 17,832 rps 36% The read()+write() path on both sides. xin's is ~3x slower on its own.
Static, 1 MB, sendfile on 5,712 rps 24,210 rps 24% How you would actually deploy nginx. sendfile accounts for ~1.4x of the ~3x gap.
Reverse proxy no number The harness could not discriminate: the backend's own ceiling sat below every proxy result. See D11.

77% is an upper bound, not a best case. nginx's ceiling was still climbing at two samples, so the true figure is 77% or worse, never better. Syscall standing on the 3-byte workload: nginx ~7 per request, xin ~16.

The interesting one

The correction that ran the other way.

The claim on record was that xin serves the small-static workload at 77.4% of nginx. Fourteen paired runs to re-check it spanned 68% to 113%, on a box carrying a load average between 13 and 43 from other work. The honest reading was that the figure was not refuted, it was not measurable here.

Declining to quote a number is usually the right instinct, and this log entry was originally written to record it as such — a fifth benchmark correction in a session that already had four.

It was not the end of the work. Rather than measuring more, the method changed to measuring when idle: a script armed to fire atomically the moment the box went quiet, because reacting to an idle notification is already too slow — the window closes in seconds.

Two clean pairs came back. xin: 341,030 and 331,655 — a 3% spread, reproducible. nginx: 357,121 and 443,483 — a 24% spread, not. Since contention only ever subtracts, best-of-each is the right estimator: 341,030 / 443,483 = 76.9%, reproducing the original claim almost exactly. The 90–107% readings were contention suppressing nginx more than xin.

So this one was not a fifth lie. The number held, and the lesson was measure-when-idle rather than measure-more.

The record

Four corrections, one session, one shape.

Every one of them was a harness measuring something adjacent to what its name claimed. They are in the public decision log rather than quietly fixed, because the pattern is the finding.

  1. D11

    The proxy benchmark that measured the backend

    A run showed xin at 122.8k req/s against nginx's 120.7k. It was not reported as a win: the backend's own ceiling measured 113k, and all three proxies came in above it. The run measured the backend.

    rule A benchmark that cannot discriminate is worse than none, because it gets quoted. The harness had printed a warning about exactly this, and it had already been ignored once.

    cost No proxy number at all, then or now.

  2. D18

    The static-file win that did not exist

    The owner was told static serving had reached 90% of nginx. The number came from the direct benchmark mode, which serves return 200 and never touches a filesystem. A real static mode reading an actual file through root gave 95,957 req/s against nginx's 428,545 — 22.4%.

    rule A benchmark mode's name is not its definition. The durable fix was making the harness print PASS/FAIL against explicit thresholds and fail any proxy run landing within 1.5x of the backend ceiling.

    cost sendfile turned out to be a large win rather than a small one, and was reprioritised.

  3. D27

    The syscall trace that counted zero

    Syscall counts are now taken by launching under strace, never by attaching — because attaching silently produced empty output twice, and an empty trace parses as “zero syscalls” rather than as a failure.

    rule The failure mode a measurement has when it breaks should not look like a good result.

    cost Current standing on a 3-byte file: nginx ~7 syscalls per request, xin ~16 — two statx where nginx does one fstat, double the epoll_ctl, double the accept4. Handed on as leads to verify, explicitly not as diagnoses.

  4. D42

    The static benchmark that never measured static serving

    bench/run.sh static serves a 3-byte index.html. At that size the workload is header generation and syscalls; file I/O is noise. The mode does not exercise the path its name claims, and the harness's own note about trailing nginx “until sendfile lands” was never being tested.

    rule A benchmark must state its workload in its output, so the reader sees “3-byte file” next to the number.

    cost On a 1 MB file the gap is ~3x, and only ~1.4x of it is sendfile. The plain read()+write() path is itself ~3x slower — a different problem from the one the design documents attribute it to.

Why a compatibility project publishes this.

xin's entire claim is that it will tell you the truth about what your config does. That claim is worth exactly as much as this project's willingness to tell the truth about itself when the truth is unflattering — so the benchmark page is the load-bearing one, and it is the one where being caught shading a number would end the argument.

The performance gap is real, it is understood, and it is being worked on. It is also not the reason anyone would adopt this: the reason is that a config either runs correctly or refuses to load, and you find out which in staging.

The full decision log ↗ The divergence catalog Back to the pitch