How we will measure, published before we measured.

The claim we intend to test

That a structure search, given the same data and a fixed compute budget, finds a model that is at least as accurate as a competently tuned baseline on held-out data, and cheaper to serve. Two axes, one run, and no permission to trade the second away silently. A method that wins on accuracy while quietly tripling inference cost has failed this claim, not passed it.

The protocol

1. Datasets are chosen before anything is run, and the list is published

Between 30 and 50 public tabular datasets covering the three supported tasks, chosen for spread across row count, column count, class balance and column type mix — and fixed in writing, with a version or checksum for each, before the first run. Any dataset dropped after the list is fixed is reported as dropped, with the reason, and counted in the denominator. Selecting the set after seeing the scores is the single easiest way to manufacture a win rate, and it is the thing a reader has no way to detect from the results table.

2. The baseline is a real one

Two baselines per dataset. The first is a regularised linear or logistic model, because a method that cannot beat one is not interesting and because it is the honest default on small data. The second is a gradient-boosted tree ensemble tuned under the same compute budget jAIn is given, using a standard tuner rather than defaults — a comparison against an untuned baseline is a comparison against nothing. Where an incumbent production model is available, it is reported alongside as a third column and never in place of the second.

3. Compute is equalised, and stated

Every method on a dataset gets the same wall-clock budget on the same hardware, and both are published: instance type, core count, memory, whether an accelerator was present, and the budget in minutes. Inference cost is measured as median and 95th-percentile latency per prediction on a single CPU core at a stated batch size, because that is the number that decides whether a model can be deployed. A search given twice the budget of its baseline has measured its budget, not its method.

4. The held-out split is held out

One split, made before the run, from data no method sees during search or tuning. Model selection happens on cross-validation folds inside the training portion only; the held-out set is scored exactly once, at the end, for every method. Where a dataset ships with its own canonical split, that split is used and said so. Nothing is re-run after seeing a held-out score — a protocol that permits a second attempt permits an unlimited number.

5. Metrics are fixed per task, in advance

AUC for binary classification, macro F1 and accuracy for multiclass, RMSE and MAE for regression — reported together rather than whichever flatters the run. Differences are reported with a confidence interval from repeated splits, and a difference inside that interval is reported as a tie rather than as a win. Absolute scores are published beside the differences so anybody can check the arithmetic against their own numbers.

6. Every number carries a receipt

A result is a claim about a run, and a claim about a run that cannot be located is a claim about nothing. Each published figure will carry a reproducibility identifier resolving to the run's record: dataset and version, split seed, budget, hardware, library versions, the structure that was found, and the scores of every method on that dataset. The product cannot write that record yet — it is the gap named on when not to use VEOX — and no figures are published here until it can.

7. Losses are published with wins

The output of the proving run is a win, tie and loss distribution across the whole fixed dataset list, not a set of case studies. Every dataset appears in it. Where the search loses, the row says so and the record is available like any other. A benchmark you can only read the top of is marketing with a table in it, and where the method does not pay is already the most useful thing this site publishes.

What we will not do

• No competitor comparison tables without a receipt behind every cell. Running someone else's product badly and publishing the result is the easiest chart on the internet to draw and the least honest. Until there are records for both sides, this site compares approaches — where an evolutionary search helps and where gradient boosting is already enough — and not products.
• No figures without provenance. The two research briefs on this site carry figures that VEOX ran and scored and nobody has reproduced; the pages that quote them say exactly that above the numbers. That note is the minimum standard, not the target.
• No stand-in numbers, anywhere. Figures presented as measurements when they were invented were removed from this site in full, and scripts/build.sh now fails the build on the vocabulary that carried the worst of them. Illustrations that remain are labelled as generated in the same view as the illustration.
• No re-running until it wins. See point 4. The seed and the split are fixed before the run and published with it.

Why publish this now

Because it is worth less later. A protocol released with its own results can be shaped by them, and no reader can tell whether it was. Released first, it can be criticised by someone who has no idea whether it will favour us — and if the criticism is right, it is cheap to fix now and expensive to fix after the run. If you think one of the seven points above is too weak, or measures the wrong thing, tell us before the run.

When the results exist they will appear here and on the product page, under this protocol, with the losses in the same table as the wins.