Let tally accept replace a preset complexity class that the growth no longer fits

CLAUDE.md

@@ -6,9 +6,9 @@ Tally benchmarks PHP code by counting instructions, cycles, and memory under Cod
 
 - `src/Attribute` – the attributes benchmark classes use: `Expected` (a method without a data provider), `ExpectedFor` (one data set), `Complexity` and its presets (how the instructions grow), `Warmup`, `Before` and `After` (hook methods), `DataProvider`, and the tolerance policies `RelativeTolerance`, `MetricTolerance`, and `DurationTolerance`.
 - `src/Expression` – the expression language of complexity classes, all pure: `Lexer` (tokens with offsets), `Parser` (recursive descent into the `Node` tree, sealed), `Printer` (the canonical form, parsing it gives the same tree), `TypeChecker` (finds what the parameter types reveal, through `ValueType`), `Evaluator`, and the `MathFunction`s of the `FunctionLibrary`. Every mistake is an `ExpressionException` that quotes the expression and underlines the part that is wrong; `ExpressionErrors` words them the same for the checker and the evaluator.
-- `src/Complexity` – `ComplexityClass`, the interface of `#[Complexity]` and its presets (`#[Linear]` and the others, which compose a `Complexity`), and `GrowthPoint`, where a case is on its growth curve. `Discovery\GrowthPoints` computes them at discovery, `Comparator::compareGrowth()` compares a case with its base. These cases have no numbers to accept.
+- `src/Complexity` – `ComplexityClass`, the interface of `#[Complexity]` and its presets (`#[Linear]` and the others, which compose a `Complexity`), and `GrowthPoint`, where a case is on its growth curve. `Discovery\GrowthPoints` computes them at discovery, `Comparator::compareGrowth()` compares a case with its base. These cases have no numbers to accept: `PresetFit` (pure) finds the lowest preset the measured instructions of a method grow within, which `tally accept` writes instead of the method's preset through the `ComplexityAttribute` of its `GrowthPoint`.
 - `src/Quantity` – `QuantityFormat` formats and parses numbers with units (`3.119B`, `44.49MiB`).
-- `src/Expectation` – where expected metrics are stored. `ExpectationSource` is the interface: `read()` the expectation, `file()` the file it lives in, the pure `edits(code, metrics)` that returns the `CodeEdit`s that write new expectations into the file's code, and `location()`, which throws `ExpectationException` when the source cannot be rewritten. `AttributeExpectations` implements it for `#[Expected]` and `#[ExpectedFor]` through `ExpectedAttributeRewriter`, which reads a file once into an `AttributeIndex` of every method's attributes, with `Tokens`, the token list it walks, and locates and edits each case in the index; `DirectorySuiteLoader` indexes each file once for all its cases, because tokenizing it for every case made locating and rewriting quadratic in the cases of a file. `ExpectationChanges` plans the rewrites of many cases (pure): it applies the edits of all cases of a file in one pass, one `FileChange` per file with every case in it, in a `ChangePlan` together with the cases that failed. `ExpectationFiles` reads and writes the files.
+- `src/Expectation` – where expected metrics are stored. `ExpectationSource` is the interface: `read()` the expectation, `file()` the file it lives in, the pure `edits(code, metrics)` that returns the `CodeEdit`s that write new expectations into the file's code, and `location()`, which throws `ExpectationException` when the source cannot be rewritten. `AttributeExpectations` implements it for `#[Expected]` and `#[ExpectedFor]` through `ExpectedAttributeRewriter`, which reads a file once into an `AttributeIndex` of every method's attributes, with `Tokens`, the token list it walks, and locates and edits each case in the index; `ComplexityAttribute` replaces the complexity class of a method through `ExpectedAttributeRewriter::complexityEdits()`, in the same index; `DirectorySuiteLoader` indexes each file once for all its cases, because tokenizing it for every case made locating and rewriting quadratic in the cases of a file. `ExpectationChanges` plans the rewrites of many cases and complexity classes (pure): it applies the edits of all of them in a file in one pass, one `FileChange` per file with every change in it, in a `ChangePlan` together with the cases that failed. `ExpectationFiles` reads and writes the files.
 - `src/Discovery` – `DirectorySuiteLoader` finds and loads `*Bench.php` files (I/O), `CaseExpander` turns one method into its cases (pure, works on reflection); `hooks()` finds a class's `#[Before]` and `#[After]` methods once per class, because scanning them for every benchmark method made loading a class quadratic in its size. `AttributeFactory` creates every attribute Tally reads: it checks the arguments against the attribute's constructor first (names, count, types through `ValueTypeCheck`, as under strict_types), so a mistake is an `InvalidBenchmarkException` naming the argument, not a PHP error. Everything a benchmark author can get wrong, including a file that does not load and a data provider that throws, ends as an `InvalidBenchmarkException` that names the benchmark or file; PHP errors with stack traces are bugs. `ArgumentResolver` checks that a method takes the arguments of a data set (a hook the leading ones) and `ConstructorArguments` that a constructor takes only `Meter` and `ScratchDirectory`; discovery checks both, so a mismatch fails `tally list` instead of the worker.
 - `src/Selection` – `CaseSelector` implementations: `FilterSelector`, `ShardSelector` (longest-processing-time sharding by expected instructions). Both keep a case compared by its growth with its base: the filter adds the base, the shards keep a method's cases together.
 - `src/Runner` – running one case in this process. `CaseRunner` calls the hooks uncounted, prepares the counter, calls the warm-up uncounted, then counts a `#warmup` span, an empty `#baseline` call, and one call of the benchmark, and subtracts the baseline's memory. The `#warmup` span exists because the first counted span costs more than later ones. `CodSpeedCounter` starts Callgrind's instrumentation once, when it is prepared, stops it when it is finished after the benchmark's span, so the after methods and the end of the process run at Valgrind's quickest, and only toggles collecting for each span: starting the instrumentation empties the simulated caches, and doing it for every span made the cycles of small benchmarks depend on the cache lines the hooks touched. `MeterSession` is the `Meter` the benchmark object gets in its constructor, one per case: every call `CaseRunner` makes is a span of it with its own `Counter` (`CodSpeedCounter` over `InstrumentHooks` for the counted ones, `IdleCounter` for hooks and warm-up), and it measures memory above the start of the span through `Memory`. `ExtensionInstrumentHooks` calls the `tally_hooks` extension, the only C that Tally calls.

README.md

@@ -290,17 +290,29 @@ Benchmarks failed: 2 slower, 1 unchanged
 Allowed difference: instructions grow at most as O(n) from the smallest data set, ±10 % of each metric
 
 Slower than expected
-These benchmarks do more work than expected. Find out why before merging. If it is the price of a fix, loosen the complexity class.
+These benchmarks do more work than expected. Find out why before merging. If it is the price of a fix, accept the new complexity class.
 
   Benchmark                            Metric         Expected   Measured     Change   Allowed
   App\PairsBench::benchPairs@size-20   instructions     93.44K     178.7K    +91.2 %     ±10 %
   App\PairsBench::benchPairs@size-40   instructions     186.9K     699.7K   +274.4 %     ±10 %
 
 Reproduce locally
   vendor/bin/tally run --filter='App\PairsBench::benchPairs@size-40'
+
+Accept the new numbers
+  vendor/bin/tally run --output=tmp/tally
+  vendor/bin/tally accept tmp/tally
+```
+
+`tally accept` replaces a preset with the lowest one, in the order of the table above, that the measured instructions grow within as `tally run` compares them, keeping the size as written:
+
+```
+slower  App\PairsBench::benchPairs: O(n) to O(n^2)
+
+Accepted the complexity class of 1 benchmark.
 ```
 
-`tally accept` does not write anything for these cases; change the attribute instead. A method needs a data provider with at least two data sets of different sizes, and cannot also have `#[ExpectedFor]` or expectations from its data provider. `--filter` adds the base of each selected case, and `--shard` keeps the cases of a method in one shard, because each is compared with its base.
+This also tightens a preset the instructions grow well within, such as `#[Linear]` to `#[Logarithmic]`, listed as faster. The presets are compared at the sizes of the data sets only, so `#[Exponential]` replaces a preset that allows more growth at the largest one, as every polynomial of a high enough degree does. Accept needs the instructions of every data set of the method. It leaves `#[Complexity]` with a formula of its own as it is, and fails for a method no preset bounds. A method needs a data provider with at least two data sets of different sizes, and cannot also have `#[ExpectedFor]` or expectations from its data provider. `--filter` adds the base of each selected case, and `--shard` keeps the cases of a method in one shard, because each is compared with its base.
 
 Mistakes in the expressions are found when the benchmarks are loaded, before anything is counted:
 
@@ -395,7 +407,7 @@ Compares the results of `tally run --output` with the expected numbers again, wi
 
 ### `tally accept <results>`
 
-Makes the results of `tally run --output` the expected numbers of the cases that are new, slower, or faster. It rewrites the `#[Expected]` and `#[ExpectedFor]` attributes, or adds them, with the imports they need, and leaves the rest of the file as it is.
+Makes the results of `tally run --output` the expected numbers of the cases that are new, slower, or faster. It rewrites the `#[Expected]` and `#[ExpectedFor]` attributes, or adds them, with the imports they need, and leaves the rest of the file as it is. It replaces the preset [complexity class](#complexity) of a method whose instructions grew more than it allows, or well within a lower preset, and removes the import of the replaced class when nothing else uses it.
 
 ```
 slower  App\SortBench::benchSort@size-10

bench/Expectation/ExpectationChangesBench.php

@@ -43,7 +43,7 @@ public static function newCases(): iterable
 	#[ExpectedFor('cases-200', instructions: '21.63M', cycles: '26.19M', memory: '291.0KiB')]
 	public function benchPlan(array $accepted, array $code): void
 	{
-		(new ExpectationChanges())->plan($accepted, $code);
+		(new ExpectationChanges())->plan($accepted, [], $code);
 	}
 
 }

src/Attribute/Polynomial.php

@@ -32,6 +32,11 @@ public function __construct(string $size, private int $degree)
 		$this->complexity = new Complexity($size, 'n^' . $degree);
 	}
 
+	public function degree(): int
+	{
+		return $this->degree;
+	}
+
 	#[Override]
 	public function size(): string
 	{

src/Cli/AcceptCommand.php

@@ -5,10 +5,17 @@
 use Override;
 use RuntimeException;
 use Tally\BenchmarkCase;
+use Tally\CaseId;
 use Tally\Comparison\Comparator;
 use Tally\Comparison\Status;
+use Tally\Complexity\ComplexityClass;
+use Tally\Complexity\GrowthPoint;
+use Tally\Complexity\PresetFit;
+use Tally\Expectation\ComplexityAttribute;
 use Tally\Expectation\ExpectationChanges;
 use Tally\Expectation\ExpectationFiles;
+use Tally\Measurement;
+use Tally\Metric;
 use function array_filter;
 use function array_map;
 use function count;
@@ -20,7 +27,8 @@
 /**
  * Writes the measured numbers of the cases that changed as their expectations: into #[Expected]
  * attributes, or wherever a data provider keeps them. Cases within their tolerance keep their
- * numbers, so accepting does not churn them.
+ * numbers, so accepting does not churn them. A preset complexity class, like #[Linear], is
+ * replaced by the preset that bounds the measured growth most tightly.
  */
 final class AcceptCommand implements Command
 {
@@ -33,7 +41,8 @@ public function __construct(
 		private ResultsArguments $resultsArguments,
 		private Comparator $comparator,
 		private ExpectationFiles $files,
-		private ExpectationChanges $changes
+		private ExpectationChanges $changes,
+		private PresetFit $fit
 	)
 	{
 	}
@@ -69,22 +78,47 @@ public function execute(Input $input, Output $output): int
 		$cases = $this->selectedCases->load($input, $this->selectedCases->configuration($input), $this->resultsArguments->paths($input));
 
 		$changed = [];
+		$growing = [];
 		foreach ($cases as $case) {
 			$measurement = $measurements[$case->id->toString()] ?? null;
 			$comparison = $this->comparator->compareCase($case, $measurements);
+			if ($case->growth !== null) {
+				$growing[$case->growth->base->toString()][] = [$case, $case->growth, $comparison->status, $measurement];
+				continue;
+			}
 			$status = $comparison->status;
-			if ($measurement === null || !$comparison->acceptable || $status === Status::Ok || $status === Status::NotMeasured) {
+			if ($measurement === null || $status === Status::Ok || $status === Status::NotMeasured) {
 				continue;
 			}
 
 			$changed[] = [$case, $measurement->metrics, $status];
 		}
 
+		$failed = false;
+		$complexities = [];
+		foreach ($growing as $methodCases) {
+			$change = $this->complexityChange($methodCases);
+			if ($change === null) {
+				continue;
+			}
+			if ($change[2] === null) {
+				$output->error(sprintf('Cannot accept %s: no preset complexity class bounds how its instructions grow, write a #[Complexity] that does.', $change[0]->toString()));
+				$failed = true;
+				continue;
+			}
+
+			$complexities[] = $change;
+		}
+
 		$plan = $this->changes->plan(
 			array_map(static fn (array $change): array => [$change[0], $change[1]], $changed),
-			$this->files->read(array_map(static fn (array $change): BenchmarkCase => $change[0], $changed)),
+			array_map(static fn (array $change): array => [$change[0], $change[1], $change[2]], $complexities),
+			$this->files->read([
+				...array_map(static fn (array $change): string => $change[0]->expectations->file(), $changed),
+				...array_map(static fn (array $change): string => $change[1]->file(), $complexities),
+			]),
 		);
-		$failed = $plan->failures !== [];
+		$failed = $failed || $plan->failures !== [];
 		foreach ($plan->failures as $id => $failure) {
 			$output->error(sprintf('Cannot accept %s: %s', $id, $failure));
 		}
@@ -104,14 +138,61 @@ public function execute(Input $input, Output $output): int
 		foreach ($accepted as [$case, , $status]) {
 			$output->write(sprintf("%s  %s\n", str_pad($status->value, self::STATUS_WIDTH), $case->id->toString()));
 		}
-		$output->write(sprintf(
-			"%sAccepted the numbers of %d %s.\n",
-			$accepted === [] ? '' : "\n",
-			count($accepted),
-			count($accepted) === 1 ? 'case' : 'cases',
-		));
+		$acceptedComplexities = array_filter($complexities, static fn (array $change): bool => in_array($change[0]->toString(), $written, true));
+		foreach ($acceptedComplexities as [$id, $attribute, $replacement, $status]) {
+			$output->write(sprintf(
+				"%s  %s: %s to %s\n",
+				str_pad($status->value, self::STATUS_WIDTH),
+				$id->toString(),
+				$attribute->complexity->describe(),
+				$replacement->describe(),
+			));
+		}
+		$what = [];
+		if ($accepted !== [] || $acceptedComplexities === []) {
+			$what[] = sprintf('the numbers of %d %s', count($accepted), count($accepted) === 1 ? 'case' : 'cases');
+		}
+		if ($acceptedComplexities !== []) {
+			$what[] = count($acceptedComplexities) === 1 ? 'the complexity class of 1 benchmark' : sprintf('the complexity classes of %d benchmarks', count($acceptedComplexities));
+		}
+		$output->write(sprintf("%sAccepted %s.\n", $accepted === [] && $acceptedComplexities === [] ? '' : "\n", implode(' and ', $what)));
 
 		return $failed ? Application::EXIT_ERROR : Application::EXIT_SUCCESS;
 	}
 
+	/**
+	 * The preset that replaces the complexity class of a benchmark method, when the instructions of
+	 * all its cases were counted: slower when its class allows less than they grew, faster when a
+	 * tighter preset bounds them.
+	 *
+	 * @param non-empty-list<array{BenchmarkCase, GrowthPoint, Status, Measurement|null}> $cases
+	 * @return array{CaseId, ComplexityAttribute, ComplexityClass|null, Status}|null null when it keeps its class, the class null when no preset bounds them
+	 */
+	private function complexityChange(array $cases): ?array
+	{
+		$points = [];
+		$status = Status::Faster;
+		foreach ($cases as [, $growth, $caseStatus, $measurement]) {
+			$instructions = $measurement?->metrics->get(Metric::Instructions);
+			if ($instructions === null) {
+				return null;
+			}
+			$points[] = [$growth->size, $instructions];
+			if ($caseStatus !== Status::Slower) {
+				continue;
+			}
+
+			$status = Status::Slower;
+		}
+
+		[$case, $growth] = $cases[0];
+		$attribute = $growth->attribute;
+		$fitted = $this->fit->fit($attribute->complexity, $points, $case->tolerance);
+		if ($fitted === $attribute->complexity) {
+			return null;
+		}
+
+		return [new CaseId($case->id->class, $case->id->method, null), $attribute, $fitted, $status];
+	}
+
 }

src/Cli/ApplicationFactory.php

@@ -3,6 +3,7 @@
 namespace Tally\Cli;
 
 use Tally\Comparison\Comparator;
+use Tally\Complexity\PresetFit;
 use Tally\Discovery\ArgumentResolver;
 use Tally\Discovery\AttributeFactory;
 use Tally\Discovery\CaseExpander;
@@ -77,7 +78,9 @@ public function create($stdout, $stderr, string $workingDirectory, array $enviro
 		$rewriter = new ExpectedAttributeRewriter();
 		$functions = new FunctionLibrary([new Log(), new SquareRoot(), new Exponential(), new Factorial()]);
 		$expressionErrors = new ExpressionErrors($closestName);
-		$growthPoints = new GrowthPoints(new Parser(new Lexer()), new TypeChecker($functions, $expressionErrors), new Evaluator($functions, $expressionErrors));
+		$parser = new Parser(new Lexer());
+		$evaluator = new Evaluator($functions, $expressionErrors);
+		$growthPoints = new GrowthPoints($parser, new TypeChecker($functions, $expressionErrors), $evaluator);
 		$selectedCases = new SelectedCases(
 			new DirectorySuiteLoader(new CaseExpander($rewriter, new AttributeFactory(new ValueTypeCheck()), $argumentResolver, $growthPoints), $constructorArguments, $rewriter, $relativePath),
 			$workingDirectory,
@@ -119,7 +122,7 @@ public function create($stdout, $stderr, string $workingDirectory, array $enviro
 					$workingDirectory,
 				),
 				new CompareCommand($selectedCases, $resultsArguments, $comparator, $reportOutput, $environment, $workingDirectory),
-				new AcceptCommand($selectedCases, $resultsArguments, $comparator, $expectationFiles, $expectationChanges),
+				new AcceptCommand($selectedCases, $resultsArguments, $comparator, $expectationFiles, $expectationChanges, new PresetFit($parser, $evaluator, $comparator)),
 			],
 			[new WorkerCommand(
 				$selectedCases,

src/Cli/ReportOutput.php

@@ -92,7 +92,7 @@ private function newExpectations(array $cases, array $measurements, string $work
 			$accepted[] = [$case, $measurement->metrics];
 		}
 
-		$plan = $this->changes->plan($accepted, $this->files->read(array_map(static fn (array $accepted): BenchmarkCase => $accepted[0], $accepted)));
+		$plan = $this->changes->plan($accepted, [], $this->files->read(array_map(static fn (array $accepted): string => $accepted[0]->expectations->file(), $accepted)));
 
 		return new NewExpectations($this->patch->render($plan, $workingDirectory), $plan->failures);
 	}