Benchmarks PHP code by counting instructions, CPU cycles, and memory instead of measuring time.
final class SortBench
{
#[Expected(instructions: '59.00K', cycles: '94.43K', memory: '10.53KiB')]
public function benchSortHundred(): void
{
$numbers = range(100, 1, -1);
sort($numbers);
}
}Timing benchmarks don't work in CI. Runners differ from each other in CPU model, clock speed, and cache sizes, and share the hardware with other jobs, so the same code takes a different time on every run. The noise is bigger than most regressions worth catching.
Tally runs each benchmark once under CodSpeed's CPU simulation, which is built on Valgrind. The simulation counts executed instructions and estimates cycles from simulated cache misses. With the same PHP build, these numbers barely move between runs: in two CI runs of Tally's own benchmarks, instructions differed by at most 0.4 % and cycles by at most 0.8 %, and memory not at all. So the numbers are committed next to the benchmark and checked in every pull request, like the expected output of a test.
- Installation
- Writing benchmarks
- Attributes
- Complexity
- Running
- Commands
- Configuration
- In CI
- Contributing a benchmark
- Storing expectations elsewhere
- Lifecycle of a case
- Agent skills
composer require --dev phpstan/tallyTally needs PHP 8.2 or later with the POSIX extension, on Linux on x86_64 or aarch64. Nothing else to set up: each release includes, in codspeed/ for both platforms,
- a build of CodSpeed's Valgrind, with its license and where its source is. Tally starts it with the simulated caches of
codspeed run --mode=simulation, so it counts the same instructions as CodSpeed, but needs neither thecodspeedCLI nor a CodSpeed account, and uploads nothing. - builds of Tally's
tally_hooksextension (ext/tally_hooks) for PHP 8.2 to 8.6, thread safe or not. It tells Callgrind when to count, with the client requests of Valgrind'scallgrind.h.tally runloads the build for your PHP into the process that counts a case with-d extension=, so it needs nophp.inichange and nothing else in your other processes.
A debug build of PHP, or another PHP whose extension API differs from the official one, needs its own build of tally_hooks. Build it with phpize and load it in php.ini, then tally run uses it instead:
build/build-tally-hooks.sh /usr/local/lib/php/tally_hooks.so # in vendor/phpstan/tally
echo 'extension=/usr/local/lib/php/tally_hooks.so' > /usr/local/etc/php/conf.d/tally_hooks.iniReleases are tagged on the release/1.x branch, which CI builds from 1.x with the builds committed. A clone of 1.x, or dev-1.x from Composer, has no builds: run make codspeed, which builds Valgrind and tally_hooks for your PHP and needs gcc, make, autotools, and phpize.
The counts depend on the PHP build and the system libraries. Numbers measured on a laptop differ from those of a CI runner, so take the expected numbers from CI.
A benchmark is a public, non-static method whose name starts with bench, in a file whose name ends with Bench.php, in the class named like the file, such as StrBench in StrBench.php. Other classes in the file, such as helpers, are not benchmarks. Tally creates the class with the constructor arguments described below. A bench* method that is static or not public is an error, so is a path without any benchmarks.
use Tally\Attribute\DataProvider;
use Tally\Attribute\ExpectedFor;
final class SortBench
{
/**
* @return iterable<string, array{list<int>}>
*/
public static function shuffled(): iterable
{
foreach ([10, 1000] as $size) {
$numbers = range(1, $size);
mt_srand($size);
shuffle($numbers);
yield 'size-' . $size => [$numbers];
}
}
/**
* @param list<int> $numbers
*/
#[DataProvider('shuffled')]
#[ExpectedFor('size-10', instructions: '4.563K', cycles: '20.96K', memory: '1.023KiB')]
#[ExpectedFor('size-1000', instructions: '955.3K', cycles: '1.119M', memory: '59.88KiB')]
public function benchSort(array $numbers): void
{
sort($numbers);
}
}Each data set of a data provider is a case with the ID Class::method@name. A case without a data provider has the ID Class::method. The data provider runs before counting, so building the input is not counted. Keep the input the same on every run: a random order must come from a fixed seed.
Numbers are written with units: K, M, B, and T for counts, B, KiB, MiB, and GiB for memory, with four significant digits.
A benchmark method takes the arguments of its data set. Tally creates one object of the class for each case, and its constructor may take these, by type, in any order:
-
Tally\Meterturns counting off and on:public function __construct(private Meter $meter) { } public function benchExclude(): int { $numbers = $this->meter->exclude(static fn (): array => range(1, 10_000)); return array_sum($numbers); } public function benchPauseAndResume(): string { $this->meter->pause(); $text = str_repeat('tally ', 10_000); $this->meter->resume(); return mb_strtoupper($text); }
Paused code still runs under the simulation, only its cost is not collected. Memory allocated while paused does not count either. The meter works in every call Tally makes on the object, so hooks can use it too; in them, it counts nothing anyway. Using it in the constructor is an error.
-
Tally\ScratchDirectoryis an empty directory of the case's own, for files the benchmark reads or writes.path()returns it. Hooks see the same directory, so a#[Before]method can write a file the benchmark reads:public function __construct(private ScratchDirectory $scratch) { } #[Before] public function writeFile(): void { file_put_contents($this->scratch->path() . '/numbers.json', json_encode(range(1, 1_000))); } public function benchDecode(): void { json_decode(file_get_contents($this->scratch->path() . '/numbers.json'), true); }
The directory is the same in both processes that run the case, see lifecycle, and Tally removes it when
tally runfinishes.
A constructor parameter of any other type, and a method parameter of one of these types, is an error.
A method with #[Before] runs before every case of every benchmark in the class, and a method with #[After] after it, also when the benchmark throws. Neither is counted. Hooks run on the same object as the benchmark, so a #[Before] method can prepare properties the benchmark uses. Several hooks run in the order they are declared. They may be private.
A hook takes the case's data set arguments, from the first one, as many as it declares:
/** @var array<int, int> */
private array $index;
#[Before]
private function buildIndex(int $size): void
{
$this->index = array_flip(range(1, $size));
}
/**
* @return iterable<string, array{int}>
*/
public static function sizes(): iterable
{
yield 'size-10' => [10];
yield 'size-1000' => [1000];
}
#[DataProvider('sizes')]
public function benchLookup(int $size): bool
{
return isset($this->index[$size]);
}To give two benchmarks different hooks, put them in different classes.
What a benchmark returns is ignored.
examples/ has a benchmark for each feature.
All attributes are in the Tally\Attribute namespace.
| Attribute | On | Does |
|---|---|---|
#[Expected(instructions: '…', cycles: '…', memory: '…')] |
method | The expected numbers of a method without a data provider. |
#[ExpectedFor('data set', instructions: '…', cycles: '…', memory: '…')] |
method | The expected numbers of one data set. Repeat it for each data set. The name must be a string literal, so tally accept can find it. |
#[DataProvider('method')] |
method | A public static method that returns an iterable of argument lists, keyed by the data set name, or of Tally\DataSet objects. |
#[Before] |
method | Runs the method before every case of the class, not counted. See hooks. |
#[After] |
method | Runs the method after every case of the class, also when it throws, not counted. |
#[Warmup(calls)] |
class, method | Calls the benchmark this many times before counting, so the counted call does not include autoloading or filling caches. |
#[RelativeTolerance(share)] |
class, method | Each metric may differ from its expected value by this share, such as 0.05 for ±5 %. |
#[MetricTolerance(instructions: …, cycles: …, memory: …, cyclesAtLeast: …)] |
class, method | Each metric may differ by its own share, and the cycles at least by cyclesAtLeast cycles. |
#[DurationTolerance(tolerances: […], instructionsPerMillisecond: …, cyclesPerMillisecond: …, memory: …)] |
class, method | Allows a larger share the quicker the benchmark is. See below. |
#[Complexity(size: '…', formula: '…')] |
method | Instead of expected numbers, how the instructions may grow with the size of the data sets. See complexity. |
#[Constant(size)], #[Logarithmic(size)], #[Linear(size)], #[Linearithmic(size)], #[Quadratic(size)], #[Polynomial(size, degree: k)], #[Exponential(size)] |
method | #[Complexity] with the formula 1, log(n), n, n * log(n), n^2, n^k, 2^n. |
A policy on the method replaces the one on the class. A benchmark without one gets the tolerance of the configuration, which is by default
new MetricTolerance(instructions: 0.02, cycles: 0.05, memory: 0.05, cyclesAtLeast: 2000)The same code counts the same instructions to a few dozen from run to run, so ±2 % of instructions is a real change. The cycles also depend on which data the simulated caches hold: when unrelated code changes, a benchmark of a few thousand instructions can gain or lose a dozen cache misses, more than a thousand cycles. So the cycles may differ by ±5 %, but at least by 2,000 cycles. Change the default with withTolerance().
A benchmark with a complexity class and no policy of its own gets ±10 % instead, from withGrowthTolerance(): the code around the part that grows makes the instructions differ from what the formula predicts by a few percent.
Without #[Warmup], the counted call is the first one. The difference can be large:
public function benchCold(): int|false
{
// 74.63K instructions, mostly compiling the regular expression
return preg_match('~^(?<user>[a-z]+)@(?<domain>[a-z]+)\.cold$~', 'tally@example.cold');
}
#[Warmup(1)]
public function benchWarm(): int|false
{
// 797 instructions, the compiled expression is cached
return preg_match('~^(?<user>[a-z]+)@(?<domain>[a-z]+)\.warm$~', 'tally@example.warm');
}DurationTolerance is for suites whose benchmarks take from microseconds to seconds, like analysing whole files. A change elsewhere moves a quick benchmark by a larger share than a slow one. The expected instructions or cycles are converted to the milliseconds they take natively, and the share allowed comes from the range they fall in:
#[DurationTolerance(
tolerances: [0 => 0.5, 100 => 0.25, 500 => 0.2, 2000 => 0.1],
instructionsPerMillisecond: 5_800_000,
cyclesPerMillisecond: 7_400_000,
memory: 0.1,
)]Here a benchmark under 100 ms may differ by 50 ms, one from 100 ms by 25 %, and so on. Each range allows at least what the range below allows at its upper end. Memory may differ by 10 %.
Expected numbers catch any change. Sometimes what matters is how the cost grows: a lookup that must stay constant, a sort that must not turn quadratic. A complexity class checks that, with no numbers to keep up to date:
/**
* @param list<int> $numbers
*/
#[DataProvider('lists')]
#[Linearithmic('numbers')]
public function benchSort(array $numbers): void
{
sort($numbers);
}The data set of the smallest size is the base. Each other data set may count at most the base's instructions times the growth of the formula between the two sizes, plus the tolerance. For n * log(n) from 100 to 10000 elements, that is 200 times. Growing less passes: the class is an upper bound. Cycles and memory are not checked.
The size is an expression of the method's parameters. An array's size is its count, a number is the size itself. A path picks it from inside an argument:
#[DataProvider('orders')]
#[Linear('order.items')]
public function benchTotal(array $order): intPaths take keys by name (order.items), by number (rows[0]), or by a quoted string (input['a b']).
The formula is an expression of n with + - * / ^, parentheses, and the functions log (natural), sqrt, exp, and factorial. Any row of a table of time complexities can be written, such as #[Complexity(size: 'number', formula: 'sqrt(n)')] for trial division. It must be above 0 at every size of the data sets. The presets are this attribute with a fixed formula.
When the instructions grow faster than the class allows:
Tally counts 3 cases under CodSpeed's CPU simulation, 16 at once.
.SS 3 / 3 (100%)
Benchmarks failed: 2 slower, 1 unchanged
Allowed difference: instructions grow at most as O(n) from the smallest data set, ±10 % of each metric
Slower than expected
These benchmarks do more work than expected. Find out why before merging. If it is the price of a fix, accept the new complexity class.
Benchmark Metric Expected Measured Change Allowed
App\PairsBench::benchPairs@size-20 instructions 93.44K 178.7K +91.2 % ±10 %
App\PairsBench::benchPairs@size-40 instructions 186.9K 699.7K +274.4 % ±10 %
Reproduce locally
vendor/bin/tally run --filter='App\PairsBench::benchPairs@size-40'
Accept the new numbers
vendor/bin/tally run --output=tmp/tally
vendor/bin/tally accept tmp/tally
tally accept replaces a preset with the lowest one, in the order of the table above, that the measured instructions grow within as tally run compares them, keeping the size as written:
slower App\PairsBench::benchPairs: O(n) to O(n^2)
Accepted the complexity class of 1 benchmark.
This also tightens a preset the instructions grow well within, such as #[Linear] to #[Logarithmic], listed as faster. The presets are compared at the sizes of the data sets only, so #[Exponential] replaces a preset that allows more growth at the largest one, as every polynomial of a high enough degree does. Accept needs the instructions of every data set of the method. It leaves #[Complexity] with a formula of its own as it is, and fails for a method no preset bounds. A method needs a data provider with at least two data sets of different sizes, and cannot also have #[ExpectedFor] or expectations from its data provider. --filter adds the base of each selected case, and --shard keeps the cases of a method in one shard, because each is compared with its base.
Mistakes in the expressions are found when the benchmarks are loaded, before anything is counted:
App\PairsBench::benchPairs(): #[Complexity] formula: Unknown function "lg". Did you mean "log"?
n * lg(n)
^^
App\PairsBench::benchPairs(): #[Complexity] size: Unknown variable "input". The variables are "numbers".
input.numbers
^^^^^
vendor/bin/tally run --filter=SortBench benchmarks/Tally counts 3 cases under CodSpeed's CPU simulation, 16 at once.
.S. 3 / 3 (100%)
Benchmarks failed: 1 slower, 2 unchanged
Allowed difference: ±2 % instructions, ±5 % cycles but at least ±2,000, ±5 % memory
Slower than expected
These benchmarks do more work than expected. Find out why before merging. If it is the price of a fix, accept the new numbers.
Benchmark Metric Expected Measured Change Allowed
App\SortBench::benchSort@size-10 instructions 3.490K 4.190K +20.1 % ±2 %
cycles 16.74K 16.62K -0.7 % ±11.9 %
memory 832B 832B 0.0 % ±5 %
Reproduce locally
vendor/bin/tally run --filter='App\SortBench::benchSort@size-10'
Accept the new numbers
vendor/bin/tally run --output=tmp/tally
vendor/bin/tally accept tmp/tally
The commands under "Reproduce locally" and "Accept the new numbers" run Tally the way you ran it, such as tools/tally/vendor/bin/tally when Tally has a Composer project of its own.
The progress bar shows one character per case: . unchanged, S slower, F faster, N new, E not measured.
A case is slower when a metric exceeds its expected value by more than the tolerance, and faster when it is lower by more. A case that throws, exits, or times out is not measured. A case without expected numbers is new. Slower and not measured cases fail the run. Faster and new cases pass, but their numbers should be accepted, so a later slowdown back to the old numbers does not go unnoticed.
Exit codes: 0 passed, 1 failed, 2 invalid usage, invalid benchmark, or CodSpeed not set up.
tally <command> [options] [arguments]
Every command takes the paths of the benchmarks as arguments: directories, searched recursively for *Bench.php, and files. Without paths, it takes them from the configuration. These options select cases:
| Option | |
|---|---|
--config=<file> |
Configuration file, tally.php in the working directory by default. |
--filter=<filter> |
Only cases whose ID contains this text, or matches it as a regular expression like /@size-10$/. A filter that matches no case is an error. |
--shard=<shard> |
Only the cases of one shard, like 2/4. See sharding. |
Every command takes --help.
Lists the IDs of the cases, one per line, in a stable order.
Counts the cases under CodSpeed and compares them with the expected numbers.
| Option | |
|---|---|
--jobs=<n> |
How many cases run at once, the number of CPUs by default. Valgrind runs each case on one CPU. |
--output=<directory> |
Keep the results there, for tally compare and tally accept. |
Before counting, run checks that every case is valid and that its expected numbers can be rewritten, and exits with code 2 if not.
Compares the results of tally run --output with the expected numbers again, without counting. Useful after editing expectations, or to compare the results of several shards together.
| Option | |
|---|---|
--table |
Also print every case with its metrics and status. |
<results> is searched recursively, so the results of several runs can be downloaded into one directory.
Makes the results of tally run --output the expected numbers of the cases that are new, slower, or faster. It rewrites the #[Expected] and #[ExpectedFor] attributes, or adds them, with the imports they need, and leaves the rest of the file as it is. It replaces the preset complexity class of a method whose instructions grew more than it allows, or well within a lower preset, and removes the import of the replaced class when nothing else uses it.
slower App\SortBench::benchSort@size-10
Accepted the numbers of 1 case.
tally.php in the working directory returns a Tally\Configuration:
<?php declare(strict_types = 1);
use Tally\Attribute\RelativeTolerance;
use Tally\Configuration;
return Configuration::create()
->withPaths([__DIR__ . '/benchmarks'])
->withIniEntries(['memory_limit=-1'])
->withTimeLimits(60, 30 * 60)
->withTolerance(new RelativeTolerance(0.05))
->withNamedArguments(true);Every value Tally uses when a benchmark or the command line does not give one is chosen in Configuration::create().
| Method | |
|---|---|
withPaths(list<string>) |
Directories and files with benchmarks, used when a command gets no paths. |
withIniEntries(list<string>) |
PHP settings the cases run with, passed as -d. |
withTimeLimits(native, counted) |
Seconds a case may run natively, before it is counted, and under the simulation, which is about 50 times slower. 60 and 1800 by default. A case that runs longer is killed and reported as not measured. |
withTolerance(TolerancePolicy) |
The tolerance of benchmarks without a policy on their method or class: any of the tolerance attributes, or your own TolerancePolicy. MetricTolerance with ±2 % instructions, ±5 % cycles but at least 2,000, and ±5 % memory by default. |
withGrowthTolerance(TolerancePolicy) |
The tolerance of benchmarks with a complexity class and without a policy on their method or class. RelativeTolerance(0.1) by default. |
withNamedArguments(bool) |
Whether tally accept and the patch of new expectations write #[ExpectedFor(dataSet: 'size-10', …)] instead of #[ExpectedFor('size-10', …)]. Off by default. |
withEnvironmentVariables(list<string>) |
The environment variables the cases run with; tally run leaves out the others. The environment is on the stack of the counted process, so a variable that differs from run to run, such as the branch name in CI, moves its memory layout, and with it the counts of benchmarks whose allocations depend on it. PATH, HOME, TMPDIR, LANG, LC_ALL, PHPRC, and PHP_INI_SCAN_DIR by default; list them too when you add one. |
withStrippedExtensions(list<string>) |
The extensions the cases run without, even when PHP's ini files load them. PHP often loads them enabled, which multiplies the counts: Xdebug in its default develop mode and PCOV with pcov.enabled=1 count a loop of method calls 6.5 and 3.4 times over. A setting cannot unload an extension, and the cases do not get the -d options of tally run. When an ini file loads one, tally run starts the cases with a php.ini made of PHP's ini files with those lines commented out. A case fails when PHP loads one some other way, such as built in. xdebug, pcov, uopz, blackfire, tideways, tideways_xhprof, xhprof, spx, excimer, memprof, and opentelemetry by default; leave out one that your benchmarks need. |
The counts depend on everything that runs the benchmark:
- the PHP version, including the patch release,
- how PHP was built, which extensions are loaded, and the PHP settings, such as whether opcache and JIT are on; the cases run without debuggers and profilers, see
withStrippedExtensions(), - the system libraries, such as glibc, which come with the runner's operating system image,
- the environment variables the cases keep, see
withEnvironmentVariables().
Expected numbers measured in one environment only fit that environment. So:
- use a fixed runner image, like
ubuntu-24.04, notubuntu-latest, which moves to a new Ubuntu release from time to time, - keep the PHP setup the same: version, extensions, and
ini-values, - expect to accept new numbers for every benchmark when you change any of these.
setup-phpinstalls the latest patch release of a PHP version, so a new PHP release can change the numbers too, usually of many cases at once.
jobs:
benchmarks:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- uses: shivammathur/setup-php@v2
with:
php-version: "8.5"
extensions: posix
- run: composer install --no-interaction --no-progress
- run: vendor/bin/tally run --output=tmp/tally-results
- if: always()
uses: actions/upload-artifact@v4
with:
name: tally-results
path: tmp/tally-resultsIn GitHub Actions, Tally also writes the report to the job summary and annotates the benchmarks that are slower, faster, new, or not measured, with paths relative to GITHUB_WORKSPACE, so Tally can run in a subdirectory of the repository. The report's accept command downloads the tally-* artifacts of the run:
gh run download 123456789 -R owner/repo -p 'tally-*' -D tmp/tally
vendor/bin/tally accept tmp/tally--shard=2/4 runs the second of four shards. The shards are balanced by expected instructions, from the most expensive case, each to the shard with the lowest total so far. A new case without numbers counts as much as the most expensive one. The same cases always give the same shards.
strategy:
matrix:
shard: [1, 2, 3, 4]
steps:
# …
- run: vendor/bin/tally run --shard=${{ matrix.shard }}/4 --output=tmp/tally-results
- if: always()
uses: actions/upload-artifact@v4
with:
name: tally-results-${{ matrix.shard }}
path: tmp/tally-resultsEach shard reports its own cases in the log, without annotations and job summary. To see them together, download all artifacts into one directory and run tally compare, which writes the annotations and the job summary.
A contributor without CodSpeed adds a benchmark without numbers and opens a pull request. CI measures it, and the report ends with a patch that adds the expected numbers:
--- a/benchmarks/StrBench.php
+++ b/benchmarks/StrBench.php
@@ -3,10 +3,13 @@
namespace App;
use Tally\Attribute\DataProvider;
+use Tally\Attribute\Expected;
+use Tally\Attribute\ExpectedFor;
final class StrBench
{
+ #[Expected(instructions: '122.3K', cycles: '203.9K', memory: '24.00KiB')]
public function benchImplode(): void
{
implode(',', range(1, 1000));The patch is in the CI log and in the job summary. The contributor saves it and runs git apply, then pushes. A new case does not fail the run.
Expected numbers do not have to be attributes. A data provider can yield Tally\DataSet objects with their own Tally\Expectation\ExpectationSource:
#[DataProvider('scripts')]
public function benchScript(string $file): void
{
require $file;
}
/**
* @return iterable<string, DataSet>
*/
public static function scripts(): iterable
{
$header = new ScriptHeader();
foreach (glob(__DIR__ . '/scripts/*.php') as $file) {
$name = basename($file, '.php');
yield $name => DataSet::named($name, [$file])
->withExpectations(new ScriptExpectations($header, $file, $header->read(file_get_contents($file))));
}
}Here each script keeps its numbers in a comment at its top:
<?php declare(strict_types = 1);
/*
* instructions: 679.5K
* cycles: 854.8K
* memory: 36.30KiB
*/An ExpectationSource has four methods:
| Method | |
|---|---|
read(): ?Metrics |
The expected numbers, null for a new case. |
file(): string |
The file that keeps them. |
edits(string $code, Metrics $measured): list<CodeEdit> |
The edits of the file's code that write the measured numbers instead, each a Tally\Expectation\CodeEdit of an offset, a length, and the text that replaces those bytes. Takes code and returns edits without reading or writing the file, so tally accept writes the edited code and tally run shows it as a patch. Tally applies the edits of all cases of a file in one pass: they must not overlap, equal edits, such as the same import, are made once, and insertions at one offset go in the natural order of their text. |
location(): SourceLocation |
Where reports point to. Throws ExpectationException when the case's numbers cannot be rewritten, so tally run fails before counting. |
examples/Scripts is the whole example.
tally run loads the benchmarks, then runs each case in two processes of its own: first natively, which fails quickly when the case is broken, then under CodSpeed's Valgrind, which counts it. Several cases run at once, see --jobs.
tally run loads the whole suite, and each process of a case loads only the file of the case's class, so the data providers of a file run twice for each of its cases, and once in tally run. Every tally command runs them too. Keep them cheap and do expensive preparation in a #[Before] method, which runs only for the case that needs it. A benchmark class finds the other classes it uses through the autoloader, not because another *Bench.php file declared them.
In each of the two processes of a case, Tally:
- loads the file of the case's class and calls the data providers of its benchmarks, not counted,
- creates the benchmark class, once, with its
MeterandScratchDirectory, - calls the
#[Before]methods, not counted, - starts CodSpeed's CPU simulation, with empty caches,
- calls the benchmark as many times as
#[Warmup]says, not counted, but simulated, so the calls fill the simulated caches, - counts a warm-up span, because the first counted span costs more than later ones, and throws it away,
- counts an empty call through the same path as the benchmark, the baseline,
- counts one call of the benchmark,
- stops the simulation, also when the benchmark throws,
- calls the
#[After]methods, not counted.
The simulation runs from step 4 to step 9, and only collecting the counts is turned on for each span, so the simulated caches stay warm from one span to the next. codspeed run starts it for each benchmark, which empties the caches, so the cycles of a benchmark of a few thousand instructions depend on which cache lines the code just before it happens to touch. Tally counts the same instructions as CodSpeed, and fewer cycles for small benchmarks.
The native process runs the same steps without the simulation, so hooks and warm-up calls run twice per case, once in each process. The scratch directory is shared between the two, so a #[Before] method that writes a file finds it already there in the second process.
Instructions and cycles come from Callgrind's profile of the benchmark's span. Memory is the peak above the memory used when counting started, minus that of the baseline.
Cycles are Valgrind's estimate: instructions, plus 10 for each first-level cache miss and 100 for each last-level cache miss of the simulated cache. The cache sizes are the same on every machine.
Tally comes with Agent Skills for coding agents such as Claude Code and Codex, in skills/:
tally-rewindfinds the commits of a project that fixed performance problems, turns each into a Tally benchmark locked at the fix, and replays it on every release since. It then makes today's code beat the best numbers the code ever had, keeping only changes that make a real workload faster in wall-clock time. It adds the benchmarks to the project, and sets up Tally and its CI job when the project has none.
Install them as a plugin, for all your projects:
claude plugin marketplace add phpstan/tally
claude plugin install tally@phpstan
codex plugin marketplace add phpstan/tally
codex plugin add tally@phpstanOr into one project, in .agents/skills/, with skills, which links them into .claude/skills/ for Claude Code:
npx skills add phpstan/tally -a claude-code -a codexOr with Composer and llm/skills, which copies the skills of the packages a project requires into .agents/skills/ after every composer install and update, so they stay in the version of Tally the project has:
composer require --dev llm/skillsClaude Code reads .claude/skills/, so for it add a skills.json next to composer.json, then run composer skills:update:
{
"aliases": [".claude/skills"]
}MIT