Repository navigation
Ship the tally-rewind skill, installable as a plugin or with Composer - #10
Merged
Merged
Conversation
The report's "Reproduce locally" and "Accept the new numbers" commands always said vendor/bin/tally, which does not exist in a project that keeps Tally in a Composer project of its own, such as tools/tally/. bin/tally now passes how it was run, and the commands use that. Amp-Thread-ID: https://ampcode.com/threads/T-01a1d9be-190b-7065-bc04-21b38804941c
Every instantiable class in a *Bench.php file was a benchmark class, so a helper next to a benchmark, such as a value object whose constructor takes data, failed tally list because Tally could not create it. Helpers belong in the bench file when the benchmark runs at old commits whose autoloader does not know them, as in tally-rewind. Now the class named like the file, such as StrBench in StrBench.php, is the benchmark class, as the dead code detector's usage provider already assumed. Other classes of the file are not checked. Amp-Thread-ID: https://ampcode.com/threads/T-01a1d9be-190b-7065-bc04-21b38804941c
tally-rewind finds a project's past performance fixes, turns each into a Tally benchmark locked at the fix, replays it on the releases since to find the best numbers the code had, and makes today's code beat them with a wall-clock gain on a real workload. It adds the benchmarks to the project, with the Tally setup and its CI job. scripts/measure.sh measures bench files at commits of the history, each in a worktree, with the Tally version the project has installed, installed apart from the project so that old commits need not fit it. The repository is a plugin marketplace that Claude Code and Codex both read (.claude-plugin/marketplace.json), and composer.json declares the skills for llm/skills, so a project gets the skills of the Tally version it requires. Amp-Thread-ID: https://ampcode.com/threads/T-01a1d9be-190b-7065-bc04-21b38804941c
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tally now comes with an Agent Skill,
tally-rewind, that people can install as a Claude Code or Codex plugin, withnpx skills, or with Composer. This PR also has two Tally fixes that came up while trying the skill on 11 PHP projects.Commits
vendor/bin/tally, which does not exist when Tally lives in a Composer project of its own, such astools/tally/. Nowbin/tallypasses how it was run, and the commands use that.*Bench.phpfile. So a helper next to a benchmark, such asfinal class Money { __construct(public int $cents) }, failedtally list. Five of the ten trial runs hit this. A benchmark that runs at old commits needs its helpers in the bench file, because the autoloader at those commits doesn't know them. NowStrBenchinStrBench.phpis the only benchmark class in its file, which matches whatBenchmarkUsageProvideralready assumed.tally-rewindskill and how it is installed. The skill finds a project's past performance fixes, turns each into a Tally benchmark locked at the fix, and replays it on the releases since to find the best numbers that code ever had. Then it makes today's code beat those numbers, keeping only changes that make a real workload faster in wall-clock time. It also adds the benchmarks to the project, with the Tally setup and a CI job.scripts/measure.shmeasures bench files at several commits, each in its own worktree, with the Tally version the project has installed.Installing the skill
.claude-plugin/marketplace.jsonis a marketplace namedphpstanwith the plugintally, which is the repository root. Codex reads this file too.extra.skillsincomposer.jsondeclares the skills forllm/skills, so a project gets the skills of the Tally version it requires.Testing
tally-rewindin its prompt. The GitHub shorthandphpstan/tallyworks only after this is merged.measure.shwas tested on a small repo with a slow version and its fix, at several commits in one run. Also tested: Tally intools/.tally, a commit whose dev dependencies don't install,TALLY_REWIND_PREPARE, and the error cases. Shellcheck reports nothing apart from its notes about PHP code in single quotes.make lint,make cs,make phpstan,make tests(1083 tests), andmake coverage(100 %) pass. Infection killed all 80 mutants in the changed files.