Skip to content

Add a live tokenizer and local Codex guidance - #386

Draft
saioai wants to merge 22 commits into
mainfrom
codex/f29-tokenizer-codex-20261008
Draft

saioai wants to merge 22 commits into
mainfrom
codex/f29-tokenizer-codex-20261008

Conversation

@saioai

@saioai saioai commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

The CLI has a server request-token counter, but lacks local token inspection and a Codex setup guide.
This adds an offline token editor, script commands, and copyable Codex instructions.
For example, Hello, world! produces four tokens with the default tokenizer.

What changes

  • Bare tokenizer opens a live editor using the image generation theme.
  • The visible View row switches between Text, Token IDs, and Bytes. Tokenizer selects the vocabulary.
  • Options use the image picker's row navigation, choice menus, focus markers, and Escape behavior.
  • The main screen shows token count and selection. Enter opens Details for IDs, byte offsets, and complete hexadecimal data.
  • Text and Token IDs support ordinary inspection. Bytes adds the input byte count for exact-byte debugging.
  • count and inspect accept text, files, or stdin. Fixed Unicode 16 rules preserve results across Go versions.
  • Help groups these commands under Local tools and lists tokenizer subcommands. codex prints setup instructions and official destinations.

Text editing preserves whitespace, line endings, and Unicode bytes. View changes preserve selection without recomputing tokens.
Editing clears stale results immediately and cancels superseded computation.
Pipes, CI, dumb terminals, and explicit text format receive guidance without consuming editor input.

These utilities need no API credentials. They never save drafts, change settings, install software, or launch agents.
Only explicit codex --open launches a browser handler, with named desktop settings and an unchanged parent environment.
The tokenizer accepts at most 1 MiB. Counts exclude request structure, multimodal input, billing, and context-limit guarantees.
Existing responses input-tokens count behavior remains unchanged.

Commands

These commands are new:

openai tokenizer                                    # live terminal editor
openai tokenizer count --text "Hello, world!"        # count literal text
openai tokenizer inspect --file prompt.txt           # IDs and exact byte boundaries
printf 'Hello' | openai tokenizer count              # read stdin
openai --format json tokenizer inspect --file prompt.txt
openai tokenizer count --text "Hi!" --encoding cl100k_base
openai tokenizer encodings
openai tokenizer licenses
openai codex                                        # instructions, without installing
openai codex --destination config                    # print one official URL
openai codex --destination docs --open                # explicitly open that URL

The default tokenizer is o200k_base; cl100k_base is also available. Model names are not guessed.
Choose --text or --file; either leaves piped stdin unread. Combining both fails; --file - explicitly selects stdin.
Use count or inspect for JSON. Existing error-format options remain available.

Tab moves between text, options, and tokens. Left/Right on View switches representations immediately; Enter opens its chooser.
Up/Down moves through options or choices. Enter opens a chooser or applies its highlighted choice.
Escape cancels a chooser and returns to text. In Tokens, arrows select tokens; Enter opens Details.
Escape closes Details and returns to Tokens.
While editing, Enter inserts a newline. Down at the end opens options; arrows within multiline text move the cursor.
F1 shows all controls. Ctrl+C exits.

Failure behavior remains explicit:

  • Invalid or oversized paste preserves the draft. Failed computation preserves it and offers retry.
  • Cancellation stops computation and restores terminal settings. Blocked output cannot hold the editor's input loop.
  • Write failures return nonzero. Scripts must reject incomplete output and check producer status, including with pipefail.
  • Failed output prevents browser launch. Launcher failures retain the URL; check whether it opened before retrying.

Shell redirection can truncate existing files. The input and recovery guide includes a tested temporary-file replacement recipe.

Code

pkg/custom owns commands, presentation, and input routing.
The new internal/tokenizer package owns local encoding and validates complete byte reconstruction in both output modes.
The small startup change bypasses irrelevant API URL setup for parsed local commands. Generated API commands retain their behavior.

The editor uses one cancellable preview child and one output child.
An input bridge preserves paste bytes before decoding; bounded output keeps writes outside keyboard handling.
The UI reuses terminal libraries and image frame helpers.

Vocabulary comes from tiktoken-go/tokenizer v0.7.0; BPE comes from pkoukk/tiktoken-go v0.1.8.
The adapter uses existing regexp2 v1.11.5; google/uuid v1.6.0 is indirect.
Bundled notices and offline regeneration cover the dependencies and pinned Unicode data.

Tested

The controls update passed 113 focused results and 113 race results under Go 1.26, without skips.
Independent review exercised 240 bounded frames, recovery messages, chooser cancellation, and view changes without recomputation.

After alignment with main 6fcc478, 522 helper/help results and 801 public routing/help results passed, without skips.
Composition checks found six local help pages missing global-option guidance. The corrected templates pass those same checks.
Another 72 native Bash/zsh cases copied and executed the help footer using spaced paths and varied PATH contents.
All passed, with no network requests or isolated state changes.

Eight native editor cases passed across Bash/zsh, including exact paste, both tokenizers, canceled choices, resize, and colorless output.
They verify exact token IDs, offsets, and hex, plus Ctrl+C status 130, restored terminal settings, and reaped helpers.
One attempt stopped during harness startup before CLI execution. Bounded setup diagnostics were added; the affected case then passed.
Recorder SIGINT/SIGTERM checks returned 130/143 within 28 ms and left no owned processes.
Independent validator review passed 40 checks, including real-capture regression cases and negative controls.

Editor checks use 4542772; help execution, the final native case, and recordings use e990b0b.
e990b0b changes help templates only; 7fc322e changes recorder validation only.
9fb91c1 adds a recorder cleanup comment. Editor runtime remains identical.
Vet and the trusted main custom-code budget passed. The budget remains 21/1,000 lines.

Earlier evidence remains scoped to unchanged source:

  • The adapter audit checked 810,264 reference results across Go 1.26/1.27; production review at 8a8c21e verified equivalent behavior.
  • 8a8c21e evidence includes 544 public checks and 30 native Bash/zsh cases, including offline operation and blocked-output cancellation.
  • Browser checks at 03aa64c/83d018c passed 177 focused results, 19 inert-launcher cases, and 129 Linux compatibility results.
  • At 30a5e91, parent-death and saturated-pipe checks passed 20 repetitions each, plus 12 focused race results.

The lifecycle tests reject a removed lifeline at their original three-second deadlines.
Module verification, deterministic regeneration, Windows amd64 compilation, and Linux 386 compilation passed on earlier source.
Native Linux/Windows editor and browser behavior remains unverified. Replay does not establish graphical-terminal appearance.
Long unbroken input remains expensive; the editor stays responsive during computation.

All 20 executed CI checks passed at 9fb91c1; three conditional jobs skipped.
Both trusted budget statuses passed. Hosted code review found no major issues; security review found none.

Demo

Actual CLI binaries on macOS arm64 with Go 1.27, identical synthetic text, and no API calls.
Before is the previous interactive editor at 30a5e91; after is e990b0b, with the same runtime as 7fc322e.
Both editors exit through Ctrl+C with status 130.
Bash PTY replays use Menlo/Apple Color Emoji, truecolor for dark/light, and NO_COLOR for colorless output.
All four comparisons passed transcript validation. Independent review accepted every static state and the three published GIFs.
The light theme uses a static screenshot; its incremental replay renderer left faint background edges.

Tokenizer views and picker controls

Before:

Before: previous tokenizer editor

After:

After: visible View and Tokenizer controls

Token IDs · Bytes · Details · View chooser · Tokenizer chooser · Alternate tokenizer · Controls

40-column editor · Light theme · NO_COLOR

The Codex guide is unchanged. Its reviewed recordings remain scoped to 510dc817:
80-column guide · Light guide, 40 columns.

Recording recipe · Recorder

@saioai

saioai commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

@codex review for failure handling, tokenization correctness, output integrity, and local-only behavior.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-08T19:04:54.991609Z 9fb91c1 Manual request
🔒 Security Review ✅ Completed 2026-10-08T19:06:08.088463Z 9fb91c1 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Castiron custom code

Evaluated main: 6fcc478086507571286727084a7d7b162a37ab51.

✅ No new custom-code files detected.

2 mixed files remain; 0 existing customizations changed.

Compared 6fcc47808650 → 9fb91c176c9f. Generated baselines verified.

2 existing customizations unchanged
  • pkg/cmd/adminorganizationcertificate.go
  • pkg/cmd/audiovoice.go

A changed generated baseline means this report cannot reliably identify which handwritten lines changed.

Inspect the custom-code diff

Download the exact patch produced by this run (requires repository access):

gh run download 37830097130 --repo openai/openai-cli \
  --name castiron-custom-code-37830097130-1 --dir /tmp/castiron-custom-code-37830097130-1
git apply --stat /tmp/castiron-custom-code-37830097130-1/custom-code.patch
cat /tmp/castiron-custom-code-37830097130-1/custom-code.patch

Or reproduce it from an SDK checkout containing the vendored reporter:

git fetch --no-tags origin 6fcc478086507571286727084a7d7b162a37ab51 9fb91c176c9f70d09cf806648c2f17b1654a292a
python3 scripts/castiron/custom_code_report.py report \
  --base 6fcc478086507571286727084a7d7b162a37ab51 \
  --head 9fb91c176c9f70d09cf806648c2f17b1654a292a --fetch --require-head-hash --public \
  --out /tmp/castiron-custom-code-9fb91c176c9f
cat /tmp/castiron-custom-code-9fb91c176c9f/custom-code.patch

This is the current full custom patch for mixed files, not an attribution of only the handwritten lines changed by this PR.

Full report and patch

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6e761e7105

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/custom/codex.go
@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 6e761e7105

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

saioai commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

@codex review the latest scoped follow-up and its regression coverage.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9e9cc403a8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread go.mod
…9-tokenizer-codex-20261008

# Conflicts:
#	pkg/custom/codex_test.go
@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 9e9cc403a8

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@saioai saioai changed the title Add local tokenizer and Codex setup helpers Add a live tokenizer and local Codex guidance Oct 8, 2026

saioai commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

@codex review the current head for tokenizer accuracy, terminal cancellation, exact paste handling, and preserved script behavior.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 510dc817a1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 510dc817a1

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

saioai commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

@codex review the current head, including the tokenizer parent-help correction and command-discovery regression coverage.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 69be356689

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/custom/codex.go Outdated
@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 69be356689

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

saioai commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

@codex review the latest scoped browser-launch follow-up and its regression coverage.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 03aa64c8f9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/custom/codex.go
@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 03aa64c8f9

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

saioai commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

@codex review the final scoped Linux desktop compatibility correction and its regression coverage.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 83d018cc2c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 83d018cc2c

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

saioai commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

@codex review the test-only lifecycle correction. Production code and deadlines are unchanged; both tests reject a disabled lifeline branch.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 30a5e9186f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 30a5e9186f

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

saioai commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

@codex review the updated tokenizer controls at 7fc322e. Check chooser cancellation, exact byte details, script-mode preservation, and compatibility with current main help.

Comment thread scripts/demos/tokenizer-codex/editor_driver.py Fixed

saioai commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

@codex review the final head 9fb91c1. It adds only the requested cleanup comment above the controls update; executable Python ASTs remain identical.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 9fb91c176c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 9fb91c176c

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant