Skip to content

parse! raises ArgumentError: invalid byte sequence instead of a ParseError, even for non-option arguments #161

Open

Description

@veralvx

Summary

When an argument isn't valid in its encoding (UTF-8 under a UTF-8 locale), OptionParser#parse! raises ArgumentError: invalid byte sequence in UTF-8 from a regexp match instead of an OptionParser::ParseError. It happens for an option (--name=\xff), for an option's separate value (--name \xff), and for a non-option argument that optparse would otherwise return untouched, such as a file name in Latin-1 (caf\xe9.txt). A program that rescues OptionParser::ParseError crashes with a backtrace instead, and so does ARGV.options { ... }, which documents that it rescues ParseError and prints the message. ri crashes the same way.

Reproduction

opt.rb:

require "optparse"
parser = OptionParser.new do |o|
  o.on("-n", "--name=NAME", "a name")
  o.on("-v", "--verbose")
end
begin
  rest = parser.parse!(ARGV, into: opts = {})
  p [:ok, opts, rest]
rescue OptionParser::ParseError => e
  p [:parse_error, e.class, e.message]
end

Run in bash, with a UTF-8 locale (LANG=C.UTF-8):

$ ruby -v
ruby 4.0.9 (2026-10-23 revision 3e93b23afc) +PRISM [x86_64-linux]
$ ruby -roptparse -e 'puts OptionParser::Version'
0.8.1
$ ruby opt.rb --bogus
[:parse_error, OptionParser::InvalidOption, "invalid option: --bogus"]
$ ruby opt.rb --name=$'\xff'
.../lib/ruby/4.0.0/optparse.rb:1738:in 'Regexp#===': invalid byte sequence in UTF-8 (ArgumentError)
	from .../lib/ruby/4.0.0/optparse.rb:1738:in 'block in OptionParser#parse_in_order'
$ ruby opt.rb --name $'\xff'
.../lib/ruby/4.0.0/optparse.rb:566:in 'Regexp#match': invalid byte sequence in UTF-8 (ArgumentError)
$ ruby opt.rb $'caf\xe9.txt'
.../lib/ruby/4.0.0/optparse.rb:1738:in 'Regexp#===': invalid byte sequence in UTF-8 (ArgumentError)
$ ruby opt.rb -- $'caf\xe9.txt'
[:ok, {}, ["caf\xE9.txt"]]
$ ruby -roptparse -e 'ARGV.options { |o| o.parse! }' $'caf\xe9.txt'
.../lib/ruby/4.0.0/optparse.rb:1738:in 'Regexp#===': invalid byte sequence in UTF-8 (ArgumentError)
$ ri $'caf\xe9'
.../lib/ruby/4.0.0/optparse.rb:1738:in 'Regexp#===': invalid byte sequence in UTF-8 (ArgumentError)

(First backtrace lines only; ... is the Ruby installation's prefix.) lib/optparse.rb on master (39a3901) is the same as the 0.8.1 that Ruby 4.0.9 ships. Under the C locale, Ruby tags such arguments as ASCII-8BIT and they parse.

Expected

  • For an option or an option's value with invalid bytes: an OptionParser::ParseError, e.g. OptionParser::InvalidArgument, which the class tree describes as the "errors on parsing".
  • For a non-option argument, I'd propose returning it untouched ([:ok, {}, ["caf\xE9.txt"]]): optparse doesn't otherwise look inside non-options, the argument may be a valid file name in another encoding, and after -- it is already returned untouched. Raising OptionParser::InvalidArgument for it would still be better than an ArgumentError.

Cause

OptionParser#parse_in_order classifies each argument with a case over regexps, the first being the long-option one (L1738, when /\A--([^=]*)(?:=(.*))?/m). A regexp match on a string with invalid bytes raises ArgumentError, so every such argument fails there, option or not. An option's value given as the next argument fails the same way in Switch#parse_arg (L566). Nothing turns either into a ParseError.

RubyGems rejects such arguments up front: Gem::GemRunner#validate_encoding raises Gem::OptionParser::InvalidArgument for any argument that isn't valid_encoding?. The crash that prompted it was in its own config code, but the check also keeps such arguments away from its copy of optparse.

Activity

  1. ihamzatarar commented on Nov 16, 2026

    @ihamzatarar

    I have opened a PR for this.

    Two things turned up while fixing it that are worth recording here.

    There are two sites, not one. Besides the case arg in parse_in_order, Switch#parse_arg matches the switch's own pattern against the value, so patching only the parser loop leaves --name=\xFF and -n\xFF still raising.

    String#rindex is why the second site cannot simply convert back. On a string that is not valid in its encoding it does not raise; it quietly fails to find a substring the string does contain. parse_arg calls arg.rindex(s, 0) immediately after the match, so matching in binary and restoring the encoding before that line turns the ArgumentError into a spurious InvalidArgument instead. Working in binary through that method avoids it.

    The change is two guards, one at each site: use arg.b when the argument is not valid in its encoding. Option syntax is ASCII, so the structure still parses and the bytes are passed on as they arrived. Only arguments that raise today are affected, so nothing that currently works can see a difference.

    With -n/--name=NAME and -v/--verbose defined, on Ruby 3.4.6:

    argv before after
    --bogus InvalidOption InvalidOption
    caf\xE9.txt ArgumentError returned in the rest, bytes intact
    --name=\xFF ArgumentError name set to \xFF
    --name \xFF ArgumentError name set to \xFF
    -n\xFF ArgumentError name set to \xFF
    --verbose caf\xE9.txt ArgumentError verbose, and the file name in the rest
    café.txt returned in the rest unchanged

    On the question of a ParseError: passing the bytes through seemed closer to what Ruby already does, since it hands arguments to ARGV as the bytes it was given without validating them, and File.open takes them back. A ParseError would also be an improvement over a backtrace, and the PR says I am happy to do that instead. Either way ARGV.options { ... } and ri stop crashing, since both rescue ParseError and neither rescues ArgumentError.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions