Skip to content

fix: read digit-grouped numbers outside of Measure #404

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pengzhendong merged 1 commit into from
Nov 6, 2026
Merged

fix: read digit-grouped numbers outside of Measure #404

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion tn/chinese/rules/cardinal.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.

from pynini import accep, cross, string_file
from pynini import accep, closure, cross, string_file
from pynini.lib.pynutil import add_weight, delete, insert

from tn.processor import Processor
Expand Down Expand Up @@ -66,6 +66,10 @@ def build_tagger(self):
| rmzero**8))

number = digits | ten | hundred | thousand | ten_thousand | hundred_million
# 15,000 => 一万五千, 1,000,000 => 一百万. It costs more than a comma between
# two numbers (100), so that 1,2345 doesn't become 1,234 followed by 5.
grouped = closure(self.DIGIT, 1, 3) + (accep(",") + self.DIGIT**3).plus
number |= add_weight(grouped @ self.build_rule(delete(","), self.DIGIT, self.DIGIT) @ number, 120)
number = sign.ques + number + (dot + digits.plus).ques
number @= self.build_rule(cross("二百", "两百") | cross("二千", "两千") | cross("二万", "两万") | cross("二亿", "两亿"),
"[BOS]").optimize()
Expand Down
6 changes: 6 additions & 0 deletions tn/chinese/test/data/normalizer.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,3 +52,9 @@ B2B => B to B
撥打119 => 拨打幺幺九
他打了120分 => 他打了一百二十分
给110打个电话 => 给幺幺零打个电话
跑了15,000公尺 => 跑了一万五千公尺
卖出3,791台 => 卖出三千七百九十一台
全长1,000,000 => 全长一百万
价格是$1,000 => 价格是一千美元
选1,2,3 => 选一,二,三
1,2345 => 一,两千三百四十五
Loading