tokenizers 1.0.0rc5

Source: pypi.org

Read original source

At a glance

Reworked tokenizers Python bindings, over the pipeline encode path (not for release yet)