Extra Credit
tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models
$
pip install tiktokenOfficial docs outline
Extra Credit builds an outline of tiktoken's docs the first time your work uses it. Until then, start at the official documentation.
More in Python packages
All 9,553- fonttools Tools to manipulate font files
- werkzeug The comprehensive WSGI web application library.
- prompt_toolkit Library for building powerful interactive command lines in Python
- mako A super-fast templating language that borrows the best ideas from the existing templating languages.
- orjson Fast, correct Python JSON library supporting dataclasses, datetimes, and numpy
- opentelemetry-exporter-otlp-proto-http OpenTelemetry Collector Protobuf over HTTP Exporter
- threadpoolctl threadpoolctl
- isodate An ISO 8601 date/time/duration parser and formatter
- pyopenssl Python wrapper module around the OpenSSL library
- nodeenv Node.js virtual environment builder
- tokenizers
- cloudpickle Pickler class to extend the standard pickle.Pickler functionality