tokenizers
Fastest implementation of the most used tokenizers in AI.
- Category
- Rust crates
cargo add tokenizers Official docs outline
docs.rsExtra Credit builds an outline of tokenizers's docs the first time your work uses it. Until then, start at the official documentation.
More in Rust crates
All 4,945-
objc-sys Raw bindings to the Objective-C runtime and ABI
-
maybe-uninit MaybeUninit for friends of backwards compatibility
-
konst Const equivalents of std functions, compile-time comparison, and parsing
-
fslock A library to use files as locks
-
wry Cross-platform WebView rendering library
-
quoted_printable A simple encoder/decoder for quoted-printable data
-
serde-xml-rs xml-rs based deserializer for Serde (compatible with 1.0)
-
serde_regex A serde wrapper that (de)serializes regex as string
-
rust-embed-impl Rust Custom Derive Macro which loads files into the rust binary at compile time during release and loads the file from the fs during dev
-
rmcp Rust SDK for Model Context Protocol
-
thin-vec A Vec that takes up less space on the stack.
-
tree-sitter-language The tree-sitter Language type, used by the library and by language implementations