@huggingface/tokenizers
🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers
npm install @huggingface/tokenizers 🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers
npm install @huggingface/tokenizers