Back

Version 4

6/11/2023
// Demo of tokenizer to mimic behavior of https://platform.openai.com/tokenizer
// Tokenizer uses "gpt-3.5-turbo" model by default but this demo uses davinci to match the playground
const TokenizerDemo = (async () => {
const tokens = await @zzz.Tokenizer(
"Hello Val Town",
"text-davinci-003",
);
console.log(JSON.stringify(tokens));
return `Tokens: ${tokens.length}`;
})();
Updated: October 23, 2023