Tokens
Tokens are nothing but a group of characters.
Last updated
Was this helpful?
Tokens are nothing but a group of characters.
Last updated
Was this helpful?
1 token is equal to 4 characters.
For example, the following sentence "Artificial intelligence is transforming the world." has 6 tokens.
Tokens are utilized by both the input (what is sent to the AI model) and the output (what is received from the AI model).
For instance, when you train an AI model with the website , here's how many tokens would be used:
Training – Total number of characters extracted from the website (Counted once, as you only train once)
Input tokens – All the tokens used to send the question to OpenAI are counted here. They include:
Chunks – When a question is asked to the AI model, the system would fetch a chunk that closely matches the question.
Input question – Question asked by the user
Prompt – System prompt and the instructions defined
Output tokens – Size of the response received from OpenAI.
Token calculation varies based on the size of your AI model, the brevity of your prompts/instructions, and the length of the responses from the AI model.