Tokens

Tokens are nothing but a group of characters.

1 token is equal to 4 characters.

For example, the following sentence "Artificial intelligence is transforming the world." has 6 tokens.

Calculating the Tokens:

Tokens are utilized by both the input (what is sent to the AI model) and the output (what is received from the AI model).

For instance, when you train an AI model with the website www.reply.cx, here's how many tokens would be used:

• Training – Total number of characters extracted from the website (Counted once, as you only train once)

• Input tokens – All the tokens used to send the question to OpenAI are counted here. They include:

  • Chunks – When a question is asked to the AI model, the system would fetch a chunk that closely matches the question.

  • Input question – Question asked by the user

  • Prompt – System prompt and the instructions defined

• Output tokens – Size of the response received from OpenAI.

Token calculation varies based on the size of your AI model, the brevity of your prompts/instructions, and the length of the responses from the AI model.