Hi guys.
Welcome to this session.
So in this session we'll talk about what are tokens and token limits.
So tokens are basically these are LM processes the text using tokens.
So the large language models like ChatGPT which we are using, they process the text which we see using
these tokens which are common sequences of characters found in a text.
These models learn to understand relationships between tokens and then predicting the next one in a
sequence.
Like in this example, you can see the sentence which we're given is.
Today is a beautiful sunny day that has utilized seven tokens and has 35 characters.
So seven tokens is each word which we get to see gets to use one token.
And then also the period at the end is given a specific token.
Now you can count the total number of tokens in your piece of text by going to the open AI platform's
tokenizer section, and you can see for yourself how many it might tokenize.
So now looking at the token limits, this is going to be the maximum number of tokens that a model or
an API actually processes.
Now this takes into consideration both the input tokens which is our prompts, and the output tokens,
which is the results which we get from the tool.
Now, for example, GPT four has a context window of 128,000 tokens it can provide now, which is almost
equivalent to 500 book pages.
Now we use the max token parameters in the API to limit the output tokens.
So which is basically done for the purpose of making sure that the output which we are giving are concise
and readable.
Now we also want to control the API cost.
We want to save computational power and also ensure that the results are below a certain length.
So these are the this is the tokens which is being used at the back end.
Every time a user is prompting the AI tool and it gets the responses.