Probabilistic Chatting – How Text Completions and Chat Completions Work Behind the Scenes

Introduction

The previous post introduced transformers and how they impacted recent natural language processing (NLP) developments. In this post, we’ll take a look at how this sort of transformer model works with completion and chat completion interactions.

For the sake of simplicity, this post will assume GPT models and Open AI’s text completion and chat completion capabilities. However, the general concepts apply to other similar models.

Probabilistic text completion

The transformer models have been trained on a large body of text gathered from the internet. In the case of GPT-3, the model was trained on a large volume of crawled internet content, books, and Wikipedia. As mentioned in the previous post, during the training the models did not know the specifics about the documents in the training set.

From this large training set, the models learn to predict the next word in a sentence. The models use patterns learned from the training set to determine what would should come next. To do this, statistical patterns about words and phrases are used. As an example, given input like “The sky is”, the model might predict “blue” as the next word in the sentence.

When you provide a prompt to the model, the patterns learned during training are used to generate a response. It uses a probabilistic method to decide the mostly likely next word based on the input and on the words generated up to that point. It then repeats this process to generate full sentences.

Randomness (temperature)

It turns out that always taking the highest probability word each time ends up generating responses that are less interesting (maybe even less human-like?). These sorts of models expose a control variable (“temperature” in the case of the GPT models) to allow different levels of randomness through in the predicted responses.

As an example, consider the prompt “My favorite animal is a”.

With a temperature setting of 0, the most likely word will always be selected. Running this 10 times would yield:

My favorite animal is a dog.

My favorite animal is a dog.

My favorite animal is a dog.

My favorite animal is a dog.

My favorite animal is a dog.

My favorite animal is a dog.

My favorite animal is a dog.

My favorite animal is a dog.

My favorite animal is a dog.

My favorite animal is a dog.

With a temperature setting of 1, the most likely word will often be selected, but other results will be allowed through. Running this 10 times yielded:

My favorite animal is a chimpanzee.

My favorite animal is a dog.

My favorite animal is a dolphin.

My favorite animal is a cat.

My favorite animal is a dog.

My favorite animal is a dog.

My favorite animal is a tiger.

My favorite animal is a dog.

My favorite animal is a dog.

My favorite animal is a cat.

Pushing the temperature to its maximum level adds a lot of variety (at a cost we’ll get to later). Running this 10 times yielded:

My favorite animal is a dragon.

My favorite animal is a tiger.

My favorite animal is a hummingbird.

My favorite animal is a ferret.

My favorite animal is a owlfish.

My favorite animal is a sea turtle.

My favorite animal is a horse.

My favorite animal is a human.

My favorite animal is a dog.

My favorite animal is a calico cat.

If we look beyond just the next word in the response, you’ll can see the level of creativity (and sometimes hallucinations) in the responses to “How do you determine the age of a tree?”.

With a temperature of 0, the response will be well-grounded in information found in the training set.

The age of a tree can be determined by counting the number of annual growth rings visible on a cross-section of the tree trunk. The number of rings indicates the age of the tree, with each ring representing one year of growth.

With a temperature of 1.5, there are elements present that are valid, but the response includes both typos and nonsensical phrases.

The age of a tree can generally be measured by counting the annual growth rings (located towards the base of the trunk or inside disturbance (dead wood) can be teaken for tree sage). Treegrowth reports can accuractemy describe the exact centimeters, height grows annually furthermore improvement variesdepending a young moderate too about old state. Generally for exogeneous tree(evertiage beings) growth willbe slow but stem core samplesfor these yields more reltables results. Photosreadingod centiry grandtimes is bit tougheroroviclesetting informnetchrise offter directionu luchrequired. Twot common misconceptionspicutltaranicate age frostier rings.Genrally increaseoriom desession occures the velocity ozingoinelps influrenbus modiletangdate depirs fluct althy conutive rimrieorithaslly eraxtended varcevoiy when agen gaging treescaresul checking is required thaquelmednumolrect toye leky genrallocalityget side themae stands om sorueddi”cringe counquences an depene evbasitter dobi-aentthenction ob thesum appricertlexamplesbica nutsafetyuesasmasessapocirc nonstle

The tradeoff here is between more randomness (and usually more perceived creativity) vs. consistency or “truthfulness”.

Text completion interactions

While there are plenty of dials that can be turned, at a basic level, all that is needed to use the text completion capabilities is a prompt. For example, a prompt like “What are the detailed instructions for producing maple syrup from scratch?” might yield:

1. Collect sap: The first step in producing maple syrup is to collect the sap from maple trees. This is typically done in the late winter or early spring when the temperatures are cold but above freezing. The best time to collect sap is when the temperature is below freezing at night and above freezing during the day.

2. Boil the sap: Once you have collected the sap, you need to boil it down to concentrate the sugar and reduce the water content. This can be done over a fire or with a large pot on a stove. Boiling will take several hours, and you need to keep an eye on the sap to make sure it doesn’t boil over.

3. Filter the syrup: Once the sap has been boiled down, you need to filter out any impurities. This can be done with a cheesecloth or other fine mesh filter.

4. Bottle the syrup: Once the syrup has been filtered, it is ready to be bottled. You can use mason jars or other containers to store the syrup.

5. Enjoy! Finally, you can enjoy your homemade maple syrup. It is best served warm over pancakes, waffles, or other breakfast foods.

There can be lots of nuance involved in communicating effectively with this sort of AI model to achieve your desired results (a process called prompt engineering).

Chat completion interactions

Chat models are a special type of use-case for the multi-modal models. They are designed to operate over a series of messages back and forth, rather than on a single block of text. When you provide the history of messages along with the input, the model evaluates that entire history as it generates a response.

The same basic concept of probabilistic word generation is used and the same randomness levers are available.

This chat-optimized interaction pattern supports different use cases (such as revising a previous response, accepting corrections, etc). As an example, look at the following exchange:

Summarize the process of photosynthesis.

=========

Photosynthesis is the process by which plants, algae, and some bacteria convert sunlight into chemical energy in the form of glucose. It can be summarized in the following steps:

1. Light absorption: Chlorophyll, a green pigment found in chloroplasts of plant cells, absorbs light energy from the sun.

2. Water splitting: Water molecules are split into oxygen gas, protons (H+), and electrons (e-) in a process called photolysis. The oxygen gas is released into the atmosphere as a byproduct.

3. Electron transport: The energized electrons are transported through a series of protein complexes and molecules called the electron transport chain. This process generates ATP, the energy currency of the cell.

4. Carbon fixation: Carbon dioxide gas from the atmosphere is converted into organic compounds (sugars) in a process called the Calvin cycle. This cycle uses ATP and NADPH (a molecule generated during electron transport) to fix carbon into organic molecules.

5. Sugar synthesis: The organic molecules produced in the Calvin cycle are used to synthesize glucose, which is the primary product of photosynthesis.

Overall, photosynthesis is a complex biochemical process that allows plants and other organisms to harness the energy of the sun and convert it into usable chemical energy.

=========

Re-write the summary down to a 4th grade reading level.

=========

Plants use a special process called photosynthesis to turn sunlight into food. First, they take in sunlight using a green part called chlorophyll. Then, they take water and split it into oxygen and small particles called electrons. These electrons travel through the plant and create energy. Next, the plant uses carbon dioxide from the air to make sugar, which is their food. Finally, the plant releases oxygen back into the air. So, plants use sunlight, water, and air to make their own food!

=========

Re-write the summary for a college-level biology class.

=========

Photosynthesis is the process by which green plants and some other organisms use the energy from sunlight to produce organic compounds, mainly sugars, that are used as a source of energy for the organism. In plants, photosynthesis occurs in specialized organelles called chloroplasts. The process can be divided into two stages: the light-dependent reactions and the light-independent reactions. During the light-dependent reactions, light energy is captured by pigments in the thylakoid membranes of the chloroplasts and converted into chemical energy in the form of ATP and NADPH. These energy-rich molecules are then used in the light-independent reactions, also known as the Calvin cycle, where carbon dioxide is fixed into organic molecules like glucose. The Calvin cycle uses the ATP and NADPH produced during the light-dependent reactions to drive the chemical reactions that result in the production of sugars. Photosynthesis is a complex biochemical process that is vital for the survival of most living organisms on Earth as it provides the energy and organic molecules needed for growth and reproduction.

What to go deeper?

If you want a deeper (though still approachable) explanation of how these models are constructed, how prompts are processed semantically, and the probabilistic weighing of results and randomness, see this article.

Next time: API examples

In the next post, we’ll look at API interactions for both text completion and chat completion capabilities as we work towards how autonomous agents leverage this functionality.