Chatting With A LLM - Using the Open AI APIs

This is the third post in a series that looks at how autonomous agents use generative AI models.

Introduction

The previous post discussed how the text completion and chat completion capabilities work generally. It also showed some examples of prompts and responses.

In this post, we’ll see how to use Open AI’s API to perform text completion and chat completion operations programatically. The post will add code bit-by-bit. If you want to see the consolidated result, you can look here.

Basics

The code examples below will use the openai Python package. You’ll also need an Open AI API key.

import openai
import os

openai.api_key = os.getenv('OPENAI_API_KEY')

Text Completions

There are multiple models available for text completion operations. For this example, we’ll use text-davinci-003.

completion_model = 'text-davinci-003'

The API reference for completions shows a number of parameters than can be used. We’ll focus on just a few of them though:

model – The ID of the model to use for completions
prompt – The prompt upon which to operate (we’ll just be using strings in this example)
temperature – The temperature setting controls randomness (discussed with examples in the previous post)
max_tokens – The number of tokens to generate with the completion (see this for an explanation of how tokens are calculated)
n – The number of completions to generate

A basic interaction can consist solely of the model and prompt parameters.

completion = openai.Completion.create(model=completion_model, prompt='What are the three primary colors?')
print(completion.choices[0].text)

The three primary colors are red, blue, and yellow.

Alternatively, you can request multiple completions:

completion = openai.Completion.create(model=completion_model, prompt='How do you get from Indianapolis to Chicago?', n=4, max_tokens=1024)
for choice in completion.choices:
  print(choice.text)

To get from Indianapolis to Chicago, you can take a road trip and drive on I-65 N for about 2 hours and 45 minutes (176 miles). Alternatively, you can save time and take an Amtrak train which takes 2 hours and 20 minutes (137 miles).

The easiest way to get from Indianapolis to Chicago is by car. The drive is about 3 hours and 40 minutes and can be completed via I-65. Other options include taking the Amtrak line from Indianapolis to Chicago, which takes 3 hours and 15 minutes, or taking a bus from Indianapolis to Chicago, which typically takes between 4 and 5 hours.

You can drive from Indianapolis to Chicago in approximately three hours via Interstate 65 North, or you can take a train from Indianapolis’ Amtrak Station to Chicago’s Union Station.

The quickest way to get from Indianapolis to Chicago is by car, as the distance is only about 176 miles and the drive takes about 2.5 hours. You can also take the train, which takes about 3.5 hours, or fly, which takes around 1 hour.

You can use the temperature setting to adjust responses from a scale of predictable to creative to non-sensical:

prompt = 'Tell a (very) short fictional story about foxes in the wild'
for temperature in [0, 0.8, 1.5]:
  completion = openai.Completion.create(model=completion_model, prompt=prompt, temperature=temperature, max_tokens=256)
  print(f'Temp {temperature}: {completion.choices[0].text}')

Temp 0:

Once upon a time, there were two foxes living in the wild. They were the best of friends and spent their days playing and exploring the forest together. One day, they stumbled upon a clearing filled with wildflowers and decided to take a nap in the sun. As they dozed off, they heard a rustling in the bushes. Suddenly, a third fox appeared! The two friends were so excited to meet their new companion and they all spent the rest of the day playing and exploring the forest together. From that day on, the three foxes were inseparable.

Temp 0.8:

Once upon a time, two foxes lived in the wild. They were both old and wise, but had very different personalities. One was a brave and independent hunter, while the other was more timid and shy.

One day, the bold fox ventured out in search of food. But while out, he stumbled upon a strange stone that had strange symbols carved into it. He decided to investigate, but soon realized it was a trap.

The timid fox, meanwhile, had been watching from afar. Seeing his friend in danger, he quickly leaped into action and sprung the trap, allowing the other fox to escape. From then on, the two foxes were inseparable, and their friendship and loyalty lasted a lifetime.

Temp 1.5:

A group of foxes lived in the woods near a small farmer’s village many years ago. Always clever, they would hide in the fallen orange leaves as they moved directionally to find food. On warm summer days they could nearly stay still close to sunny wobbly spots until the evenings. Often at night their age wise ears would almost succeed in stolen sweethorns from the always stockedfarmers homes, quickly after the childrens schalding screams its goals. Toadstools however were much easier story challenges needing watched only once for bun burrs steeps and pinecone threes standing as they could give was something seasonal subtlet pleasant luckyhunt flavorings.

Text completions for chat

Before looking at the chat completion capability itself, you might ask whether you could cobble together chat functionality using just the text completion API.

You could call the completion API in a loop like this:

while (True):
  print('Prompt: ')
  prompt = input()

  if prompt == "":
    break

  completion = openai.Completion.create(model=completion_model, prompt=prompt, max_tokens=250)
  answer = completion.choices[0].text

  print(f'Answer: {answer}')

However without any sense of shared history, the results wouldn’t be great:

Prompt:
What are the capitals of states on the west coast?

Answer:

Oregon – Salem
Washington – Olympia
California – Sacramento
Idaho – Boise
Alaska – Juneau
Hawaii – Honolulu

Prompt:
Sorry. I meant the east cost.

Answer:

The East Coast of the United States stretches from the Atlantic seaboard of Maine, New Hampshire, and Massachusetts in the north all the way down to the Florida peninsula in the south. Major cities along the East Coast include Boston, Philadelphia, Washington D.C., Miami, and New York City.

You might be tempted to keep history yourself and provide it to the text completion API:

history=[]
while (True):
  print('Prompt: ')
  prompt = input()

  if prompt == '':
    break

  formatted_history = '\n'.join(history)
  formatted_prompt = f'Given previous conversation history of:\n{formatted_history}\n\n{prompt}'

  completion = openai.Completion.create(model=completion_model, prompt=formatted_prompt, max_tokens=250)
  answer = completion.choices[0].text

  history.append(f'User: "{prompt}"')
  history.append(f'Agent: "{answer}"')

  print(f'Answer: {answer}')

Which does show better results:

What are the capitals of the west coast?
Answer:

The capitals of the west coast of the United States are Sacramento, California; Olympia, Washington; and Salem, Oregon.

Prompt:
Sorry, I meant the east coast.

Answer:

No problem! The capitals of the east coast of the United States are Boston, Massachusetts; Providence, Rhode Island; Hartford, Connecticut; Albany, New York; Trenton, New Jersey; Dover, Delaware; Annapolis, Maryland; and Richmond, Virginia.

However, there is a better way.

Chat completions

That last example showed an interactive chat dialog using the text completion API. There is a purpose built chat completion API that could be used instead.

For chat interactions, this examples will assume the gpt-3.5-turbo model (which is optimized for chat interactions).

chat_model = 'gpt-3.5-turbo'

When interacting with the chat model, you provide a list of messages describing the conversation up to the current moment. Messages in that list are setup using one of three roles:

system – Used to define the role for the AI model (optional)
user – Provided by the user
assistant – Responses from the chat completion API

history=[]
while (True):
  print('Prompt: ')
  prompt = input()

  if prompt == '':
    break

  history.append({ 'role': 'user', 'content': prompt })

  chat = openai.ChatCompletion.create(model=chat_model, messages=history)
  response = chat.choices[0].message

  history.append(response)

  print(f'Answer: {response.content}')

Prompt:
What are the capitals of the west coast?

Answer: The capitals of the west coast in the United States are:

1. Sacramento, California
2. Salem, Oregon
3. Olympia, Washington

In Canada, the capital of the west coast British Columbia is Victoria.

Prompt:
And the east?

Answer: The capitals of the east coast in the United States are:

1. Augusta, Maine
2. Concord, New Hampshire
3. Boston, Massachusetts
4. Providence, Rhode Island
5. Hartford, Connecticut
6. Albany, New York
7. Trenton, New Jersey
8. Harrisburg, Pennsylvania
9. Dover, Delaware
10. Annapolis, Maryland
11. Richmond, Virginia
12. Raleigh, North Carolina
13. Columbia, South Carolina
14. Atlanta, Georgia
15. Tallahassee, Florida

In Canada, the capital of the east coast Nova Scotia is Halifax and the capital of Newfoundland and Labrador is St. John’s.

Next time – autonomous agents

In the next post, we’ll look at how autonomous agents use this sort of chat capability (and potentially other AI capabilities) to complete tasks on their own.

Chatting With A LLM – Using the Open AI APIs