Text Generation with OpenAI
Hello everyone and welcome back to the Text Generation with OpenAI demos. This follows lesson 4, Advanced chat completion techniques. In this video, you will create JSON sample data for unit tests.
Demo
Since you’re now more familiar with system prompts, tool use, and other advanced techniques for chat completion, you should try more use cases of this. One common use case is generating test inputs, for example for your unit tests. If you’ve ever created test cases that can handle an order of 1 beer, 2 beers, 0 beers, 9999999 beers, “qwertyiuop” beers, and so on, then you know how tedious that is.
In this demo, you’ll use chat completion to create JSON data for unit tests of your order-taking app. You’re launching your app in Slovakia. You want your app to be able to handle all cases that might consist of invalid inputs. Your order-taking app requires the details of the order:
- Full name of the person who ordered.
- Name of the order.
- Quantity of the order.
- Type of order, either in person or via delivery.
Go and start from a fresh ipynb file.
First, you need a structure of your JSON. From the requirements above, you might use a structure like this:
{
fullName: <name of person who ordered>,
itemName: <name of the item ordered>,
quantity: <number of items ordered>,
type: <pickup or delivery>
}
Although OpenAI just released a new feature that you could also use, called Structured Outputs here, you won’t use it for now.
In order to generate JSON that follows that format, you could use JSON mode from chat completion.
Then, in JupyterLab, again, make sure that you have included the API key in your environment. Then add the following code in the first cell of your notebook file:
import os
import openai
openai.api_key = os.environ["OPENAI_API_KEY"]
model = "gpt-4o-mini"
from openai import OpenAI
client = OpenAI()
Run the cell to initialize the client object.
This is the same as the starter code you used in the instruction section of this lesson. Now, use similar code to what you used to instruct the chat completion to generate this JSON.
Create a new cell and add:
# 1
SYSTEM_PROMPT = (
"You generate sample JSON data for unit tests. You must return a response in JSON format:"
"{"
" fullName: <name of person who ordered>,"
" itemName: <name of the item ordered>,"
" quantity: <number of items ordered>,"
" type: <pickup or delivery>"
"}"
)
# 2
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
]
# 3
response = client.chat.completions.create(
model=model,
messages=messages,
response_format={ "type": "json_object" }
)
# 4
print(response.choices[0].message.content)
By adding this, you are:
- Initializing the system prompt. The instruction is to generate JSON for unit tests. The format of the JSON is also included.
- Setting messages as an array. For now, it contains only the system prompt.
-
Calling chat completion in JSON mode while providing
messages. - Printing the output response.
Run the cell, and you should see a generated JSON that looks like this:
{
"fullName": "John Doe",
"itemName": "Margherita Pizza",
"quantity": 2,
"type": "delivery"
}
This looks pretty good. Notice how you haven’t used any messages with the user role yet. If you want to generate more sample inputs, for example, you could add that as a user message. You could also add it in the system prompt but imagine this code as a service that allows you or your team to generate JSON sample data.
Modify the messages array to the following:
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": "Generate 5 examples"},
]
Run the cell and check if it has generated five examples. You should get an output like here.
{
"orders": [
{
"fullName": "Alice Johnson",
"itemName": "Margherita Pizza",
"quantity": 2,
"type": "delivery"
},
{
"fullName": "Michael Smith",
"itemName": "Caesar Salad",
"quantity": 1,
"type": "pickup"
},
{
"fullName": "Emily Davis",
"itemName": "Cheeseburger",
"quantity": 3,
"type": "delivery"
},
{
"fullName": "Chris Brown",
"itemName": "Spaghetti Carbonara",
"quantity": 1,
"type": "pickup"
},
{
"fullName": "Jessica Williams",
"itemName": "Chocolate Cake",
"quantity": 2,
"type": "delivery"
}
]
}
They all look different, which is good for unit tests. However, you might want to generate inputs that would test the rare cases. Also, notice that in the output above, the response contained an extra orders key. If you want to guarantee the format of the response is a certain way, OpenAI provides Structured Outputs. You should try it in your free time.
You could decide to instruct the chat completion to generate very different JSON samples, like in the beer joke earlier. You can choose to put it in the system prompt or in user messages. But again, imagine your code will be a service, you would preferably provide the service that already has the instruction to generate variants in a certain way. Also, you can allow users to modify the output using their own instructions.
Try this first. Change the system prompt to this code:
SYSTEM_PROMPT = (
"You generate sample JSON data for unit tests."
"Generate as diverse variants as possible."
"You must return a response in JSON format:"
"{"
" fullName: <name of person who ordered>,"
" itemName: <name of the item ordered>,"
" quantity: <number of items ordered>,"
" type: <pickup or delivery>"
"}"
)
Here, you simply inserted the string "Generate as diverse variants as possible." in between the previous prompt.
Run the cell again. You should see an output that can be very similar to what you already got earlier. See here:
{
"orders": [
{
"fullName": "Alice Johnson",
"itemName": "Pepperoni Pizza",
"quantity": 2,
"type": "delivery"
},
{
"fullName": "Mark Smith",
"itemName": "Caesar Salad",
"quantity": 1,
"type": "pickup"
},
{
"fullName": "Emily Davis",
"itemName": "Spaghetti Bolognese",
"quantity": 3,
"type": "delivery"
},
{
"fullName": "Jacob Wilson",
"itemName": "BBQ Chicken Wings",
"quantity": 5,
"type": "pickup"
},
{
"fullName": "Sophia Brown",
"itemName": "Vegan Burger",
"quantity": 4,
"type": "delivery"
}
]
}
Notice that the quantity could be zero, negative, string, or something completely unexpected. Since you already tried setting the prompt to generate diverse variants, try to make it clearer now.
SYSTEM_PROMPT = (
"You generate sample JSON data for unit tests."
"Generate as diverse variants as possible."
# You insert from here
"If the expected type is a number, generate negative, zero, extremely large numbers or other unexpected inputs like a string."
"If the expected type is an enum, generate non-enum values."
"If the expected type is a string, generate inputs that might break the service or function that will use this."
# You end insert to here
"You must return a response in JSON format:"
"{"
" fullName: <name of person who ordered>,"
" itemName: <name of the item ordered>,"
" quantity: <number of items ordered>,"
" type: <pickup or delivery>"
"}"
)
Here, you added the code surrounded by the comments above to the system prompt.
Run the cell, and you should see a similar output to here:
{
"testCases": [
{
"fullName": "John Doe",
"itemName": "Pizza",
"quantity": 1,
"type": "pickup"
},
{
"fullName": "Jane Smith",
"itemName": "Burger",
"quantity": -5,
"type": "delivery"
},
{
"fullName": "Alice Johnson",
"itemName": "Sushi",
"quantity": 0,
"type": "pick-up"
},
{
"fullName": "Bob Brown",
"itemName": "Salad",
"quantity": 1000000,
"type": "delivery"
},
{
"fullName": "Charlie Black",
"itemName": "Tacos",
"quantity": "two",
"type": "unknown"
}
]
}
Notice that now the quantity has negative, zero, positive, large numbers and string numbers. That’s great! Being very explicit about what you expect as an output helps generate results, just like when you try to spell out your name when you order in coffee shops. :]
Look at the type, which has misspellings and unknown value, too, now, including the two valid values. However, both the fullName and itemName values above didn’t generate very different results.
The user of this service might want to modify the output itself. You would want to allow them to submit follow-up changes to the generated JSON.
You might have seen earlier when you were parsing the tool_calls object, that you can add to the messages array and call the chat completion again. This is the same technique that’s used when users have follow up inquiries to previous outputs.
Add a new cell and into it, the following code:
# 1
user_follow_up = "Use names that come from Slovakia"
# 2
messages.append({"role": "user", "content": user_follow_up})
# 3
response = client.chat.completions.create(
model=model,
messages=messages,
response_format={ "type": "json_object" }
)
# 4
print(response.choices[0].message.content)
You’re doing the following:
- Initializing the follow-up message from the user. Here, they want the names to be of Slovakian origin.
-
Appending the follow-up to the
messagesarray. -
Calling the chat completion in JSON mode once more, but this time with more entries in the
messagesarray. - Printing the response.
Run the cell, and you should get a similar output to here:
{
"orders": [
{
"fullName": "Ján Novák",
"itemName": "Bryndzové Halušky",
"quantity": 3,
"type": "pickup"
},
{
"fullName": "Tatiana Horváthová",
"itemName": "Kapustnica",
"quantity": -5,
"type": "delivery"
},
{
"fullName": "Marek Kováč",
"itemName": "Treska",
"quantity": 0,
"type": "pickup"
},
{
"fullName": "Lucia Sliacka",
"itemName": "Pirohy",
"quantity": 9999999,
"type": "gobble"
},
{
"fullName": "Peter Hrušovský",
"itemName": "<script>alert('Hello')</script>",
"quantity": "two",
"type": "delivery"
}
]
}
This meets the requirement of the user to generate names and even item names from Slovakia. Notice how one item’s name looks like code. It looks like script injection. This is very helpful for unit testing or even security testing your code.
Hope that stirs your imagination on how you can use chat completion.