Multimodal Integration with OpenAI

This lesson explores the capabilities and applications of DALL-E for image variations and editing using the OpenAI API. You will learn how to create variations of images, and edit existing images. This lesson also covers the integration of text and image generation in a single application.

To create variations of an image, write and run the following code:

# Create variations of an image with DALL-E 2

# Define the path to the logo image
logo_path = "images/kodeco.png"

# Open the image
with open(logo_path, "rb") as f:
    # Call the API to create variations
    response = client.images.create_variation(
      model="dall-e-2",
      image=f,
      n=4,
      size="512x512"
    )

# Display images in a grid
image_urls = [img.url for img in response.data]
display_images_in_grid(image_urls)

You use the client.images.create_variation function to create variations of an image. The image parameter should be the file object of the original image.

The display_images_in_grid function was defined in the previous demo to display images in a grid.

Editing Images with DALL-E API

To edit an image using DALL-E 2, you first need to prepare the original image and a mask image with transparent areas. First, checkout the image you want to edit:

# Display the original image

# Image path
cat_ceo_image_path = "images/cat_ceo.png"

# Open the image
img = Image.open(cat_ceo_image_path)

# Display the image
plt.imshow(img)
plt.axis('off')
plt.show()

As mentioned in the instruction segment, you need to create a mask image with transparent areas indicating the parts of the image you want to edit. Checkout add_transparency_in_gimp video in Materials section to learn how to create a mask image in GIMP.

Now, you can edit the image using the mask image:

# Edit an image using DALL-E 2

# Define the paths to the original image and mask image
original_image_path = "images/cat_ceo.png"
mask_image_path = "images/cat_ceo_mask.png"

# Define the prompt for the edit
image_prompt = "Show a dog CEO."

# Call the API to edit the image
with open(original_image_path, "rb") as image_file, open(mask_image_path,
  "rb") as mask_file:
    edit_response = client.images.edit(
        image=image_file,
        mask=mask_file,
        prompt=image_prompt,
        n=1,
        size="1024x1024"
    )

Next, download and display the edited image:

# Download and display the edited image

# Retrieve the image URL
image_url = edit_response.data[0].url

# Download the image
response = requests.get(image_url)

# Create an image object
img = Image.open(BytesIO(response.content))

# Display the image
plt.imshow(img)
plt.axis('off')
plt.show()

Here, you use the client.images.edit function to edit an image. Provide the original image and mask image file objects along with the prompt for the desired edit. Now, you’ve successfully edited an image using the DALL-E API and displayed a dog CEO!

So far, you have learned how to use both text generation and image generation capabilities of the OpenAI API. Now, combine these two features to create a recipe generator that provides a recipe description along with an image of the food. Add the following code:

# Combine text and image generation

# Function to generate a recipe with an image of the food
def generate_recipe(food: str) -> str:

    # Generate ingredients
    completion = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "You're an expert in culinary
              and cooking."},
            {
                "role": "user",
                "content": f"Provide recipe of {food}."
            }
        ]
    )

    # Extract the recipe description
    recipe_description = completion.choices[0].message.content

    # Prompt for DALL-E image generation
    dalle_prompt = f"a hyper-realistic image of {food}"
    # DALL-E model and image size
    dalle_model = "dall-e-3"
    image_size = "1792x1024"

    # Image Generation
    response = client.images.generate(
      model=dalle_model,
      prompt=dalle_prompt,
      size=image_size,
      n=1,
    )

    # Retrieve the image URL
    image_url = response.data[0].url

    # Download the image
    response = requests.get(image_url)

    # Open the image
    img = Image.open(BytesIO(response.content))

    # Displaying the image
    plt.imshow(img)
    plt.axis('off')
    plt.show()

    # You can also save the image if you want

    # Return the recipe description
    return recipe_description

• First, you use the client.chat.completions.create function with the model and prompt to generate a recipe description for a given food.
• Next, for image generation, you use the client.images.generate function with the DALL-E model, prompt, and size parameters.
• Then, you retrieve the image URL and display the image.
• Finally, you return the recipe description.

Now, generate a recipe for Chicken Tikka Masala and display the result:

# Generate a recipe for Chicken Tikka Masala
chicken_tikka_masala_recipe = generate_recipe("Chicken Tikka Masala")
print(chicken_tikka_masala_recipe)

Test the application with another dish, such as Spaghetti Bolognese:

# Generate a recipe for Spaghetti Bolognese
spaghetti_bolognese_recipe = generate_recipe("Spaghetti Bolognese")
print(spaghetti_bolognese_recipe)