Azure Content Safety

In this segment, you’ll get hands-on experience on image moderation in content safety studio. You’ll also learn how to customize the image moderation response in the studio.

Explore Image Moderation in Content Safety Studio

Start by opening Content Safety Studio in the browser. Next, click on the Moderate image content card to navigate to the page.

You’ll notice that most of the parts of the pages are similar to the text moderation content pages; so you won’t have much trouble navigating! But for the sake of completeness, here is a quick overview:

• Introduction: This section provides a brief overview of the page. You can also find the link for documentation and sample code for using the Image Moderation API in this section.
• Try it out: This section contains the acknowledgment and sub-sections you’ll cover soon.
• Select a sample or upload your own: This section provides sample images you can use to test Image Moderation API and analyze the results.
• Test: This is the main section where actual tests are performed. On the left side, it contains an image preview area where the selected images that you want to analyze will be shown. On the right side, it includes options to configure filters to customize the API.
• View results: Your test results will appear here once the image is analyzed.
• Next steps: Finally, this section reveals the next steps that you might like to try.

Similar to the Moderate text content page, Moderate image content also offers two types of tests:

• Run tests on a single image.
• Run tests on many images.

Testing Simple Image for Moderation

Using Run tests on a single image feature, you can test whether a single image is safe or unsafe. In the “Select a sample or upload your own” section, select any unsafe image that you would like to run through the moderation API. You’ll see the selected image displayed in the preview.

Now, let’s understand and configure the filters for moderation:

• Enable/disable the category: You can turn the harm category on/off for whichever image you want to analyze. For testing purposes, deselect the Hate category for the selected image.
• Update the Category Sensitivity: Next, update the threshold level for each category to modify the sensitivity permitted while analyzing the image. Setting the threshold to “Low” will flag even slightly harmful content as inappropriate. On the other hand, setting the threshold to “High” will only block hazardous content. For testing purposes, change the Violence category threshold to low.

Finally, click the Run test button to analyze the selected image.

The moderation results show that the selected image is blocked because the moderation API detected elements of Violence and Self-harm. Feel free to try other sample images from the options provided above to better understand the moderation results for different types of images.

Testing Bulk Test for Moderation

Next, you’ll use the Run a bulk test feature, which is located next to the Run a simple test tab. This feature is particularly useful when you want to test the moderation API on multiple images in a single request. It allows you to assess how your API will perform on a variety of images, which can be beneficial when you’re dealing with a large dataset, or when you want to compare the performance of different filter and configurations.

This feature is helpful for quickly testing whether your adjusted filters can effectively moderate content based on your needs.

You’ll use the provided sample data for this demo, but you can also test your image dataset. You’ll need to share the images and their labels in zip format to do so. Click on How to create and format a file for more details.

Now, back to the testing. Select Dataset with AI-Generated Content card, which contains a set of AI-generated samples. These images are useful for testing the performance of your API on a variety of content types.

Next, in the filter configuration, re-enable the Hate category and set its threshold level to Low. Finally, click the Run test to start the analyses for the images and wait for it to finish the analyses.

Select Dataset with AI-Generated Content card to select the sample images on which you’ll perform the test. Next, in the filter configuration, re-enable the “Hate” category and set its threshold level to Low. Finally, click the Run test to start the analyses for the images and wait for it to finish.

The results are broken into four parts. Let’s understand them one by one.

First, you have the percentage of images allowed and blocked by the moderation API. In this case, 42.4% are allowed, and the remaining 57.6% are blocked.

Next, you have precision, recall, and F1 metrics:

• Precision: The precision value for the performed bulk analysis is 0.20 (or 20 percent). This means of all the harmful content that was flagged, only 20% was harmful, and the remaining safe.
• Recall: Next up, you have the recall value. This is 0.40 (or 40 percent) for the performed bulk image test, which means the moderation system could only identify 40% of harmful content, and the remaining 60% of images were marked as safe.
• F1 Score: Finally, you have the F1 Score, which is the harmonic mean of Precision and Recall. It is 0.30 in this case. The higher the F1 score, the better the moderation system will perform.

At the bottom of all these metrices, you have Severity details per record. This provides a detailed view of image analysis results for each image and where the judgment shared by the model was wrong concerning what was expected, based on the labels.

Finally, you have Severity distributed by category, which provides information about the severity distribution for each category in the dataset. Here are a few notable observations from the graphs:

• Only one high-severity level image has been identified for the violence category.
• The provided sample database contains no harmful content in the Hate and Self-harm categories.

To improve the moderation results, change the Violence category threshold to Medium and the Sexual category threshold to Low. Now, re-run the analysis by clicking on the Run test button.

This time, the moderation results looks much better. Great job :]

In the next segment, you’ll learn about the content moderation API for image moderation and its implementation in Python, so that you can integrate Azure’s solution into your platform.