Understanding Image Moderation Using Azure Content Safety
Image Moderation Overview
Similar to text moderation, image moderation focuses on reviewing images across various platforms, ensuring they are not harmful and don’t violate platform guidelines.
For any platforms which allow users to upload images, the moderation system should analyze the uploaded content. If an image is found to be harmful in any way (like containing disturbing images of violence, blood, etc.), or it contains something outside the scope of what’s allowed on the platform, the image can be flagged - or even blocked.
On the other hand, if a service enables AI-generated user images, they’ll need a moderation system in place to ensure any generated images are within the bounds of what is acceptable, and that the AI doesn’t generate anything inappropriate or controversial. If this happens, the generated image will be discarded instead of being shared with the end user.
Importance of Image Moderation
In today’s digital world, images are equally as important as text forms of data. Images now play a crucial role in communication, marketing, and user engagement - so any failure to properly moderate user-posted and generative images can have severe consequences for your platform and, in extreme situations, for society.
Consider a scenario where your platform offers generative AI features. If the platform generates a racially inappropriate image and shares it with the user, it could swiftly offend large segments of society. This could lead to a sudden loss of trust and respect for your platform, damaging its reputation.
The same concern applies to user-generated content. The only difference in this case is that the user uploads the image rather than it being generated by AI. Any impacted people may perceive your platform as unsafe, or incapable of maintaining a secure environment. This again could cause reputational damage and ultimately make users leave the platform.
Bearing this in mind, it’s crucial for platforms to integrate robust image moderation systems into their infrastructure. These systems should be capable of performing effectively and consistently — key areas where image moderation is vital include social media platforms, e-commerce platforms, gaming, and virtual worlds.
Understanding Image Moderation services Offered by Azure Safety Content
Azure Content Safety offers AI-enabled image moderation solutions, allowing you to detect inappropriate images in real-time, and can scale itself to handle large amounts of requests if required.
Key features of its image moderation solution include:
- Multi-category Classification: Similar to the Azure text moderation solution, image moderation also has four distinct harm categories - Hate, Sexual, Violence, and Self-harm. The classification model supports multi-labeling, meaning an image can be flagged for multiple categories.
- Customization Thresholds: The moderation system assigns a severity rating to every harm category. The severity level is meant to indicate the severity of the image’s potential harm. A higher severity score means that the content is more harmful. Unlike text moderation, image moderation only has trimmed version of severity scale, which are 0, 2, 4 and 6.
- Creating Custom Categories: Similar to the text moderation solution, you can create custom categories by training your moderation AI models to identify these categories on your own data.
You can learn more about the content safety categories and their severity levels for image content on the Harm categories in the Azure AI Content Safety page. This section provides a comprehensive overview of the default categories offered by the Azure AI Content Safety team. It also helps you understand how to interpret the severity levels for each category - helping you decide the appropriate severity level threshold that suits your needs.
In the next section, you’ll use Content Safety Studio to customize and test your image moderation API.