Understanding Azure Content Safety Text Moderation API

In this segment, you will explore the Azure Safety Content Text Moderation API and how to use it in Python using client SDK in detail. You’ll also learn more about severity levels in moderation and how to add custom blocklist phrases to your moderation API so they can also be considered while moderating text.

Understanding Azure Safety Content Text Moderation API

For text moderation, Azure Safety Content provides two types of API:

contentsafety/text:analyze: This is a synchronous API for analyzing potentially harmful text content. At the moment of creating this module, it supports four categories: Hate, Self-Harm, Sexual, and Violence.
contentsafety/text/blocklists/*: These are also a set of synchronous APIs that allow you to create, update, delete blocklist terms that can be used with text API. Usually, the default AI classifiers are sufficient for most content safety needs, but if you need to screen some terms specific to your use case, you can make use of it as well.

Putting the focus back on analyzing text Rest API — it takes five input parameters in the request body:

text: This is the primary parameter and consists of the text you want to analyze. A single request can process up to 10k characters per request. The longer text needs to be split into multiple requests.
blocklistNames: This is an optional parameter. Using this parameter, you can also supply a list of blocklist names that you defined.
categories: If you want to analyze your text on specific categories, you can provide those categories in a list format. This is also optional, and if it is not assigned, a default set of analysis results for the categories will be returned.
haltOnBlocklistHit: This is an optional parameter. When set to true, further analyses of harmful content will be stopped in cases where blocklists are hit and response is shared back, or else, it will complete the analyses even if the blocklists are hit.
outputType: This is an optional parameter. This allows you to define the granularity of the severity scale. By default, its value will be FourSeverityLevels, that is, output analyses will contain severity in 4 levels - 0,2,4,6. If instead, the EightSeverityLevels value is provided, the output analyses will contain severity in 8 levels - 0,1,2,3,4,5,6,7.

A sample request body for text analyses can look something like this:

{
  "text": "A sample example",
  "blocklistNames": ["block_terms"],
  "categories": ["Hate", "SelfHarm", "Violence"],
  "haltOnBlocklistHit": false,
  "outputType": "EightSeverityLevels"
}

The response body for a successful API call has the following parameters:

blocklistsMatch: A list of json data containing information about block matches if found any. {FPE: Not sure what “if found any” is - maybe “if any are found”?}
categoriesAnalysis: This is again a list of TextCategoriesAnalysis JSON that contains the categories (like Hate, SelfHarm, etc.) and their severity level as determined by the moderation API.

A sample response body for a successful API call can look something like this:

{
  "blocklistsMatch": [],
  "categoriesAnalysis": [
    {
      "category": "Hate",
      "severity": 0
    },
    {
      "category": "SelfHarm",
      "severity": 3
    },
    {
      "category": "Sexual",
      "severity": 0
    },
    {
      "category": "Violence",
      "severity": 2
    }
  ]
}

While the API can be called directly, thankfully, Azure also provides SDKs for several languages (Python, JavaScript, Java, .NET) to simplify integration for devs.

Instead of making raw HTTP calls, you’ll use the Azure AI Content Safety client library for Python in this module. You can learn more about the API at Text Operations - Analyze Text and Text Blocklists.

Understanding the Azure AI Content Safety Client Python Library

The first step to using a content safety client is to create an instance of it. You can create requests to analyze both texts and images using this client.

A sample code to create the safety client will look like this:

from azure.core.credentials import AzureKeyCredential
from azure.ai.contentsafety import ContentSafetyClient

# Create an Azure AI Content Safety client
endpoint = "https://<my-custom-subdomain>.cognitiveservices.azure.com/"
credential = AzureKeyCredential("<api_key>")
content_safety_client = ContentSafetyClient(endpoint, credential)

To create ContentSafetyClient, you need two objects:

endpoint: This is the endpoint, where the analysis request will be made.
credential: You provide API keys used for authenticating your request. This is of type AzureKeyCredential.

Once you’ve created the client, you can then use it to create requests to analyze text content:

# Construct request
request = AnalyzeTextOptions(text="Your input text")

# Analyze text
response = client.analyze_text(request)

In the code above, you’re passing your request to the client using AnalyzeTextOptions object.

Understanding AnalyzeTextOptions

AnalyzeTextOptions object is used to construct the request for text analyses. It also allows you to customize text analysis requests to suit your specific needs. It has the following properties:

text (required): This holds the texts that need to be analyzed. The text size should not exceed 10k characters. In case you have longer text, you must split the text and make separate calls for each chunk.
categories (optional): You can use this property to specify specific categories for which you want to analyze your harmful content. If not specified, the moderator API should analyze content for all categories. It accepts a list of TextCategory. At the moment of writing this module, the possible values include - TextCategory.HATE, TextCategory.SEXUAL, TextCategory.VIOLENCE, and TextCategory.SELF_HARM.
blocklist_names (optional): You can provide the names of blocklists you created to block specific terms and phrases for the use case. It accepts the blocklists as a list of strings.
halt_on_blocklist_hit (optional): Similar to Rest API’s halt_on_blocklist_hit. When set to true, it halts the further analyses of text in cases where blocklists are hit.
output_type (optional): This allows you to define the granularity of severity scale. If no value is assigned, the default value will be "FourSeverityLevels". It can either take value as string or object of type AnalyzeTextOutputType. At the moment of writing this module, the possible value of AnalyzeTextOutputType include AnalyzeTextOutputType.EIGHT_SEVERITY_LEVELS, AnalyzeTextOutputType.FOUR_SEVERITY_LEVELS.

A sample AnalyzeTextOption definition can look like this:

# Create AnalyzeTextOptions
analyze_text_request = AnalyzeTextOptions(
    text="This is the text to analyze.",
    categories=[TextCategory.HATE, TextCategory.VIOLENCE],
    blocklist_names=["block_list"],
    halt_on_block_list_match=True,
    output_type=AnalyzeTextOutputType.EIGHT_SEVERITY_LEVELS
)

Processing Analyses Response

Once the analyses of text content is finished, you can use the response received from the method client.analyze_text to decide whether to approve the content or block it.

analyze_text has a return type of AnalyzeTextResult. Since it’s a JSON response converted into object, the class has the following properties:

blocklists_match: It holds value of type list[TextBlocklistMatch], where TextBlocklistMatch allows you access to the following values:
- blocklist_name: Name of the blocklist that was detected.
- blocklist_item_id: Id of the matched term within the blocklist.
- blocklist_item_text: Term that is detected in the requested text content.
categories_analysis: Holds value of type list[TextCategoriesAnalysis], where TextCategoriesAnalysis allows access to the following values:
- category: Category name for which moderation API has analyzed the text.
- severity: Severity level provided by the moderation API for the above category.

You can handle process the AnalyzeTextResult response in the following way:

# 1. Analyze image
try:
    response = client.analyze_image(request)
except HttpResponseError as e:
    print("Analyze image failed.")
    if e.error:
        print(f"Error code: {e.error.code}")
        print(f"Error message: {e.error.message}")
        raise
    print(e)
    raise

# 2. extract result for each category
hate_result = next(item for item in response.categories_analysis if 
  item.category == TextCategory.HATE)
self_harm_result = next(item for item in response.categories_analysis if 
  item.category == TextCategory.SELF_HARM)
sexual_result = next(item for item in response.categories_analysis if 
  item.category == TextCategory.SEXUAL)
violence_result = next(item for item in response.categories_analysis if 
  item.category == TextCategory.VIOLENCE)

# 3. print the found harmful category in the text content
if hate_result:
    print(f"Hate severity: {hate_result.severity}")
if self_harm_result:
    print(f"SelfHarm severity: {self_harm_result.severity}")
if sexual_result:
    print(f"Sexual severity: {sexual_result.severity}")
if violence_result:
    print(f"Violence severity: {violence_result.severity}")

Although the code might look tricky and complicated, it’s simple. Here’s the break down:

Send the analyze request and store the result in the response variable. If any exception occurs while performing the analysis, catch the error using the try-except block, print the error, and rethrow the exception to the parent function that called this code.
Using the next function, we iterate through the response.categories_analysis to extract TextCategoriesAnalysis for every individual harmful category.
Suppose any of the following category result lists is non-empty, then print the corresponding category name with its severity level.

Add custom blocklist phrases

If required, you can further customize the text moderation API results to detect blocklist terms that meet your platform needs. You’ll first need to add the blocklist terms to your moderation resource. Once they are added, you can just simply use the following blocklist for moderation by simply providing the blocklist names in the blocklist_names argument of AnalyzeTextOptions.

To add a blocklist, you’ll have to first create a block list client similar to a content safety client:

from azure.core.credentials import AzureKeyCredential
from azure.ai.contentsafety import BlocklistClient

# Create an Azure AI blocklist client
endpoint = "https://<my-custom-subdomain>.cognitiveservices.azure.com/"
credential = AzureKeyCredential("<api_key>")
client = BlocklistClient(endpoint, credential)

Next, to add the block list you can use the following code:

# 1. define blocklist name and description
blocklist_name = "TestBlocklist"
blocklist_description = "Test blocklist management."

# 2. call create_or_update_text_blocklist to create the block list
blocklist = client.create_or_update_text_blocklist(
    blocklist_name=blocklist_name,
    options=TextBlocklist(blocklist_name=blocklist_name, 
      description=blocklist_description),
)

# 3. if block list created successfully notify the user using print function
if blocklist:
    print("\nBlocklist created or updated: ")
    print(f"Name: {blocklist.blocklist_name}, Description: {blocklist.description}")

Then, you will also like to add some terms and phrases to your blocklist that so that they can be used to mark text content inappropriate during text moderation if the following terms and phrases were found:

# 1. define the variable containing blocklist_name and block items
#    (terms that needs screened in text)
blocklist_name = "TestBlocklist"
block_item_text_1 = "k*ll"
block_item_text_2 = "h*te"

# 2. create the block item list that can be passed to client
block_items = [TextBlocklistItem(text=block_item_text_1), 
  TextBlocklistItem(text=block_item_text_2)]

# 3. add the block item list inside the blocklist_name using the 
#    function AddOrUpdateTextBlocklistItemsOptions
try:
    result = client.add_or_update_blocklist_items(
        blocklist_name=blocklist_name, options=AddOrUpdateTextBlocklistItemsOptions(
          blocklist_items=block_items)
    )
    # 4. print the response received by the server on successful addition
    for block_item in result.blocklist_items:
        print(
            f"BlockItemId: {block_item.blocklist_item_id}, Text: {block_item.text}, 
              Description: {block_item.description}"
        )
# 5. Catch exception and notify the user if any error happened during
#    adding the block terms
except HttpResponseError as e:
    print("\nAdd block items failed: ")
    if e.error:
        print(f"Error code: {e.error.code}")
        print(f"Error message: {e.error.message}")
        raise
    print(e)
    raise

You can learn more about adding block lists and other text blocklist management APIs at Manage text blocklist.

Lesson 1: Introduction to Azure Content Safety

Lesson 2: Text Moderation With Azure Content Safety

Lesson 3: Image Moderation Using Azure Content Safety

Lesson 4: Advanced Content Moderation Strategies

Understanding Azure Content Safety Text Moderation API

Understanding Azure Safety Content Text Moderation API

Understanding the Azure AI Content Safety Client Python Library

Understanding AnalyzeTextOptions

Processing Analyses Response

Add custom blocklist phrases

All videos. All books.
One low price.

Understanding Azure Safety Content Text Moderation API

Understanding the Azure AI Content Safety Client Python Library

Understanding AnalyzeTextOptions

Processing Analyses Response

Add custom blocklist phrases

Sign up/Sign in

All videos. All books. One low price.

All videos. All books.
One low price.