Azure Content Safety

In this segment, you’ll make your moderation solution more robust by adding human-in-the-loop functionality.

Add Human-in-the-loop

To implement human-in-the-loop functionality, you will first update analyze_text and analyze_image function code to address the cases where the current way of classifying content as safe or unsafe for a particular category may not be correct at times.

Head to starter/text_analysis.py and replace the conditions below # Check for inappropriate content with the following:

violations = {}
if hate_result and hate_result.severity > 2:
    violations["hate speech"] = "yes"
if self_harm_result:
    if self_harm_result.severity > 4:
        violations["self-harm"] = "yes"
    elif self_harm_result.severity > 3:
        violations["self-harm"] = "likely"
if sexual_result:
    if sexual_result.severity > 1:
        violations["sexual"] = "yes"
    elif sexual_result.severity > 2:
        violations["sexual"] = "likely"
if violence_result and violence_result.severity > 2:
    violations["violent references"] = "yes"

In this updated code, you introduced a few more checks to capture the scenarios where the possibility of a harmful category found is "likely" but not guaranteed. For example, the self-harm check now has a new condition where the severity level is between 3 and 4. Then, there might be a possibility that the text content may be harmful and should not be allowed to be published. The same is the case with the sexual check.

For the sake of learning and demo purposes, you’ll only update and introduce the likely case for text content, and not for image content. In the actual production-ready system, you might have to implement such checks for other content types as well.

Update check_content_safety Function

Next, head back to starter/business_logic.py and update code below the if statement that checks for any form of violation is found in the moderation results of text and image analysis:

# 1
text_violation_flags = text_analysis_result.values()
image_violation_flags = image_analysis_result.values()

# 2
if "likely" in text_violation_flags or "likely" in image_violation_flags:
    return {'status': "re-evaluation needed"}

# 3
status_detail = f'Your post contains references that violate our
  community guidelines.'

if text_analysis_result:
    status_detail = status_detail + '\n' + f'Violation found in text: {','
      .join(text_analysis_result)}'
if image_analysis_result:
    status_detail = status_detail + '\n' + f'Violation found in image: {','
      .join(image_analysis_result)}'

status_detail = status_detail + '\n' + 'Please modify your post to adhere to
  community guidelines.'

# 4
return {'status': "violations found", 'details': status_detail}

In this code:

1. You’re collecting the values of text_analysis_result and image_analysis_result dictionaries into text_violation_flags and image_violation_flags, respectively. Recall that these dictionaries contain the value that provides extra information about the harmful category, such as whether the detected harm category is likely intolerable or certainly intolerable.
2. Here, you check if text_violation_flags and image_violation_flags list contains "likely" value. If yes, then the moderation system feels that a human evaluator should also review the content. Hence, the function returns with the "re-evaluation needed" response, instead of approving or rejecting the publishing request. Typically, you’ll write additional logic to send and notify a human reviewer to look at and evaluate the content. In parallel, the user is asked to wait sometime for the content to be evaluated by humans.
3. The rest of the below code stays the same. You define a new variable status_detail and append the harmful category to the string in human-readable format when detected, so that the user can be informed and requested to update the post to adhere to community guidelines.
4. Finally, you return the result of the safety check so that the user can be informed about the violation found in the content and request them to update the content to address the shared concerns.

Re-run the App to Test Moderation System

Open the terminal in the VSCode and re-run the web app using the following command:

streamlit run app.py

This time, add text that is potentially harmful but will need further evaluation. Add an image as well, and try to publish the content by clicking the Submit button.

You’ll get the following warning: "Possible community guidelines violation found. Your post has been sent to our team for evaluation!".

Next, try to add both image and text content that are potentially harmful and try to submit the publish request once again.

Appeal the Platform for Re-evaluation

This time you get the following message - “Post Upload Failed”. At the bottom of the message, you can also find an option to submit an appeal request. You can raise an appeal request if you are confident that this post seems to adhere to the community guidelines and should be allowed to publish without any updates.

In scenarios like these, it’s always a good call to provide an Appeal option to the platform, for any users that think their content was rejected or flagged by mistake and should be allowed. This request should be then sent to the human evaluators, who will evaluate and make the final call to either enable the content, or provide a response back to the users explaining to them why the content isn’t a good fit for publication.

That’s it for this demo. Please continue toward the next segment to conclude the lesson!