Safety classifiers employ several techniques to flag harmful content, ensuring that AI systems, especially Large Language Models, can identify and mitigate unsafe or offensive material. Here are some of the specific techniques used:
1. Machine Learning Models: Safety classifiers often utilize machine learning models like DistilBERT, which can be fine-tuned on datasets containing examples of safe and harmful prompts. These models learn to classify input prompts as safe or harmful based on patterns in the training data[3].
2. Adversarial Prompt Shield (APS): APS models are trained on diverse datasets such as Wikipedia Toxic Comments (WTC), Build-It Break-It Fix-It (BBF), Bot-Adversarial Dialogue (BAD), Anthropic Red-Team Attempts (Red), and Bot-Adversarial-Noisy-Dialogue (BAND) to enhance robustness against adversarial prompts. These models help in identifying and mitigating harmful content by being exposed to a wide range of potential attacks[1].
3. Configurable Content Filters: These filters assess content based on predefined harm categories such as hate speech, harassment, sexually explicit content, and dangerous content. They assign scores for both the probability and severity of the content being harmful, allowing for customizable thresholds for blocking content[2].
4. Few-Shot Learning: This technique allows AI systems to adapt quickly to new types of harmful content by leveraging a general understanding of topics and learning from minimal labeled examples. It enables the system to respond to evolving forms of harmful content more efficiently[5].
5. Unified Datasets for Sensitive Content: Researchers create unified datasets that cover a broad range of sensitive categories, including conflictual language, profanity, sexually explicit material, drug-related content, self-harm, and spam. These datasets help in training models that can detect multiple types of harmful content under a single framework[4].
6. Severity and Probability Scores: AI classifiers use both probability and severity scores to evaluate the potential harm of content. The probability score reflects how likely the content is to be harmful, while the severity score indicates the magnitude of harm. These scores are often discretized into levels such as negligible, low, medium, and high[2][6].
7. Post-processing for Fairness: Techniques like fairness-aware post-processing are used to debias safety classifiers. This is crucial because classifiers trained on imbalanced data can learn societal biases, leading to unfair outcomes. Debiasing methods help ensure that the classifiers are more equitable in their assessments[8].
Citations:
[1] https://aclanthology.org/2024.woah-1.12.pdf
[2] https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/configure-safety-filters
[3] https://github.com/aounon/certified-llm-safety
[4] https://arxiv.org/html/2411.19832v2
[5] https://about.fb.com/news/2021/12/metas-new-ai-system-tackles-harmful-content/
[6] https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/harm-categories
[7] https://safety.google/content-safety/
[8] https://arxiv.org/html/2409.13705v2