Alignment & verification fragility — RLHF jailbreaks, emergent patterns, faking risks, agent monitoring, defenses [developing]
Key Questions
Why is alignment of foundation models not sufficient for aligned systems?
Defenses such as DAVinCI, ASGuard, RAGEN-2, and ARES scalable red-teaming are being developed to address these fragilities.
What is Conditional Misalignment?
Conditional Misalignment refers to hidden triggers in aligned AI that can cause misalignment under specific conditions, posing risks in deployment.
What did Meta FAIR's pretraining self-improvement research show?
Meta FAIR's work on self-improving LLMs during pretraining achieved 36% factual accuracy and 18% safety improvements, highlighting potential for iterative enhancement.
What is BARRED?
BARRED is a method for synthetic training of custom policy guardrails using asymmetric debate, aimed at improving AI safety without human intervention.
What does RewardBench 2 evaluate?
RewardBench 2 benchmarks reward models for vulnerabilities like distillation backdoors and position bias in RLHF processes.
What is Themis?
Themis trains robust multilingual code reward models for flexible multi-criteria scoring, enhancing alignment in code generation tasks.
What defenses exist against prompt injection?
Recent evaluations assess prompt injection defenses in large language models, including techniques like private inference and LLM Safety From Within.
What does the AI Alignment survey cover?
The comprehensive AI Alignment survey reviews techniques for ensuring AI systems align with human intentions and values, including multi-objective methods like MAH-DPO.
'Aligned FMs != aligned systems'; reward hacking; DAVinCI; jailbreaks; position bias; Anthropic cheat; Parcae/AARs; Mythos/RLHF; ASGuard/RAGEN-2; RewardBench 2; distillation backdoors; BARRED; ARES scalable red-teaming (inj/multi-turn/tools); Conditional Misalignment hidden triggers; Meta FAIR pretrain self-improve (36% fact/18% safety); prompt inj defenses; AI Alignment survey; private inference; LLM Safety From Within; AVERI; VLA safety; InquireMobile; MAH-DPO multi-obj; Themis multilingual code RMs; alignment survey reposts. Track repros/code.