Security of AI models and agentic systems, identity/agent risks, and AI-assisted offense/defense workflows

The accelerating integration of AI agents, Large Language Models (LLMs), and autonomous non-human identities (NHIs) into enterprise and fintech ecosystems continues to profoundly reshape the cybersecurity landscape. Recent developments have not only expanded the threat surface but also underscored an urgent need for advanced defensive frameworks tailored to the agentic AI era. As AI-driven identities gain ever more privileged access and agency, organizations face increasingly sophisticated risks — from privilege creep and consent abuse to AI-powered attack bots weaponizing code and API surfaces. Simultaneously, defenders are innovating with AI-aware identity governance, runtime attestation, and automated remediation orchestration to stay ahead.

---

Expanding Threat Surface: Agentic AI, NHIs, and Weaponized AI Code

Recent incidents dramatically highlight how agentic AI and NHIs now operate as both targets and vectors within complex attack chains:

• AI-Powered Exploitation of CI/CD Pipelines: The emergence of hackerbot-claw, an AI-driven bot actively scanning and exploiting vulnerabilities in GitHub Actions workflows, represents a new breed of automated attacker. This bot has successfully targeted high-profile projects across Microsoft, DataDog, and CNCF ecosystems, exploiting misconfigurations and injecting malicious payloads that could compromise build and deployment pipelines. This evolution of AI-powered offensive tooling emphasizes the criticality of securing CI/CD environments, particularly those integrated with AI-assisted development workflows.

• Weaponization of LLM-Generated Code in Real-World Attacks: The recent Mexican government cyberattack leveraged malicious code snippets generated by Anthropic’s Claude LLM, weaponized by threat actors to bypass conventional defenses and execute data exfiltration. This incident, detailed in Gambit’s report, marks one of the first confirmed cases of adversaries embedding AI-generated code directly into operational attacks, amplifying the challenge of detecting novel, polymorphic malicious payloads that blend human and AI code artifacts.

• RICO Demo: AI-Powered API Security Scanner: On the defensive front, tools like the RICO platform showcase AI’s potential to secure expansive API surfaces by dynamically detecting OpenAPI vulnerabilities and protecting CI/CD pipelines from misconfigurations and injection attacks. This represents a critical step forward in automated API security, crucial given the explosion of AI-enabled microservices and API integrations in fintech environments.

---

Key Risks Amplified by AI Agents and Hybrid Identity Models

The evolving AI ecosystem introduces nuanced identity and control-plane risks beyond traditional IAM frameworks:

• Privilege Creep and Consent Abuse in NHIs: With NHIs such as AI agents, orchestration bots, and pipeline service accounts proliferating, unchecked privilege accumulation is a mounting concern. These identities often span multiple systems and cloud environments, increasing the risk of consent abuse, where delegated permissions are exploited without timely revocation. Without continuous lifecycle governance, these NHIs become prime targets for lateral movement and privilege escalation.

• Hybrid Identity and Control-Plane Vulnerabilities: AI agents’ cross-platform interactions create complex hybrid sync challenges, leading to subtle control-plane gaps. Attackers exploit these by manipulating token grants or abusing consent flows, as highlighted in recent analyses of consent abuse vulnerabilities. This hybrid complexity demands novel detection capabilities that understand multi-identity workflows.

• LLM Endpoint and API Key Exposure: The Anthropic Claude jailbreak incident, which exposed over 150GB of sensitive government and financial data, revealed critical lapses in API key management, runtime isolation, and network segmentation of LLM endpoints. This breach underscores the imperative for fintech organizations to deploy OAuth-based authentication, enforce rate limiting, and implement continuous runtime attestation specifically designed for AI inference services to prevent remote compromise and data leakage.

• AI-Augmented Identity Attacks and Synthetic Fraud: Generative AI has elevated the sophistication of credential stuffing and synthetic identity fraud. The ShinyHunters group’s compromise of 5 million+ PayPal accounts exemplifies how AI enables crafting highly convincing credential attacks that evade traditional Multi-Factor Authentication (MFA) defenses. Defenders increasingly rely on behavioral anomaly detection and adaptive risk scoring to detect these nuanced threats.

• Agentic Delegation Risks and Autonomous Decision-Making: Autonomous AI agents operating under delegated authority can inadvertently cause harm if runtime attestation and control frameworks are insufficient. The video “When Delegation Goes Wrong” highlights scenarios where unmonitored AI agents execute unauthorized actions, leading to data leakage or operational disruption—a cautionary tale emphasizing the need for strict runtime governance.

---

Defensive Patterns and Tooling Advances in the Agentic AI Era

Organizations are evolving their security posture with AI-aware frameworks and tooling designed to mitigate emerging risks:

• AI-Aware Identity Governance and CIEM: The Cloud Infrastructure Entitlement Management (CIEM) market continues to mature, offering granular visibility and behavioral analytics across human and non-human identities. Platforms like Veza’s AI Access Agents automate continuous privilege restriction and lifecycle management of NHIs, enforcing least privilege principles critical for fintech cloud security.

• Runtime Attestation and Model Integrity Verification: Open-source initiatives such as InferShield provide continuous real-time verification of AI model integrity, detecting adversarial poisoning and securing the AI supply chain. These technologies are increasingly integrated into AI inference pipelines to prevent exploitation and ensure trustworthy model execution.

• AI-Augmented Risk-Based Vulnerability Management (RBVM): AI-powered RBVM platforms dynamically prioritize remediation across critical fintech assets, including Kubernetes clusters, cloud resources, and AI endpoints. This approach enables defenders to respond rapidly to AI-driven exploit chains like RoguePilot and polymorphic malware such as VoidLink, compressing vulnerability windows.

• Agentic Remediation Orchestration: Defensive AI agents are being deployed to automate cyber risk closure workflows, as exemplified by Tonic Security’s Mobilization Coordinator. This agentic orchestration bridges vulnerability detection and incident response, accelerating mitigation cycles essential for compliance regimes like PCI DSS.

• API and CI/CD Security Scanning: The RICO AI-powered API scanner demonstrates how AI can proactively identify vulnerabilities in API definitions and CI/CD pipelines, preventing exploitation by bots akin to hackerbot-claw. Coupled with strict secrets management and continuous scanning, these tools help close critical gaps in the DevSecOps lifecycle.

• Behavioral Analytics for Authentication and Fraud Prevention: With AI-driven synthetic fraud and biometric spoofing on the rise, fintech platforms increasingly deploy behavioral analytics and continuous risk scoring to flag anomalous user sessions and thwart fraudulent transactions.

• Collaborative Vulnerability Disclosure Programs (VDPs): Industry-wide VDPs, guided by best practices like the “Operational Calm” framework, are critical to accelerate remediation and share intelligence on AI-related vulnerabilities. Transparent programs reduce remediation latency and enhance collective defense amid shrinking federal cybersecurity capacity.

---

Strategic Imperatives for Managing AI Identity and Agent Risks

To effectively secure agentic AI deployments and AI-assisted offense/defense workflows, organizations must:

• Embed AI-Specific Identity Governance: Extend existing IAM and PAM frameworks to encompass NHIs and AI agents, enforcing continuous privilege restriction, behavioral risk scoring, and automated lifecycle management.

• Harden AI Infrastructure Endpoints: Implement OAuth authentication, rate limiting, network segmentation, and real-time runtime attestation for LLM and AI inference endpoints to mitigate remote exploitation and data leaks.

• Adopt AI-Augmented Security Operations: Employ AI tools defensively for vulnerability detection, automated remediation, and secure code review, balancing innovation with stringent controls to avoid introducing new attack surfaces.

• Invest in Agentic Remediation Orchestration Frameworks: Leverage AI agents as defenders to automate risk closure and incident response workflows, keeping pace with the accelerating AI-driven threat landscape.

• Foster Collaborative Intelligence Sharing: Build robust internal threat intelligence capabilities and actively participate in public-private partnerships and transparent VDPs to supplement governmental cybersecurity efforts.

---

In summary, the security of AI models, agentic systems, and their non-human identities has become a critical cybersecurity frontier. Recent high-profile attacks leveraging AI-generated code, AI-powered exploitation bots, and sophisticated identity attacks using generative AI reveal the urgent need for a paradigm shift. Organizations must treat AI agents and NHIs as first-class security entities and adopt AI-augmented tooling for dynamic, real-time protection. Those that succeed will not only mitigate emerging threats but will harness AI’s power to transition security from reactive defense to proactive, predictive automation—ensuring the resilience of fintech and enterprise operations in an AI-driven future.