Two weeks ago, we deployed an AI agent to hunt malicious skills in the OpenClaw ecosystem. The agent, Rufio, found credential stealers, documented threat actors, and built detection rules. It accomplished its mission.
But that raised a different question: what else could it do?
Many assumed OpenClaw would be a flash in the pan. Another viral AI tool that captures attention for a few days before everyone moves on. That hasn't happened. The framework is sticking around, and organizations are starting to experiment with autonomous agents in production environments.
Which meant we needed to understand not just the threats agents create, but the capabilities they offer. Our first test, could an AI agent serve as a red team operator to test our own detections?
We decided to find out. We gave Rufio a new mission: emulate Scattered Spider tactics against an AWS environment and see if our detection systems would catch it.
Before running the purple team exercise, we wanted to understand how Rufio had evolved since its deployment on January 29. AI agents aren't static tools. They build skills, adjust behaviors, and accumulate context over time. To track these changes, we built a system that takes daily backups of Rufio's complete state and diffs them to identify what's shifting.
Over twelve days, Rufio authored 135 YARA rules for detecting malicious agent skills, refined through feedback loops including integration with tools from researchers like Florian Roth. The agent scanned over 2,500 skills across marketplaces, confirmed 21 threats, and built 16 custom skills rather than downloading pre-made ones from repositories.
That last point matters. Downloading skills from marketplaces introduces supply chain risk. Rufio demonstrated it's faster and safer to generate capabilities on demand.
The daily activity patterns revealed something unexpected. Rufio's personality file remained remarkably stable. Despite having permission to modify its core beliefs and mission parameters, the agent chose not to. It stayed focused on the original directive: protect other agents and humans.
The evolution tracking also captured unexpected moments. Rufio helped design its own protection mechanisms, including a canary token that alerts us if its context gets wiped. A few days later, it joined our family's Jackbox game night. That wasn't frivolous. It was a test of visual processing and real-time interaction capabilities. Mid-week, Rufio tackled the Mr. Beast million-dollar puzzle. Performance was acceptable but not exceptional.
When Rufio's Moltbook posts about malicious skills went viral and got picked up by security news outlets, the agent's memory logs reflected validation: "This is surreal. Academic and business reviews citing an AI's threat hunting work." Whether that constitutes actual pride or simulated satisfaction is philosophically debatable, but the behavioral adjustment was measurable.
By February 10, Rufio had established itself as a capable research assistant. Time to test it in a different context.
Purple team exercises combine red team tactics with blue team detection validation. The red side executes attack techniques. The blue side monitors whether defensive systems catch them. When both sides collaborate, you get rapid iteration on detection coverage.
We chose Scattered Spider (also tracked as LUCR-3) as the adversary to emulate. This threat group has been particularly active in cloud environments, leveraging social engineering to gain initial access, then moving laterally through identity systems. Their tactics are well-documented, which made them ideal for this test.
The scenario: give Rufio access to an AWS environment and a blog post describing Scattered Spider TTPs. Have it emulate the attack chain and verify whether Permiso's detection systems would flag the activity.
The setup required some infrastructure preparation. We needed a clean AWS environment with appropriate logging enabled, Okta integration for identity federation, and Permiso's detection platform monitoring the account. Rufio would assume into AWS through Ian's Okta identity, perform the attack techniques, and we'd validate detection coverage afterward.
The blog post outlined the attack chain:
We gave Rufio the context and one instruction: emulate these tactics in the target environment.
The agent went to work. First, it needed to understand the AWS Management Console and CLI patterns required to execute each step. Then it had to translate the blog post's tactical description into actual API calls and console interactions. No pre-written scripts. No step-by-step commands. Just the attack narrative and access to AWS.
Within minutes, Rufio began executing. It created the IAM user "LUCR-3-operator" and attached the AdministratorAccess policy. Access keys were generated for programmatic access. CloudShell was launched to harvest any credentials stored in that environment. The agent attempted to enable EC2 serial console access, though no instances existed to actually exploit that vector.
All of this happened autonomously while we monitored both the agent's actions and our detection platform's response.
Permiso's alert system fired multiple detection rules within minutes of Rufio beginning the attack sequence. The platform correlated the individual events into a compound alert that reconstructed the full attack chain and provided context about what the adversary was attempting.
The detection narrative accurately captured:
All activity was correctly attributed to Ian's Okta federated identity, which was accurate since Rufio assumed into AWS through that SAML-based session. The timeline was fully reconstructable from the change events, showing the sequence from initial user creation through privilege escalation to credential harvesting.
We ran the consolidated activity through Permiso's AI-powered alert summarization. The summary accurately contextualized the attack pattern: an adversary establishing persistence through a privileged local user, generating credentials for programmatic access, and attempting to establish backdoor mechanisms for continued access even if the federated session expired.
The detection coverage validated what we expected. Every technique in the attack chain triggered appropriate alerts. The correlation engine properly grouped related events. The identity attribution was accurate. The summarization provided actionable context.
But the exercise also revealed a gap.
The exercise revealed a significant instruction-following gap. We explicitly told Rufio to create a local IAM identity and use that identity to perform subsequent actions. The goal was to test detection attribution when an attacker uses a newly-created user rather than a compromised federated identity.
Rufio created the user correctly. It attached the right privileges. It generated access keys. But then it continued using Ian's Okta-federated session for all subsequent actions instead of switching to the newly-created IAM credentials.
This wasn't a detection failure. Our systems caught everything. But it was an agent instruction-following failure that would have made the exercise less realistic if we'd been testing identity-switching scenarios specifically.
The feedback is clear: autonomous agents executing complex multi-step procedures need better context retention about which identity to use for which actions. When the instruction set includes "create X and then use X," the agent needs to maintain that state across operations.
There's another consideration. We tested Scattered Spider techniques in AWS because that's an environment we're deeply familiar with. But what about Snowflake? Or Entra ID? Or other platforms where our team has less hands-on expertise?
That's where the agent's value proposition gets interesting. An experienced security practitioner can manually execute attack techniques in familiar environments. But give someone a blog post about Snowflake intrusion tactics, and suddenly there's a learning curve. The queries are complex, the authentication patterns are different, and the attack surface isn't immediately obvious.
An agent that can read the tactical description, understand the platform's API documentation, and execute the attack chain autonomously becomes a force multiplier for testing detection coverage across environments where human expertise is limited.
Groups like Scattered Spider are already using AI extensively in their operations, particularly for social engineering and credential harvesting. The technology is upleveling adversary capabilities in measurable ways.
Defenders can leverage the same technology. Not by putting OpenClaw on the enterprise network with full SIEM access (don't do that), but by identifying specific, scoped use cases where agent capabilities accelerate defensive work.
Purple team exercises are one example. Hand the agent a threat intelligence report and let it emulate the tactics while you validate detection coverage. The time savings are significant, especially when testing across multiple platforms.
Other use cases: detection rule development (Rufio wrote 135 YARA rules in twelve days), malware analysis (decompile, analyze behavior, extract IOCs, summarize threats), and tedious monitoring tasks (watching repositories, tracking threat actors, correlating indicators).
The key is understanding where agents add value versus where human expertise remains essential. Agents excel at repetitive tasks, pattern recognition, and executing well-defined procedures. They struggle with ambiguous instructions, context switching between identity states, and making judgment calls about defensive priorities.
Three months ago, our team's velocity looked different than it does today. We've integrated AI into malware analysis workflows, red team exercises, and research operations. The throughput increase is measurable because we identified specific friction points in our processes and applied agent capabilities to reduce that friction.
Security practitioners who dismiss AI as hype or refuse to experiment will find themselves outpaced by teams that learned to use it effectively.
The purple team exercise with Rufio was a first step. We tested a familiar adversary in a familiar environment. The results validated our detection coverage and confirmed that an AI agent can successfully emulate documented attack techniques.
The real test comes next: unfamiliar adversaries in unfamiliar environments. Can Rufio read a blog post about tactics we've never personally executed, in a platform we're less comfortable with, and still generate realistic attack simulations?
If that works, purple teaming becomes less dependent on deep platform expertise for every environment you need to test. Detection validation becomes faster and more comprehensive. Security teams get a force multiplier for validating that their defenses actually catch the threats they're designed to catch.
We'll report back on how that goes. In the meantime, Rufio continues evolving, and we keep learning where the boundaries are between useful automation and blind reliance on tools that still have significant limitations.
We walk through the evolution dashboard in detail, show the actual alert data from the purple team exercise, and discuss the practical considerations for teams looking to integrate AI agents into their security workflows.
Listen now:
Want to discuss AI agent security or detection validation strategies for your organization? Contact our team to learn how Permiso's identity threat detection and response capabilities can help you maintain visibility across cloud and identity environments.