CISO at F500 Company on automating security operations with AI agents

We spoke with a CISO from a Fortune 500 company who is prioritizing AI agent adoption to automate security operations.

The conversation explores their implementation of AI in SOC alert triage, their human-in-the-loop approach to agent autonomy, and their evaluation process for AI tools that maintains the same rigorous security standards used for traditional software.

Key points via Sacra AI:

How does AI agent adoption fit into your broader mandate as a security leader? For example, where does it stand in the priority stack?

It is very high priority because we need to automate a lot of functions in the security organization. We want to automate the blue team operations, which includes detection response and incident response. We also want to have agents and automation in the areas of vulnerability scanning, vulnerability management, source code scanning, SCA, and remediation. These are the areas that come immediately to mind.

Can you walk me through one area where you're actively piloting or using agents today? What's the setup look like? And what's the agent actually doing?

One of the areas we're focusing on right now is our SOC (security operation center). At the tier 1 level, we're using agents to look at various alerts, identifying those that are duplicates and those that are false positives based on historical data. That's one of the first cases we've been implementing.

What's the underlying stack or setup for that tier 1 SOC use case? For example, are you using an off-the-shelf solution, building internally, or plugging into something like a SOAR platform?

We're building internally at this stage, not using a third-party startup yet. We're building on top of Jira, GitHub, Bitbucket, and Splunk through our internal automation. For our overall AI platform and LLM, we're using OpenAI.

How do you interface with OpenAI in that setup? Are you using their public API, routing through Azure OpenAI, or something else?

We have an enterprise license with OpenAI and ChatGPT, so we use their APIs directly.

When it comes to that tier 1 SOC alert triage use case, how are you handling things like agent autonomy? Is the agent allowed to take any direct actions based on a conclusion, or is it more assisting an analyst who remains in the loop for decisions?

One of our general principles is that we always want to be human augmented or human assisted. We don't want full automation with all decisions being made by AI. At this stage, we're still using humans to validate and make the final decisions on all the recommended actions, tasks, and closures. We'll maintain human oversight until we have more historical data and confidence in the system.

What kind of telemetry or audit trail do you capture around those agent suggestions and human decisions? Is that being logged or structured in any way right now?

Yes, we log everything. We generate logs from each of the agents so we can review them, build analysis on the data, and implement direct monitoring of agent activities.

Which specific failure modes around these kinds of agents do you most want to avoid? Is it wrong decisions, data exposure, downtime, or something else?

We're not really worried about data exposure or downtime because these agents operate within our cybersecurity team, so it's very controlled and restricted. What we don't want is incorrect decisions—that's the primary concern. Ultimately, we want the system to take actions, make decisions, and remediate issues, but we don't want errors, false positives, or false negatives. That's our main concern above anything else.

Over the next 12 months, where would you feel comfortable allowing an agent to take action with no human in the loop? Are there any tasks or workflows you think might be ready for that soon?

We would certainly consider the scenarios we just described with the SOC tier 1 analysts—specifically closing false positives and closing duplicate alerts.

Let's shift to the intake and evaluation process. Walk me through your standard process when someone wants to bring in an AI tool, especially something agentic. What are the key steps from request to deployment?

We treat it just like any other software purchase, acquisition, or build. We go through the full procurement process, which includes security review, licensing review, and legal review. We fully analyze how that piece of software is going to be used, what permissions it has, and what data it will touch. The first part of the process involves treating it the same way whether it's built internally or externally. We complete this full procurement and evaluation process before any agent can be used and deployed.

During that evaluation process, what are the first 3 specific security checks you look at, especially for AI agents? Are there any reviews or controls that differ from traditional software tools?

That's a good question. The evaluation process is not generally different from traditional software. However, we take a closer look at what permissions that particular agent has, what role it will perform, and what access authorizations or actions it can execute. So while the process isn't fundamentally different from other software, we recognize that agents can perform unintended or potentially malicious actions if not carefully restricted and monitored.

Do you enforce any kind of sandboxing or gating mechanism before agents are exposed to real systems or data? What does that test or staging environment look like?

Ideally, whenever possible, we review and test the agent in a test data environment where it's not using production data or production systems. This allows us to see how it operates, if it's meeting its requirements and specifications. We also use that same environment to perform activities like penetration testing and prompt injection testing to verify that it would be safe to use in the production environment.

Who typically initiates or requests a new AI tool or agent in your organization? Is it coming from security, from engineering, from business units? What's the usual entry point?

Well, historically, it would always come from engineering. But what has changed over this past year is that almost everyone in all roles, all disciplines, and all parts of the company wants to use agents for their various functions. The world has opened up very quickly for every role and discipline.

This transcript is for information purposes only and does not constitute advice of any type or trade recommendation and should not form the basis of any investment decision. Sacra accepts no liability for the transcript or for any errors, omissions or inaccuracies in respect of it. The views of the experts expressed in the transcript are those of the experts and they are not endorsed by, nor do they represent the opinion of Sacra. Sacra reserves all copyright, intellectual property rights in the transcript. Any modification, copying, displaying, distributing, transmitting, publishing, licensing, creating derivative works from, or selling any transcript is strictly prohibited.