How do I evaluate AI SOC vendors?

Start with your operating model, existing stack, and gaps. Select the Shift Map zones you need, from data ingestion and detection engineering through investigation, response, and feedback. Compare candidates against the same requirements. For each capability, record how it is delivered, how much human involvement remains, and the evidence supporting the claim. Keep vendor declarations, demos, and validation in your environment distinct.

How do I compare AI SOC platforms when every vendor claims the same things?

Separate delivery, autonomy, and evidence. A pre-built capability is different from a workflow the customer must build or a service delivered by a partner. Then record whether the system assists, asks for approval, acts subject to a human veto, or runs end to end. Compare these dimensions within the same scope instead of treating a capability checkbox as proof of equivalent support.

What should go into an AI SOC RFP?

Ask vendors to identify the lifecycle zones they cover, current capability delivery, customer engineering requirements, autonomy, guardrails, and available evidence. Keep roadmap and preview separate from generally available features. Use the Research Briefing Profile for vendor declarations and a focused demonstration; validate important claims in your own environment before a purchase decision. Ask what happens when a decision is wrong, who can review or reverse it, and what is logged.

Can I use ASEF to evaluate a single tool, not a whole platform?

Yes. Put only the zones you care about in scope, for example detection engineering or investigation, and ASEF grades those in depth, down to their subdomains, and leaves the rest out of the verdict. You can score a data pipeline, a detection engineering tool, an investigation copilot, or a standalone response engine on its own. This matters as more of the stack decouples and buyers assemble it from separate parts.

Practitioners use ASEF to define needs, shortlist products, and evaluate them consistently. Product teams can review their own capabilities against the same criteria. Research briefings use aligned delivery and autonomy vocabulary, but a vendor RFI or forty-minute demo is not equivalent to a validated practitioner evaluation. Current ASEF has one product evaluation workflow, not a separate Builder mode or maturity ranking.

How does ASEF evaluate AI capabilities?

Delivery records whether a capability exists and what is needed to use it. Autonomy records human involvement: 0 is manual, 1C is assistance, 1G requires human approval, 1A allows a human veto, and 2 runs end to end. Autonomy can be delivered through automation, LLM or agent reasoning, or both; it is not an LLM-only score. Platform and Trust uses a separate functional scale of None, Limited, and Full. Unassessed support remains unknown.

How is ASEF different from ARMM?

ARMM focuses on response. ASEF covers the full SecOps lifecycle and draws on ARMM's response action catalogue, but current ASEF evaluates delivery, autonomy, and evidence rather than Builder maturity tiers. You do not need a separate ARMM assessment. Platform and Trust applies across the map, and PICERL can measure operational outcomes when baseline and follow-up data exist.

Does ASEF score or rank vendors?

ASEF supports capability-level evaluation and profile-based comparisons, not a universal best-vendor ranking or one blended maturity score. Results show coverage, delivery, autonomy, and evidence within the selected scope. Practitioner ratings belong to the evaluator; vendor-declared research profiles and analyst observations remain distinguishable from independent environment validation.

Yes. The framework reference and the guide are public at secops-unpacked.ai/asef/guide.

AI SecOps Evaluation Framework (ASEF)

ASEF, the AI SecOps Evaluation Framework, is a vendor-neutral framework for evaluating AI-enabled tools across the full security operations lifecycle: data ingestion, detection engineering, investigation and triage, response, remediation, and feedback, with Platform and Trust assessed across every zone. It supports AI SOC evaluation as well as individual SecOps tools.

Contributors: Anton Chuvakin · Rafal Kitab · ASEF v3.0.3 · View changelog

Evaluate any tool on the map, not only a whole platform.

ASEF lays out the full security operations lifecycle as one Shift Map: data ingestion and processing, detection engineering, investigation and triage, then response, remediation, and the feedback loop, with a Platform and Trust layer running across all of it. Every product you might buy sits somewhere on that map.

You do not have to evaluate a whole AI SOC platform to use it. Score a data pipeline, a detection engineering tool, an investigation copilot, or a standalone response engine on its own. Put the zones you care about in scope, and the framework grades only those, in depth, down to their subdomains, and leaves the rest out of the verdict.

Buy the whole stack, or decouple it.

You can now buy this whole stack from one vendor, or assemble it from separate parts. Both are real strategies, and the choice keeps getting harder as more of the stack decouples. The question shifts from which platform to which tool for which job.

That is why ASEF grades tools, not suites. It gives you one consistent ruler to hold against a single-vendor platform and against a specialized tool for one stage, so both get measured against your gap, not against a vendor's framing. You evaluate each product for the exact job you need it to do, and compare them on the same sheet.

Separate delivery, autonomy, and evidence.

First record delivery: out of the box, configured, customer-built, partner-delivered, preview, roadmap, or not supported. A shipped pre-built capability is not the same as a workflow or integration the customer must engineer. Unknown delivery, preview, and roadmap do not count as generally available coverage.

Then record human involvement. Autonomy level 0 means a manual process, not a missing capability. Higher levels may use automation, LLM or agent reasoning, or both. Record evidence separately, distinguishing a vendor claim or published RFI from documentation, an observed demonstration, or validation in your environment. Platform and Trust uses functional support, not autonomy.

One evaluation workflow, different evidence contexts.

Start by mapping your needs or go directly to a known product. Evaluate the selected lifecycle zones and Platform and Trust, then compare profiles against the same requirements. Results keep delivery, coverage, autonomy, and evidence visible instead of reducing a product to one maturity ranking.

The Research Briefing Profile uses aligned delivery and autonomy vocabulary for vendor declarations and a forty-minute demo. A published vendor profile can inform your evaluation, but does not prove every underlying capability or replace validation in your environment. Product teams can use the same criteria to understand how their products will be evaluated.

The SecOps Shift Map

Data ingestion and processing
Detection engineering and SecOps resilience
Investigation and Triage
Response, Remediation, and Feedback loop
Platform and Trust (cross-cutting)

The autonomy scale

0, Manual: A manual process with no autonomy. Delivery status separately records whether the capability exists.
1C, Collaborator: The system assists, the human acts.
1G, Guide: The system proposes, the human approves.
1A, Approver: The system acts, the human can veto.
2, Automated: Runs end to end, no human in the loop.

The six-stage journey

Frame the gap: define the SOC gap and the org path (no SOC, MDR, mature).
Map the shift: place capabilities on the Shift Map across the lifecycle zones.
Evaluate capabilities: record delivery, autonomy, and evidence separately for each in-scope capability.
Verify claims: distinguish vendor declarations and demos from validation in your environment.
Compare profiles: compare candidate tools side by side on the same scoring sheet.
Decide and measure: make the call and track PICERL metrics after rollout.

Frequently asked questions

How do I evaluate AI SOC vendors?: Start with your operating model, existing stack, and gaps. Select the Shift Map zones you need, from data ingestion and detection engineering through investigation, response, and feedback. Compare candidates against the same requirements. For each capability, record how it is delivered, how much human involvement remains, and the evidence supporting the claim. Keep vendor declarations, demos, and validation in your environment distinct.
How do I compare AI SOC platforms when every vendor claims the same things?: Separate delivery, autonomy, and evidence. A pre-built capability is different from a workflow the customer must build or a service delivered by a partner. Then record whether the system assists, asks for approval, acts subject to a human veto, or runs end to end. Compare these dimensions within the same scope instead of treating a capability checkbox as proof of equivalent support.
What should go into an AI SOC RFP?: Ask vendors to identify the lifecycle zones they cover, current capability delivery, customer engineering requirements, autonomy, guardrails, and available evidence. Keep roadmap and preview separate from generally available features. Use the Research Briefing Profile for vendor declarations and a focused demonstration; validate important claims in your own environment before a purchase decision. Ask what happens when a decision is wrong, who can review or reverse it, and what is logged.
What is ASEF?: ASEF, the AI SecOps Evaluation Framework, is a vendor-neutral framework for evaluating AI-enabled tools across the full security operations lifecycle: data ingestion, detection engineering, investigation and triage, response, remediation, and feedback, with Platform and Trust assessed across every zone. It supports AI SOC evaluation as well as individual SecOps tools.
Can I use ASEF to evaluate a single tool, not a whole platform?: Yes. Put only the zones you care about in scope, for example detection engineering or investigation, and ASEF grades those in depth, down to their subdomains, and leaves the rest out of the verdict. You can score a data pipeline, a detection engineering tool, an investigation copilot, or a standalone response engine on its own. This matters as more of the stack decouples and buyers assemble it from separate parts.
Who can use ASEF?: Practitioners use ASEF to define needs, shortlist products, and evaluate them consistently. Product teams can review their own capabilities against the same criteria. Research briefings use aligned delivery and autonomy vocabulary, but a vendor RFI or forty-minute demo is not equivalent to a validated practitioner evaluation. Current ASEF has one product evaluation workflow, not a separate Builder mode or maturity ranking.
How does ASEF evaluate AI capabilities?: Delivery records whether a capability exists and what is needed to use it. Autonomy records human involvement: 0 is manual, 1C is assistance, 1G requires human approval, 1A allows a human veto, and 2 runs end to end. Autonomy can be delivered through automation, LLM or agent reasoning, or both; it is not an LLM-only score. Platform and Trust uses a separate functional scale of None, Limited, and Full. Unassessed support remains unknown.
How is ASEF different from ARMM?: ARMM focuses on response. ASEF covers the full SecOps lifecycle and draws on ARMM's response action catalogue, but current ASEF evaluates delivery, autonomy, and evidence rather than Builder maturity tiers. You do not need a separate ARMM assessment. Platform and Trust applies across the map, and PICERL can measure operational outcomes when baseline and follow-up data exist.
Does ASEF score or rank vendors?: ASEF supports capability-level evaluation and profile-based comparisons, not a universal best-vendor ranking or one blended maturity score. Results show coverage, delivery, autonomy, and evidence within the selected scope. Practitioner ratings belong to the evaluator; vendor-declared research profiles and analyst observations remain distinguishable from independent environment validation.
Is ASEF open to use?: Yes. The framework reference and the guide are public at secops-unpacked.ai/asef/guide.