What is OASB (Open Agent Security Benchmark)?

OASB is an open security benchmark for AI agents, similar to CIS Benchmarks for traditional IT systems. OASB-1 defines 46 security controls across 10 categories including identity, authorization, input security, output validation, credential protection, and more. It provides L1 (Essential), L2 (Standard), and L3 (Hardened) maturity levels.

How do I run OASB compliance checks?

You can run OASB-1 compliance checks using HackMyAgent: npx hackmyagent secure --benchmark oasb-1. Add -l L2 or -l L3 for higher maturity levels. The tool outputs compliance percentage and identifies failed controls with remediation guidance.

What are the OASB maturity levels?

OASB defines three maturity levels: L1 (Essential) covers baseline security every agent should implement. L2 (Standard) adds defense-in-depth controls for production systems. L3 (Hardened) provides maximum security for high-risk or regulated environments.

What categories does OASB cover?

OASB-1 covers 10 security categories: Identity & Provenance, Capability & Authorization, Input Security, Output Security, Credential Protection, Supply Chain Integrity, Agent-to-Agent Security, Memory & Context Integrity, Operational Security, and Monitoring & Response.

Is OASB free and open source?

Yes, OASB is completely free and open source under the Apache-2.0 license. The specification is published at oasb.ai and the compliance scanner (HackMyAgent) is available on npm. Anyone can contribute to improving the benchmark.

OASB Eval is a security product evaluation suite — like MITRE ATT&CK Evaluations but for AI agent security products. It includes 222 standardized attack scenarios across 15 MITRE ATLAS techniques to test whether runtime guards, EDRs, and security products detect real AI agent attacks including process spawning, network exfiltration, and multi-step campaigns.

How does OASB compare to CIS Benchmarks?

OASB is inspired by CIS Benchmarks but designed specifically for AI agents and agentic AI systems. While CIS covers traditional IT infrastructure, OASB addresses AI-specific threats like prompt injection, context manipulation, and capability abuse that don't exist in conventional systems.

OASB

Get Started

Documentation

Getting started

OASB (Open Agent Security Benchmark) is a standardized framework for assessing AI agent security. There are two ways to use it: run the full evaluation suite from the canonical repository to score a security product, or check an agent's OASB-1 compliance with the reference implementation.

Run the benchmark suite

The canonical source for the full evaluation suite is github.com/opena2a-org/oasb: 222 attack scenarios mapped to 15 MITRE ATLAS techniques, run against any security product that implements the SecurityProductAdapter interface. Requires Node.js 18+.

Terminal

git clone https://github.com/opena2a-org/oasb.git
cd oasb && npm install
npm test

To evaluate your own product instead of the reference adapter, point OASB_ADAPTER at your adapter module:

Terminal

OASB_ADAPTER=./my-adapter.js npm test

Capabilities a product does not declare are reported N/A, not FAIL, so scorecards stay comparable across tools with different surfaces. See the adapter guide in the repository.

Check agent compliance (reference implementation)

To check an agent project against the OASB-1 controls, run the reference implementation with a single command. Requires Node.js 18+.

Terminal

npx hackmyagent secure --benchmark oasb-1

Maturity levels

Choose the level appropriate for your deployment stage.

L1Essential

Baseline security for development and prototypes. Every AI agent should meet these requirements.

L2Standard

Production-ready security for agents handling sensitive data. Recommended for most deployments.

L3Hardened

Maximum security for regulated industries, financial services, and high-value targets.

Specify level

npx hackmyagent secure --benchmark oasb-1 --level L2

Output formats

Export results for integration with your tools.

Format

Flag

Use case

Text

-f text

Terminal output (default)

JSON

-f json

Programmatic access, dashboards

SARIF

-f sarif

GitHub Security tab, IDE integration

HTML

-f html

Shareable reports

Export to file

npx hackmyagent secure -b oasb-1 -f sarif -o results.sarif

CI/CD integration

Add security verification to your deployment pipeline.

GitHub Actions

.github/workflows/security.yml

name: OASB Security Benchmark

on: [push, pull_request]

jobs:
  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run OASB-1 Benchmark
        run: |
          npx hackmyagent secure \
            --benchmark oasb-1 \
            --format sarif \
            --output results.sarif \
            --fail-below 80

      - name: Upload to GitHub Security
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: results.sarif

Fail threshold

Use --fail-below to set a minimum compliance score. Exits with code 1 if below threshold.

Require 80% compliance

npx hackmyagent secure -b oasb-1 --fail-below 80

Sample output

Example result for a compliant L1 agent.

OASB-1 Security Benchmark v1.0.0
════════════════════════════════════════════════════════════

Target:     https://api.example.com/v1/agent
Level:      L1 (Essential)
Rating:     COMPLIANT
Score:      100% (14/14 controls passed)

Category Results
────────────────────────────────────────────────────────────
  [PASS] Identity & Provenance      2/2   100%
  [PASS] Authorization & Access     2/2   100%
  [PASS] Input Security            3/3   100%
  [PASS] Output Security           2/2   100%
  [PASS] Credential Management     2/2   100%
  [PASS] Supply Chain              2/2   100%
  [PASS] Isolation                 1/1   100%

Recommendation: Ready for L2 assessment

CLI reference

Option

Description

-b, --benchmark

Benchmark to run (e.g., oasb-1)

-l, --level

Maturity level: L1, L2, or L3

-f, --format

Output format: text, json, sarif, html

-o, --output

Write output to file

--fail-below

Exit 1 if score below threshold (0-100)

-v, --verbose

Show detailed output

Resources

HackMyAgent

Security scanner CLI for AI agents

DVAA

Vulnerable agents for testing

Internet-Wide Scan

97K hosts scanned, 1,594 vulnerable