Stop Trusting AI Vulnerability Scanners That Can't Prove Their Exploits

If your security pipeline generates 500 vulnerability alerts on a single web application, you do not have an AppSec tool. You have an unmanageable queue of noise that developers will inevitably ignore. Traditional scanners drown teams in theoretical risks, while modern "AI-assisted" wrappers often hallucinate CVEs based on superficial pattern matches. The missing link in automated offensive security has never been finding candidates. It has always been the verification layer.

Enter pentest-ai (ptai). Instead of asking engineers to trust probabilistic language models or noisy heuristic output, ptai introduces an uncompromising architectural invariant: an issue is treated as an unconfirmed hypothesis until an execution engine reproduces it.

The Machine Oracle: Deterministic Proof Over Probabilistic Output

Most static and dynamic analyzers flag a vulnerability based on string matching or signature heuristics. ptai treats raw scanner output simply as candidate telemetry.

A candidate finding is only elevated to the VERIFIED state if a dedicated machine oracle successfully reproduces the exploit $N$ out of $N$ times against the target.

[ Ingest Probes / External Scanners ]
               │
               ▼
       [ Candidate Finding ]
               │
               ▼
   ┌───────────────────────┐
   │ Machine Oracle Engine │ <── Re-executes exploit N/N times
   └───────────────────────┘
               │
      [ Deterministic Pass? ]
         /           \
      (Yes)          (No)
       /               \
      ▼                 ▼
[ VERIFIED Badge ]   [ Suppressed / Filtered ]
(Proof Capsule Gen)

This verification model mirrors how TruffleHog validates leaked secrets by actively hitting identity providers rather than guessing. If third-party scanners like Nuclei, Nikto, or ZAP spit out warnings, ptai intercepts, filters, and suppresses that data until its own oracles prove execution.

Crucially, verification never relies on an LLM saying "this looks vulnerable." It comes from named, deterministic machine oracles. ptai currently ships with eight specialized oracles covering:

Reflection
Open-redirect
IDOR / BOLA
Error-disclosure
MCP-exposure
SQL Injection (boolean and blind)
Out-of-band (OAST) interactions for blind SSRF and XXE via collaborator infrastructure

Every confirmed vulnerability outputs a portable, non-destructive proof capsule that allows developers to replay the exact state transition locally.

The Numbers: Benchmarking Precision on Modern Targets

Theoretical architecture is meaningless without baseline performance. To keep performance claims reproducible, ptai includes a clean-app zero-FP gate and a honeypot test harness directly inside its repository.

When pitted against the OWASP Juice Shop Single Page Application (SPA), the variance between traditional heuristic scanners and ptai's verification engine becomes evident:

Tool	Findings	Critical + High	OWASP Top 10 Buckets	FP Rate
ptai 0.13.0	88	46	5	0%
ZAP 2.17.0	593	0	1	47%
Nuclei 3.8.0	1	0	1	0%
HexStrike v6.0	11	0	1	-

ZAP generated 593 alerts with a 47% false positive rate and zero confirmed Critical/High issues on this surface. Nuclei caught one finding. ptai extracted 88 verified findings, 46 of which were Critical or High across 5 OWASP Top 10 categories, maintaining a 0% false positive rate across verified output.

In a controlled honeypot evaluation with 20 seeded bugs, ptai captured 19 out of 20 with 100% precision, zero false positives, and 4 complete bug classes fully validated by the machine oracle. The oracle gate acts as a high-pass filter that raises precision without collapsing recall.

Multi-Phase Agent Architecture

The orchestration engine decouples reasoning from raw execution. The phase lifecycle operates deterministically, running identical probes regardless of whether an upstream LLM API key is present.

Agent	Phase	Role
`recon`	1	Port scanning, DNS & subdomain enum, service fingerprinting
`web`	2	Authenticated OWASP Testing Guide v4 execution
`api_security`	2	OpenAPI, GraphQL, and REST surface analysis (OWASP API Top 10)
`browser`	2	Playwright-based DOM analysis, XHR capture, security header grading
`ad`	3	Active Directory enumeration, Kerberoasting, BloodHound pathfinding
`cloud`	4	AWS, Azure, GCP IAM, misconfigurations, K8s RBAC, serverless checks
`credential_tester`	4	Password spraying, credential stuffing, MFA bypass testing
`privesc`	5	Local and lateral privilege-escalation path suggestions
`vuln_scanner`	5	Cross-cutting vulnerability aggregation against Findings DB
`exploit_chain`	6	Correlating isolated bugs into multi-step attack graphs
`poc_validator`	7	Non-destructive proof-of-concept generation per finding
`detection`	8	Generating Sigma, SPL, and KQL detection rules for blue teams
`report`	9	Exporting Markdown, HTML, PDF, SARIF, JUnit, and compliance maps
`llm_redteam`	Optional	OWASP LLM Top 10 security probes
`social_engineer`	Optional	Phishing corpus and pretext analysis
`mobile`	Optional	Static and dynamic checks for Android and iOS
`wireless`	Optional	Wireless reconnaissance and handshake capture

Engineering Highlights: Beyond Basic Scanners

Several engineering decisions distinguish ptai from conventional security tooling:

Decoupled LLM Architecture: The LLM handles phase transitions and correlates data, but finding identification comes from a deterministic probe library. No LLM key? The phase loop and probes still execute.
Native Context Preservation for Authentication: Unlike simple crawlers that fail past an identity barrier, ptai manages active sessions and token refreshes. Authentication profiles reference environment variables, Vault paths, or op://* schemas rather than plaintext strings. Every wrapped sub-process inherits this session context.
Zero Cloud Lock-in: The runtime is MIT-licensed, stores state strictly on local disk, and supports air-gapped terminal operations via local Ollama models.
Declarative Orchestration: Security workflows can be checked into version control as shareable YAML playbooks.

Integration Paths and Tool Provisioning

You can install ptai via pip:

bash
pip install ptai

From here, the system supports three distinct operational models depending on your execution environment.

1. Model Context Protocol (MCP) Workflows

For engineers using Claude Code, Cursor, Codex, or Claude Desktop, ptai acts as an MCP server. This allows you to leverage existing assistant subscriptions without providing direct Anthropic API keys to the CLI.

bash
# Add directly to Claude Code
claude mcp add pentest-ai -- ptai mcp

# Auto-configure for Cursor, VS Code Copilot, or Claude Desktop
ptai setup --mcp

2. Standalone Terminal / CI/CD Deployment

For headless build pipelines or secure environments, run the engine natively. A built-in $10 USD spend guardrail (PTAI_PRICE_LIMIT) prevents runaway LLM loops during automated runs.

bash
# Direct execution with Anthropic
export ANTHROPIC_API_KEY=sk-ant-...
ptai start https://your-target.com

# Fully offline execution via Ollama
export PENTEST_AI_LLM_PROVIDER=ollama
ptai start https://your-target.com

3. Tool Tier Provisioning

ptai acts as an orchestrator across 200+ security utilities. These can be provisioned automatically on first execution or pre-baked in modular tiers:

bash
# Core tier: Essential baseline (~6 tools, ~30s)
ptai setup --tier core

# Recommended tier: Adds fuzzers, web crawlers, password tools (~5 mins)
ptai setup --tier recommended

# Full tier: Comprehensive environment installation (~30 mins)
ptai setup --tier full

# Explicit package targeting or interactive selection
ptai setup --per-tool wpscan,dalfox,paramspider
ptai setup --wizard

Out-of-Band (OAST) Validation Dynamics

Testing for blind vulnerabilities (blind SSRF, blind SQLi, blind XXE, SSTI, Log4Shell) requires out-of-band callbacks. By default, ptai sends payloads configured to hit ProjectDiscovery's oast.fun infrastructure.

To preserve operational privacy, ptai encrypts payload markers with a client-side keypair before transmission. Only your local process holds the private key required to decrypt and confirm the callback payload.

While the payload remains private, metadata such as source IP, timestamp, and network protocol touches the public collaborator edge. In high-assurance or strict-compliance networks, you can route OAST traffic to a self-hosted Interactsh instance or bypass out-of-band checks entirely:

bash
ptai start https://your-target.com --no-oast

Production Guardrails and Ethical Operations

Because ptai generates live exploit traffic, it requires strict boundary enforcement. Running offensive operations against targets without prior written authorization is illegal.

The framework provides specific safeguards to keep scans safe and scoped:

Strict scope file evaluation to block out-of-bounds domains and IPs
Non-destructive default configurations for all candidate PoC validation steps
Adaptive rate-limiting when operating in stealth mode

Whether integrated into GitHub Actions to fail pull requests via SARIF outputs, utilized by red teams targeting Active Directory vectors, or driven via Claude Code over MCP, ptai shifts automated pentesting from vague heuristic alerting to deterministic, verifiable execution.