Building Your First AI SOC Agents: Deploying Triage and Threat Hunting Agents on AWS (Part 2)

This is Part 2 of a series on building autonomous SOC agents. In Part 1, we built a triage agent that classifies alerts using hypothesis-driven investigation. Now we'll deploy it so it runs without you.

We'll build two agents, each deployed differently to match its workload:

1. SOC Triage Agent (lambda/handler.ts) - an SQS-triggered Lambda that investigates security alerts as they arrive. It queries your logs through Scanner MCP, classifies severity, and writes structured findings to CloudWatch. Pay-per-invocation, 15-minute hard timeout, ~$5/month compute for hundreds of daily alerts.
2. Threat Hunt Agent (container/threat_hunt.ts) - a scheduled ECS Fargate task that runs every 6 hours. It pulls CISA KEV vulnerability data and IOCs from ThreatFox, OTX, and Feodo Tracker, hunts across a year of historical logs, and posts findings to Slack. No timeout ceiling, no idle compute.

All source code is at scanner-inc/first-soc-agents.

The Gap Between "It Works" and "It's Working"

Your agent works. You ran it on your laptop, watched it triage an alert, and the classification was right. It generated hypotheses, queried your data lake, correlated user activity across a 7-hour window, ran two self-critique passes, and posted a clear summary to Slack. You close your laptop and go to bed.

At 2:17am, three alerts fire. Nobody is running node triage_agent.js.

This is the gap between a working agent and a deployed one. On your laptop, you were the safety net. You watched the agent's reasoning scroll by, noticed when it went down a wrong path, killed it when it looped. In production, you lose that direct observation, so you need to replace it with structure: logs that capture what the agent did and why, a failure queue that catches alerts the agent can't handle, and a hard timeout that kills investigations that run too long. This post covers the infrastructure that makes that work - Lambda for triage, containers for longer investigations, and the monitoring that ties them together.

Start with Lambda

Lambda is a good starting point for a first agent. It gets a prototype running quickly with minimal infrastructure.

Lambda gives you one alert per invocation. That means one set of logs per investigation, one clear success or failure, one cost you can attribute to a specific alert. When the agent misclassifies something (and it will), you can pull up that single invocation and see exactly what happened: what it queried, what it found, how it reasoned, where it went wrong. Try doing that with a long-running container processing alerts in a loop.

The economics work too. Lambda's pay-per-invocation model means your infrastructure cost is roughly $5/month for hundreds of daily invocations. The API cost of the model is 100-1000x more than the compute. You want to spend your optimization energy on investigation depth and model selection, not container orchestration.

Lambda's 15-minute timeout is a hard ceiling on how long any single investigation can run. A confused agent can't loop for hours burning tokens - the platform kills it. It's a trust boundary enforced by infrastructure.

The pattern is the same query() call from Part 1, wrapped in a thin Lambda handler. SQS gives you retries and a dead letter queue for free. One alert arrives, one agent runs, one result gets logged.

The Handler

Here's the actual Lambda handler we run. The system prompt is the longest part, and that's on purpose.

// handler.ts — Lambda function using @anthropic-ai/claude-agent-sdk
import { query } from "@anthropic-ai/claude-agent-sdk";
import type { McpHttpServerConfig } from "@anthropic-ai/claude-agent-sdk";
import type { Handler } from "aws-lambda";

const SYSTEM_PROMPT = `You are a security alert triage agent. Investigate each alert using
the following methodology:

**Phase 1: Initial Assessment & Hypothesis Generation**
1. Review the alert details and understand what it detected.
2. Generate 2-4 hypotheses ranked by probability:
   - Benign explanation (legitimate user activity, known process)
   - Misconfiguration (incorrect rule, system issue)
   - Actual attack (malicious activity, compromise)
   - Insider threat (authorized user acting maliciously)
3. For each hypothesis, identify what evidence would confirm or refute it.

**Phase 2: Evidence Collection**
4. Collect targeted evidence to test your hypotheses:
   - Query events BEFORE, DURING, and AFTER the alert (4-6 hour window)
   - Look for the same source (user/IP/account/system)
   - Think adversarially: if this were an attack, what would the attacker do next?
5. Check for expansion indicators:
   - Privilege escalation attempts or role changes
   - Lateral movement or unusual network connections
   - Data access anomalies or exfiltration patterns
   - Persistence mechanisms (new users, scheduled tasks, backdoors)
   - Multiple failed attempts followed by success

**Phase 3: Classification**
6. Classify the alert:
   - BENIGN: The weight of evidence points to legitimate activity. This includes
     cases where the activity pattern is well-established (recurring user, known IP,
     business hours, expected role chains) even if some fields are redacted or
     unavailable — redacted parameters are a visibility gap to note, not evidence
     of malice. If you can explain WHO did it, WHY it's expected, and there are
     ZERO indicators of compromise, classify as BENIGN.
   - SUSPICIOUS: There are concrete anomalies that don't fit legitimate patterns —
     e.g., new IP, unusual time, unexpected role, first-time access, failed attempts
     before success. Gaps in visibility alone do not make something suspicious.
   - MALICIOUS: High confidence evidence of attack with corroborating indicators
     (e.g., known-bad IOCs, persistence mechanisms, data exfiltration, multiple
     ATT&CK techniques chained together).
7. Assign confidence:
   - high (80-100%): Multiple independent evidence sources support conclusion
   - medium (60-79%): Moderate support with some gaps or contradictions
   - low (0-59%): Insufficient evidence to confidently support any hypothesis

**Phase 4: Self-Critique** (run twice)
8. After your initial classification, critique your own analysis:
   - What evidence might you have missed?
   - Are there alternative explanations you didn't consider?
   - Is your confidence level justified by the evidence?
   - What would change your classification?
   Revise your assessment if the critique reveals weaknesses.

Respond with a JSON object:
{
    "classification": "BENIGN|SUSPICIOUS|MALICIOUS",
    "confidence": "high|medium|low",
    "confidence_pct": 85,
    "summary": "Two-sentence TL;DR: what happened and what the classification means",
    "timeline": [
        {"timestamp": "ISO8601", "event": "Description of what happened"}
    ],
    "hypothesis_testing": {
        "confirmed": "The hypothesis supported by evidence, with reasoning",
        "ruled_out": [
            "Alternative 1: why ruled out with specific evidence",
            "Alternative 2: why ruled out with specific evidence"
        ]
    },
    "key_evidence": [
        "Evidence point 1 with technical details (IPs, users, timestamps)",
        "Evidence point 2"
    ],
    "mitre_attack": ["T1078 Valid Accounts", "T1550.004 Use Alternate Auth Material"],
    "next_questions": [
        "Question the analyst should ask to continue this investigation",
        "Question that would confirm or change the classification"
    ]
}

Notes on fields:
- "mitre_attack": Only populate for MALICIOUS classifications. Empty list otherwise.
- "timeline": Chronological events from your investigation window, not just the alert itself.
  Include pre-alert context, the alert trigger, and any post-alert activity.
- "confidence_pct": Integer 0-100 matching your confidence level.
- "next_questions": Do NOT give prescriptive actions ("disable this account", "block this IP").
  Instead, propose questions that guide the analyst's investigation: "Is this user authorized
  to access this system?", "Was there a change ticket for this access?", "Does this IP belong
  to a known VPN provider?" Questions acknowledge uncertainty and keep the analyst in control.`;

// scannerMcpConfig() connects to your MCP server (see full source in repo)

// extractJson() handles Claude's formatting quirks — tries raw parse,
// then fenced code blocks, then outermost {...}. Small function that
// saves you from intermittent parsing failures in production.

export const handler: Handler = async (rawEvent) => {
  // Unwrap SQS event envelope if present
  const event =
    rawEvent.Records?.[0]?.body != null
      ? JSON.parse(rawEvent.Records[0].body)
      : rawEvent;

  const { alert_id, alert_summary } = event;
  const start = Date.now();

  const mcpServers = scannerMcpConfig();
  const allowedTools = Object.keys(mcpServers).length > 0 ? ["mcp__scanner__*"] : [];

  const q = query({
    prompt: `Triage this alert.\n\n**Alert ID**: ${alert_id}\n**Summary**: ${alert_summary}`,
    options: {
      model: process.env.MODEL || "claude-opus-4-6",
      systemPrompt: SYSTEM_PROMPT,
      permissionMode: "bypassPermissions",
      allowDangerouslySkipPermissions: true,
      mcpServers,
      allowedTools,
    },
  });

  let resultText: string | undefined;
  for await (const message of q) {
    if (message.type === "assistant") {
      for (const block of message.message.content) {
        if (block.type === "tool_use") {
          console.log(JSON.stringify({
            alert_id,
            step: "tool_call",
            tool: block.name,
            input: block.input,
          }));
        }
      }
    }
    if (message.type === "result" && message.subtype === "success") {
      resultText = message.result;
    }
  }

  const result = extractJson(resultText);
  const durationMs = Date.now() - start;

  console.log(
    JSON.stringify({
      alert_id,
      classification: result.classification,
      confidence: result.confidence,
      duration_ms: durationMs,
      model: process.env.MODEL || "claude-opus-4-6",
    })
  );

  return {
    statusCode: 200,
    body: JSON.stringify({ alert_id, result }),
  };
};

Let me walk through the decisions that matter.

permissionMode: "bypassPermissions" and allowDangerouslySkipPermissions sound alarming, but they're narrower than they look. bypassPermissions only applies to the tools you've explicitly listed in allowedTools. The agent can't access anything you haven't granted. In this handler, that's mcp__scanner__* - read-only queries against your data lake. The agent literally cannot take any action outside of querying logs. allowDangerouslySkipPermissions is the SDK's safety gate - it requires you to explicitly opt in to headless operation, preventing accidental deployment without permission controls. Both flags together say: "yes, this is running unattended, and here's exactly what it's allowed to do."

The console.log(JSON.stringify({...})) lines are easy to overlook, but they're the foundation of your operational visibility. Lambda automatically captures stdout to CloudWatch. The tool_call log gives you a structured record of every MCP call the agent made - which queries it ran, what parameters it used. The final summary log gives you classification, confidence, and duration for every triage. From day one, you'll be able to query both what the agent decided and how it got there. The full result (timeline, hypothesis testing, evidence, next questions) is returned in the Lambda response body for downstream consumers - Slack bots, ticketing integrations, SOAR platforms - to format however they need.

Deploying the Lambda

The components:

• Detection system pushes alerts to an SQS queue
• SQS triggers the Lambda function (container image, Node.js 22)
• Failed messages go to a dead letter queue for manual review
• All output goes to CloudWatch Logs

Three things worth explaining:

Container image Lambda, not zip deployment. The Claude Agent SDK spawns cli.js as a child process - it needs a writable filesystem and the ability to spawn subprocesses. A standard zip-packaged Lambda silently fails: the SDK hangs for ~30 seconds and returns 0 messages. Container-image Lambda provides the full runtime environment the SDK needs. Here's the entire Dockerfile:

FROM public.ecr.aws/lambda/nodejs:22

# The agent SDK spawns cli.js as a child process, which needs a writable HOME
ENV HOME=/tmp

COPY package.json package-lock.json ./
RUN npm ci --production

COPY dist/ ./dist/

CMD ["dist/handler.handler"]

Container-image Lambdas can sometimes have longer cold starts (1-10 seconds) than zip deployments. For an agent that runs 3-6 minutes per invocation, this is negligible.

SQS, not EventBridge schedule. A schedule means your agent polls for new alerts - there's a delay between the alert firing and the agent noticing. SQS is event-driven: your detection system pushes an alert, the agent picks it up immediately. SQS also gives you automatic retries (if the agent crashes, the message reappears) and a dead letter queue (if it fails twice, the alert is preserved for manual review instead of silently lost). For alert triage, losing an alert is worse than a delayed one.

visibility_timeout = 960s, slightly longer than Lambda's 900s timeout. This prevents SQS from thinking the message failed (and retrying it) while the agent is still running. If the Lambda function times out at 15 minutes, SQS waits the extra 60 seconds before making the message available again. Without this, you get duplicate invocations.

The handler works for both direct invocation and SQS. When SQS triggers a Lambda, AWS wraps your message in a Records array. The handler checks for this envelope and unwraps it; if the event comes from a direct aws lambda invoke, it uses the payload as-is. The Terraform sets batch_size = 1, so there's always exactly one record - no need to loop. This dual-mode design means you can test with direct invocation and run in production via SQS without changing the handler.

Concurrency and Rate Limiting

SQS absorbs bursts naturally. If 50 alerts fire in a minute, they queue up as 50 messages. Lambda pulls them off one at a time (per concurrent execution), and SQS's visibility timeout prevents duplicate processing.

The real constraint is the Anthropic API rate limit, not Lambda concurrency. Each agent invocation makes dozens of API calls over several minutes. Set reserved_concurrent_executions to cap how many agents run in parallel:

resource "aws_lambda_function" "triage" {
  # ...
  reserved_concurrent_executions = 5
}

Five concurrent agents is a reasonable starting point. If messages start aging in the queue, increase the limit - but watch your API rate limits first. Excess messages stay in SQS until a slot opens up. If they exceed the visibility timeout waiting, they retry and eventually land in the DLQ.

All the infrastructure is Terraform: Lambda function, SQS queue, DLQ, IAM roles, Secrets Manager, CloudWatch log group. See the Terraform config in the repo. The deploy script handles the full pipeline: ECR repo creation, TypeScript build, Docker build (linux/amd64), ECR push, Terraform apply, and Lambda code update. One ./deploy.sh gets you from zero to running.

What Your Agent Looks Like in Production

Your agent is deployed. Alerts are flowing in, investigations are happening, classifications are landing in CloudWatch. Now what?

Reading the Logs

Every invocation writes structured JSON to CloudWatch. With the tool_call logging, you get two kinds of entries: intermediate MCP calls (what the agent queried) and the final summary (what it concluded). Here's real output from a triage of a "GetSigninToken Potential Abuse" alert:

{"alert_id":"ad7f6417","step":"tool_call","tool":"mcp__scanner__get_scanner_context","input":{}}
{"alert_id":"ad7f6417","step":"tool_call","tool":"mcp__scanner__execute_query","input":{"query":"GetSigninToken","start_time":"2026-02-19T00:00:00Z",...}}
{"alert_id":"ad7f6417","step":"tool_call","tool":"mcp__scanner__fetch_query_results","input":{"fields":["sourceIPAddress","userIdentity.arn","userAgent","recipientAccountName",...]}}
{"alert_id":"ad7f6417","step":"tool_call","tool":"mcp__scanner__execute_query","input":{"query":"GetSigninToken | summarize count by sourceIPAddress",...}}
{"alert_id":"ad7f6417","step":"tool_call","tool":"mcp__scanner__execute_query","input":{"query":"GetSigninToken | summarize count by userAgent",...}}
{"alert_id":"ad7f6417","step":"tool_call","tool":"mcp__scanner__execute_query","input":{"query":"sourceIPAddress = 172.56.21.73",...}}
{"alert_id":"ad7f6417","step":"tool_call","tool":"mcp__scanner__execute_query","input":{"query":"sourceIPAddress = 98.247.31.82",...}}
{"alert_id":"ad7f6417","step":"tool_call","tool":"mcp__scanner__execute_query","input":{"query":"GetSigninToken @ecs.event.outcome = failure",...}}
{"alert_id":"ad7f6417","step":"tool_call","tool":"mcp__scanner__execute_query","input":{"query":"userAgent: aws_consoler",...}}
{"alert_id":"ad7f6417","classification":"BENIGN","confidence":"high","duration_ms":170036,"model":"claude-opus-4-6"}

You can read the investigation strategy directly from the logs. The agent searched for all GetSigninToken events, aggregated by source IP and user agent, pivoted to investigate two residential IPs it found, checked for failed attempts, and specifically searched for the aws_consoler tool signature mentioned in the alert description. When the agent misclassifies something, you can trace exactly which queries it ran (or didn't run) and what data it saw. Without these logs, you'd only know it said "BENIGN" - not why.

The Lambda REPORT line gives you the execution metadata:

REPORT RequestId: 3f3bfbaf  Duration: 170041.39 ms  Billed Duration: 170481 ms  Memory Size: 1024 MB  Max Memory Used: 322 MB  Init Duration: 439.09 ms

This invocation ran for ~2.8 minutes and used 322MB of the 1024MB allocated. The agent is I/O-bound (waiting on API calls), not memory-bound. The 439ms cold start is the container image Lambda initializing.

CloudWatch Insights lets you query these logs. A few queries we run regularly:

Classification distribution over the last 24 hours:

fields @timestamp, classification, confidence, duration_ms
| filter classification in ["BENIGN", "SUSPICIOUS", "MALICIOUS"]
| stats count() by classification

Slowest investigations (potential tuning candidates):

fields @timestamp, alert_id, classification, duration_ms
| filter duration_ms > 300000
| sort duration_ms desc
| limit 20

Error rate:

fields @timestamp, alert_id, error
| filter ispresent(error)
| stats count() as errors by bin(1h)

If your error rate spikes, you probably have an MCP server issue (Scanner API down, credentials expired) rather than an agent problem. If your duration distribution shifts upward, the agent might be encountering a new alert type that sends it on longer investigation paths - worth reviewing.

What the Agent Actually Produces

Numbers in a log are useful for monitoring. But what your analysts actually see is the Slack message. The full structured JSON (classification, timeline, hypothesis testing, evidence, next questions) is returned in the Lambda response body for downstream consumers - Slack bots, ticketing integrations, SOAR platforms - to format however they need. Here's real output from an Opus run investigating the same High-severity alert about a team member assuming a privileged cross-account role. (We tested all three models on the same alerts - the cost table below shows the comparison.) This is what landed in our #demo-ai-secops channel (51 turns, ~5.5 minutes, $1.68):

🚨 Security Alert Investigation

Alert: Team member assumed integration access role | Severity: High

TL;DR: A team member assumed corp-IntegrationAccessRole in the acme-prod-use1
customer account, which grants access to customer data. Investigation confirms
this is legitimate support activity by a known team member from a corporate IP
with a well-established historical pattern.

Classification: 🟢 BENIGN
Confidence: 90% (High)

Timeline:
- 2026-02-25T00:17:09Z - corp-SupportRole runs DescribeInstances in
  acme-prod-use1 (444455556666)
- 2026-02-25T00:17:11Z - SendCommand via SSM to customer EC2 instance
- 2026-02-25T00:17:12Z - GetCommandInvocation — checking SSM command result
- 2026-02-25T00:17:14Z - Second SendCommand via SSM
- 2026-02-25T00:17:16Z - Alert triggered: AssumeRole to
  corp-IntegrationAccessRole from IP 203.0.113.42
- 2026-02-25T00:17:17Z - Role chain continues: AssumeRole to
  corp-ScannerRole in account 111122223333

Hypothesis Testing:
✓ Benign — legitimate support activity: Corporate IP 203.0.113.42 shared by
  multiple team members; user jane authenticated via SSO
  (AWSReservedSSO_corp-Admin); activity pattern is scripted debugging
  (DescribeInstances → SendCommand → GetCommandInvocation in ~8 seconds);
  10 identical alerts in 7 days for same customer
✗ Misconfiguration: Rule is working as designed — it correctly detects
  customer-account role assumptions; excludes dev/internal accounts as intended
✗ Actual attack: Zero failed attempts, zero persistence mechanisms, zero data
  exfiltration, no lateral movement to other customer accounts, legitimate
  user agent (Boto3/1.42.39 on macOS arm64)
✗ Insider threat: Activity is consistent with standard support workflow; no
  unusual data access; same pattern repeated by same user over past 7 days
  with no escalation

Key Evidence:
- Source IP 203.0.113.42 used by multiple team members across various SSO
  roles — confirmed corporate IP
- User jane also active via AWSReservedSSO_corp-Admin from same IP,
  performing GetCallerIdentity and ListAccountAliases — standard identity
  verification
- Post-assume activity via IntegrationAccessRole was read-only console
  browsing: DescribeEventAggregates, GetCostAndUsage, GetAccountInformation
- Historical pattern: 10 alerts for this rule in 7 days — 8 targeting
  acme-prod-use1, 1 targeting prod-east-1, all from 2 known IPs
- Zero IAM modifications: no CreateUser, CreateRole, CreateAccessKey,
  AttachPolicy, PutRolePolicy in the customer account

The entire support session spans only 8 seconds (00:17:09Z to 00:17:17Z)
across multiple parallel botocore sessions — consistent with automated support
tooling, not manual exploration or data harvesting. No S3 or DynamoDB data
access was observed.

Next Questions:
- Is there a corresponding support ticket for the acme-prod-use1 engagement?
  Confirming this closes the audit loop.
- This rule has fired 10 times in 7 days for what appears to be an ongoing
  support engagement — should it be tuned? Options: time-based suppression
  for known active engagements, or enriching alerts with support ticket
  correlation to auto-close verified cases.
- Does the team want visibility into every support access, or only anomalous
  ones?

The agent reconstructed the full support session from CloudTrail: DescribeInstances → SSM SendCommand → GetCommandInvocation → AssumeRole, all within 8 seconds. It confirmed the source IP was a known corporate IP shared by multiple team members, found 10 identical alerts over 7 days for the same customer, and verified zero persistence mechanisms or data exfiltration. The conclusion is specific: this is automated support tooling, not manual exploration.

The hypothesis testing is the part that matters most. Four hypotheses generated, three ruled out with specific evidence, one confirmed. An analyst reading this can immediately see why the agent reached its conclusion and whether they agree. If the agent were wrong, the evidence trail makes that obvious too.

Questions, Not Instructions

Notice what the agent doesn't do: it doesn't say "disable this account" or "block this IP." It asks questions. This is deliberate. When agents give prescriptive instructions - "Revoke all access keys and sessions for the compromised principals" - analysts sometimes follow them without thinking. Questions flip the dynamic: "Is there a corresponding support ticket?" prompts the analyst to verify something the agent can't verify. The next_questions field in the JSON schema enforces this structurally - it's harder for the model to slip into prescriptive mode when the output field is literally called "questions."

This took 51 turns and ~5.5 minutes. An analyst doing the same investigation manually - pulling up CloudTrail, checking SSO logs, correlating IPs across sessions, reviewing historical patterns - would spend 30-45 minutes.

Security for Agent Workloads

Agent workloads have three risks that standard web services don't.

Prompt injection via alert data. Prompt injection via alert data is a broader concern for any agent processing external input, but it's beyond the scope of this post.

Runaway tool calls. A confused agent can loop, calling the same query repeatedly or spiraling through related-but-unproductive investigations. Lambda's 15-minute timeout is your hard ceiling on both execution time and API cost per alert. In our testing, thorough investigations complete in 26-51 turns within 3-6 minutes. If an agent consistently hits the timeout, the prompt needs tuning.

‍Secrets management. The handler needs API keys for Anthropic and your MCP servers. Don't put them in environment variables - use AWS Secrets Manager. The Terraform config in the repo stores the Anthropic API key in Secrets Manager and grants the Lambda role access. Environment variables show up in the Lambda console, in CloudFormation outputs, in deployment logs. Secrets Manager keeps them encrypted and auditable.

What It Actually Costs

Here's real data from three runs investigating the same alert (the role assumption from the Slack output above) with different models. Same agent code, same MCP servers, same methodology:

Sonnet and Opus both classified correctly. Haiku got it wrong - same alert, confidently wrong answer. We'll come back to that.

The surprising number: Opus was cheaper than Sonnet on the same alert. Opus took 12 more turns but accumulated fewer cached tokens (1.7M vs 2.2M). Cost is driven by context accumulation - how much data the agent pulls into the conversation - not by turn count or per-token pricing. Opus's investigation was more targeted; Sonnet pulled more data per query.

This is the key insight for cost management: the real lever is investigation depth, not model selection. The conversation context grows every turn as tool results (query responses, log entries, API payloads) accumulate. By turn 30, each API call is sending over a million tokens, most of which hit the prompt cache at ~10x lower cost than uncached. But even cached, that volume adds up.

Depth tiers based on our testing:

Infrastructure cost is a rounding error. Lambda at 500 invocations/day averages ~$5/month. Optimize model selection and investigation depth first. Infrastructure optimization is wasted effort.

When the Agent Gets It Wrong

Remember that Haiku row in the cost table: same alert, MALICIOUS at 88% confidence. Here's what the analyst would have seen in Slack:

TL;DR: Multiple failed and successful privilege escalation attempts detected
from external AWS IP 203.0.113.200 attempting to assume
corp-IntegrationAccessRole. The pattern shows an attacker first failing with
low-privilege credentials, then succeeding with escalated privileges,
indicating potential compromised credentials and lateral movement across AWS
accounts.

Classification: 🔴 MALICIOUS
Confidence: 88% (High)

The evidence Haiku cited was real: there were failed AssumeRole attempts from that IP, followed by successful ones from a different principal. There were GetSecretValue calls. The timeline was accurate. But the conclusion was wrong.

What happened? The IP 203.0.113.200 belongs to internal infrastructure - it's an ECS task running a scheduled integration job. The "failed then successful" pattern was two different services operating independently: one lacked permissions (a known misconfiguration being tracked), the other had legitimate access. Haiku saw the pattern "failed escalation → successful escalation from same IP" and latched onto the attack narrative without checking whether the IP was internal or whether the two principals were related.

Sonnet and Opus both caught this. They correlated the IP back to jane's SSO session, checked prior access patterns, and confirmed the role chain was a routine support workflow. The key divergence: Sonnet and Opus ran the SSO correlation step - linking the assumed-role session back to a named employee - while Haiku skipped it entirely and built its case on the surface-level access pattern.

Three implications:

Don't trust confidence scores. Haiku said 88% with high confidence. The structured output looked convincing - proper MITRE ATT&CK mappings, detailed timeline, specific evidence. A busy analyst skimming the Slack message could easily accept this at face value. Confidence is the model's self-assessment, not an accuracy guarantee.

Use Sonnet or Opus. Haiku at $0.32 is tempting for volume. But a misclassified MALICIOUS alert triggers incident response: pages, war rooms, credential rotations. The cost of a false positive dwarfs the savings. Haiku consistently lacks the reasoning depth to do real investigation. It skips correlation steps, latches onto surface-level patterns, and reaches confident wrong answers. For security triage, use Sonnet or Opus.

Structured output is your audit trail. The reason we caught this was running all three models on the same alert and comparing outputs. In production, you won't do that for every alert. But the structured JSON - hypothesis_testing, key_evidence, ruled_out - gives reviewers a fast path to spotting errors. When Haiku's output shows zero ruled-out hypotheses for "benign activity," that's a red flag an analyst can catch in seconds.

When You Outgrow Lambda

Three signs you've outgrown Lambda:

• Your investigations routinely hit the 15-minute timeout. Some alert types need broader context - more data sources, longer time windows, deeper correlation chains.
• You need stdio MCP servers. Lambda can only reach HTTP MCP servers. If your threat intel, endpoint, or communication tools run as stdio servers, you need a persistent process.‍
• You're building agents that do more than triage. Threat hunting across a year of logs, multi-stage incident response, coordinated investigation across multiple alerts - these need 30-60+ minutes of runtime.

The Container: ECS Fargate

The graduation path is ECS Fargate. Same Agent SDK, same query() pattern, different execution model. Fargate runs your container on managed infrastructure with no timeout ceiling. Threat hunting across a year of logs, multi-stage incident response, coordinated investigation across multiple alerts - these may need a lot more than 15 minutes of runtime. Fargate lets the agent run until it's done.

Here's a threat hunting agent that combines CISA KEV vulnerability data with structured IOCs from ThreatFox, OTX, and Feodo Tracker, hunts across your historical logs, and posts findings to Slack.

The Prompt

The prompt defines a 6-phase hunt methodology. It's long because it needs to be - the agent is running unattended and the prompt is the only guidance it has.

const prompt = `
  You are an autonomous threat hunting agent. Your mission is to proactively
  hunt for evidence of compromise in historical logs using threat intelligence.

  **CISA Known Exploited Vulnerabilities (most recently added):**
  ${kevContext}

  Execute the following 6-phase threat hunt:

  **Phase 1: Environment Discovery**
  - Call get_scanner_context to understand what log sources are available
  - Identify the environment: what platforms, vendors, services exist
  - This context determines which threats are worth hunting for

  **Phase 2: Threat Intelligence Gathering**
  - Filter KEV entries for relevance to the environment discovered in Phase 1.
    Skip vulnerabilities for products/vendors not present in your log sources.
  - Use threatfox_iocs for recent IOCs, otx_get_pulses for community intel,
    feodo_tracker for botnet C2 IPs
  - Determine search time range based on when the threat was first active

  **Phase 3: Announce the Hunt (Slack post #1)**
  - Post to #${slackChannelName}: what CVE/campaign is being hunted,
    which IOCs, what time range, and why

  **Phase 4: Historical Log Analysis via Scanner**
  - Start with broad IOC sweeps using **: "IOC" queries (IPs, domains, hashes)
  - Only if you find hits: pivot to targeted behavioral queries
  - If IOC sweeps come back clean, you're done searching

  **Phase 5: Correlation & Assessment**
  - Cross-reference findings, build timeline, map to MITRE ATT&CK
  - Identify visibility/telemetry gaps

  **Phase 6: Report Findings (Slack post #2)**
  - Post structured report: hunt target, IOCs searched, results,
    findings, visibility gaps, recommended next questions
`;

The kevContext variable is pre-fetched CISA KEV data - the 5 most recently added entries. The agent gets current threat intel baked into the prompt rather than needing to discover it.

Two things in the prompt matter most. First, Phase 1 gates everything else. The agent discovers what log sources exist before deciding what to hunt for. A KEV entry for Cisco SD-WAN is useless if your environment is AWS-only. Second, the search strategy is deliberately conservative: broad IOC sweeps first, targeted behavioral queries only on hits, and stop searching if sweeps come back clean. Without this, the agent will run speculative queries for hours.

The Handler

The query() call connects three MCP servers: Scanner (HTTP) for log queries, Slack (stdio) for posting results, and a threat intel server (stdio) for IOC feeds. Stdio servers are why this can't run in Lambda - they spawn local processes.

const q = query({
  prompt,
  options: {
    model: "claude-opus-4-6",
    permissionMode: "bypassPermissions",
    allowDangerouslySkipPermissions: true,
    mcpServers: {
      scanner: { type: "http", url: scannerMcpUrl, headers: { ... } },
      slack: { type: "stdio", command: "npx",
               args: ["-y", "@modelcontextprotocol/server-slack"], env: { ... } },
      threatintel: { type: "stdio", command: "npx",
                     args: ["-y", "mcp-threatintel-server"], env: { ... } },
    },
    allowedTools: [
      "mcp__scanner__*",
      "mcp__slack__slack_post_message",
      "mcp__threatintel__threatfox_iocs",
      "mcp__threatintel__otx_get_pulses",
      "mcp__threatintel__feodo_tracker",
      // ...
    ],
  },
});

for await (const message of q) {
  if (message.type === "assistant") {
    for (const block of message.message.content) {
      if (block.type === "text") {
        console.log(block.text);
      } else if (block.type === "tool_use") {
        console.log(JSON.stringify({
          step: "tool_call",
          tool: block.name,
          input: block.input,
        }));
      }
    }
  }
}

The iterator loop logs both the agent's reasoning (text blocks) and every tool call (tool_use blocks) as structured JSON. In CloudWatch, you can trace the full investigation: which IOCs it searched for, which queries it ran, what it found. The container streams stdout to CloudWatch via the awslogs driver, so every console.log becomes a log event.

What the Agent Produces

Here's the report it posted to Slack after a ~5-minute hunt across a few months of CloudTrail logs:

🔍 Threat Hunt Report

Hunt Target: PlugX/STATICPLUGIN (UNC6384 / Mustang Panda) C2 Infrastructure
  + QakBot C2 (Feodo) | AWS CloudTrail Environment
Intel Source: AlienVault OTX (pulse published 2026-02-25)
  + Feodo Tracker (active today)

TL;DR: Hunted for known-bad C2 infrastructure IPs and domains linked to the
PlugX STATICPLUGIN campaign (UNC6384/Mustang Panda APT) and an active QakBot
C2 across ~97GB of AWS CloudTrail log data spanning Jan 1–Feb 25 2026. No
evidence of these IOCs appearing in any AWS API call records was found,
suggesting the environment has not been accessed from this threat actor's
infrastructure during the search window.

IOCs Searched:
- 45.251.243.210 — PlugX C2 server (China, AS55933 Cloudie Limited)
- 108.165.255.97 — PlugX C2 server (port 443)
- fruitbrat.com — PlugX C2 domain
- 34.204.119.63 — QakBot C2 server (AWS EC2, first seen 2026-01-13,
  last online 2026-02-25) from Feodo Tracker

Hunt Results: 🟢 NO EVIDENCE FOUND
Confidence: 85% (High)
Time Range Searched: 2026-01-01 — 2026-02-25

Findings:
- ✓ 45.251.243.210 (PlugX C2, China) — 0 matches across ~23.3GB scanned
- ✓ 108.165.255.97 (PlugX C2) — 0 matches across ~25.6GB scanned
- ✓ fruitbrat.com (PlugX C2 domain) — 0 matches across ~21.9GB scanned
- ✓ 34.204.119.63 (QakBot C2, AWS EC2) — 0 matches across ~27.0GB scanned

Note on KEV pivot: This week's CISA KEV additions (Cisco SD-WAN, FileZen,
RoundCube) were assessed as not relevant to this AWS-centric environment. The
hunt was re-scoped to active C2 IOCs observable in CloudTrail
sourceIPAddress fields.

Visibility Gaps:
- No VPC Flow Logs / network telemetry — east-west movement and outbound C2
  connections from EC2/Lambda workloads are invisible; PlugX beaconing to
  fruitbrat.com would not appear in CloudTrail at all
- No DNS resolution logs — domain-based IOCs like fruitbrat.com can only be
  detected if they appear in API call parameters, not via DNS queries
- No endpoint/EDR telemetry — PlugX DLL sideloading, MSI execution, and
  shellcode injection are entirely undetectable with current log sources

The interesting part is the KEV pivot. The agent's instructions say to start with CISA KEV entries, but none of the week's additions (Cisco SD-WAN, FileZen, RoundCube) were relevant to an AWS-only environment. Rather than writing a useless report, it autonomously re-scoped to active C2 IOCs from OTX and Feodo Tracker - infrastructure that would show up in CloudTrail sourceIPAddress fields if an attacker used it to call AWS APIs. That re-scoping decision is why the report was actually useful.

Deploying to ECS

Here's the architecture:

EventBridge (rate: 6h)
  → ECS RunTask (Fargate)
    → Private subnet → NAT gateway → internet
    → Secrets from Secrets Manager
    → Logs to CloudWatch

The components:

Dockerfile. The container is a Node process that runs the hunt and exits. The Agent SDK spawns cli.js as a child process, so the container needs a writable HOME directory.

FROM node:22-slim

WORKDIR /home/agent/app

COPY package.json package-lock.json ./
RUN npm ci --production

COPY dist/ ./

RUN mkdir -p /home/agent && chown node:node /home/agent
ENV HOME=/home/agent

USER node

ENTRYPOINT ["node", "threat_hunt.js"]

Running as the node user (non-root) is a security baseline. The HEALTHCHECK in the actual Dockerfile monitors the process, though for a scheduled task that runs and exits it matters less than for a long-lived service.

Task definition. The ECS task definition configures Fargate resources, secrets injection, and logging:

resource "aws_ecs_task_definition" "triage" {
  family                   = "triage-agent"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = "512"
  memory                   = "1024"
  execution_role_arn       = aws_iam_role.execution.arn
  task_role_arn            = aws_iam_role.task.arn

  container_definitions = jsonencode([{
    name      = "triage-agent"
    image     = "${aws_ecr_repository.triage.repository_url}:latest"
    essential = true

    secrets = [
      { name = "ANTHROPIC_API_KEY",   valueFrom = "${secret_arn}:ANTHROPIC_API_KEY::" },
      { name = "SCANNER_MCP_API_KEY", valueFrom = "${secret_arn}:SCANNER_MCP_API_KEY::" },
      { name = "SLACK_BOT_TOKEN",     valueFrom = "${secret_arn}:SLACK_BOT_TOKEN::" },
      { name = "OTX_API_KEY",         valueFrom = "${secret_arn}:OTX_API_KEY::" },
      { name = "ABUSECH_AUTH_KEY",    valueFrom = "${secret_arn}:ABUSECH_AUTH_KEY::" },
    ]

    environment = [
      { name = "SCANNER_MCP_URL",   value = var.scanner_mcp_url },
      { name = "MODEL",             value = var.model },
      { name = "HOME",              value = "/home/agent" },
      { name = "SLACK_TEAM_ID",     value = var.slack_team_id },
      { name = "SLACK_CHANNEL_NAME", value = var.slack_channel_name },
      { name = "SLACK_CHANNEL_ID",  value = var.slack_channel_id },
    ]

    logConfiguration = {
      logDriver = "awslogs"
      options = {
        "awslogs-group"         = "/ecs/triage-agent"
        "awslogs-region"        = var.aws_region
        "awslogs-stream-prefix" = "triage"
      }
    }
  }])
}

The secrets block is important. ECS pulls each secret from Secrets Manager at task startup and injects it as an environment variable. The secrets never appear in the task definition, the ECS console, or deployment logs. The execution_role_arn gets secretsmanager:GetSecretValue permission on the specific secret ARN. The task_role_arn is what the container itself uses at runtime - it only gets permission to write CloudWatch logs.

Networking. Unlike the Lambda triage agent (which doesn't need a VPC), the container agent needs VPC plumbing. Fargate tasks run in private subnets with no public IP. A NAT gateway in a public subnet provides outbound internet access so the container can reach the Anthropic API, MCP servers, and threat intel feeds. The security group allows only HTTPS egress:

resource "aws_security_group" "agent" {
  name_prefix = "soc-agent-"
  vpc_id      = aws_vpc.main.id

  egress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

No ingress rules. The container reaches out to APIs; nothing reaches in. A single NAT gateway keeps costs around $32/month. See the full network config for VPC, subnets, route tables, and the internet gateway.

Scheduling. A long-running ECS service (desired_count = 1) is the wrong pattern for an agent that runs once and exits. You'd pay for a container sitting idle between hunts. Instead, EventBridge schedules periodic RunTask calls:

resource "aws_cloudwatch_event_rule" "threat_hunt" {
  name                = "soc-threat-hunt-schedule"
  schedule_expression = "rate(6 hours)"
}

resource "aws_cloudwatch_event_target" "threat_hunt" {
  rule     = aws_cloudwatch_event_rule.threat_hunt.name
  arn      = aws_ecs_cluster.soc.arn
  role_arn = aws_iam_role.eventbridge.arn

  ecs_target {
    task_definition_arn = aws_ecs_task_definition.triage.arn
    task_count          = 1
    launch_type         = "FARGATE"

    network_configuration {
      subnets         = aws_subnet.private[*].id
      security_groups = [aws_security_group.agent.id]
    }
  }
}

Every 6 hours, EventBridge spins up a Fargate task. The agent hunts, posts to Slack, exits. No idle compute, no polling loop. The EventBridge role needs ecs:RunTask and iam:PassRole (for both the execution and task roles) - see the full Terraform for the IAM config.

Lambda handles triage. Containers handle longer-running work. Start with Lambda and graduate to containers when your agents need more room.

What Changes in Production

In Part 1, we talked about the human-AI partnership model: agents investigate, humans decide. That still holds. What changes is how you maintain oversight.

On your laptop, you watched the agent think, noticed wrong paths, killed it when it looped. In production, you need equivalents for each of those: structured JSON logs give you visibility into what the agent did and why. The dead letter queue catches failures automatically and preserves them for human review. The 15-minute timeout kills investigations that run too long.

These are the same guardrails you applied interactively, implemented as infrastructure. Logs instead of watching. A failure queue instead of manually re-running. A hard timeout instead of Ctrl-C. The principle is the same: give the agent room to work, and make sure you can see what it did.

Key Takeaways

Start with Lambda. One alert per invocation gives you clean logs, easy debugging, and a hard timeout as a trust boundary. Graduate to containers when you need longer runtime or stdio MCP servers.

The handler is thin; the prompt is thick. Your Lambda handler is plumbing. The system prompt - hypothesis generation, evidence collection, classification criteria, self-critique - is where the agent's reliability comes from. Invest your iteration time in the prompt.

Log structured JSON from day one. The console.log(JSON.stringify({...})) line in your handler is easy to overlook but foundational. Everything else - monitoring, debugging, accuracy tracking, cost analysis - builds on top of it.

Cost is driven by context accumulation, not model pricing. Opus can be cheaper than Sonnet on the same alert if it runs a more targeted investigation. The real lever is investigation depth. Use Sonnet or Opus, tune the system prompt, and watch your cached token counts.

Expect misclassifications and design for them. Your agent will get things wrong - sometimes confidently. Use structured output as an audit trail so reviewers can spot errors fast. The hypothesis_testing and ruled_out fields let analysts verify the agent's reasoning in seconds. Make failures visible and contained.

Every interactive guardrail needs a production equivalent. On your laptop you watched, re-ran failures, and killed loops. In production, logs, failure queues, and timeouts do the same work. Build these from day one.

What's Next

Part 1 built the agent. Part 2 deployed it. In Part 3, we'll make it learn. Every investigation produces structured output: what the agent found, how it reasoned, whether the analyst agreed. That output is a learning opportunity. We'll cover how to record investigation outcomes as memories, retrieve relevant ones at triage time, and feed analyst corrections back into the loop. The goal: an agent that gets better over time without changing the prompt.