How to build custom agent skills for security operations

AI agents are quickly becoming part of everyday security operations, helping teams with alert triage, threat hunting, log analysis, and detection engineering. Connecting an agent to your security data lake or SIEM through a Model Context Protocol (MCP) server gives it efficient, direct access to search your logs. What the connection can’t carry is the knowledge your team has built up about how that access should be used: which indexes are expensive to search, how a detection rule should be tested before it ships, and what a finished triage report needs to include.
Agent skills are how you hand that knowledge to an AI agent. A skill is a set of written instructions, packaged as a file the agent loads when it’s relevant and that brings your team’s conventions into each agentic workflow. In this post, you’ll see how Scanner’s MCP server and agent skills work together, walk through three of the twelve open-source skills we’ve published for SOC automation, learn what we found while building them, and write a custom skill of your own that you can adapt for your security team.
MCP gives access. Agent skills give judgment.
Scanner’s MCP tools are built around a context economy, where each call returns just enough for the agent to take its next step without flooding the context window:
get_scanner_contextreturns a condensed orientation, with a docs table of contents, a compact index list, and the available source types, so the agent can get its bearings without loading everything up front.get_docspulls detail on exactly one topic at the moment it’s needed, whether that’s query syntax, aggregations, or common mistakes to avoid.get_top_columnsreturns the most frequent column names for just the indexes the agent is about to query, and because it’s backed by metadata rather than a scan, it costs nothing in scanned bytes.execute_queryreturns a summary of field names, top values, counts, and distributions along with aresult_handle, andfetch_query_resultsthen pulls back only the rows and fields the agent actually needs.
In practice, a query that matches 5,000 S3 access events doesn’t have to land in the agent’s context as 5,000 rows. The agent reads the summary, spots the 50 events that look wrong, and fetches just those 50 with the three fields that matter, which saves nearly all of the tokens without compromising the signal you’re gaining.
What agent doesn’t know is everything specific to your environment: which of your indexes are large and which are cheap to search, what backtest window your team considers reasonable, or that every new detection starts in Staging until a person promotes it to Active.
That kind of knowledge usually lives in the head of whoever has been running the system longest, and an agent you’ve just wired up is often not given this important context. A skill gives this knowledge somewhere to live so that agents can use it: a markdown file with YAML frontmatter that loads when the agent decides it’s relevant. The skills you create can carry your call order, your conventions, and the judgment calls you’ve already made so the agent doesn’t have to rediscover them.
What we built
We’ve published 12 skills in scanner-inc/agents, covering the main areas of security work: SOC operations, detection engineering, log ingestion, and reporting. Rather than walk through all of them, we’ll focus on three that show the range of what a skill can do, from guiding an open-ended investigation, to enforcing a multi-step engineering workflow, to taking an alert from ID to verdict.
investigate: plan before you query
investigate is the skill to reach for when you have an open-ended question about your logs and don’t know where to find the answer. You ask in plain language, such as “are there any failed Okta logins from new countries?” or “did anyone exec from a public S3 bucket today?”, and the agent works out which indexes to search, writes the queries, and reports back what it found.
The querying itself isn’t what makes the skill useful, but rather it’s the structure the skill imposes before the first query runs that makes the difference. The agent first restates your question to itself in a single line, so that it’s working from a precise version of what you asked rather than a loose one, and if the question is too ambiguous to restate, it asks you a clarifying question instead of guessing.
Next, the agent drafts a plan. To ground the plan in your actual environment, it’s limited at this stage to Scanner’s read-only MCP tools, which return information about your indexes and fields without searching any log events. It then prints three to six bullets, each naming an action and a target, so you can see which index or data source it intends to consult and why. A plan from investigate looks like this:
- Query @index=global-cloudtrail for all lambda:* events in the past 6 hours, grouped by eventName
- Check EventBridge PutRule and PutTargets calls in the same window to see how the Lambda was wired to a schedule
- Cross-reference source IP 31.41.59.26 against threat intelThe skill gets plans into this shape by showing the agent examples. Its instructions include a plan like the one above alongside a rejected version of the same step, written with the full query syntax that belongs in execution rather than planning:
- Query @index=global-cloudtrail | stats count() as events by eventName, userIdentity.arn | where events > 5 over the last 6 hours to find anomalous Lambda callsEvery bullet in the preferred plan can be read at a glance. Because the plan is printed before any queries run, you can spot a wrong direction within seconds and redirect the agent before it goes further. Pairing a preferred example with a rejected one is an effective way to steer an agent, and it’s a technique worth borrowing when you write your own skills.
Once the plan is out, the agent executes it. Before running any aggregation that covers more than a few hours, it checks how much data the target index holds per day and sizes the time window to fit your team's scan budget, so a question phrased as “over the last week” gets an answer scoped to what's reasonable to search, along with a note explaining the narrower window. The final report scales with the question, too: a small question like “did this user log in today?” gets a short answer rather than a report padded to fill a template.
write-detection: the loop a prompt can’t run
write-detection takes a description of something you want to catch, such as “alert when someone disables CloudTrail logging,” and turns it into a working Scanner detection rule. Because writing a good detection is an iterative process, it’s challenging to pack this into a single prompt.
This skill walks the agent through it in eight steps:
- Restate the hypothesis so it’s clear exactly what behavior the rule should catch.
- Discover the relevant log schema through the MCP tools rather than guessing at field names.
- Run the first filter clause on its own against real data, to confirm it matches something before building on it.
- Draft the rule as YAML.
- Write two to four tests using real events sampled from your logs, including at least one event the rule should not match, so the rule is checked against what your data actually looks like.
- Validate the rule and run the tests locally with
scanner-cli validateandscanner-cli run-tests. - Backtest the rule against historical data and report how often it would have fired per day. Because Scanner stays fast on rare-event searches, a highly selective rule can be backtested 30 to 90 days, or more, while a broader rule gets a window sized to the scan budget. If a Medium or higher rule would fire more than about ten times a day, the agent proposes ways to tune it before hand-off.
- Hand the finished rule off for review.
Several of the conventions built into those steps are choices rather than requirements. Scanner’s MCP is deliberately flexible about how detections get created, routed, and deployed, because every team runs detection engineering a little differently, and the product shouldn’t force one workflow on all of them. Our team settled on a specific set of conventions over time, shaped by lessons like not wanting a brand-new rule to page someone at night or a change to go into production without review. Those choices are ours to make, so the skill is where we write them down for the agent:
- Every new rule starts in Staging. Staging lets you watch how a rule behaves against real traffic, by reviewing its matches in the
_detectionsindex, before it’s promoted. A rule that looks right on paper can still fire far more often than expected once it meets production data, so the skill’s hand-off tells you to watch a new rule for a few days first. Promoting it to Active is a decision for a person to make once they trust the results, so the skill never makes that change on its own. - Severity determines whether a rule produces an alert or a signal. Rules rated Medium and above send alerts to your notification destinations. Rules rated Low or Informational send nothing on their own, because their job is to produce signals that feed correlation rules, which look for patterns across many low-level events. Without that split, every low-confidence rule would land in front of a security engineer, and alert fatigue would bury the ones that matter.
- Hand-off always goes through the GitHub sync flow. The agent commits the rule and pushes it to the main branch, and the Scanner GitHub app runs validation and unit tests server-side before syncing the rule into the live environment, so a rule only reaches production once it passes. That keeps every detection in version control, with a history of who changed what and a validation gate in front of production.
Your team’s conventions will look different, and that’s the point. Whatever rules you’ve settled on through experience are exactly what a skill should carry, because they’re the part no tool can infer on its own.
triage-alert: from alert ID to verdict
triage-alert handles one of the most common jobs in a SOC: an alert has fired, and someone needs to decide if it matters. Here, you give the agent an alert ID by typing /triage-alert followed by the ID, and the skill supports the agent through a full investigation resulting in a written verdict.
Before running any queries, the agent briefly explains what it's about to do and confirms the investigation is read-only.
The first step is finding the alert. Most of the time that’s an alert that fired minutes ago, but triage also happens in hindsight: a new threat-intel report names an IP address you saw last month, or an active incident sends you back to check whether an earlier alert was part of the same activity. The skill tells the agent to search the detections index across the last 180 days. If nothing comes up, the scope widens to a full year. If nothing still comes up, the skill tells the agent to stop there and to deliver a “not found” response rather than invent an answer. It’s important to tell an agent explicitly what to do when it comes back empty in order to keep it honest.
Once the alert is in hand, the agent works through the investigation in five phases:
- Generate hypotheses about what could explain the activity, including both innocent and malicious explanations.
- Collect evidence from the surrounding logs, anchored to the time window in which the detection fired.
- Classify the alert as BENIGN, SUSPICIOUS, or MALICIOUS, along with a confidence level for that call.
- Critique its own reasoning before committing, checking whether the evidence actually supports the verdict or whether an alternative explanation was dismissed too quickly.
- Write the report using a fixed template, so every triage reads the same way regardless of which alert it covers or who requested it.
The full procedure for those phases lives in a separate reference file rather than in the main skill file, so it loads into the agent’s context only once an alert has actually been found. That keeps the skill light until the moment more detail is needed.
triage-alert also shows how skills can lean on each other. When an investigation turns up an IP address or file hash worth checking, the skill hands it to lookup-ioc, a separate skill that queries several threat-intelligence sources in parallel and returns a single combined result. Once the report is finished, the agent offers to render it with report-as-html, a skill whose job is to provide a clean and shareable template for every report.
What we learned building skills
Shared references beat repetition. Every Scanner skill that runs queries needs to manage query cost (how much data each search scans) so that an agent doesn’t launch a query far broader than the question requires. Rather than repeating those rules in each skill, we wrote them into a shared file, shared/query_cost_control.md, and pointed our other skills at it. Keeping the rules in one place also means that if they ever need to change, there’s a single file to update instead of one in every skill.
Progressive disclosure keeps skills from crowding the agent’s context. A common worry with skills is that loading detailed instructions will fill up the agent’s context window and leave less room for the actual work. Skills avoid this by loading in three stages, much like Scanner’s MCP tools do. The skill’s name and short description are always available to the agent, letting it know the skill exists. The full instructions load only when the agent decides the skill is relevant. Any supporting files the skill bundles load only when a step actually calls for them. Our write-vrl skill, which writes and tests log transformation code, shows how far this can go. It always loads two short reference files covering Scanner’s conventions, but its library of production transformations for common log sources stays out of context until the agent finds one that matches the logs in front of it. The skill also bundles the full Elastic Common Schema, a 2,116-field list of standard field names, and tells the agent never to read it at all. Instead, the agent searches the file from the command line whenever it needs to confirm a single field, so the schema is available on demand without entering the agent’s context. A good rule of thumb is to keep the main skill file under 500 lines and move deeper material into a references/ folder.
The description decides when a skill gets used, so write it around the situations that should trigger it. An agent chooses whether to load a skill based almost entirely on its description, and in our experience agents lean toward answering directly when a description is vague about when it applies. The descriptions that worked best for us state plainly when to use the skill and list the actual phrases people type when they need it, such as “triage this alert” or “look at alert 123.” It helps just as much to say what a skill is not for. The description for migrate-detection, which translates a single detection rule from another platform such as Splunk, Sigma, or Elastic into Scanner’s format, states explicitly that it doesn’t crawl vendor repositories or recommend which rules are worth porting. Deciding what to port is recommend-detections’ job, so spelling that out keeps the agent from reaching for the wrong skill.
Skills work best when they know about each other. Before write-detection starts drafting anything, it checks whether the request is really a different job, such as migrating an existing rule, tuning a rule that fires too often, or writing a correlation rule that combines several signals, and it hands the work to the matching skill if so. recommend-detections takes the same approach from the other direction: it produces a prioritized list of recommendations, from new rules to write to existing rules to tune or migrate, and each item ends with the exact command for the skill that would act on it. A set of focused skills that can pass work between them has worked better for us than one large skill trying to cover every Scanner task at once, because each skill can be written, tested, and improved on its own, and a change to how we tune detections never risks breaking how we triage alerts.
Build your own skills
Scanner’s skills are a useful starting point, but you can make a bigger impact by building skills that capture how your team works. The steps below outline one way to approach writing one for your team.
- Start with what you find yourself re-explaining. Rather than asking what an agent could do in general, look for the instructions you give it over and over: the format your tickets follow, which details belong in a closure note, or who gets paged for which kind of alert. Anything you’ve typed into a prompt more than a few times is a good candidate for a skill.
- Keep it scoped to a single job. A skill that produces one kind of output is easier for the agent to recognize as relevant and is easier for you to test, so if a skill starts covering two different tasks, it’s usually a sign it should be split in two.
- Write the frontmatter carefully. The frontmatter is the short block at the top of the file containing the skill’s name and description, and the description is what the agent reads when deciding whether or not to use the skill. Describe what the skill does, list the phrases people actually use when they need it, and say what it isn’t for.
- Write the body as a procedure. Lay out the steps in the order the agent should take them, along with your team’s conventions and the mistakes you’ve already seen it make. Where you can, show the agent an example of the output you want next to one you don’t. If your skill queries Scanner, point the agent to
get_docs("best_practices_and_mistakes")rather than restating the query language yourself, so the skill remains up to date with our documentation. - Test it against a real failure. Find a past conversation where the agent got something wrong, such as an invalid query, a field name that doesn’t exist, or a rule file that failed validation, and run the same request again with the skill loaded. If the same mistake comes back, the skill needs to be more specific about that step.
- Keep improving it. A skill is never really finished; each new mistake the agent makes points to the next instruction worth adding.
Here’s a complete skill draft to show how these steps look in practice. It doesn’t depend on anything specific to Scanner’s internals, and most security teams have some version of this task:
---
name: shift-handoff
description: Write the end-of-shift handoff note for the SOC, formatted to
our team's conventions and ready to paste into the #soc-handoff Slack
channel. Use when the user types /shift-handoff, says "write the
handoff", "end of shift note", "handoff for nights", or asks to
summarize what happened this shift for the next team. Also use when a
shift is ending and the user asks what the next shift needs to know.
Does NOT triage alerts or run investigations -- it summarizes work that
already happened. If the user wants an alert investigated, route to
/triage-alert instead.
---
# shift-handoff
## Inputs
Ask for the shift window if it isn't obvious (default: the last 8 hours).
Call `get_scanner_context` if you haven't already this session to confirm the name of the detections index (commonly `_detections`), then pull open and recently closed alerts from it. Keep the time range to the shift window. This is a small index, so query cost is not a concern here.
## Team standards
- **Lead with what's still open.** The next shift reads the first three
lines and nothing else if it's busy. Anything unresolved goes at the top.
- **Every open item names an owner.** If nobody owns it, say "UNOWNED" --
do not leave it blank and do not guess.
- **Ticket IDs, always.** Format `SOC-1234`. Never describe an item without
its ticket.
- **Severity words match Scanner's**: Critical / High / Medium / Low /
Informational. Don't substitute "urgent" or "P1".
- **No speculation.** If a verdict isn't in the ticket, write "pending" --
the handoff is a record, not an analysis.
## Format
Keep the note under 25 lines and follow this structure exactly:
*Shift handoff -- <date>, <start>-<end>*
*Still open*
• SOC-1234 · High · <one-line summary> · Owner: <name or UNOWNED>
*Closed this shift*
• SOC-1229 · Medium · <one-line summary> · BENIGN
*Watch for*
• <anything the next shift should keep an eye on, or "Nothing notable">
## Mistakes to avoid
- Don't re-triage. If an alert was closed BENIGN, report it closed BENIGN
even if you'd have called it differently.
- Don't pad. A quiet shift produces a three-line handoff, and that's
correct.
- Don't include alert volume stats unless something anomalous happened --
the daily posture report already covers those.
None of the rules in this skill are complicated, but together they capture conventions that would otherwise have to be explained to the agent at the end of every shift.
Skills deserve the same care as the rest of your security tooling. They should be version-controlled, reviewed before changes go in, and owned by someone on the team, just like your detection rules.
Wrapping Up
An agent connected to Scanner through MCP already has an efficient way to search your logs. What it doesn’t have is everything your team has learned about how to use that access well: which queries are worth running, how a detection should be built and shipped, and what a finished report or handoff should look like. Skills are where that knowledge goes, so every agent starts with it instead of having it explained again in each conversation.
The twelve skills we’ve built are available at github.com/scanner-inc/agents. If you’re deciding where to begin, investigate is a solid first choice, since it works for any question about your logs and its plan-before-query step shows how a skill changes an agent’s behavior. The detection-engineering skills are worth adding once your team’s rule conventions are settled enough to write down, because a skill that encodes a convention your team is still debating will only spread whichever version it happened to capture.
The skills are packaged as a Claude Code plugin and expect Scanner MCP to already be configured, and most of them also need your Scanner API URL, API key, and Team ID set as environment variables. The repository README covers where to find each value. Because the plugin marketplace lives in the repository’s skills/ folder rather than its root, the usual one-line install won’t find it, so clone the repository first:
git clone https://github.com/scanner-inc/agents.git
Then, from inside the cloned agents folder, run these in Claude Code:
/plugin marketplace add ./skills
/plugin install scanner-soc-skills@scanner-soc-skills
The skills load automatically, and running /triage-alert with any alert ID makes a quick first test.
If you want to get your team’s own conventions written down, one approach is a short internal hackathon: everyone brings one thing they find themselves re-explaining to an agent, and leaves with a working skill that captures it. The skills themselves are the obvious result, but the list of things people were re-explaining may be the more useful one, since it shows exactly where your team’s knowledge has been living outside any system.
