Your CI/CD pipeline is your production environment's root of trust

If an attacker compromises one laptop, they gain access to one user.

If they compromise one server, they gain access to one workload.

If they compromise your CI/CD pipeline, they gain access to every future deployment.

Attackers understand this.

Security teams should too.

In 2020, the SolarWinds attack showed what that access is worth. Attackers didn't breach thousands of organizations one at a time, but got into a single build environment and inserted malicious code into signed software updates. 

SolarWinds happened to be a vendor, so its blast radius reached its customers. But the mechanism doesn't require a vendor at all. Any organization running a CI/CD pipeline, whether a bank, a hospital, a retailer, or an internal IT team, hands an attacker the same leverage over its own production, even without downstream customers.

The lesson from SolarWinds wasn't that a vendor got breached. It was that modern CI/CD systems have become a root of trust for nearly every organization, regardless of industry or whether they ship software to anyone else.

Today's pipelines have access to source code, secrets, cloud credentials, artifact repositories, signing keys, and production environments. In many organizations, CI/CD systems are among the most privileged assets in the company. Yet, despite this, most security teams spend significantly more time monitoring production than they do monitoring the systems responsible for deploying it. Every CI/CD compromise tells a story:

• A pipeline was modified.
• A secret was accessed.
• An artifact was created.
• A deployment occurred.

Every one of those actions generated telemetry.

Every deployment leaves clues. The question is whether you can analyze them before they become production incidents.

Why CI/CD systems are a high-value target

Modern software delivery pipelines sit at the intersection of nearly every critical component of your infrastructure. A typical deployment runs through CI/CD tools like GitHub, GitLab CI, Jenkins, or CircleCI, but touches Vault, Kubernetes, cloud providers, artifact registries, and Infrastructure-as-Code platforms along the way.

A CI/CD pipeline is your organization's deployment mechanism. If production is where your business runs, CI/CD is how it gets there.

The pattern behind every pipeline compromise

SolarWinds, Codecov, Uber, and XZ Utils all demonstrated a common pattern:

1. Access was established.
2. Build or deployment systems were modified.
3. Secrets or credentials were accessed.
4. And from there, whatever the attacker came for: artifacts produced, software deployed, or simply the keys to everything the pipeline could reach.

‍Codecov is the cleanest illustration of step three. An attacker extracted a service account credential from a public Docker image and used it to modify the Bash Uploader script sitting in cloud storage. Between January 31 and April 1, 2021, the malicious version was active 108 separate times. Each one a short interval, each one doing real damage. Any CI job that ran the script while it was active shipped its environment variables, things like AWS keys, deploy keys, and API tokens, to the attacker. Codecov only found out because a customer noticed the script's checksum didn't match the published hash. Two months of intermittent credential harvesting, caught by a diligent user rather than by monitoring.

‍XZ Utils shows how patient step one can be. An account operating as "Jia Tan" spent roughly two years making legitimate contributions to a compression library bundled with nearly every Linux distribution, earned co-maintainer trust, and then used that position to land a backdoor in liblzma that reached sshd on distributions patching OpenSSH for systemd notification. Andres Freund found it in March 2024. He found it not through a security review, but because failed ssh logins on a Debian test machine were burning unexplained CPU, and logins had gotten about half a second slower. A performance anomaly, not an alert. The industry got lucky on timing: the backdoored versions had barely reached stable distributions when he published.

Uber's 2022 breach is the version of this that never reaches a customer. A Lapsus$-affiliated attacker got a contractor's corporate password (Uber's assessment: likely purchased after malware on the contractor's personal device exposed it), then wore down MFA with repeated approval prompts until one was accepted. From there, they moved through several employee accounts into elevated access on internal tooling. According to the attacker's own account and subsequent reporting, the pivot came from PowerShell scripts on an internal share holding hardcoded credentials for a privileged access management system — a detail Uber's own statement doesn't confirm, though Uber does confirm it rotated keys across many internal services in response. Uber states the attacker didn't reach production systems or customer data, and found no evidence the codebase was modified. What the attacker did get was internal: Slack messages, data from a finance invoicing tool, the HackerOne dashboard. Nothing was built. Nothing shipped. And the damage was entirely Uber's own.

That last one is worth sitting with, because it's the more common shape. Most organizations will never poison an update that reaches 18,000 customers. They don't have 18,000 customers downloading signed binaries. But every organization with a pipeline has secrets in it, and those secrets reach production.

Every step above generates logs. Notice how few of these were caught by looking at them: a checksum mismatch reported by a customer, a latency regression chased by a curious engineer. The pattern doesn't change much from incident to incident. Only the attacker changes, and how far down the list users bother or are able to go.

What telemetry should you collect?

At a minimum, security teams should collect from these pipeline components:

• GitHub Actions workflow execution and changes
• GitLab pipeline execution and administrative actions
• Jenkins job creation and configuration changes
• CircleCI user activity and pipeline execution
• Vault secret access
• Kubernetes service account activity
• CloudTrail IAM and infrastructure changes
• Artifact registry publishing and downloads

If an event could change what gets deployed to production, it should be logged.

Collecting CI/CD telemetry with Scanner

Scanner uses federated indexing to create a high-performance index over data wherever it resides. It connects directly to your object storage, SaaS platforms, and streaming systems such as Kafka, eliminating the need to move or duplicate your data.

Collecting telemetry is only half the solution. CI/CD investigations rarely involve a single event. Analysts need to correlate pipeline executions, secret access, administrative changes, cloud activity, and deployments across large time windows.

Scanner maintains an optimized index, typically requiring approximately 10-15% of the original uncompressed raw data footprint and allowing analysts to investigate months of telemetry with significantly lower latency and compute requirements.

Why Scanner is well suited for CI/CD investigations

CI/CD compromises are often slow.

The XZ Utils campaign ran for roughly two years before the backdoor landed. SolarWinds shipped backdoored updates for about nine months, from March to December 2020, before anyone caught it. Codecov's uploader was tampered with for two months before a customer spotted the checksum.

Security teams routinely ask:

• What changed six months ago?
• Has this runner ever existed before?
• When was this secret first accessed?
• Has this user ever deployed to production?

Every one of those questions is really the same request: search a very large volume of data, very quickly. The reason teams don't ask them isn't lack of curiosity but rather the fact that on most platforms a six-month lookback is a query you schedule and walk away from, or one you simply can't afford to run. So retention gets cut to 30 or 90 days, and the window that would have contained the first anomalous runner registration is gone before anyone thinks to look.

For years, defenders have been forced to ration visibility. Scanner exists to change this. Scanner's optimized indexing architecture is designed for exactly this type of workload. Analysts can pivot across large datasets without repeatedly paying the cost usually associated with scanning raw telemetry, which is what makes a year or more of CI/CD history a practical thing to query rather than a line item you cut to control cost. The historical window stops being a budget decision or a performance hurdle.

These investigations are increasingly being performed by autonomous agents and AI-assisted analysts.

Agents thrive on context, and are hungry for large amounts of rapidly accessible telemetry. Asking an agent to repeatedly scan hundreds of terabytes of raw logs could introduce both latency and cost.

An optimized index is not merely about storage/performance optimization. It is an optimization for the analysts and agents consuming the data.

Detection rules every security team should have

• Pipeline modifications
• New runner registrations
• Secret access anomalies
• Administrative changes
• Artifact anomalies
• Deployments outside normal behavior

Nobody should be writing these from scratch. Scanner's detection rules for GitHub ship out of the box and are published openly in a public repository. Scanner's users get them on day one, and everyone else can read them as plain queries to see what several of the categories above actually look like once written down.

That list looks short, but it expands fast. "Pipeline modifications" isn't one rule, it's a different rule per platform, per repo convention, per branch protection model. Same for secret access: Vault, cloud secret managers, and CI-native secret stores each need their own logic, and each needs variants for first-time access, off-hours access, and access by an identity that has never touched that path.

Then multiply that by deployment volume. A modern enterprise isn't deploying weekly, it's deploying continuously, dozens or hundreds of times a day, and every one of those deployments fans out into many underlying events: workflow triggered, runner assigned, dependencies fetched, secrets pulled, image built, artifact pushed, signature applied, manifest updated, rollout started, health check passed. A single deploy can generate dozens of log events across five or six systems. CI/CD telemetry is high-cardinality, high-volume, and bursty in a way that punishes detection architectures that evaluate rules serially or rely on index querying.

This is where the economics of detection matter more than the rules themselves. Scanner can evaluate thousands of detection rules in parallel against the data stream. Teams stop rationing detections, pruning the noisy-but-useful ones, skipping coverage for a platform because the rule budget is already spent. They can instead cover every CI/CD system they actually run, at the granularity the platform deserves.

The practical effect is that detection coverage becomes a function of what you want to catch, not what your pipeline volume will let you afford to evaluate.

That headroom is also what makes agent-authored detections practical rather than theoretical. Agents are good at precisely the work humans are slow at: reading months of pipeline telemetry and noticing that a particular runner label only ever appears on jobs that also pull a production secret, or that one service account's behavior changed shape three weeks ago and nobody filed a ticket. An agent can spot the pattern, describe the scenario, and propose the rule that would catch it next time.

But a proposed rule is worth nothing if adding it triggers a prioritization conversation. That's the trap most teams are in, the bottleneck isn't ideas about what to detect, it's the need to evaluate one more thing. Remove that constraint and the two capabilities compound: the agent surfaces patterns worth encoding, parallel evaluation makes encoding them nearly free, and the detection library grows from what the data actually showed you rather than from what someone thought to write down at onboarding.

Threat hunting with Scanner

01:58 AM  User "build-admin" logs in.

02:01 AM  GitHub workflow modified.

02:03 AM  Pipeline accesses Vault secret.

02:05 AM  Artifact signed.

02:08 AM  Production deployment begins.

02:13 AM  Deployment successful.

09:15 AM  Security incident declared.

Every line above is a log event. Every line could have been collected. Every line could have been correlated.

Here's the unsettling part: read individually, not one of those events is alarming. A workflow was edited. A pipeline pulled a secret. A deployment succeeded. That is a description of Tuesday. It's also a description of an on-call engineer patching a broken build at 2 AM, which happens in every organization that ships software.

What's notable is the shape. One identity modified the pipeline, pulled a credential, and shipped to production, all inside fifteen minutes, with no review interval between the workflow change and the secret access. Separation of duties collapsed into a single actor and a single quarter-hour.

The trick is scoping that shape narrowly enough to be worth alerting on. "Three CI/CD event types from one actor in fifteen minutes" describes every deployment your service account performs, it's a deployment detector, not a detection. What makes the timeline above unusual is narrower: a human account modified a workflow by pushing directly rather than merging a reviewed PR, and production shipped minutes later. In an organization with branch protection and code review, that combination should be rare and raise a flag:

actor.type:"user" and workflow.change_method:"direct_push" and (event_type:"vault.secret_access" or event_type:"github.workflow_modified" or event_type:"deployment.started")
| stats countdistinct(event_type) as distinct_event_types, min(@scnr.time_ns) as first_ns, max(@scnr.time_ns) as last_ns by actor.username
| eval window_minutes = (last_ns - first_ns) / 1000000000 / 60
| where distinct_event_types >= 3 and window_minutes <= 15
| table actor.username, distinct_event_types, window_minutes

Even scoped that tightly, it's a trigger rather than a verdict. Emergency fixes happen, and some of them will look exactly like this.

The fastest way to resolve that is also the least technical: ask the engineer. A message to the actor: “did you push this workflow change at 2 AM?” disambiguates most of these in minutes, far quicker than any amount of log analysis. Most of the time the answer is yes, there's a ticket behind it, and the alert closes.

The interesting case is when the answer is no, or nobody can account for it. That's the moment the question changes, and it becomes a much bigger one than "is this alert real." It becomes: how long has this been happening, and what did it already touch?

actor.username:"build-admin" and event_type:"github.workflow_modified"
| stats count() as modifications, min(@scnr.datetime) as first_seen, max(@scnr.datetime) as last_seen by repo.name, workflow.path
| table repo.name, workflow.path, modifications, first_seen, last_seen

Run that over twelve months rather than twelve hours. If first_seen comes back as tonight, the exposure is bounded and you're dealing with an attempt you caught early. If it comes back as five months ago, you are not investigating an alert anymore, you are scoping an incident that has been running since March, and every deployment in between is now in question. Run the same pattern against vault.secret_access by secret path and you learn which credentials were reachable during that period, which tells you what has to be rotated.

That second outcome is the one worth designing for, because it's exactly what happened in each case mentioned earlier in this post: Codecov, SolarWinds, and XZ Utils all had months or years of quiet access before anyone looked. In none of those cases was the hard question "is this alert real". By the time anyone asked, the answer was obvious. The hard question was what has already been lost, and answering it requires history that goes back further than the compromise.

Short retention would have answered the first question and not the second. The trigger tells you where to look; history tells you how bad it is.

The malware wasn't deployed by an attacker. It was deployed by your own pipeline.

The pipeline is already monitored, but not for this

Here's the thing that makes CI/CD monitoring different from most security gaps: the telemetry usually already exists, and in many cases someone is already looking at it.

Engineering organizations monitor their pipelines closely. They track build durations, failure rates, queue times, flaky tests, deployment frequency, and lead time to production. There are alerts. There is genuine operational maturity here.

But all of it is instrumented for efficiency, not for security. A build that takes 40% longer than usual raises a flag. A build that quietly pulled an extra dependency, or ran a step that wasn't in the committed workflow file, or reached for a secret it had never touched before, does not because nobody wired that question into the framework.

The result is that CI/CD often ends up in an awkward middle ground: rich in telemetry, actively watched, and still effectively unmonitored from a security standpoint. Two teams look at the same event stream and only one of them is asking whether it was supposed to happen.

Most organizations have mature strategies for monitoring production systems. Far fewer have mature strategies for monitoring the systems responsible for deploying them.

If your CI/CD pipeline is compromised, your infrastructure may simply report that a perfectly legitimate deployment occurred.

Final thoughts

The next major CI/CD compromise won't begin in production.

It will begin in a repository, a pipeline, a runner, or a build system.

The organizations that respond fastest won't necessarily be the ones collecting the most telemetry. They'll be the ones capable of asking questions across months of data and receiving answers in seconds, and of running a large number of detections against everything moving through their pipelines rather than the handful they could afford to prioritize.

Telemetry is table stakes. The ability to interrogate lots of it quickly is the differentiator.