FloQast Optimizes High-Volume Security Log Analysis and Threat Detections Using Scanner

Before implementing Scanner, FloQast struggled with several key infrastructure limitations. Their existing SIEM solution from Panther made ingesting certain log sources prohibitively expensive, particularly for high-volume sources like EDR and VPC Flow Logs. This cost barrier created significant visibility gaps in their security operations, as critical log sources remained effectively inaccessible for analysis.

The team also faced technical challenges with data analysis across different sources due to disparate storage locations and formats. Their existing infrastructure created data silos that complicated investigations and limited their ability to derive insights from their log data.

Additionally, their current setup wasn't providing the scalability needed for their growing operations.

After evaluating multiple solutions including Datadog SIEM and Cribl Search, FloQast implemented Scanner. The selection was based on Scanner's technical architecture and approach to data management.

Braden King, Security Engineer at FloQast, explained the need for more log source coverage, sharing that "several of our high volume sources were not returning enough value in Panther, but we wanted to have logs available for investigation and ideally some basic detections."

Technical Implementation

The deployment process centered around infrastructure as code using Terraform for consistent and repeatable deployments. The team implemented flexible S3 import rules to manage log ingestion efficiently, while role-based access control provided granular security management. Detection-as-code implementation with GitHub CI/CD integration enabled version control and collaborative development of detection rules.

Data Architecture

Scanner's approach to log management introduced several technical advantages to FloQast's infrastructure. The architecture allows log data to remain in FloQast's own S3 buckets, ensuring complete data sovereignty. The query engine operates without requiring rigid schema definitions, allowing for more flexible data analysis. Selective log ingestion capabilities minimize data duplication and associated costs. The system's flexible data structures readily adapt to changing log formats, reducing maintenance overhead.

Search Performance

Scanner's core search engine leverages a specialized inverted index architecture designed specifically to work with data in S3, enabling fast query performance at scale. For high-speed threat hunting, needle-in-haystack searches—like scanning for specific IP addresses, domains, or other indicators of compromise—can process 100TB of uncompressed log data in just 10 seconds.

The system automatically searches across all log sources simultaneously, eliminating data silos and enabling security engineers to quickly correlate activity across EDR logs, VPC flow logs, and other sources with a single query. This unified search capability is particularly valuable during incident investigations, where rapid cross-source analysis can reveal the full scope of potential security events.

Enhanced Visibility

Scanner significantly improved FloQast's security visibility, particularly for EDR logs which typically have a 30-day retention limit in manufacturer portals.

The team gained comprehensive access to historical EDR logs beyond the standard 30-day window, enabling deeper historical analysis with a new retention of 12+ months. VPC flow log analysis became practical and cost-effective, providing insights that were previously unavailable. The system opened up access to log sources that were formerly impractical to analyze, enhancing their threat hunting and investigation capabilities.

Technical Operations

The implementation delivered substantial operational improvements to FloQast's security infrastructure. "The system has already proved its value by successfully detecting changes to the public role in a Snowflake deployment, demonstrating the effectiveness of their new detection capabilities," Braden King shared.

Scanner provided robust support for Snowflake security monitoring and efficiently processes high-volume log sources that were previously cost-prohibitive to analyze. Detection capabilities now extend to previously inaccessible data sources, significantly expanding their security coverage.

With Scanner's schemaless data lake approach, investigation workflows have been streamlined, allowing analysts to conduct effective investigations without requiring detailed knowledge of field names.

Infrastructure Benefits

Scanner's architecture transformed FloQast's log management infrastructure.

• Data ingestion costs were reduced through selective processing mechanisms that eliminate unnecessary data duplication.
• Storage requirements decreased as Scanner's optimized indexing approach occupies less space than traditional solutions.
• Operational efficiency improved through better search performance and more intuitive data access.
• Infrastructure management overhead was minimized as Scanner's architecture automatically scales with data volume.
• Data sovereignty was maintained by keeping logs within FloQast's own S3 buckets.

The integration process leveraged several key technical components that ensured a smooth deployment. As Braden King explained about Scanner's detection rule system, "It's pretty intuitive and even going to DaC using YML was close enough to Sigma that it just makes sense. The query language is very logical and easy to use. It just works."

Terraform modules provided consistent and repeatable deployments across environments. The S3 import rule configuration system offered flexibility in managing log ingestion paths. The RBAC implementation enabled granular access control while maintaining security. The system's support for detection-as-code in both the UI and GitHub workflows facilitated efficient detection development and management.

FloQast continues to expand their use of Scanner across different teams. The security team is focusing on expanding detection coverage to additional log sources and enhancing their investigation capabilities through advanced query techniques.

Work continues on optimizing log ingestion processes and developing more sophisticated detection-as-code implementations to address emerging threats.

The implementation revealed several key technical insights about modern security infrastructure.

• The value of schemaless architectures for log management became evident through significantly reduced maintenance overhead and increased flexibility.
• Maintaining data sovereignty while enabling advanced analysis proved crucial for compliance and security requirements.
• The importance of flexible query capabilities for security investigations was demonstrated through improved investigation speeds and depth.
• Infrastructure-as-code approaches to security tooling showed clear advantages in deployment consistency and maintenance efficiency.

FloQast's experience with Scanner demonstrates the speed of Scanner's engineering team and their dedication to solving problems for users as quickly as possible. As Braden King observed, "Seeing the development of the product over the last 6 months or so was incredibly impressive. It went from a 'that's cool but I need XYZ to consider it' to 'TAKE MY MONEY'. I think there are a lot of pain points in detection and response that Scanner will help to address."

FloQast's implementation demonstrates how modern data lake architectures can transform security operations - enabling them to search years of high-volume EDR and VPC flow logs in seconds while significantly reducing costs, a combination that was previously impossible with traditional SIEM platforms.