Filtering & Processing
Learn how to use routing and processing rules to shape your telemetry data before it egresses your network. Drop structural noise, sample events, and route traffic to the right destinations.
What is a Pipeline?
A pipeline defines a unique ingestion path on your Gateway (e.g., /ingest/pipe_123). It acts as a set of instructions telling the Gateway how to handle the events that arrive at that path. Each pipeline contains an ordered list of Processors and a list of Destinations.
Pipelines can automatically parse and process a variety of inbound event formats natively, including JSON, Logfmt, Raw Text, Apache Access / Error, Nginx Access, and Syslog (RFC 3164). Processors dynamically adapt to the selected event type to ensure robust transformations.
Pipeline Authentication
By default, pipelines accept all traffic sent to their ingest path. For production deployments or pipelines exposed to the public internet, you can enable Pipeline Authentication to secure your ingest paths.
When enabled, the Gateway enforces a strict token check. Data forwarders must provide a valid Authorization: Bearer <token> header. When rotating tokens, SRVANT automatically provisions a 24-hour grace period where both the old and new tokens remain active to ensure zero downtime during credential swaps.
Rate Limiting
Protect your downstream systems and control costs by enforcing a maximum Requests Per Second (RPS) limit on your pipelines. When traffic exceeds the configured threshold, the Gateway will automatically reject excess requests with a 429 Too Many Requests response, ensuring your infrastructure remains stable during unexpected traffic spikes.
Data Routing
Routing rules evaluate your events in real-time to determine if they should be dropped or sampled. The SRVANT Gateway evaluates these rules sequentially against every incoming event before any data processing occurs.
1. Sampling
The Sampling processor randomly keeps a defined percentage of events and discards the rest. This is ideal for high-volume, low-value events where you only need a statistical representation of the data rather than every single occurrence.
2. Routing
The Filter/Drop processor allows you to discard entire events if they match a specific regular expression. This is incredibly useful for eliminating structural noise like ELB health checks, debug messages, or spammy heartbeat requests.
When parsing rigid event formats like Apache Access, Nginx, or Syslog, the exists conditional operator is disabled. Since these formats have a static, predictable structure, a field either always exists or never exists, making dynamic existence checks unnecessary.
Data Processing
Data processing rules allow you to mutate, scrub, and reshape your events before they are forwarded to their final destinations.
1. Field Mutation
Field Mutation allows you to rename, flatten, inject or delete specific JSON keys. This drastically reduces payload size before egress by dropping redundant structural data.
{"op": "delete", "target_field": "kubernetes.labels"},
{"op": "delete", "target_field": "unnecessary_metadata"}
]
2. PII Scrubber
The PII Scrubber scans your payloads and automatically redacts sensitive information like Social Security Numbers, Credit Cards, or custom regex patterns before the events are sent to downstream providers.