Pipelines & Routing
Learn how to construct pipelines to ingest your log data, apply sampling or dropping rules, and route traffic to your Primary and Filter sinks.
What is a Pipeline?
A pipeline defines a unique ingestion path on your Agent (e.g., /ingest/pipe_123). It acts as a set of instructions telling the Agent how to handle the events that arrive at that path. Each pipeline is configured with a Primary Sink, an optional Rejection Sink, and an ordered list of Routing Rules (Sampling & Filtering).
Pipelines can automatically parse a variety of inbound event formats natively, including Logfmt, Apache Access / Error, Nginx Access, and Syslog (RFC 3164).
Pipeline Authentication
By default, pipelines accept all traffic sent to their ingest path. For production deployments or pipelines exposed to the public internet, you can enable Pipeline Authentication to secure your ingest paths.
When enabled, the Agent enforces a strict token check. Data forwarders must provide a valid Authorization: Bearer <token> header. When rotating tokens, SRVANT automatically provisions a 24-hour grace period where both the old and new tokens remain active to ensure zero downtime during credential swaps.
Rate Limiting
RPS Caps
Protect your downstream systems and control costs by enforcing a maximum Requests Per Second (RPS) limit on your pipelines. When traffic exceeds the configured threshold, the Agent will automatically reject excess requests with a 429 Too Many Requests response, ensuring your infrastructure remains stable during unexpected traffic spikes.
Sinks & Filter Routing
Every pipeline must have a Primary Sink, which is the default destination for all accepted traffic.
Primary Sink
The primary destination for your log data (e.g., Datadog, Splunk, Elasticsearch). If a log passes through all routing rules successfully, it will be forwarded here.
Filter Sink (Filter-Fail)
The Filter Sink receives any event that is intentionally dropped by your pipeline's routing rules (e.g., via Sampling or Filter Drop). This allows you to route "dropped" structural noise to cheap, cold storage like AWS S3 or GCP Cloud Storage for compliance purposes, without paying Datadog ingestion fees.