Policies & Processing
A Policy is a reusable set of data processing rules that can be attached to one or more Pipelines. This allows you to enforce centralized compliance rules, like PII redaction, across all of your log streams.
Event Filter
Event Filter rules evaluate your events in real-time to determine if they should be dropped before any heavy processing occurs. If dropped, they are forwarded to the Filter Sink (if configured) or permanently discarded.
1. Event Sampling
Event Sampling randomly keeps a defined percentage of events and discards the rest. This is ideal for high-volume, low-value events where you only need a statistical representation of the data.
2. Event Filter Rules
Filter events matching specific conditions. This allows you to discard entire events if they match a regular expression or string comparison. This is incredibly useful for eliminating structural noise like ELB health checks, debug messages, or spammy heartbeat requests.
When parsing rigid event formats like Apache Access, Nginx, or Syslog, the exists conditional operator is disabled. Since these formats have a static, predictable structure, a field either always exists or never exists, making dynamic existence checks unnecessary.
Data Processing
Data processing rules allow you to mutate, scrub, and reshape your events before they are forwarded to their final sinks. When a Policy is attached to a Pipeline, its processors run immediately after filtering and sampling rules are applied.
1. Transform Fields
Transform Fields allows you to rename, flatten, inject or delete specific JSON keys. This drastically reduces payload size before egress by dropping redundant structural data.
{"op": "delete", "target_field": "kubernetes.labels"},
{"op": "delete", "target_field": "unnecessary_metadata"}
]
2. Redact Sensitive PII Data
The PII Scrubber scans your payloads and automatically redacts sensitive information like Social Security Numbers, Credit Cards, or custom regex patterns before the events are sent to downstream providers.