How to Implement Wildcard Search in Google StackDriver Logging Filters


2 views

Many developers face frustration when discovering StackDriver Logging doesn't natively support wildcard characters (*, ?) in filter expressions. This becomes particularly problematic when:

  • Searching for logs with dynamic resource names
  • Filtering logs with variable prefixes/suffixes
  • Querying across multiple similarly-named services

While not perfect, these approaches can achieve wildcard-like functionality:

1. Using Regular Expressions

The ~ operator enables regex matching:

resource.type="gce_instance" AND logName~"projects/.*/logs/cloudaudit"

2. Multiple OR Conditions

For known variations, chain conditions with OR:

resource.type="k8s_container" AND (
  container.name:"service-a" OR
  container.name:"service-b" OR
  container.name:"service-c"
)

3. Advanced Logs Explorer Queries

In GCP's Logs Explorer, use:

logName:"projects/[PROJECT_ID]/logs/compute.googleapis.com%2Factivity_log"

When implementing these workarounds:

  • Regex patterns impact query performance
  • Multiple OR conditions may hit query length limits
  • Always scope queries with time ranges
  1. Structure log names consistently (e.g., service-name.log)
  2. Use standardized log formats (JSON recommended)
  3. Consider log routing for complex filtering needs

Google StackDriver Logging's advanced filters don't natively support wildcard characters (* or ?) in the same way that regular expressions do. This limitation often frustrates developers who need to search logs with partial matches or pattern-based filtering.

While you can't use literal wildcards, StackDriver supports several methods to achieve similar results:

1. Using Partial String Matching

For simple prefix/suffix matching, you can use the : operator:

resource.type="gce_instance" AND
logName:"projects/your-project-id/logs/syslog" AND
textPayload:"error*"

2. Regular Expression Matching

StackDriver supports RE2 syntax regular expressions with the =~ operator:

resource.type="gce_instance" AND
logName=~"projects/.*/logs/.*error.*" AND
textPayload=~"(?i).*connection.*failed.*"

3. Combining Multiple Conditions

For cases where you need OR-like wildcard behavior:

resource.type="gce_instance" AND
(textPayload:"connection error" OR
textPayload:"connection failed" OR
textPayload:"connection timeout")

For more complex scenarios, consider these approaches:

Log-Based Metric Filters

Create custom metrics that match patterns:

filter = 'resource.type="gce_instance" AND
textPayload=~"Failed to process .* file: .*"'

Exporting to BigQuery

For extensive wildcard searches, export logs to BigQuery where you can use full SQL wildcards:

SELECT * FROM project.dataset.cloudaudit_googleapis_com_*
WHERE protoPayload.methodName LIKE '%SetIamPolicy%'

Remember that complex pattern matching can impact query performance. For production systems:

  • Limit time ranges for pattern searches
  • Use more specific resource.type filters
  • Consider log exclusions for noisy patterns

Here's a complete example for filtering multiple error types across services:

resource.type=("gce_instance" OR "cloud_function") AND
(textPayload=~"(?i).*(error|fail|timeout|exception).*" OR
jsonPayload.message=~"(?i).*(error|fail|timeout|exception).*") AND
timestamp>="2023-01-01T00:00:00Z"