VictoriaMetrics LogsQL | Courtesy of Grok

logsql-tutorial.md

Comprehensive LogQL Commands Tutorial

Introduction

LogQL (LogsQL) is a query language used within VictoriaMetrics to search, filter, and analyze log data. It is designed for efficient log retrieval, enabling users to extract specific log entries based on time, content, metadata, or numeric calculations. This tutorial provides a detailed overview of LogQL commands—primarily filters and mathematical operations—along with examples to help you construct effective queries. The content is structured for both beginners and advanced users, serving as a reference for future use.

Basic Structure of a LogQL Query

A LogQL query consists of:

• Filters: Criteria to select logs, such as time ranges, words, or field values. At least one filter is required.
• Optional Actions: Operations like stats, fields, or eval applied to filtered logs, separated by the pipe (|) operator.

Example:

_time:5m error

• _time:5m: Selects logs from the last 5 minutes.
• error: Filters logs containing the word "error".

Filters can be combined using logical operators (AND, OR, NOT) to create complex queries. Field-specific filters use the format field_name:filter, e.g., log.level:error.

Available LogQL Filters

LogQL offers a wide range of filters to narrow down log searches. Below is a comprehensive list, organized in a table for clarity, with examples and descriptions. These filters are the core "commands" of LogQL, as they define how logs are selected.

Filter Type	Syntax Example	Description
Time Filter	_time:5m	Filters logs by timestamp range (e.g., last 5 minutes). Supports durations, absolute times, and offsets.
Day Range Filter	_time:day_range[08:00, 18:00)	Filters logs within a daily time range (e.g., 8 AM to 6 PM UTC).
Week Range Filter	_time:week_range[Mon, Fri]	Filters logs within a weekly day range (e.g., Monday to Friday).
Stream Filter	{app="nginx"}	Filters logs by stream labels (e.g., application name).
_stream_id Filter	_stream_id:0000007b000001c850d9950ea6196b1a4812081265faa1c7	Selects logs by unique stream ID.
Word Filter	error	Searches for specific words in the log message. Case-sensitive by default.
Phrase Filter	"ssh: login fail"	Searches for exact phrases in the log message.
Prefix Filter	err*	Matches words or phrases starting with a specific prefix (e.g., "err").
Substring Filter	~"ampl"	Matches substrings using regular expressions.
Range Comparison Filter	response_size:>10KiB	Compares numeric, IPv4, or string values (e.g., greater than, less than).
Empty Value Filter	host.hostname:""	Matches logs without the specified field.
Any Value Filter	host.hostname:*	Matches logs with non-empty values for the field.
Exact Filter	log.level:="error"	Matches exact field values. Case-sensitive by default.
Exact Prefix Filter	="Processing request"*	Matches field values starting with a prefix.
Multi-exact Filter	log.level:in("error", "fatal")	Matches field values that are one of the specified values.
contains_all Filter	contains_all(foo, "bar baz")	Matches logs containing all specified words or phrases (equivalent to AND).
contains_any Filter	contains_any(foo, "bar baz")	Matches logs containing at least one word or phrase (equivalent to OR).
Subquery Filter	`user_id:in(_time:1d AND path:admin	fields user_id)`
Case-insensitive Filter	i(error)	Makes word, phrase, or prefix filters case-insensitive.
Sequence Filter	seq("error", "open file")	Matches logs where words or phrases appear in a specific order.
Regexp Filter	`~"err	warn"`
Range Filter	request.duration:range(4.2, Inf)	Filters numeric field values within a specified range.
IPv4 Range Filter	user.ip:ipv4_range("127.0.0.0/8")	Filters IPv4 addresses within a range or CIDR notation.
String Range Filter	user.name:string_range(A, C)	Filters string field values within a range (inclusive of lower bound).
Length Range Filter	len_range(5, 10)	Filters logs by the length of the message or field.
value_type Filter	user_id:value_type(uint64)	Matches logs with fields of a specific value type (e.g., uint64).
eq_field Filter	user_id:eq_field(customer_id)	Matches logs where two fields have identical values.
le_field Filter	duration:le_field(max_duration)	Matches logs where one field value does not exceed another.
lt_field Filter	duration:lt_field(max_duration)	Matches logs where one field value is smaller than another.

Filter Notes

• Field Specificity: Filters applied without a field name target the log message (_msg). To filter a specific field, use field_name:filter, e.g., log.level:error.
• Case Sensitivity: Most filters are case-sensitive. Use i() for case-insensitive matching, e.g., i(error) matches "ERROR" or "error".
• Logical Operators: Combine filters with AND, OR, and NOT. For example, _time:1h AND log.level:error or _time:1h OR log.level:error.

Mathematical Operations and Functions

LogQL supports mathematical operations and functions for numeric fields, typically used with the math or eval keywords. These are useful for calculations, such as converting units or aggregating values.

Operation/Function	Syntax Example	Description
Addition	arg1 + arg2	Adds two arguments.
Subtraction	arg1 - arg2	Subtracts the second argument from the first.
Multiplication	arg1 * arg2	Multiplies two arguments.
Division	arg1 / arg2	Divides the first argument by the second.
Modulus	arg1 % arg2	Returns the remainder of the division of arg1 by arg2.
Power	arg1 ^ arg2	Raises arg1 to the power of arg2.
Bitwise AND	arg1 & arg2	Performs bitwise AND on two arguments.
Bitwise OR	arg1 or arg2	Performs bitwise OR on two arguments.
Bitwise XOR	arg1 xor arg2	Performs bitwise XOR on two arguments.
Default Value	arg ault arg2	Returns arg2 if arg1 is non-numeric or NaN.
Absolute Value	abs(arg)	Returns the absolute value of arg.
Ceiling	ceil(arg)	Returns the smallest integer greater than or equal to arg.
Exponential	exp(arg)	Computes (e) raised to the power of arg.
Floor	floor(arg)	Returns the largest integer less than or equal to arg.
Natural Logarithm	ln(arg)	Computes the natural logarithm of arg.
Maximum	max(arg1, ..., argN)	Returns the maximum of the given arguments.
Minimum	min(arg1, ..., argN)	Returns the minimum of the given arguments.
Random Number	rand()	Returns a pseudo-random number in the range [0, 1).
Rounding	round(arg)	Rounds arg to the nearest integer.

Example:

_time:5m | eval (duration_secs * 1000) as duration_msecs

• Filters logs from the last 5 minutes and converts duration_secs to milliseconds, aliasing the result as duration_msecs.

Minimal Example

Query:

_time:1h log.level:error

Explanation:

• _time:1h: Selects logs from the last hour.
• log.level:error: Filters logs where the log.level field is exactly "error".

Use Case: This query is useful for monitoring error logs in a system over a recent time period.

Advanced Example

Query:

_time:1d response_size:>10KiB error

Explanation:

• _time:1d: Selects logs from the last day.
• response_size:>10KiB: Filters logs where the response_size field exceeds 10 KiB.
• error: Filters logs containing the word "error".

Use Case: This query helps identify large responses associated with errors, potentially indicating performance issues.

Additional Features

• String Literals: Support double-quoted ("..."), single-quoted ('...'), or backtick-quoted (`...`) strings. Escaping is required for quotes within the same type, except in backticks.
  • Example: "some \"field\":123" or `some "field":123`.
• Comments: Start with # and extend to the end of the line.
  • Example: error # find logs with error word.
• Numeric Values: Support suffixes like K (10^3), KiB (2^10), and underscores for readability (e.g., 1_234_567).
• Duration Values: Use suffixes like s (seconds), m (minutes), h (hours), e.g., 1h33m55s.
• Query Options: Modify query behavior with options(opt1=v1, ..., optN=vN).
  • Example: options(concurrency=2) _time:1d | count_uniq(user_id) limits the query to 2 CPU cores.

Critical Analysis

While LogQL is powerful, its reliance on a large number of specialized filters may introduce complexity for new users. The documentation assumes familiarity with concepts like stream labels and regular expressions, which could be a barrier. Additionally, the case-sensitive default for most filters might lead to unexpected results if not explicitly addressed with i(). Users should verify field names and data types in their logs, as mismatches can cause queries to fail silently. The mathematical operations are robust but limited to numeric fields, which may not cover all use cases (e.g., string manipulation). Comparing LogQL to other log query languages like Loki’s LogQL or Elasticsearch’s Query DSL, VictoriaMetrics’ version is optimized for its ecosystem but may lack some advanced aggregation features found in competitors.

Conclusion

LogQL provides a flexible and efficient way to query logs in VictoriaMetrics, with a rich set of filters and mathematical operations. By mastering the filters listed above and understanding how to combine them, users can effectively analyze log data for monitoring, debugging, and performance optimization. The examples provided should serve as a starting point, while the comprehensive filter table offers a reference for more complex queries.