TraceQL: Preview the new query language coming in Grafana Tempo 2.0

Tags

Top Highlights

• With our new trace querying language, you can interactively extract and search through your traces so you can diagnose and respond to root causes faster.

• TraceQL: Why Grafana Labs added a query language for traces

• Distributed traces contain a wealth of information that can help you track down bugs, identify root cause, analyze performance, and more.

• And while tools like auto-instrumentation make it easy to start capturing data, it can be much harder to extract value from that data.

• Today, the Tempo metrics-generator can aggregate traces into service graphs and span metrics, and you can use exemplars to navigate from a spike in API latency to a trace that contributed to that spike — but traces can do so much more.

• Traces are the flow of events throughout your components.

• They have a tree structure with a root, branches, and leaves; arbitrary key-value data at any location; and, of course, timestamps.

• With existing tracing solutions, including Tempo 1.0, it can be challenging to find what you’re looking for if you don’t know the exact trace, or if you need broader context about a chain of events.

• Also, what if we want to use this structure to go beyond just isolating individual events? What if we want to find entire sequences of events?

• Unfortunately, none of the existing query languages make it easy to express these complex questions, so we are adding a new query language designed from the ground up to work well with traces.

• TraceQL fits perfectly in the Grafana LGTM stack. For example, look at this query shown at ObservabilityCON:

• { .db.statement =~ "INSERT.*"} | avg(duration) > 1s

• This query will find traces that made inserts into the database, and where the average duration of those inserts was greater than one second.

• But TraceQL is data-type aware, so you get flexible, powerful operations for dealing with text, integers, and other data types.

• Plus, Tempo 2.0 will incorporate a new storage format based on the open source columnar Apache Parquet project.

• Tempo will write trace data in Parquet, which is supported by a huge range of databases, data pipelines, and analytics tools.

• Combining this with TraceQL will help you unlock your data better and build the tooling and analysis that’s important to your organization.

• Grafana Tempo 2.0 will be available in the coming weeks, but you will be able to preview the new search experience in Grafana 9.3.1.

• We’re excited to see what users do with TraceQL and how it improves their tracing journey.

• We’ll continue to iterate, including adding the ability to find and analyze traces based on their structure, more advanced processing pipelines, and aggregations. But for now, check out Grafana Tempo 2.0 in the coming weeks and let us know what you think!