The future of latency profiling in Go
rakyll.orgPM for tracing at Google here.
As the blog post mentioned, we're collaborating with partners on a common RPC context proposal for tracing systems [1]. Some of the major contributors are currently on vacation, but there'll be be another round of comments and updates once they get back.
We're also working with the same partners on a project called Census - a set of tracing instrumentation libraries that all vendors can use and contribute to, though we're still in the very early days on this effort. We'll have more to announce later in the year (still need to publish additional libraries, set up a website, etc.), but you can follow the progress on GitHub here [2].
[1] https://github.com/TraceContext/tracecontext-spec/pull/1/fil... [2] https://github.com/census-instrumentation
OMG, Census! It's one of the frameworks you miss the most once you no longer can use it. There is some preliminary version of it hidden deep inside gRPC, but it's great to hear that it's becoming a standalone project.
What's the difference between this and opentracing? AFAIK opentracing is built by the people who created Dapper at Google, and the project also has a set of standards for HTTP transport.
They're different in scope and goals. Census is composed of a context propagation format (which the blog post mentioned), a single distribution of language-specific libraries that include instrumentation hooks for popular web / RPC frameworks and metrics + trace exporters for various backends, and an optional agent that you can run locally to view metrics and RPC stats.
not having a tool like jvisualvm frustrates me.
Using Opentracing and uber's Jaeger client is awesome. It's easy to implement, and Jaeger provides multiple different mechanisms for sampling traces (always on, guaranteed throughput etc.).
It's awesome to be able to see a UI giving you diagrams like this: https://www.dropbox.com/s/b2uhx77urpnuk9g/Screenshot%202017-... for your API.
Definitely would recommend - it beats old-school log tracing.
Do I miss anything? OpenTracing mentions "A vendor-neutral open standard for distributed tracing." It reads like it's for implementors - is it useable stand alone + Jaeger as UI?
How can it be used - can you point me to a tutorial/blog post?
Hey frik, former Uber engineer here. I never worked on Jaeger, but I loved using it at Uber.
If you want to get started, the documentation is at http://jaeger.readthedocs.io/en/latest/
We also have a blog post that describes the history of Jaegar and some of the design decisions. https://eng.uber.com/distributed-tracing/
Jaegar is compatible with the OpenTracing standard ( http://opentracing.io/documentation/pages/supported-tracers ) and is also compatible with the Zipkin backend and wire protocol ( http://zipkin.io/ ).
Thanks a lot, will try it out.
btw https://uber.github.io/jaeger/ still mentions "Backend components are implemented in Go (will be open sourced soon)" ...in the meantime the code go published as it seems: https://github.com/uber/jaeger
Mm it's a little sparse right now - had to dig into the opentracing spec, dig in to the implementations and make it piecemeal.
OpenTracing is for everyone - both tracing implementors (eg. Jaeger engineers) and for you implementing tracing in your system. As an implementor in your system you mainly care about the "vocabulary" and "standards" opentracing defines. So, if you've got two functions (A which calls B) opentracing defines:
- Syntaxes to say that the span for "B" is a "childOf" "A". This gives you the graph I had above.
- Syntaxes to attach errors and other information to spans in a standard way (this lets any opentracing-compatible UI display/search errors etc): https://github.com/opentracing/specification/blob/master/sem...
OpenTracing also allows you to define functions that follow each other: https://github.com/opentracing/specification/blob/master/spe....
Plus, defines abstract ways of passing tracing contexts through service boundaries (ie. from client => API server => other microservices) via strings and HTTP headers.
Also, the best part of opentracing (in go) is that you can use it directly to set these fields (https://github.com/tonyhb/personal-starter/blob/master/pkg/a...). You can pull in the "github.com/opentracing/opentracing-go" library and use it to create and propagate your spans and contexts, and set errors using the default vocabulary (https://godoc.org/github.com/opentracing/opentracing-go/ext).
Been meaning to sort my blog out and write some posts on this as the implementation documentation is sparse. Some time soon.
Before Jaeger was out I started using this, although I'm considering switching over to using Jaeger: https://github.com/opentracing/opentracing-go
It should be pretty clear how to use it once you read through the examples.
Jaeger requires using Cassandra as a datastore (for now), which isn't easily supported at my company, so I went with Zipkin and used MySQL as a datastore.
For 1)., if you use a LLVM backed language, XRay supposedly should be it: https://llvm.org/docs/XRay.html
LLVM is a future consideration for Go. One of the main reasons I don't want to tackle (1) right now if the possibility of benefiting from XRay even though it doesn't sort the case out for the vast majority: the gc users.
what is the latency caused by the call to trace the latency?