Tracing Goroutines in Realtime with eBPF
sazak.ioInteresting to see an ebpf approach to this idea, some time ago I did the same with Dtrace: https://gaultier.github.io/blog/detecting_goroutine_leaks_wi...
However I did not think of observing the ‘castgstatus’ runtime function.
Always nice to be able to compare the DTrace and Ebpf approaches, they both have different strong points.
that is actually sick.
how common is it for go devs to experience leaking goroutines ? id like to think go is a lot less shoot yourself in the foot here since they provide a framework for concurrency/parallelism rather than you working with the tiny pieces of it and building out the architecture yourself, but ive only needed to use goroutines once and it was a pretty problem-free experience.
It depends how much the code uses manually spawned goroutines, and how complex the lifecycle of these goroutines is… in big codebases such as kubernetes, docker, etc, it has been a problem. There have been research papers and blogs about this, but most Go developers are not aware of this issue it seems.
> how common is it for go devs to experience leaking goroutines
About as often as leaking memory in C++
F. again i have minimal experience actually ever needing it in go, but guessing this is just generally the exercise of managing the lifecycle of a goroutine well in your code? proper handling so things dont get orphaned in buffer, fire and forget woopsies, etc.
early on i do feel like go kinda advertised batteries included concurrency but i kinda wished they advertised the foot-shooting-mechanisms and gaps in the abstraction a little more. overall i prefer to have enough control to choose how to manage the lifecycle. mem leaks bum me out and kill my steam, at least from my experience with c/cpp.
So cool! Does it support multiple Go versions?
I would say that this is limited to a targeted Go version and architecture. For example, the filter checks for the goroutine pointer on `r28` which is correct for arm64 but not universally true.
Any changes to the struct layouts, stack, or heap layouts would also cause failures in these lookups. E.g., in Go 1.17, many functions now use direct register mappings for arguments rather than always placing arguments in the stack.
Would need to thoroughly vet compatability with each new Go version before using something like this in production.
Cool! is there something similar for async rust ?
https://docs.rs/tokio-console/latest/tokio_console/ comes to mind.
I'm not aware of eBPF-based dynamic instrumentation for either tokio or other async runtimes, but I'm also not (in need to be) debugging this on a daily basis.