Orbit: C/C++ Performance Profiler
github.comAs a xoogler who's wanted to use this and invest in it, I'm curious about the state it's in. Google has been investing in perfetto, this is different, but has some redundancy. It would would be nice to know this isn't abandonware.
Using some scripts/parsers to take DTrace/perf/Instruments/ETW data and transfer it to perfetto was one of the most exciting moments of my performance engineering career. It’s such a powerful thing compared to every single other workflow I’ve ever used.
It just shows contention in a way that so hard to see otherwise.
If this tool packages some of that in an easier to use package it’s going to be a great tool for some.
"It just shows contention in a way that so hard to see otherwise."
Could you elaborated on this? I've never used it myself.
You start with a trace event you care about. You see it’s too slow. The first step is that you see explicitly that it’s not because the code is slow, you constantly get descheduled because there are thread state indicators above your event.
Then you navigate to the CPU tracks find the thread(s) that were running instead of the one you care about and directly inspect their stacks. Sometimes almost nothing is running. Maybe contention is not on even on the CPU.
So the “parsing styles was slow” conclusion you would get from only looking at histograms turns into “parsing styles was slow because I couldn’t get my fonts and that was slow because IndexDB was hogging the hard drive on another process”
Edit: I should mention the user provided trace events are very important here. You get flow arrows really underlining task posting and IPC response as well as user interactions. Entry/Exit style uprobes on function are great but I found won’t get you all the way there on a large application.
Thanks!
I'm curious how does it work on Windows?
Looks like it may have been stadia related.
Ah, I was wondering why Google would develop a profiler with Wine/Proton support.
Stadia was such a monumental technical achievement, it was painful to see how they killed it without even seeing what it could be used for. Idiocy.
Stadia was dead-on-arrival (regardless of any technical merits), the uncertainty, money and amount of work required on games leaves developers skittish about unreliable platforms (or companies). Especially with something as a new techonology like with streaming games where people were more or less rightfully concerned about latency.
All their product killings had already gotten someone to register the killedbygoogle.com domain on 2018-09-12 , that's a month before Google Stadia was even in closed beta. People were already joking about when they'd close Stadia when it was released.
Adding insult to injury, everyone saw the absolute fuckup with the Terraria developer. You have an indie developer (that iirc was _featured_ in the Stadia promotions) who loses his work Google accounts making it impossible to work and in frustration announces that the game is dead (apparently his account and the game were re-instated but after a month people had lost track and the image damage was done).
And yes, Google had some big-name contracts outside of just indies, but viability of a platform is always evaluated by it's entire scope (and iirc EA wasn't on board from day 1).
Google has a huge image issue with being humanly unreachable, we accepted with when Google was a cool perky upstart with a do-no-evil creed that hadn't messed up enough yet, but it's far harder to swallow from a big company that we know are only doing it's automated account management to save money (and is so insulated that it takes a _month_ to unfreeze an account of someone that is even supposed to be your partner!).
https://www.polygon.com/2021/3/18/22338396/terraria-google-s...
Idiocy was expecting game developers would blindly jump into GNU/Linux, Vulkan, with a stone age development experience versus what game console devkits and Windows look like.
I wasn't surprised given how bad Google Android talks at game conferences tend to be, so they put up with Google's lack of game development bones.
Android is only different because game developers don't have a option if they want to cover both top mobile platforms.
Android GDK is still a joke versus what Apple puts out, let alone Microsoft, Sony and Nintendo.
>stone age development experience
simply not true. Stadia had full integration into MSVS in build and debug flows. Even 'lets show a build to management' flows there were better than Windows (i was in shock myself). And for long time during Covid Stadia builds were the easiest way to get playtests up and running at home.
But I agree with your other point that investing into porting and Vulkan renderers was a huge waste of dev resources.
I found the Stadia developer experience to be pretty solid. The visual studio integration was top notch and worked, with a few minor exceptions, without a hitch. Having worked with all major gaming platforms to come out in the last two decades, Stadia was certainly not the worst.
Sure, if one wanted to rewrite their engine.
Shortly before Stadia was ramped down, there was a talk about Stadia team starting the Stadia Porting Toolkit, as they weren't getting enough studios interested in porting their XBox/Windows engines into Stadia, versus what they would get out of GeForce Now and Cloud Game Pass.
Android development seems to be universally acknowledged as a backwater -experience compared to other platforms.
While it certainly has helped to spread the spectrum of managed languages used across an OS stack, something that Microsoft failed at with Longhorn due to internal politics between DevDiv and WinDev, it has definitly plenty of warts, starting by the expectation to wait for the first set of bug fixes afer each "stable" SDK/IDE release, somehow all the previews, canary and beta are never enough.
Imho the thing that killed stadia was the pricing model. Buying games but not owning them outright* is a recipe for failure, especially for a company known for killing off services (and I say that as a xoogler with very mixed feelings). Stadia should have just had a monthly subscription fee.
*Yes, with DRM, copy protection, online only, etc, it's debatable whether anyone owns any game. But conventional game ownership or even steam style is still _more_ ownership than the stadia case.
They killed the user-facing product because by all accounts it was a flop. The tech is still there but they're pivoting to a game-streaming SaaS/middleware thing for B2B.
Did anything come of that?
at least looks like they dropping support for windows
"Orbit's focus has shifted to the Linux version. Windows local profiling is currently only supported partially and major features, such as dynamic instrumentation, are not yet implemented. It is possible however to profile Linux executables from a Windows UI instance. For Windows local profiling, you can still use the released binaries, but please note that they are deprecated and mostly undocumented."
It looks like all the top contributors no longer work on it much, based on github's insights.
i don't really think there is _anything_ that comes even close to tracy https://github.com/wolfpld/tracy.
on top of this, given google's penchant for dumping projects aka abandonware, this would be an easy pass.
What surprised me about Tracy was just how easy it was to include in a C++ project. Just compile and link a single .cpp file (TracyClient.cpp)
Orbit doesn't require you to change any code at all, according to their readme
Does tracy support automatic function entry/exit (FEE) traces (i.e. all functions are automatically instrumented with a start/end)? The documentation seems to indicate traces are all manual which is pretty inconvenient. There are very few cases where you even need manual instrumentation if you have FEE data from my experience.
What about vtune? I was under the impression that it is pretty much the state of art in profiling
ETW Traces are the gold standard for most uses IMO - https://randomascii.wordpress.com/2015/09/24/etw-central/.
For architecture specific profiling, Vtune is still pretty good though.
JFR is also quite good, for those on the JVM world.
Tracy is amazing. Having just integrated it into a major project, it's leaps and bounds above what we were using previously.
Considering Stadia's demise, is orbit dead as well?
> It supports native applications written in languages such as C, C++, Rust, or Go.
Seeing it supports Go, I wonder if it supports other AoT compiled languages with garbage collectors, like .NET.
Can anyone suggest a good non-sampling profiler for low latency apps? The only one I can think of is https://github.com/morganstanley/Xpedite but it's not even clear that it is being maintained anymore.
Any googlers chime in? Is this any good? Or just a dump of one of their internal unused products?
VTunes can tell you the average time that functions take to run but which tools can highlight functions that very occasionally run a lot slower than normal?
At first glance - how is this different from VTune?
Its foss, vtune is proprietary.
-
Last VTune release was three months ago, can you elaborate?
Huh? Isn't their most recent release 2024.0.1? https://www.intel.com/content/www/us/en/developer/articles/r...
There's a bunch of really good C++ profilers already. My current favourite is Tracy: https://github.com/wolfpld/tracy
Not if you are on macOS, the number of available profilers drops after every release. Same with leak detectors, valgrind doesnt work, sanitizers don't work anymore (even if you compile your own clang).
It took Apple a long time to bring ASan and TSan to macOS, but they work now, for me at least. (MSan still unavailable though).
At least in the specific case of Tracy, you can profile over the network. So, you instrument your OSX application, and it streams its data in realtime to your profiler viewer.
Though it's true that more advanced features require OS support (eg: showing how threads get moved amongst CPU cores, and so on).
Is there _any_ usable profiler on macOS? I've found Instruments to be straight-up useless (very slow, frequently gives just plain wrong results on larger projects) and pretty much every search I've tried just gives “use Instruments”.
Can't go wrong with spindump and sample.
Both address sanitizer and UB sanitizer work for me (macOS-13.6, Intel). You have to add a library search path and the appropriate `libclang_rt.*` library to your build.
Library search path is `/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/1[4-9].[0-9].[0-9]/lib/darwin`.
Apple mostly keeps their sanitizers up-to-date? Not that it’s really a profiler, to be honest.
Sanitizers only sometimes work on Windows too :)
I found that you can use the 'leaks' command to replace vlgrind memcheck
samply is pretty sweet. It's been my go-to for Deno work for the last year or so.
Tracy and Superluminal.
Superluminal is great as well. I currently prefer Tracy because it gives faster profiling feedback.
Superluminal is windows (and xbox/playstation) only too, so it's off the board immediately.
Sadly there’s no one tool to rule them all. The only choice is to use the best tool for each platform.
Superluminal is far and away the best profiler imho.
Tracy is a great instrumented profiler. But damn do I love a good sampling profiler. Tracy can sample. But I had serious issues with it on one of my projects I could never get around.
I agree, just that I haven't used nor developed for Windows for quite some time, so can't use it. For games it's a natural environment, of course.
What’s the difference between this and libprofile in GPT? Just earlier in its transition to Google abandonware?
According to the presentation video, it is a profiling tool used for Stadia Games ( https://www.youtube.com/watch?v=8V-EPBPGZPs ) that can adapt to Unity and Unreal Engine. Which seems to make sense as the original developer has experience developing on video games.
The main selling point of the tool is Dynamic instrumentation, which allows you to generate Flame charts without manual code annotation (you pick functions that you are interested into, and the profiler adds a hook to these functions in order to log the entry+exit time).
That’s what libprofiler does, too, sample the stack during runs.
Why does it sound like a remake of FlameGraph?