Octane benchmark retired
v8project.blogspot.comThis actually makes sense. When the low level stuff is at a terrible level they made Octane to make that better. Once that was fixed, optimizing for it is counter-productive.
This has a big resemblance IMO to most formal study methods including University and natural languages. Once you know the basics you can optimize for learning or for grades, depending on your goals. They are correlated (but not strongly correlated), but the time investment is not negligible so you'll have to choose either of them.
They probably got tired of FF & Edge beating them in their own benchmark. Can you imagine the conversation with some middle level manager when you were asked to explain that?
WebKit is currently king on Octane fwiw.
We still think it's a great benchmark. V8 made the mistake of focusing only on this benchmark, so they didn't know that some of their Octane optimizations were bad for other scenarios. We didn't have that problem because we have always used a broader spectrum of benchmarks.
Agreed, Octane was good as far as benchmarks go but you have to balance with a much larger set of benchmarks and always ground your work in real-world scenarios (something we focused heavily on from day one e.g. when we realized having an interpreter really helps startup on lots of real-world apps but doesn't do much for Octane score).
That said, FWIW I think Edge/Chakra gets the crown on Windows!
(I work on Chakra)
Out of curiosity: does only having to target one OS make a difference here? Is it "easier" to get it to run fast? Any worries that you don't have compared to Firefox and Chrome, even though you purely work on the JS side of things?
I'm sorry I missed your comment earlier, but hopefully replying later is better than never :)
ChakraCore is cross-platform now so we no longer get the benefit of a single platform. However back when we were only Windows there were advantages, certainly, if only because all the work spent getting everything working on different OSes can instead be invested in improving performance.
I don't think they only focused on Octane, it's possible they may have weighted it more highly, I can't speak for the v8 team (I don't work on v8), but that sounds like an oversimplification of what probably happened.
One example could be focusing on throughput instead of startup latency which could yield distorted outcomes for real world apps, or mostly app code that is mostly early-bound style OO code/monomorphic functional code vs code that is hostile to class shape analysis/PIC.
It seems in the last few years, especially with the idea of Web competing with native with stuff like asm.js for games, there was a focus on maximize throughput, but it's rare your typical interactive web app is running code that benefits from that.
Octane was V8's primary tuning benchmark. They tuned for it to the detriment of other benchmarks. Whether you call that "only focusing" on Octane or not is a matter of semantics to me.
That does seem to be a concern here. They are replacing Octane with what amounts to a-collection-of-stuff. Without a specific workset (ie. a benchmark) there is no clear way to measure one thing against another.
This seems like a mechanism to hide where you stand. Now they can just say "That's not what we find in our tests".
I'm inclined to agree that becnhmarks do become obsolete as workloads change and development targets benchmark pointscoring. The solution is to make a new standardised benchmark, or indeed many of them. If they are clear about what they are measuring then the relative perfomance of different JavaScript engines can be understood by developers to give a nuanced view of what performs well.
We find that http://browserbench.org/Speedometer/ -- which is what v8 is optimizing against now -- is extremely close to real world Discourse (EmberJS, Glimmer) performance.
We always wondered why Android was so crazy ridiculously slow at our real world JS vs. iOS starting in late 2013 onward.. getting worse and worse over 2014 and 2015.. and "gee, we optimized for the wrong thing aka Octane" was a big part of that.
http://benediktmeurer.de/2016/12/16/the-truth-about-traditio...
You can read more about the history at https://meta.discourse.org/t/the-state-of-javascript-on-andr...
JavaScriptCore (WebKit's JS engine) also optimized for Octane. But we additionally optimized for Speedometer, Sunspider (the not totally broken subset that made it into Jetstream) and JSBench. While Octane isn't the single answer to all JS performance measurement, using it in combination with better things seems better than not using it. To go back to your example, iOS both has way better real-world perf and way better Octane results than Android devices.
BTW benchmarks I'd endorse include MotionMark, Speedometer and Jetstream from http://browserbench.org/ as well as JSBench http://plg.uwaterloo.ca/~dynjs/jsbench/
These are all good at testing different aspects of browser performance including relevant testing of the JS engine.
First sunspider, now octane.
This shows that browser vendors are not really good at writing JS benchmarks representative of real world, which is not surprising considering that they are not doing real web development.
Ideally benchmarks would come from developers of popular websites like Facebook, Twitter, etc.
That's just a truism about benchmarks. Once all the low hanging fruit is picked they stop showing interesting progress and just become a garden for crazy microoptimizations. The same thing happened with C compilers 10-15 years ago.
Basically, Javascript performance is "done". It's not going to get much faster at this point, and higher performance will come from paradigm shifts (SIMD, WebAssembly, pick your fad).
Looks like someone at Google realized that Servo (Quantum) is coming
Servo uses SpiderMonkey to provide Javascript support, so retiring Octane (a Javascript engine benchmark) would have nothing to do with Servo (a layout engine).
Got it. I did'nt realize that.