Settings

Theme

New Computer Language Benchmarks Game metric: time + source code size

benchmarksgame-team.pages.debian.net

47 points by benstrumental 4 years ago · 58 comments

Reader
sidkshatriya 4 years ago

Geometric mean of (time + gzipped source code size in bytes) seems statistically wrong.

What if you shifted time to nanoseconds ? Or source code size in terms of Megabytes. The rankings could change. The culprit is the '+'

I would think Geometric mean of (time x gzipped source code size) is the correct way to compare languages together. It would not matter what the units of time or size are in that case.

[Here the geometric mean is the geometric mean of (time x gzipped size) of all benchmark programs of a particular language.]

  • ntoskrnl 4 years ago

    Yep this is correct. Adding disparate units is almost always nonsensical. You can confirm with a scientific calculator like insect:

      $ insect '5s + 10MB'
    Conversion error:

      Cannot convert unit MB (base units: bit)
                  to unit s

  $ insect '5s * 10MB'
  50 s·MB
  • tuukkah 4 years ago

    I think the summed numbers might be unitless. At least all the other numbers are relative to the fastest/smallest entry. That is, what would make sense is score(x) = time(x) / time(fastest) + size(x) / size(smallest) instead of score(x) = (time(x) + size(x)) / score(best)

  • dwattttt 4 years ago

    It's not necessarily wrong to add disparate units like this. It's implicitly weighting one unit to the other. Changing to nanoseconds just gives more weight to the time metric in the unified benchmark. You could instead explicitly weight them without changing units, if you cared about the size more you could add a multiplier to it.

    • sidkshatriya 4 years ago

      You really don’t know what weight is the right weight to balance time and gripped size. Multiplying them together sidesteps the whole issue and puts time and size on par with each other regardless of the individual unit scaling.

      The whole point of benchmarks is to protect against accidental bias in your calculations. Adding them seems totally against my intuition. If you did want to give time more weight then I would raise it to some power. Example: geometric mean of (time x time x source size) would give time much more importance in an arguably more principled way.

      • dwattttt 4 years ago

        Multiplying them is another way of expressing them as a unified value. It's not a question of accidental bias, you're explicitly choosing how important one second is compared to one byte.

        You could imagine there's a 1 sec/byte multiplier on the bytes value, saying in effect "for every byte of gzipped source, penalise the benchmark by one second".

        • sidkshatriya 4 years ago

          > You could imagine there's a 1 sec/byte multiplier on the bytes value, saying in effect "for every byte of gzipped source, penalise the benchmark by one second".

          Your explanation makes sense. However the main issue is we don’t know if this “penalty” is fair or correct or has some justifiable basis. In absence of any explanation it would make more sense to multiply them together as a “sane default”. Later, having done some research we can attach some weightage perhaps appealing to some physical laws or information theory. Even then I doubt that + would be the operator I would use to combine them.

      • igouy 4 years ago

        > Adding them…

        Read '+' as '&'.

  • igouy 4 years ago

    > The culprit is the '+'

    That annotation does seem to have caused much frothing and gnashing.

    Here's how the calculation is made — "How not to lie with statistics: The correct way to summarize benchmark results."

    [pdf] http://www.cse.unsw.edu.au/~cs9242/11/papers/Fleming_Wallace...

    • yorwba 4 years ago

      That paper is only about the reasoning behind taking the geometric mean, it doesn't have anything to say on the "time + gzipped source code size in bytes" part.

agentgt 4 years ago

I really wish they aggregated the metric of build time (+ whatever).

That is a huge metric I care about.

You can figure out it somewhat by clicking on each language benchmark but it is not aggregated.

BTW as biased guy in the Java world I can tell you this is one area Java is actually mostly the winner even beating out many scripting languages apparently.

  • igouy 4 years ago

    Do Java "build time" measurements include class loading and JIT compilation? :-)

    • kaba0 4 years ago

      Does C “build time” include the time it takes to load the binary from disk?

_b 4 years ago

I'd be interested to see "C compiled with Clang" added as another language to the benchmark games. In part, digging into Clang vs gcc benchmarks is always interesting, and in part, as Rust & Clang share the same LLVM backend, it would shed light on how much of the C vs Rust difference is from frontend language stuff vs backend code gen stuff.

IshKebab 4 years ago

With a totally arbitrary conversion of 1 second = 1 gzipped byte.

This is basically meaningless. I don't see why you'd even need to do this. You can easily show code size and performance on the same graph.

NeutralForest 4 years ago

This presentation is pretty bad, there should be more context, some kind of color scheme or labels instead of text in the background, spacing between the languages represented, other benchmarks than the geometric mean, etc.

kibwen 4 years ago

For comparing multiple implementations of a single benchmark in a single language, this sort of data would be interesting as a 2D plot, to see how many lines it takes to improve performance by how much. But for cross-language benchmarking this seems somewhat confounding, as the richness of standard libraries varies between languages (and counting the lines of external dependencies sounds extremely annoying, not only because you have to decide whether to include standard libraries (including libc), you also need to find a way not to penalize those for having many lines devoted to tests).

Thaxll 4 years ago

The thing they should change is to forbid the nonsense like:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Actually if you look at all the top net core submissions the only one fast are the one using low level intrinsics etc ...

arunc 4 years ago

Just curious, why does this benchmark not include D language? I remember seeing it a few years ago. Was it removed recently?

cpurdy 4 years ago

Predictably, "The Computer Language Benchmarks Game" once again proves the worthlessness of "The Computer Language Benchmarks Game".

This thing has been a long running joke in the software industry, exceeded only by the level of their defensiveness.

SMH.

guenthert 4 years ago

I think APL had already shown, that brevity in itself is not desirable.

Shadonototra 4 years ago

these pseudo benchmarks should be banned

hexo 4 years ago

I dont buy these results at all. Julia at second place looks like plain lie and complete nonsense, to the point I'm gonna look into this and run it myself.

After trying hard to use julia for about a year and I came to conclusion it's one of the slowest things around. Maybe the stuff changed? Maybe, but julia code still remains incorrect.

I hope they fix both things, speed (including start up speed, it counts A LOT) and correctness.

  • ChrisRackauckas 4 years ago

    Note that these benchmarks include compilation time for Julia, while it does not include compilation time for C, Rust, etc.

    • igouy 4 years ago

      Julia is presented like this —

      “Julia features optional typing, multiple dispatch, and good performance, achieved using type inference and just-in-time (JIT) compilation, implemented using LLVM.”

      Julia 1.7 Documentation, Introduction

      https://docs.julialang.org/en/v1/

      • ChrisRackauckas 4 years ago

        Yes, because it's all set for prime time in the next release.

        "Julia features optional typing, multiple dispatch, and good performance, achieved using type inference and just-in-time (JIT) compilation (and optional ahead-of-time compilation), implemented using LLVM."

        https://docs.julialang.org/en/v1.9-dev/

        So it'll be updated when v1.9 comes out? Anyways, it's a somewhat interesting thing that Julia still gets 3rd even though it's measuring compilation time.

        • igouy 4 years ago

          Tell us when it becomes achieved using ahead-of-time compilation (and optional just-in-time (JIT) compilation).

Keyboard Shortcuts

j
Next item
k
Previous item
o / Enter
Open selected item
?
Show this help
Esc
Close modal / clear selection