Netlify’s Disingenuous Survey-based Attack on Next.js (and Eleventy, too)—zachleat.com

As Netlify distances itself from Jamstack in an attempt to survive a VC-funded battle with Vercel, they have also rebranded the Jamstack Community Survey to a more generic The State of Web Development report (behind an email signup form). It should be called the Netlify Community Survey (as the results only represent Netlify customers) but I digress.

A few disclaimers before we get started:

I am the creator and maintainer of Eleventy.
From 2020–2023 I was employed by Netlify.
In 2021 I worked on the Jamstack Community Survey results website (but was not involved in the survey data collection nor the results content).

Problems

This year’s report has a few changes to the presentation of results that I think have traversed the report into territory that I called “intellectually dishonest” on Mastodon. My default in these situations is to let these things go but enough people have tagged me on this—while also drawing some damaging conclusions from the disingenuity in this report—that I feel the need to weigh in (though frankly I’m a little annoyed that I need to spend time on it at all).

My primary beef with this year’s report is on page 10 (of 13) and features a section on rendering frameworks and site generators, with one and only one chart documenting changes in satisfaction score versus change in usage.

This chart shows Astro’s stellar growth this year, the runaway winner for increased satisfaction and usage. Next and 11ty are shown in the bottom left corner, losing usage and satisfaction

This chart looks bad for Next. It also looks bad for Eleventy. But the bigger problem here is that this chart shows changes without any context of absolute scores. Next might have gotten some caveats in the text written in the section but 11ty did not.

In previous years, the report discussed absolute scores prior to deltas because this context is important.

(I also took special note that the label for Gatsby—a framework purchased by Netlify and under very recent criticism for languishing investment—was obscured behind Gridsome)

On page 32 (of 39) of the report buried deep into the Appendices section the report finally includes the real scores (conveniently unordered):

Astro has the best positive sentiment too at 87%, Next is 78% and Eleventy is 74%

Now we learn that Next.js is ranked #3 (tied with SolidStart and Remix) for satisfaction and 11ty is ranked #6 (tied with Nuxt).

Here’s what the chart looks like using absolute values for satisfaction percentages and usage, painting a story that Netlify probably doesn’t want to share:

Next.js is the clear winner here. A cluster of Nuxt, Astro, 11ty, and Sveltekit all sit together with high scores and similar usage. Gatsby is an outlier with large usage but low satisfaction.

Without ascribing malice

I am aware that Netlify has historically employed a Data Scientist with statistics and data analysis background to conduct this survey and prepare the report. This person departed the company months ago. The Data Scientist in question was very good at documenting and publishing the survey and report’s methodology and you can see one such (2022) document.

The 2023 report did not include a methodology document.

The 2022 methodology document reported how the data was collected (sources) and margins of error and included the following notable section at the end:

Conclusions we cannot draw:

We do not believe these results are indicative of preferences amongst web developers as a whole. We are also not making claims as to:

Popularity (or not) of Netlify products and services, given bias in respondents

The 2023 report name change to The State of Web Development seemingly goes against the first point.

The 2023 report also goes against the second point by eschewing any mention of bias. It (surprisingly) includes a section on the popularity of Netlify itself:

88% of Netlify customers want to use Netlify more. 79% of Netlify customers want to use Vercel more.

I was taken-aback that Netlify would publish statistics to show what percentage of Netlify customers want to use Vercel more—but apparently it’s 79%.

Conclusions

Importantly: After Eleventy’s Netlify-induced setbacks this year, we are back on our feet and we will continue to punch far above our node_modules weight-class. Many of the alternative frameworks discussed here are VC-funded with full-time developers and have millions in funding. We don’t.

Eleventy’s project goals are different: stay focused, keep our core feature set small, and ruthlessly minimize our external dependencies for a long-term future.

On Astro (since folks have asked), I will say that Astro and Eleventy share the same zero JavaScript footprint vision for the web and in that regard we are allies in the web development framework melee.

Reading the report, it’s clear that Netlify has a vested interest in elevating Astro because Astro is best poised to dethrone Next.js. If Astro wins, Netlify wins. Perhaps surprisingly I also believe that if Astro wins, Eleventy wins! It is not a zero-sum game and I know that 2022 Netlify believed this (even if 2023 Netlify does not).

Appreciate y’all—and keep building for the web!