Making Go telemetry opt-in is a mistake
twi.github.ioHere's an idea for how to maintain Go moving forward: keep making the damn tool however you want. The thing never would have existed in the first place if they had started with an industry survey. It was created to address a perceived need, by the people with the need, for themselves. This model is perfectly fine moving forward. Some industrial user wants a new Go feature or bugfix? Great. If it's enough of a problem, they can fix it and upstream a patch. That's how open source software is always supposed to work. Telemetry does nothing to improve this situation. If the Go team at google no longer has any ideas for what to work on (as must be the case if they're wasting their time on dumb crap like forced google spyware in a compiler) then they should just stop. Maybe focus on accepting PRs from people with ideas and strong enough motivation to work on them. I mean, there are 5000+ issues and 330 open PRs on the go github right now, so that should be plenty to keep them occupied. On top of that, how can literal thousands of issues not be a strong enough source of actionable usage information that they saw fit to try to get more? Do they plan on wrapping up all the open issues before looking at telemetry?
I really appreciate this point, and I think it applies very broadly. Good design comes from a coherent, individual (or group) vision. Citing examples will only incite discussions, because everyone has a different needs and ideas, but the things I most appreciate in tech and art share this common aspect.
As I see it, relying on excessive telemetry, surveys, focus-groups, etc is a bit of an indication that nobody at the top has a strong idea of where to take the project.
Which is strange coming from ~Go~ Google. The community loudly demanded features for years which ~Go~ Google said were unnecessary (dependency management, monotonic time, etc). Weird to think that telemetry would do anything to change their opinions when the project has always done what it wants.
Edit: fighting the markup trying to get strikethrough to work
> keep making the damn tool however you want. The thing never would have existed in the first place if they had started with an industry survey. It was created to address a perceived need, by the people with the need, for themselves. This model is perfectly fine moving forward.
I disagree, doing what you feel like is a great approach when you have no users, your building a tool that solves your needs and basically saying "Try this it, it works great for me, maybe it'll be good for you too!"
Once you have millions of users and billions of lines of code you need to think about how your changes affect those users. If you get it wrong there's a huge cost to that (c.f. early forays in to go packaging and dependency management). This is why go's comparability promise is such a big deal and a big part of why go is a very popular language.
> If the Go team at google no longer has any ideas for what to work on (as must be the case if they're wasting their time on dumb crap like forced google spyware in a compiler) then they should just stop.
They did stop. "Go 1" is complete.
Since 2017, the project renewed itself by moving to what is dubbed "Go 2"[1], being built around community feedback. Decisions are made based on data collected from the community. Adding telemetry to increase the amount of data available is in line with the new project scope. Go 1.13 was the first "Go 2" release[2].
Indeed, "Go 1" is still there if you want to use the tool that was built for what was needed at Google and nothing more. You don't have to move into the "Go 2" ecosystem. But if others want to put their time into that, why not? That's their prerogative.
there are 5000+ issues and 330 open PRs on the go github right now, so that should be plenty to keep them occupied
just for the sake of argument, wouldn't telemetry be useful to know which of these issues are most likely to benefit the majority of users?
whether that's worth the cost of telemetry is another issue.
You say this:
> keep making the damn tool however you want.
But then you follow up with this:
> If the Go team at google no longer has any ideas for what to work on (as must be the case if they're wasting their time on dumb crap like forced google spyware in a compiler) then they should just stop.
They want to add telemetry. So they should just add it. If you dislike it, then patch it out yourself. As you say:
>That's how open source software is always supposed to work.
> if they had started with an industry survey
They already do regular surveys.
> Some industrial user wants a new Go feature or bugfix? Great. If it's enough of a problem, they can fix it and upstream a patch.
Inbefore I go to a new job and find out that they are using outdated, custom patched go compiler.
> I mean, there are 5000+ issues and 330 open PRs on the go github right now
How do they know which ones are affecting the most users?
> forced google spyware in a compiler
go is open source, feel free to compile it yourself without the telemetry. Which distros will do if any major promises would be broken
There's a lot of 'just' handwaving here about compiling without telemetry. We just need to look as far as VSCode, which is riddled with unremovable telemetry, and the entire project of VSCodium which has to exist to provide telemetry free versions, and still cannot remove all of it. You're discounting the complete waste of human time and effort required to undo something that should simply not exist in the first place.
In terms of the open GH issues, people are pretty vocal about which ones they think are most important to fix, as is the case for most popular projects. It's simply not true that the Go team have no way of knowing which of the open issues are most important to the community.
I think that the author has kind of a Stockholm syndrome.
"Users are liars, so let's spy them directly to know what we want to know".
It is mind blowing how, as an user/the target, you can support that.
Nothing is really anonymous and your anonymous data can say a lot about you.
Telemetry coming from this IP, so company x is using go. A pattern of data coming every 2 days, so their build nodes rebuild every 2 days. That kind of build pattern is there, so they are using the xxx crypto library...
And when they say, let's trust Google, I would propose to Google to accept the opposite:
Now they will transmit to the public telemetry of their internal systems: how many users, what do they do, how many users they block, for what reason, how many build nodes they have, how many commits, how long the go team is spending looking at telemetry reports, which website are the more visited by Google employees,...
And let's see if they will accept. It's for the good of the world, why they would refuse?
I find the attitude on display here utterly gross and it makes me wonder where this person got it. Is this the common attitude inside Google or Silicon Valley proper these days?
Privacy used to be kind of sacrosanct in the industry. What happened?
I guess when surveillance driven advertising pays all the bills it has an extreme warping effect on everyone’s mentality. It’s hard to care about privacy when violating it is how you get paid.
I’m increasingly wondering if we might vastly improve our society by banning or taxing the hell out of all advertising. It’s an industry that makes the world a worse place in deep and pervasive ways that go much further than just making everything ugly and obnoxious. The industry pushes a toxic relationship between people and businesses at every level.
> Privacy used to be kind of sacrosanct in the industry. What happened?
It is difficult to get a man to understand something when his salary depends upon his not understanding it.
> I find the attitude on display here utterly gross and it makes me wonder where this person got it.
It is gross. It's also very ignorant. The author's argument basically boils down to, "I can't see an obvious major downside to this proposal, so that must mean there isn't one nor will there ever be one". How many times to we have to get burned by large companies invading our privacy and after promising not to before we wise up?
Look, I hate large corps too but this paranoia hinders open source's ability to self-cooperate.
> Telemetry coming from this IP, so company x is using go. A pattern of data coming every 2 days, so their build nodes rebuild every 2 days.
Not true. Even if Google lied about collecting IPs, the data would be sent only every ~ year with aggregated counts so no real time usage data. And even if one could see the patterns from the data, everyone will be able to, not just Google.
Let's not trust Google. But let's not shoot ourselves in the foot by refusing any automated cooperation.
Additionally, majority of distros use package managers, so if any of the major promises of the upstream would be broken, distro packages could patch it out. This isn't forced, there are several points where anyone can stop the telemetry.
How is it paranoia to simply propose that Google give us the same thing they want us to give them?
If they don't want to do that, why exactly not?
Paranoia is imagining something. This is posing a question that they or you or anyone is free to simply answer, or fail to.
> For every kilobyte of Go code available on GitHub, GitLab, and other Git forges, there are unknowable megabytes of private Go code that will never see the light of day (or maybe they will if LAPSUS$ decides to make that company a target).
> Without knowing what ports of Go are used, the Go team can’t make sure that the right time is spent on maintaining those ports.
Am I being overly pragmatic if not selfish for thinking this is absolutely fine? If you use a free tool behind closed doors to make money and don't want to opt-in to telemetry, then I couldn't care less if the go team doesn't make go work better for your use case. Meanwhile I'm developing my open source project with telemetry turned on, ?????, PROFIT.
i.e. there are 10s-100s of millions publicly available LoC, but we really need to see your private workflow without your consent.
I remember people pointed out that Google already knows this, unless you go out of your way to use a company wide GOPROXY. Every time you sync your dependencies with go mod, you tell google what you're doing.
also why mentioning a group of kids that bought stolen employee credentials and got arrested for it?
It's a joke referring to their practice of dumping companies' internal source code publicly.
> The problem of scrying into the unknowable
People really need to accept that they can't and shouldn't have full introspection into everything they're ever interested in. Even if it's a thing they care about (like their own homepage or programming language). Sure it's nice to get feedback. Sure it's nice to know that somebody uses it. But putting a tracker on everything with the argument of "will help me provide a better service", forgoing informed consent.. It's wrong when it's done on an institutional level, whether it's companies or the goverment, and it's wrong when individuals do that.
The new owners of Audacity did this and then gaslit the community about it for weeks, despite constant, almost unanimous backlash.
Please elaborate on your allegations. As I understand it, there was the original telemetry PR, which had a "Don't Send"/"Send" dialog at first startup. (A screenshot of this dialog was not added to the PR until several days after it was first posted, so many were under the impression that it was strictly opt-out.) The "Send" option was more prominent, which many people called a dark pattern, but it was still an explicit choice by the user.
After that, they resolved to only add "Don't Send"/"Send" crash reporting and opt-out update checking, which only sends the OS version, Audacity version, and IP address. (They claim to immediately anonymize the IP address.) The user is notified about the update checking at first startup.
Which part of this do you consider "doing this and gaslighting the community about it"?
> opt-out
There's your answer.
> Which part of this do you consider "doing this and gaslighting the community about it"?
Telling long time users that the new owners knew better. Everyone was able to go and check out how they ruined their other products too.
They had an army of people telling them "no" and they did it anyway.
You seem to be familiar with the fiasco so I'm not sure how we aren't seeing the same thing....
> There's your answer.
So you're saying that opt-out automatic update checking ruined their product by any reasonable definition, and they were entirely aware of this, so by claiming that automatic update checking is useful they were gaslighting their users?
If so, then from what source are we certain that the "army of people telling them 'no'" in the relevant GitHub issues is representative of how their broader user base feels about automatic update checking? And how do we know that they ought to have been similarly certain?
It's right there in the name.
But the creators (someone trying to be "owners" of sowmth5 that is just a name, lol) has the foresight to make it free and open source, so we have Tencity, the original software with a new name.
I'm not sure what it is you're saying.
(to have the audacity to do something unacceptable)
Ah thanks :D
> Unfortunately, the proposal came from a Google employee. Google has a reputation of taking user data and feeding it into piles of linear algebra in order to skew its search results to match your existing biases or whatever the hell else people do with linear algebra.
If we lived in any other age, people might tolerate this, but we don't, and therefore everyone is well within their right to mutiny, regardless of how interesting a technical solution this is.
Reminded of a post I saw the other day referencing Clausewitz to say "if your ideal military strategy is politically unachievable, it's not the ideal strategy". If going for opt-out telemetry makes your customers hate you and you're forced to retreat under a hail of fire, going opt-in is not a mistake.
There's also a philosophical problem in going too deep into "customers don't know what they want" and A/B testing everything to death: you end up using it as a substitute for dialogue. Ultimately customers are making conscious decisions. You have to make the case rather than assert that it's "better" from behind your metrics dashboard.
Yes, the author of the article makes the mistake of... not listening to or responding to what the other side thinks, just because she disagrees with them. No concern was addressed in her article.
For what it's worth I have actually listened to the arguments that the other side had made, but I didn't want to focus on that in place of the stuff that I felt was missed.
I think the Go developers argued for it very well, but there is a reason they decided to go with opt-out, so calling it a mistake makes no sense to me unless you address that reason.
My argument in TFA is that going opt-in biases the data so much that it makes the process of building that system a mistake. Of course, under the assumption that it is ethically correct to build such a system in the first place.
In that scenario, not going opt-out is not a mistake. But going opt-in probably is a mistake, because the data will be useless, so it’s a waste of time to implement.
For those conscious of the security implications of that code even existing it all comes down to whether you trust Google, I would argue at this point you definitely shouldn't. Given that if you program in Go now and have code you really wouldn't just give Google then you probably need to run all your go executions in a VM without network access. This alone is going to be ardious enough from a security point of view to make other languages more interesting.
The entire idea is bad, the defaulting reduces the impact to many but the very existence of this telemetry is enough to take more significant security defence against the tool. Once you start doing that as an organisation Go becomes legacy with a strong desire to replace it. Its definitely a mistake to make it opt in, the data will be lower quality and it will still drive security concerns.
> all comes down to whether you trust Google
Not true. The code is open source and everyone can trivially with minimal effort check that they are sending only the data they said they would send.
Why would data be useless?
I'd say there are times when bad data is worse than no data. Consider that the farther away you get from it, the parties involved are either unaware or unwilling to treat it as bad data, so they make make harmful decisions.
And opt-out or even mandatory data is without doubt quite bad to start with. And the interpretation of the data is rarely flawless or even pretends to be objective.
So if someone says that opt-in data is useless, I'd say that out-out data is just as useless.
And then we have to pretend that implementation, collection and analysis and then further actions would make it worthwhile.
Googler 1: Do we need this telemetry opt-out functionality? How many users are using it?
Googler 2: The data says 100% of users are opted in, so deprecating the obsolete and unpopular opt-out functionality is the evidence-based choice. Ignore the shrill demagogues on the mailing list saying otherwise, their anecdotes are meaningless.
Ah, are you also a Bret Devereux reader?
Correct! acoup.blog articles are always highly upvoted here, too.
I love his writing. Had just read the piece on Ukraine that you were referencing the day before, so I recognized the Clausewitz (drink!) quote :)
Respecting the users wishes isn’t a mistake. For sure it may have a few digits less of accuracy in some dashboards but who cares.
The consensus was that opt-in was the best solution, and thankfully Google went with the best solution.
While I agree that going with what the users want is the correct approach, but "a few digits less of accuracy" really undersells how much the data quality will suffer.
It will pretty much become useless (how many people are going to opt-in, realistically?) - and that's fine if it's what the community wants.
> For sure it may have a few digits less of accuracy in some dashboards but who cares.
You could argue that it's ultimately the users who suffer. Telemetry helps guide development, so it's harder for developers to know what to focus their efforts on.
But the users can still report issues.
And Golang also has the yearly survey that gives much more valuable inputs than telemetry could track IMHO.
> But the users can still report issues.
They can, but do they? Russ' very first blog post gave several examples where major issues slipped under the radar because they went unreported. The example that sticks in my mind is that they accidentally introduced a dependency on a c compiler on Mac that was not reported for over a year simply because users assumed it was meant to be a dependency.
Several counterexamples were listed in the article.
Assuming every Go user on the planet fills it out.
They don’t have to care about the super rare cases, and in practice they never did. The survey has already a more than big enough sample to take the right decisions.
Reminds me of the whole walrus operator python thing.
People were using it, they took it away (IIRC, could be wrong about that), people whined, they brought it back as an explicit operator, people really whined and the BDFL walked away.
Telemetry would have told them what exactly?
The walrus operator did not exist before 3.8, there was no real alternative before. The backlash was from people who dislike the idea and would prefer Python didn't add it.
The operator didn’t exist but the functionality did. It was just a side effect of python’s wonky variable scoping which they gave an operator.
I’m not familiar with how it works now but the example I remember was the variables escaping from list comprehension statements, like:
Or something like that, maybe I’m wrong, honestly never had a use for the walrus.x = [y for y in z] if y != z[-1]: bad_stuff_happened()Python’s scoping rules are wonky, but the walrus is generally useful in two cases. The first one is to define a variable in an `if` and only run the contents of the `if` if the variable is truey — for example, re.search returns a Match object or None:
The other would be defining a variable in the `if` clause of a list comprehension and using it in the output (which is not necessary in languages where map/filter is a first-class citizen, because you could just map before you filter):if (m := re.search("regex is(n't)? fun")): print("there was a match") # do something with m
I can’t think of a way to define a variable in either of those cases without using the walrus and without significantly extending the code (you can define the `m` object and then do `if m:` on two lines, or you can nest two list comprehensions).[y for x in xs if (y := f(x)) == 5]There's also:
which is a lot nicer than:while data := fh.read(131072): # do stuffwhile True: data = fh.read() if not data: break # do stuff
Taking it to the extreme: Would be even better if they know what you write so send the whole code.
It's just a programming language not organ donation.
Users wishes would be respected in both opt-in and opt-out scenarios.
They really didn’t wish for opt out in the discussions I read.
> Unfortunately, the proposal came from a Google employee
Not really. People hate opt-out telemetry regardless of who runs it.
Good job Russ Cox. Despite initial concerns around this the right decision was made. Opt in is the morally correct choice for any software community. Notably I will opt into this where it is appropriate.
To be honest, the fact that Google is behind Go, is whats holding me back from getting into it. Same with React and FB. I dont really have any sympathy points left for them.
> Developers using macOS are used to just installing Xcode to make the error messages go away, so they just installed Xcode and assumed it was normal behavior.
So, they don’t test go in CI on clean Mac OS installs? If this sort of issue is the rationale for the telemetry system, it sounds like they’re trying for a older, slightly-less-evil Microsoft approach where you fire QA, and just treat user machines as your test environment.
Google: "we don't want to talk to people, or hear their opinions, so we'll add telemetry to make decisions we want without including anyone else!" Also Google: removed "do no evil" so they could... Do evil... Without sounding like hypocrites.
Mozilla shows time and time again that decisions based on opt-out telemetry are worse than decisions taken without telemetry at all.
"Telemetry shows people aren't using this (privacy / internet freedom related) feature, so we can just remove it"
While I agree, Mozilla is a horrible example.
There are bugs reports with 100s of people complaining, about simple features, or broken code, and the attitude is WONTFIX.
If you won't listen to users, seething, upset, annoyed over inane issues you created? If you won't fix bugs for years, because it's your pet project/feature?
Then how are you even paying attention to telemetry correctly?
Opt-in telemetry is only useful if you will, no matter what, implement and fix what it tells you. Mozilla, predicated upon their behaviour, clearly only uses data when it agrees with prior opinions.
opt-in is useless to such an org, and opt-out isn't why it is happening.
It's them. They are broken as an org.
There are vast amounts of Go code available in the wild to study. Open source is a thing.
If the Go team wants to see what features are used, go audit open source Go code.
Since Go is also a Google thing, they could likely audit the presumably large sums of Google Go.
Yes, you will miss unknowable amounts of code this way. Who cares?
If they want to add telemetry without asking the user, then they have to fix an entire industry including Google itself. They trained the people to hard disagree with such approaches, because in the past there was way too many incidents where data was unnecessary collected and misused.
Just trying to add this telemetry with opt-out in the first place shown one thing: They don't know their audience. Because if they would, if someone would have come up with that idea, everybody in the team should have screamed: "No, hell no!". You don't have the trust of your users.
I get it, getting trust back is much more effort than collecting more data (even if it's with good intent). But hey, the industry put itself into this.
They should simply allow for people to pick their default at installation. (so at download)
Even better if the go tool were to have a `go update` with a `telemetry-off` flag or something. (and possibly a prompt to remind people otherwise they will complain again)
Problem solved.
`telemetry-on` flag. Sending data to third parties should _never_ be implicit or the default. Every PM, marketer, and mid-level manager needs to be firmly reminded that this is grossly invasive, user-hostile, impolite, and creepy.
Without an explicit agreement, it could even run afoul of wiretapping laws (unresolved in courts, as far as I’m aware).
I still think it depends on the telemetry itself. Companies knowing my location at all times? Disgusting and creepy. Company knowing I ran `go help` 200x as opposed to running `go build` 10x is fine imho.
It depends on the previously installed default.
When you 'go upgrade' your compiler, if telemetry was off it should stay off. No need to further specify in most cases.
If telemetry was on, you might want to turn it off, however.
Yes, the data won't be as good. But it will still be able to catch issues like that motivated this in the first place.
Knowing the goal allowed rsc to make this judgement.
Honestly, I think they should go with opt-in and make it clear to the community that the public dataset will be the sole source of decision making for future deprecations. If you want your favorite pet feature still enabled and no one using it is sending them telemetry then too bad.
As someone who was considering getting some Go and Rust under his belt, they've saved me a lot of time. This whole telemetry nonsense has ensured I won't waste my time investing it in Go. Great job.
Uh, what? The Go team introduced an idea, opt out telemetry with a one week delay before it was sent. Users said it wasn’t great, overwhelmingly. The Go team heard the feedback and made it opt in. What’s the problem, that’s a good thing.
People are naive, am sure he doesn’t know that rust has telemetry
I'm not aware of rust hanging any telemetry
Rustup was going to add it, some work was done, and then it was abandoned.
This was years ago. https://github.com/rust-lang/rustup/pull/1642
Interesting. I'd assume that the installer is one of the better places to get this information too.
For me every kind of telemetry, data harvesting and so on should be opt-in.
Consent is always opt-in
All these situations I blame Google not Go, sometimes people are ignorant when to differentiate Google corporation itself and the Go team