Twitter's Recommendation Algorithm
blog.twitter.comContext: I teach at Princeton and study social media and recommendation systems.
From a very quick skim of the repositories, this appears to be quite limited transparency. The documentation gives a decent high-level overview of how Tweet recommendation works—no surprises—and the code tracks that roadmap. Those are meaningful positive steps. But the underlying policies and models are almost entirely missing (there are a couple valuable components in [1]). Without those, we can't evaluate the behavior and possible effects of "the algorithm."
I work on Google Assistant Suggestions and I don't think it's very practical to open-source an algorithm like that including the models and the underlying data. Both of them can live in separate services and be frequently updated.
I am assuming that open sourcing the code aims to increase transparency about the business logic of the ranking decisions. At the same time you don't want spammers to be able to easily run experiments against a cloned version of your system.
> But the underlying policies and models are almost entirely missing (there are a couple valuable components in [1]). Without those, we can't evaluate the behavior and possible effects of "the algorithm."
Haven't gone through yet, but yeah, if that's the case, all this is, is a glorified framework to plug your own in.. Not exactly what was promised.
Did you also skim the accompanying (or rather, main) repo, https://github.com/twitter/the-algorithm ?
From a quick clone and line-count, it has:
So I don't think you did, since you posted so quickly and that's a LOT of code.235 kLOC .scala 136 kLOC .java 22 kLOC .py 7 kLOC .rsI also haven't skimmed this code except very superficially, but perhaps you should since you're out there making statements with your Princeton credentials.
(I posted this comment with the heads-up a few minutes after your comment above and then expanded it as you didn't respond.)
I think you misunderstood. He's saying the training models are not there.
For example, MostRecentCombinedUserSnapshotSource seems to be influential (such as for calculating "tweepcred"), but we can't see how it's calculated.
Wouldn’t that make them easy prey of “spam SEO”. However, given the framework isn’t it still possible to guess the models?
The spam SEO issue should be dealt/thought about _before_ engaging in the whole adventure, and having to guess how it could work if decently implemented properly defeats the "open source" spirit of it.
More credits would be given if the very idea of open sourcing the algorithm hasn't already been discussed to death with predictions of the difficult points and how it probably won't happen in any sane way.
And them be pilloried for not doing it or not fast enough. Damn if you do, damn if you don’t.
I’m starting to think the broblem with Elon is mostly personal, he’s just a proxy and default wrong.
(not that I approve of his behaviors, but I can’t enjoy this whole mobbing that he’s getting; not that he cares this I’m not worried he’s getting traumatized in any way? it’s just how it’s become an identitarian trait for a certain group that irks me.)
Makes me wonder if a way to override people SEO hacking the algorithm is to create a market of open-source algorithms that each individual can choose and then it's not trying to hack THE algorithm but having to hack many and not knowing which algorithm an individual is using.
You don't have to target each 'algorithm' all at once. You can target them one at a time. Hell you can run A/B test single out the easiest targets.
Yes but right now there is 100% of the users using the one algorithm (or chronological). If one doesn't know what percentage or which people are using which algorithm, it becomes harder to know which ones to try to hack to have the biggest result.
What about these? https://huggingface.co/Twitter
Those look older to me. They all have last updated dates for October and November 2022.
FB open source algo looks much better, right? /s
Is it valid to focus tracking a Dem/Rep split when that split is an exclusionary design for many Americans? Or is it not exclusionary in your belief? I'm curious of a social science perspective.
Ignoring the global nature of Twitter for a moment.
So why did they opensource it?
So they could pretend to be open. It's the "Open"AI model. Open-washing?
This is a very cynical take. They should be commended for publishing recommendation code at all, which no other major social network does.
Well if they say “we will open source the algorithm” and then what they really open source is a little bit of slightly relevant code that doesn’t allow us to understand the algorithm, then what we can deduce is that they are trying to weasel out of public commitments.
I can’t say for sure if that happened, but if they made a clear promise and then did something else, it’s perfectly reasonable to call that out.
Devil's advocate though: imagine you were to open source (probably with quite a short deadline) some 'algorithm' used in whatever you work on, but the rest should stay private; how would you go about that?
I don't think it's easy, there's inherently some interface(s!) where it's a hand-wavey 'get the thing from the private bit', and defining that sensibly is hard, and if you try to do it well will probably lead to a lot of meetings, scope creep, etc. - and as far as that goes it's not easy anyway, since it's highly technical and implementation-specific yet also a management/policy decision to make.
It depends on what your goal in open sourcing is. Are you looking to provide a base for others to build software on, and to provide a way for others to contribute back to your code? Then publishing the code makes sense.
Are you looking to build public trust in you and your organization? Then dumping a bunch of code with no context isn't going to help much, as it's not code but behavior that builds or destroys trust.
Are you looking to lean into a polarized partisan environment, pushing a narrative where its you and your supporters against an unfair group of "others"? Then a big splashy move high on symbolism and low on substance that will inspire lots of high profile, divisive media coverage is a great way to go.
If you were doing it in good faith, you wouldn't need to publish the actual code. Most likely you should publish an article and a flowchart explaining how the algorithm works. Publishing a partial chunk of code just creates a story that supporters who don't understand can parrot that "they opened their algorithm".
Exactly. Publishing what they have is the worst of both worlds - hopefully people will create flowcharts based off it, though, although it sounds like there will still be a low level of accuracy.
I still hear reverse-FUD about nvidia supposedly fully open-sourcing their Linux driver, when in reality they opened a tiny kernel portion of it that allows the main proprietary blob to connect to necessary kernel interfaces. You have to call out this bullshit when you see it.
Wait, what? AFAIU what you say is true, except for the part where the “main proprietary blob” does not run on the CPU. This isn’t as glorious as an actual open-source driver would be, but it does have meaningful advantages—e.g. you now have a ghost of a chance of implementing Nvidia GPU support on a non-Linux kernel, by uploading the GPU-side blob and rewriting the CPU-side shim as required. Or is the blob license-restricted from being used line that?
The "main proprietary blob" they're talking about is the userspace portion of the driver; the portion which does all of the heavy lifting. That definitely runs on your CPU. The only part they open-sourced is the kernel portion of the driver, which just exists to facilitate communication between the userspace driver and the hardware.
Hey, we can get even more cynical. Why should we trust that this code is even similar to what they run in production currently?
I can't imagine deliberately special casing Elon's account in something they made from scratch to fool people.
Let's have reasonable goals, shall we ? "Their shit doesn't stink as bad as others'" is nothing commendable, especially after souch publicity.
I say "why not both". Even if they are doing it only for good PR, we encourage it by giving them praise, because we should encourage things we want. (While remembering that they are not our friend, they are an entity we should pressure, and the way we pressure is by giving praise when they do things we like, and critcisim when they do not).
I’d give them more credit if they’d been honest and kept it secret then lie to my face and pretend they didn’t?
They should be commended for open sourcing something they don't understand because they fired all of the people whom understood it? Elon admitted as much.
This is like FB open sourcing the compiled frontend code you can see yourself using inspect.
If we commend them for this we're helping promote and encourage this faux open source virtue signaling
No, that's very different.
There is clearly a lot of information to share. It's worth considering this could be step 1 of n as opposed to assuming the worst possible intention.
It's healthy to have a normal amount of cynicism. They released it for a reason. "The goal of our open source endeavor is to provide full transparency to you, our users, about how our systems work."
Why be transparent (or try to appear transparent)? To convince people to trust your platform (or to recruit - which seems to be another goal of the post). Why would Twitter want or need to do this now? Well, there is a bit of context. This disclosure doesn't exist in a vacuum.
I love this take. Doomed if you do, doomed if you don't.
If we are willing to not assume some borderline "it's what they want you to think" conspiracy play, obviously there was always going to be a lot of highly interested and qualified people taking a very close look at this and, at some point, there was always going to be very definitive conclusion of what's the deal with what they released.
If your play was "it's some source code, hence people will think we are open, and that should be really good for us", that would make you a very special kind of idiot in this space.
That was one of Elon’s core statements when he first talked about buying Twitter. If he had gotten it out sooner there would be an easier link between the two, but if you want more context just go read the old tweets and articles from the Twitter vs Elon days.
If we can't build anything with this, is it "source"?
"Does not include batteries"
You must be new to Musk's business practices.
It's no secret that Twitter, like any other social media platform, is driven by user engagement and ad revenues. The more time we spend on the platform, the more valuable it becomes for them. With this new open-source algorithm, they're essentially crowdsourcing improvements to their system to better serve us the content we crave.
this move could be seen as a strategic PR play to boost their public image amidst the growing concerns around algorithmic bias and lack of transparency. By inviting the community to collaborate and address these issues, they're not only shifting some of the responsibility onto the users but also deflecting potential criticism.
Because they let go many of the engineers working on it?
Noone has mentioned this before - I don't know if it's really related, but afaik the European Union is thinking about requiring social media platforms to be more transparent when it comes to recommendations etc. If you can already say "hey we have a lot already online!" then maybe the laws will become less strict.
bc he have no devs anymore and thinks the community will fix it for free
PR and it was already leaked last week.
PR
> But the underlying policies and models are almost entirely missing... Without those, we can't evaluate the behavior and possible effects of "the algorithm
And neither can spammers find and test the cracks and edge cases that would allow them to break the system, that does sound reasonable to me. If they were public there would be an arms race between spammers/those wishing to game the system and Twitter engineers.
Then don’t pretend to release “the algorithm.”
They’re explaining how it works without giving the specifics. Much like the US military explains how the nuclear deterrent works without disclosing detailed plans and control codes.
It's an open algorithm, but it's not open data! (joking)
What did you expect?
I don’t know if the parent’s expectations matter here. This is more about making sure others don’t misunderstand the meaning here.
Good point. I didn't see it like that. Thanks!
Can i audit your classs for free?
Great! But nothing is going to change until people realize that the problem is the feedback loop. It's not the recommendation engine itself, it's the fact that there's no way "out" of the feed that the engine produces. It recommends you stuff, you have little choice but to engage with it, and then it trains on that information.
This is the problem with most of social media today. It is a very well known problem in ML [1], but nobody is willing to do anything about it because it's a fundamental UX change. Facebook, Twitter, YouTube, TikTok, they have defined themselves by their recommendation engines.
[1] https://towardsdatascience.com/dangerous-feedback-loops-in-m...
reminds me of a story about a guy who was given a gift, a decorative plate with a rooster on it i think it was. didn’t care for it too much, but out of politeness put it on display on an empty cabinet he had. a while later someone noticed he had it and figured he liked it, so got him a similar decorative plate with a rooster on it. again, out of politeness, he put it next to the old one. now other people started to think he just really liked roosters, and started giving him little rooster statues and nicknacks. Eventually he just has a whole display cabinet of rooster themed gifts that he never really cared for to begin with, but people just assume he likes them because people keep giving them to him.
I had a friend who this happened to with ducks. You would go to her house and there were little duck icons and duck figurines and duck themed things everywhere.
When questioned, it turns out she never cared about or liked ducks. Someone gave her one and then it became the go to present for her for every occasion, for decades.
Decades of ducks.
I am probably part of this problem - an enabler, so to say. I was hosted a nice lady with a house full of duck themed things (this was in southern England, probably not the same lol), and left her two (rather cute) duck-themed things when I moved assuming that she'd like those.
She was unequivocally happy when I did, but perhaps it was just because of the gesture.
(EDIT: also, the rooster/horse/duck example of feedback loop would make for a terrific blog post)
I have a whole shelf full of horse stuff because of this!~
this is a hilarious example for a serious problem. well done!
and in the end, he starts to like that decorative plates with roosters or at least thinks rooster on plate is a nice decoration if it is produced that much.
I think Instagram in particularly is bad in this regard. It seemingly becomes convinced that I care deeply about the subject of any post that I even momentarily linger on.
Don't you have a choice.. to not engage with it? If you didn't like it then assuming the metrics system is working correctly, this would be negative feedback to the ML model, causing said content to not be shown in the future.
Couple problems:
1. Actively supplying negative feedback is sometimes hidden behind secondary menus, making it much higher friction compared to just...scrolling past. So most users don't spend the effort. Even with a dislike button, it's unclear what the system is learning. It can't know that I don't like this particular video because it's a conspiracy theory, and to stop showing me those. These platforms often don't even support explicit categories, so how would they know?
2. It's extremely high friction to teach the algorithm you're interested in something that it doesn't suggest to you! There's the whole unknown unknowns problem: how do you teach the algorithm you're interested in something that you've never seen before?
I still think Reddit has handled this the best. No system is perfect, but Reddit's challenges are much more manageable than the quagmire that TikTok, Facebook, and YouTube have gotten themselves into. I can just unsubscribe from r/conspiracy, and I'm out. Basically impossible to teach that to YouTube without weeks of careful curation. They think they're smart enough to know what I like, but they're not and never will be.
Eh, 1 is sort of why I see implicit negative feedback as more useful here. Namely, tracking the duration a user is probably giving their attention to a given item, weighted in accordance with how long you'd expect someone to give their attention to an item based on how long it is.
For example, I might see a some specific word or pattern of words in a tweet and quickly skip to the next one. That's very low friction but also a powerful signal. There are drawbacks (e.g. the long boring video that looks like something is about to happen but never does), but that's mitigated by combining this with other signals.
With 2, it's an explore/exploit trade-off. You try to explore as far and wide as you can while trying to avoid the things the user may dislike, all while sprinkling in just enough of the stuff that you know they'll like.
You're talking like an ML engineer. :)
Yes, there are signals in human behavior that you can feed to your model. But no, it is never going to learn "on Mondays he works on his comics, so he'd prefer to see webcomic-related content" Don't sprinkle in your explore/exploit experiments. I know what I want, just let me decide based on what I'm in the mood for!
TikTok-style feeds are the absolute worst offender here, where they couldn't care less about what you think. They will serve you content, and you will either say "yes" or "no". So the only option for you as the user is to just wander through their content hyperspace. There's no structured way to jump between topics because everything lives in this formless content soup.
The other problem is that many social media platforms have you subscribe to the content streams of individuals directly. Individuals are high variance. How can you teach these engines that "I only care about this person's posts about pianos, not their terrarium hobby."
> Namely, tracking the duration a user is probably giving their attention to a given item,
I pay a lot of attention to classes of things I don't want to see at all.
Then, according to the algorithm, you do want to see them.
That's exactly the problem! I don't want to see them, but the algorithm refuses to ask me.
If they think that the length of time I spend on a post is 100% correlated with the amount that I want to see it, they really don't understand people at all. It's an embarrassingly shallow model of how how humans behave.
> It's an embarrassingly shallow model of how how humans behave.
It's a trade off. Viewing rates give back feedback 100% of the time. Asking users for a thumbs up or down gives feedback almost none of the time, and still might not be accurate.
Twitter could recreate a similar system: 1) auto-tag tweets with labels, rather than users needing to submit to subreddits 2) auto-sub most people some default set of labels 3) let them un-sub if they want.
They just don't want to.
Auto-tagging is not a business they want to be in, because they will invariably get it wrong sometimes and people will start drama over it. And they'd be right to: human communication is nuanced and you can't just "guess" what someone means within 280 characters.
People already use hashtags, what would happen if you could subscribe to them? It would create a whole new way of interacting with the platform, something closer to Reddit where people post to streams. But the fact that these streams can't be moderated (or even downvoted) might be a nonstarter.
Wait, Twitter already has this in the form of Topics. It's very coarse-grained, but still.
Regarding 1. I would expect like rate on recommended tweets to be a better signal than impressions.
Thank you! Working on a concept for a big org on what may become a large ML-based system one day. I knew about this feedback loop issue, but was too dumb to actually remember and face this problem. :) It's all over today's rec engines -- and yet, just like the things we're not shown in these systems, the problem itself seems to become invisible. Because it requires new thinking.
Worth exploring.
Reddit is a good example of an alternative system, and it works because subreddits serve as lower-variance composable streams as well as shared digital spaces. Even though your main feed is partially driven by a recommendation engine, it never contains any content that you didn't subscribe to (ignoring ads, of course). Instead, you may see a "posts from other subreddits" section that breaks up the feed explicitly to offer alternatives. If you're interested in a particular topic, you can drop into that subreddit, where you'll see pretty much the same view as everyone else. In all these cases, you're always in control.
I wouldn't be surprised if Reddit actually has the best data on their user's preferences, because they let them explicitly decide what they want rather than playing this "hot and cold" game like TikTok. Those platforms also only let you subscribe to individual accounts, so they have to infer what your interests actually are.
I feel like the Youtube one is good. You can mark videos and channels as "not interested" and Youtbe really knows me due to my account age and usage... It recommends me unknown videos and I tend to like them but also more mainstream stuff.
It doesn't matter how good they try to make their recommendation system, they will never know you like yourself. For example, when I go to the YouTube home page, there is a list of categories at the top that it's identified for me. I didn't choose these. I can't add or remove them myself if it's wrong. I just have to hope that I watch the "right videos" and it picks up on a new interest that I have.
But I already know what interests I have! I want to have videos about terrariums on my home page now, not in a week when I've watched enough. This is what I mean by recommendation systems not being good enough. They need to give the user more control over what they want to see, because they can never read my mind. Their recommendations will get even better with that information!
Isn’t the “I’m not interested in this tweet”, “Show me fewer tweets from X”, etc options working for you? They seem to have an effect on my end.
What did they learn though? They can't possibly know why I'm not interested in that tweet, because there's just not enough information. So all they'll do is stop showing me tweets from that one person. No way to express "yeah I'm interested in his tweets about music, not gardening." The problem is that mostly following people means that you can't teach the algorithm about your particular interests.
Topics are better, but there doesn't seem to be a way to make sure your tweet ends up in a particular topic. Again, you have to pray to The Algorithm that it will label tweets correctly. And if your particular interest isn't a Topic, you're out of luck.
So , all we need to do is to open source humans?
Haha no, we just need to stop pretending as if ML can read our minds. It's a tool, it's good at recommending me content, but it will never know what I'm in the mood for on a particular Friday night.
Great pull request here which improves the algorithm: https://github.com/twitter/the-algorithm/pull/17
That would be great (unweighting bluechecks) but they actually plan to go in the other direction: Starting April 15th non-bluechecks won't show up in the "For you" section (the algorithm timeline) at all. Unpaid users are being written completely out of the algo.
Inaccurate. Musk stated that people you follow will continue to show up. I only use the 'for you' feed when I'm bored and want stupid dopamine hits, I leave it on Following almost all the time. But that's on desktop, my understanding it that it keeps resetting itself for mobile users (of whom I am not one).
He only said that later; he seems to be working on a "say we'll do feature and then change it all after people yell at him for weeks" process.
Sounds like moving fast and adjusting to user feedback, usually something commendable in tech, right?
No. There's an implied willingness to listen to people in the first place when you're agile in response to feedback. You shouldn't have to bruise Elon's ego to get him to do not-stupid things.
This distinction you’re making relies entirely upon how you personally interpret the motives of the people involved. An interpretation that you’ve entirely invented for yourself.
Approximately every US news organization and Lebron James announced they weren't going to pay for checkmarks yesterday, so he announced a surprise twist where the top 10,000 famous people get free checkmarks.
…so he's now not charging people who can pay, while charging people less able to pay.
Though, the main problem is that Twitter is about as big in Japan as it is in the US, but Elon only thinks about US culture wars because his only friends are a few VCs and a retired postal worker crank named catturd2. So none of his new ideas are going to work there.
Criticizing somebody by speculating about their motives really only reflects badly upon you, especially if you’re going to accompany it with that much hyperbole. All you’re communicating is that you’ve replaced your ability to reason about a topic with some version of the fundamental attribution error.
> the main problem is that Twitter is about as big in Japan as it is in the US
It’s not even close. When you account for the value of ad impressions, the US market is worth about 3.5x its next biggest market (Japan), and it drops off very sharply after that. You can argue that Twitter’s policies should be less US-centric. But the reason they are that way in the first place is because the US market is their most valuable market by a long shot.
> Criticizing somebody by speculating about their motives really only reflects badly upon you, especially if you’re going to accompany it with that much hyperbole.
No, you can tell who he cares about because he replies to them, and because his emails were released in discovery from the earlier Twitter lawsuit and it turned out his friends put him up to it because they were mad the Babylon Bee got banned for a US culture war pronouns joke.
I certainly don't have to expect he'll make good decisions, since he has no experience with the business, is a completely atypical user, was forced to buy it in the first place, and has lost $20 billion so far by his own valuation. (May have lost a few millions more by firing that disabled Icelandic acquihire in violation of his contract and a few discrimination laws.)
The network effect is very very strong though. Even if someone makes a good competitor (Instagram is prototyping one apparently) I'd be surprised if bad decisions actually killed it.
> When you account for the value of ad impressions, the US market is worth about 3.5x its next biggest market (Japan), and it drops off very sharply after that.
I suggest reverse adjusting for how poor their ad targeting is. I've used it both places; the ads are actually good and relevant in Japan (…except for being for domestic apps, so not valid for tourists) but in the US they've always been nonsense. eg if you follow any doctors it will just assume you're also one and burn the budget of every medical ad it's got showing them to you.
More like rewarding someone for putting out a fire that they started. Not exactly a model of good governance.
Musk said from day one of his Twitter purchase that there will be mistakes. Move fast and break things as that other guy said. :)
I have had it reset a couple times on mobile a few months ago but at least recently I have not noticed it happening anymore.
Just pressing the home icon twice takes it back to For You.
Which is incredibly annoying, as it used to scroll you to the latest tweets, I have years of muscle memory for hitting the home button
I don’t see a way out of this with the GPT/AI able to create fake persona in an instant.
What does that matter? If people find the content engaging then it will be amplified. If not, it shouldn't be there in the first place. This whole "AI / Bot swarm" excuse is just smoke and mirrors for "I want more people to pay twitter".
If "bots indistinguisable from humans" find your "account that posts pro-russian propaganda" engaging it will be amplified onto your For You page.
If _accounts_ find the tweet engaging
If you don't care whether the content is generated then you don't need to use a social network at all. GPT can generate you an endless interesting feed no problem.
I believe LeBron James said recently he isn't going to waste his money on a blue checkmark, so it should be interesting to see what stays and what goes.
Most of the major news outlets are not doing so either. The Elon stans are crowing that this will be the long-overdue end of legacy media, but it strikes me that the new 'blue check twitter' might end up becoming even more of a social bubble than what it replaced. There are so many low quality accounts sporting a checkmark now that users who value substance will soon be incentivized to just block anyone they find annoying.
yeah, they, uh seem to have realized that's gonna be a problem.
https://variety.com/2023/digital/news/twitter-business-month...
and of course all this means is that the organizations most likely to be able to afford it won't have to
Good find! It'll be funny to see if the incumbents respond with 'don't do me any favors.' Also to see whether Musk's frens sulk aboit him selling out to the elites or so - their gratitude has an extremely short half-life.
NY Times, WaPo, LA Times and other major accounts too https://www.thewrap.com/ny-times-la-times-not-pay-for-twitte...
Seems dumb of them. Cost is trivial and their competition that isn’t so politically motivated will have a much further reach.
The smart move would be silent on the policy change, pay, and support rival platforms as they can. Instead they will eventually pay and look like they lost.
It's smart because Musk already blinked
The top 10,000 are getting exemptions and won't have to pay
https://variety.com/2023/digital/news/twitter-business-month...
> Seems dumb of them. Cost is trivial and their competition that isn’t so politically motivated will have a much further reach.
It's wild hubris for twitter to try to invoice/penalize the very users and organizations that make twitter anything but insolvent. There should be money exchanged here, but it should be flowing generously and most importantly in the other direction.
maybe they can make their own twitter and give themselves a big blue check mark there
For the NYT to verify their official accounts plus those of their reporters (using the Twitter Blue Affiliations feature) would be $1m annually. This, for a budget line item that has heretofore been $0. In this economy, that's a reach.
I don't think the NYT is worried about "reach."
> I don't think the NYT is worried about "reach."
LOL they are desperate for reach. Incredibly so; have you not listen to any podcast by them? They are begging people to go to their site. They get a fraction of the organic traffic they used to and nearly everything is driven from other site like Twitter, Google News, Facebook, etc. The internet age has not been kind to classic news orgs.
The NYT has been doing great recently. They're probably the legacy news company that's doing the best online of anyone.
That's on the strength of having a lot of verticals like games, recipes and Wirecutter though.
have you seen @nyttypos and (to a lesser extent) @nyt_diff? NYT online editorial standards are hilariously abysmal.
Yeah and it doesn't matter.
correctly writing words and punctuation on the page digitally printed by "the legacy news company that's doing the best online of anyone" doesn't matter at all? isn't that like the bare minimum of what their job consists of?
I was responding to a post saying they weren't getting online traffic by saying they are getting online traffic. Nothing about the quality of their content.
> I don't think the NYT is worried about "reach."
Then why are they on Twitter?
Having a presence doesn't mean you can infer they are worried about not having a presence.
How do you get the 1 million figure?
An affiliated account to a verified org is $50 per month per seat, so NYT would have to authenticate 1,647 affiliated accounts to reach 1 million dollars per year
Exactly this. Use the published number of reporters at the NYT and multiply.
Are all of these 1647 reporters (they have that many??) and posting on Twitter? That’s a lot of traffic generators or not. Surely they could just do the bulk with 100 or so.
You're suggesting the NYT further tier its reporting ranks, along with all the internal difficulty that would entail. For ex: obviously the 100 have to include the most senior reporters, who are also older and therefore the least likely to create the viral content NYT wants affiliated with their account, so immediately they probably need to look at a much larger number. For another example: social media is different for each reader, or from the other side, each reporter has a constituency. In one season, the fashion reporters are driving views, while the following season it's the European war correspondents or the economics reporters (and all of these desks have subdivisions that wax and wane in popularity).
And all that discussion so that they can spend $72k annually with Twitter, a y/o/y increase of $72k from last year. With no guarantees of reach, because the whole paid-only verification thing is an experiment that began an hour ago. Let me just say that this whole pitch is going to be...difficult... at the point in the economic cycle where we find ourselves.
The NYT has revenues of 2.1 billion. I’m sure they have a marketing budget and probably already spending money on Twitter to get traffic. This isn’t something strange.
Facebook did they same thing btw, just more gradual. For years they changed the algorithm slowly to take away reach from Pages only to offer it back as long as you paid.
Seems like a fair cost to pump spam
We’re in a time where ideology trumps revenue for some companies. You know, “Get woke, go broke.”
Everyone says that about things that made tons of money though.
Oh, I don't disagree. I wasn't necessarily (always) agreeing with it. As with anything it's a matter of degrees.
The phrase is "Better Broke Than Woke", we'll see how it works out for folks
I had to search who this person is, and I still could not care less.
Depends on how they weight the is_user_china_mouthpiece variable
So you need a blue check mark to reach non followers and Lebron won’t get one? Sounds like it’s making Twitter better already.
LeBron doesn’t get $84 of value from Twitter? Definitely not a political statement going on there.
Parent didn’t say it’s not “political”. It’s reasonable for a wealthy person to feel that a system that discriminates against the poor is not a system they want to participate in.
(Note that I use discriminate in the literal sense, as a simple statement of fact.)
But the example you give is an appeal to a universal moral good. Not partisan politics. So despite saying it’s not not political, your justification is that it’s not political.
Also, how did you get a blue check before being able to buy one?
Some people unfortunately view concern for the poor as political. However my point of mentioning politics is to say that “it’s political” is not any kind of gotcha when it was never denied as being political. Regardless of the actual justification being political or not, the “political” gotcha is nonsense.
That’s bullshit. Virtually everyone agrees poverty is a problem. Sure, the welfare state feeds the cycle of poverty, but it’s not like that was the goal.
Absolutely not bullshit. Some cynical people on the internet believe this, and that's what I thought the person I was replying to was saying. It is an extremely low bar to say "some people believe X", and I don't know why you care to question that. Even with your own reply you say "virtually all people agree" and your use of "virtually" acknowledges that not everyone agrees with you, therefore some people do believe what I say. This is anyway such a silly tangent and was not even my point.
The exception proves the rule though.
Are you sure about that? If I break out of the poverty cycle who am I going to vote for?
Republicans I guess, but the point is that doesn’t mean Democrats hate poor people. That’s just political framing.
lol you don't actually believe that's his reason do you?
LeBron started a school where poor kids can get free food and clothes. The man obviously cares.
yeah he gave some money and shows up for photo ops sometimes, sticking the taxpayers with the rest of the bill - swell guy
Twitter doesn’t get $84 of value from Lebron?
Lebron famously uses the free version of Spotify. Maybe he just does not see the value of paying for the blue mark.
He may well, but he may have also concluded that the indirect cost of having a blue tick outweighs the benefits.
Like it or not but it's the twitter that gets value from celebrities. How many people are on social networks jusy so see what their fav celebrites are doing?
It obviously goes both ways. Social media is a megaphone and ego boost for celebs.
The problem for twitter is it isn't the only game in town when it comes to social media, not by a long shot. They're not even in the top ten. They're a megaphone in a large pile of megaphones, and those other megaphones don't bite the hand that picks them up.
Twitter needs the LeBrons of the world far more than they need Twitter.
As someone who was on Twitter long before Oprah, or Elon, or Obama, or most other celebrities and politicians: I strongly disagree.
(Speaking about Twitter the product, not necessarily Twitter the company.)
That was an entirely different era. If you yearn for it, come on over to Mastodon.
Twitter the multi-billion dollar advertising company needs the big whales. I think it was pretty clear the company was under discussion.
What their fav celebrites say -v- what 'the media' says?
Followed users will still show
But i agree it is a bad idea. The worst actors have money to buy the blue marks of an army of accounts.
This is basically making it easier for authoritarian governments to abuse it
Like that wasn't already happening?
not with blue marks
I'm not in the US
It's actually non-US autharitarian governments that the biggest abusers, using social media against their own people
https://www.theguardian.com/technology/2021/apr/13/facebook-...
it's a shame we can no longer short twitter stock
Aside from the spam PRs, there is actually one PR that fixes a bug: https://github.com/twitter/the-algorithm/pull/242/files
Modern Java actually allows `10_000` for that very reason, as does Scala (https://scala-lang.org/files/archive/spec/2.13/01-lexical-sy...)
It removes the extra weight to Twitter blue tweets?
If the property names are to be believed it sets a weight multiplier to 0. So it prevents recommending them entirely.
It sets the default to zero, but apparently can range up to 100. So... what modifies it? (The answer is probably in there somewhere, but I'm sure someone will find it before I do.)
Me feed has lately been full of accounts that have blocked me. Like, I see a tweet from someone unknown, click their profile and it says I'm blocked.
So wonder if some value is wrong in one of those constants. Anyways, the blocking feature is broken..
I think it's okay to give Twitter Blue users a boost, as it's most likely not spam (unlike the 95% of my non-blue followers who are bots).
I think it makes sense for out-of-network. However I see no value for boosting among people that I have followed. If I have followed them they definitely aren't spammers (from my PoV).
I agree! Haven't thought of that.
Drive-by pull requests that break the intention of something aren't ever going to be taken by a maintainer.
Why do companies even bother to put source up on github? To put up a front that their open source? What a joke.
Yes please! I definitely put my thumbs up in there!
That will definitely do something! Good job!!
The irony is that I prefer Mastodon's sort by time and don't try to be clever approach to this expensive and futile attempt to feed me an endless stream of click bait. I objectively spend more time on Mastodon than on Twitter at this point. It's more engaging for me. It's how Twitter used to work when it was still nice to use.
If Twitter wants to put a stop to the user exodus and save lots of money in the process, here's what they could do:
1) Add an off switch to the for you feed. I'll click it right away and never turn it on again. Stop wasting minutes of CPU time on my behalf. I never asked for it. It doesn't do anything for me that I need or want.
2) Sort by time, filter by hashtag. Twitter used to be about real time information. I don't care about things that happened days or weeks ago. I don't need to see all of it. This is the core feature that made Twitter popular. Mastodon has it and it is absorbing users from Twitter by the millions. It still works. Restore this feature and make it the default.
3) Join the fediverse. That's where a lot of the former hard core users went. They still exist. They still post messages. They still engage with each other. They just don't use Twitter anymore. Allow people to follow mastodon users. Allow mastodon users to follow Twitter users. Not that hard to implement and probably would do wonders for user engagement.
They did add an off switch to the for you feed. There's a tab for following, albeit not sorted by time.
But it's not sticky and it's easy to miss when it forces you back to For You. Less of a problem on desktop because you can write a trivial extension to force Following, but not an option on mobile, unfortunately.
Following IS sorted by time, though.
Hm it Is sticky for me. On Twitter.com
I think, very recently, it was made sticky again on twitter.com, yes. I'm not sure when since I use an extension anyway, but it definitely wasn't sticky about a month or so ago.
As near as I can tell, Mastodon doesn't really have #2 in the list above. Last I heard, the social architecture was hostile to comprehensive indexing of the entire fediverse for search.
That's probably one of the biggest reasons that I have remained on Twitter even after setting up a Mastodon persona.
Mastodon searching is a lot more focused on hashtags rather than the contents of peoples posts. Consider it more "opt-in" discovery.
The existing user base is strongly in favour of a more organic social graph from exploring tags for shared interests.
Browsing tags is a very normal thing to do on the platform.
Hashtags work great on mastodon. I follow a few of them. It's all sorted by time.
Interesting:
// we only keep unfollows in the past 90 days due to the huge size of this dataset,
// and to prevent permanent "shadow-banning" in the event of accidental unfollows.
// we treat unfollows as less critical than above 4 negative signals, since it deals more with
// interest than health typically, which might change over time.
val unfollows: SCollection[InteractionGraphRawInput] =
GraphUtil
.getSocialGraphFeatures(
readSnapshot(SocialgraphUnfollowsScalaDataset, sc),
FeatureName.NumUnfollows,
endTs)
.filter(_.age < 90)
How long does the NSA record them?
From https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
(
"author_is_elon",
candidate =>
candidate
.getOrElse(AuthorIdFeature, None).contains(candidate.getOrElse(DDGStatsElonFeature, 0L))),
(
"author_is_power_user",
candidate =>
candidate
.getOrElse(AuthorIdFeature, None)
.exists(candidate.getOrElse(DDGStatsVitsFeature, Set.empty[Long]).contains)),
(
"author_is_democrat",
candidate =>
candidate
.getOrElse(AuthorIdFeature, None)
.exists(candidate.getOrElse(DDGStatsDemocratsFeature, Set.empty[Long]).contains)),
(
"author_is_republican",
candidate =>
candidate
.getOrElse(AuthorIdFeature, None)
.exists(candidate.getOrElse(DDGStatsRepublicansFeature, Set.empty[Long]).contains)),
)Only used for metrics, apparently. [0]
[0]: https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92.../** * These author ID lists are used purely for metrics collection. We track how often we are * serving Tweets from these authors and how often their tweets are being impressed by users. * This helps us validate in our A/B experimentation platform that we do not ship changes * that negatively impacts one group over others. */... Metrics tracked in AB test. So even if it's not explicitly encoded in the algo (or implicitly through some of the features plugged in), they'll pick the winning cell as long as it doesn't hurt Elon's metrics (I'm just parroting the comment you quoted).
It doesn't have to be in the algorithm for the systems to be tweaked to please Elon vanity metrics.
[I've been running lots of ML AB tests over the years, some in organizations of similar size & complexity as Twitter]
That lines up with reporting from Casey Newton a few days ago where a handful of VIPs e.g. Musk, LeBron James, AOC were being used as weather vanes to understand what the algorithm was doing.
It definitely isn't just metrics. Any algorithm change that negatively affected Musk was clearly not going live.
Do you think the code looked like that prior to Elon's purchase? I suspect that there was another name there before.
Separately, which of these groups do you think that they use as a control?
> I suspect that there was another name there before
Who ? Musk is unique in being obsessed with being liked and relevant.
All of the other social CEOs including Porag and Jake have never really cared that much. And none of them participated in contributing content anything close to what Musk does.
>Who ? Musk is unique in being obsessed with being liked and relevant.
This is a meta-level bias!
I miss Jake
Do you mean Jacky Dorsy?
You mean Jack Dorsey?
How easily we forget our temp workers ...
Good old Duck Jersey
No, Marcy D'Arcy's cousin Jacky D'Orcy
Why is Musk obsessed with being liked? There was a hoax last month about the algorithm being tweaked to make everyone view Elon’s tweets which was provably false.
It is very much likely to be the CEO a company wanting to understand his companies product.
> a hoax last month about the algorithm being tweaked to make everyone view Elon’s tweets
Didn't this just reveal that they're A/B testing on Musk's tweet performance? At the very least they're avoiding regressions, and I guess any incidental improvements to it won't be reversed, so isn't it fundamentally the same? Unless we're taking the word "everyone" literally, I guess.
I don’t remember the specifics but I think there was a case where they accidentally amplified his tweets too much and they did reverse that change. He tweeted about it but I think it was a couple months back so it would take a bit to find.
Tabloid sites (Verge, Vice, etc) reported it as making Elon’s tweets appear in everyone’s timeline all based on the same blog article.
not tabloids, not a hoax, did you go on Twitter during that period? I have him blocked and saw his tweets
Yes tabloids, both known for making up stories and sensationalism, all sourced from the same single blog article, easily confirmable looking as false at the view count for Elon’s tweets, explained by engineers as exactly what we saw here (logging output not changing output) and I don’t believe you re: seeing his tweets despite blocking him. Nobody else had that happen.
I saw several people say they were seeing his tweets despite having him blocked during that incident
They're no more tabloids than Tucker Carlsen is a news anchor.
This was a very odd comment
I don’t know why you think I’d like Tucker Carlson or why Tucker being sensational would make Verge and Vice not clickbait.
We can't comment on why, but there's no rational way to watch his behavior and assume he isn't obsessed with being liked. He constantly tweets about his own tweets performance, makes humiliating appearances on stages, and pretty much terrified that someone might do something 1/100000th as awful to him as he and his family have done to others.
he is isn't a CEO, he is a whining baby
> He constantly tweets about his own tweets performance, makes humiliating appearances on stages, and pretty much terrified that someone might do something 1/100000th as awful to him as he and his family have done to others.
Ah so he makes jokes, and is well known enough to need security precautions. This makes him “a whiny baby”. Solid argument.
The guy is literally creating the next gen intercontinental ballistic missile system and military delivery for the US, but according to HN it’s whiny for him to need security. Right.
Oh and I forgot, he also helps ukranians vs the Russians. Not a good idea if you want a risk free life.
Hard to see the difference between that or someone so obsessed with twitter that he paid a huge premium to own it.
The alcoholic bought his favorite bar.
I've enjoyed Matt Levine's metaphor in Money Stuff - an MMO-addict bought the maker of his favourite game.
Someone wants to measure their business after paying what was a reasonable figure when they offered it, but that became overpriced once the market changed.
Sounds like fairly rational behaviour.
HN isn’t a site for making unfounded personal attacks on others.
His offer included a meme number and was overvalued even in April '22 (by 54% !), claimed he was going to combat the spam bots, and then tried to back out, citing the same spam bots. The event was described as a "hostile takeover" at the time [0], a term which has stuck all the way through to most summaries of the event [1]
[0] https://variety.com/2022/digital/news/elon-musk-offering-to-...
[1] https://en.wikipedia.org/wiki/Acquisition_of_Twitter_by_Elon...
A hostile takeover has a specific meaning. It's simply a takeover attempt where a buyer approaches shareholders instead of the company's management, this is usually (if not always) because the latter fails. And it always comes with a premium and drama, precisely because it goes against the wishes of the management of a company. And the premium is also logical because presumably shareholders of a company "believe" in that company, and an outsider now wants to take it over. A good example [1] is when Inbev (massive multinational beer company) purchased Anheuser-Busch (Budweiser). It made the purchase of Twitter look downright cordial, but was probably outside of most of our bubbles.
[1] - https://en.wikipedia.org/wiki/Anheuser-Busch#Acquisition_by_...
Yea, to make people buy your company you need to offer a premium. He backed out saying he was lied to about the current level of fake accounts, which is also reasonable if that was the case.
> which is also reasonable if that was the case.
It would have been reasonable if he had performed due diligence and came to that conclusion, but he explicitly chose to waive due diligence, and then tried to renege anyway.
It is very difficult to describe anything Elon has done to Twitter since (and including) buying it as "rational".
As a person who occasionally uses Twitter, but has no interest in having an account, he has dramatically improved the experience. Before he took over it was made impossible to view replies from a user, users were spammed with popup login dialogues that could not be closed with a single click, and more. I expect Twitter probably drove more people away from the site with all this nonsense, than they coerced into registering.
I also think many of the changes to make Twitter more open, such as now open sourcing the recommendation algorithm, publishing view counts (which indirectly makes it clear when something or somebody has been shadow banned), and more are all big steps forward in creating a much better and open service for all. Really most of every change he's pursued since taking over Twitter, from my perspective at least, seems to have been big steps forward.
What would you say has been irrational?
> What would you say has been irrational?
Really? You can't even think of one single thing?
It sounds like you can’t either. Do you have an example to share?
He accused Twitter of being politically biased to the left and suppressing tweets, so he bought it to "allow free speech", except in his mind free speech apparently equates to being able to promote right wing propaganda.
As a platform, sure it can be better, and it wasn't exactly under the best management before (although the Twitter file leak actually made the old management look better than everyone thought). But lets not pretend that Musk is in this for the platform - he wants control of information to boost Republican leaders because in his mind, he is the savior of human race, so the best leadership is that which allows him to do the things he wants to do.
> He accused Twitter of being politically biased to the left and suppressing tweets, so he bought it to "allow free speech", except in his mind free speech apparently equates to being able to promote right wing propaganda.
I'm not a fan of Elon at all at this point. That said, this is more an indictment of the tech industry than Twitter alone. Most employees are some form of left, many of those employees sit on committees, have outsized wealth compared to the average citizen, donate accordingly, the executives among them have power, etc and tech plays an outsized roll on our lives now. This was made a lot more evident when Twitters head of trust and safety was interviewed and she was unable to identify, or had let situations go on, that were offensive to people you'd commonly call right where anything that was offensively to what people would commonly call left was getting combed over and at times over enforced. I'd also say that my perception of her was more that she had large gaps rather than outright hate. The biggest one that comes to mind is "#learntocode" which a bunch of journalists not only wrote articles but also used that hash tag on people working blue collar jobs, such as coal mining, that were under threat at the time. It's an isolated example, but that hash tag stayed promoted for quite a while.
This isn't to say that right-leaning oppression and left-leaninf oppression at categorically the same or have the same categorical effects. It does point to that we have larger class issues at play.
Twitter was politically biased to the left. Yes right wing ideas can be there now. Yes that’s part of free speech.
Sure, except Twitter doesn't have free speech. Twitter can and does issues bans for thing said, notably in the Kanye case, and when Elon banned the elonjet account.
It's either free speech absolutionism, or not free speech. If you claim you are for free speech in the moral sense of the word, you have to let everything be posted.
As soon as you start removing things, it becomes your version of controlled narrative. And there is nothing wrong with just that, because of course you want to minimize potential damage. But then you have to be responsible for the things you let slide.
So when Twitter allows right wing rhetoric, a.k.a conspiracy theories about Nancy pelocies husband, pushed by the CEO, or things like Fauci gain of function research, it's clearly a controlled narrative.
Damn, you got me!
If everyone in the anti-Musk crowd was this honest, we'd have much less drama on the Internet in the past half year.
Not everyone can be as honest, brilliant, insightful and just as you dear Sir.
I absolutely will criticize Elon for his behavior around the twitter buyout. He's handled layoff/firing decisions extremely poorly. He's alienated a large portion of the ad buyers. These are not unfounded attacks.
Elon does not need you going around personally trying to shut down other people's discussion. That's cult like behavior.
It was going to be tough to cut Twitter regardless of who did it. Advertisers were pulling out before the deal was even closed, mainly because of Tiktok.
How are the people that disagree with you in this thread trying to shut down your discussion?
a) Musk is a public figure therefore personal attacks are legal and normal.
b) Nothing I have said is untrue.
c) This is not a normal way of measuring the health of a business. Standard metrics include EBITDA, DAU, MAU, LTV, CAC etc.
d) This is not normal behaviour from a CEO. You don't see Mark Zuckerberg, Shou Zi Chew, Evan Spiegel etc using their products the same way Elon does.
If the “normal way” of doing things worked then Twitter wouldn’t have been in such a bad state when he took over in the first place. An unconventional approach brings more risk but can also end up being worth it.
Would I do things differently if it were up to me? Sure, but it isn’t, and I can at least appreciate that they are trying to move quickly & try different things. I’m reserving my judgement for whether or not this approach works in the long run.
Twitter wasn't in that bad of state. If anything, it just needed tweaking to unlock the value it should have had given it's place in the media. Now, Twitter is in a terrible state financially given the debt Musk loaded onto it for the purchase.
The problem is Musk is learning the hard way when taking something over is that the old crew is never as dumb as he hoped they were, and he wasn't as smart as he thought he was.
A company that is 16 years old, still not profitable, & with no concrete plans to achieve profitability is totally dysfunctional and probably wouldn’t survive a high interest rate + likely recessionary environment anyways.
> If the “normal way” of doing things worked then Twitter wouldn’t have been in such a bad state when he took over in the first place.
The normal ways of measuring the business, namely EBIDITA, DAU, MAU .., worked. They reliably reported the state of the business. Elon Musk was the only person in the financial market who looked at those measures and interpreted them as meaning “add US$1B annual debt service.”
The state of the business being…perpetually unprofitable with no plans to change that? That seems functional.
He measured the business into halves; then lost one of those halves.
That is "measuring their business" only if one of the main goals for their business is for it to promote them; because this is exactly what they measure here.
It’s the CEO being able to write something and see where it ends up after going through the pipeline. It does not increase views. I can repeat this, other can repeat this, or you can look at the code.
Or he saw something was wrong (which has now been publicly revealed to be true) and had the money to fix it
> which of these groups do you think that they use as a control?
When you run an A/B test you randomly divide your users into groups, one (treatment) getting the new behavior and one (control) getting the current production behavior. So your question doesn't make much sense?
Depends on the question. If you want to answer a question like “does change X increase engagement?” then a straight A/B test works. But if you want to answer one like “does change X increase engagement while (favoring/not favoring) group 1 over group 2?”, then an A/B test plus measuring groups 1 and 2 will not work without a control group, because without controls you don’t know if any changes to engagement for your measured groups are significant. There is some threshold of change to the engagement for the measured groups which is too small to be significant, and you should ignore results that only measure that noise.
You’re describing a multivariate test. Two groups under two different algorithm variants. Multivariate tests still require control groups.
I don't think so. If the token was "author_is_jake" for example, it would have changed to "author_is_ceo" on second pass
Our old friend Singleton looks up from the bench, hope in their eyes.
Possible, but clearly not the only possibility.
I think a lot of it is lies... I used to observe that tweeting anything remotely political went straight to trending timelines, so did tweets about crypto and NFTs...
I doubt that Twitter and most of these social platforms are driven by Ai... Maintaining and changing complex algos could not be done on a rapid pace like what occurs now... I think Twitter has moderators, and scripts that control everything, and it can be more easily adjusted to tweak what is visible on the platform based on whatever agenda they want to represent (politics, revenue, PR/damage control).
Twitter frequently bends the rules to serve celebrities, politics, and sponsors and for that they need to be able to quickly adjust scripts. True Ai is meant to function on it's own with minimal intervention, and therein a simple change to a massive logic scheme would completely FUBAR everything.
I think there are rooms full of people that filter and either promote or suppress posts on Twitter every day, namely suppressing any tweets critical of the platform and it's owner... I've noticed that at night time hours, moderation of Tweets is less restrictive as a cue to what goes on there.
There are certain topics that are not moderated as heavily as others (3d Design and other non-controversial topics), and some topics (for example music and porn) that are restricted heavily visibility-wise because they force people to run ads to rank in trending (because it generates a lot of opportunistic money for Twitter as a platform).
Users that are critical of the platform and Elon and his allies (for example) can easily be neutered by moderators and put on shadow ban for any period of time in order to preserve the illusion of calm concerning Twitter operations. Complaints about Twitter only become visible when the majority of the audience tweets about problems (as that can't be moderated out without exposing moderation).
It's pretty much all smoke and mirrors there in order to maintain order in my opinion, and it's pretty much futile and torturous to people who just want to create and seize opportunity for their business without spending tons of money on platform marketing...
There is absolutely no reason to believe there was another Single user getting this treatment before. The Elon-case was just copy & pasted as an ego-stroking hack.
i think he means trump, and i think it would've been strategically wise to give trump special treatment (tho probably not ideal)
Actually, my first thought was the previous owner, or perhaps Twitter itself. But I left the question open, since it doesn’t matter who.
Considering I had to look up who the Twitter CEO even was before Musk, I’m going to say that no, they didn’t do the same thing to themselves.
Your memory of who the previous CEO was is not evidence one way or the other.
You would think someone forcing themselves up to the top of a feed designed to catch your attention would be a little bit more memorable than “I had to look up what he was even called”. You didn’t even know what his title was, ffs.
I’m sure your simping for Elon is highly appreciated. Maybe he’ll let you taste his boots next.
The code under discussion has nothing to do with forcing anyone or anything to the top of a feed. It’s entire observable effect is to _measure changes in engagement_. Any use beyond that is speculative.
However, if you believe that it was to be used as part of a system for increasing engagement, then you are asserting that it is a system with no control groups, which is a stupid mistake that wouldn’t be made by an undergraduate taking their first statistics class. They wouldn’t make that mistake even on the first day of that class, because they took a class in statistics in high school!
We all know said code does exist and just wasn’t part of the released source. It was very clearly used, although I know it goes against the cult so you yourself can never dare utter those words.
But no, please, go off on your tangent that is not at all centered in reality.
Such a system, which certainly exists, was also made extremely quickly. We all know the timeline. We all have seen the rants and then the immediate effect of manual changes made to please the rants. I could easily see control groups going out the window when your boss expects their tweets to be prioritized yesterday.
I also like how confident you are in a company’s ability after they explicitly laid off and fired the majority of its staff. Generally speaking, when that happens, people lose confidence rather than gain it, but here you are proving us all of wrong. Gold star for you, maybe you’ll even get the other boot now!
I’m sure it feels so rewarding to simp for a man who will never even look in your general direction, let alone talk to you or know all the great things you said about him and his companies.
Yea, I’m not willing to engage in any conversation with someone who uses ad hominem attacks. I should have noticed your previous one; I can only conclude that I wasn’t paying enough attention.
> I can only conclude that I wasn’t paying enough attention.
I mean, considering you didn't even know what the title of the "previous owner" even was, you seem to do that a lot.
No. Not a chance. Elon’s claims of mismanagement at Twitter have merit, though not nearly as much as he think, but the one thing that’s undeniable is that the previous “owner” (well, the CEO), is an adult that didn’t care about this shit.
Isn't it a problem if the CEO of Twitter doesn't care about his/her tweets, and the general public doesn't care either?
The CEO of Twitter should be doing more important things than tweeting.
And far more important things than worrying whether said tweets are popular or not.
I bet the previous CEO never thought of firing an employee for a disability through a public tweet thread.
You mean, better the firing for disability is done in private? Isn't transparency the best way to ultimately eliminate such behaviors?
It is usually better for major criminals to confess and turn themselves in too. "I bet" doesn't read "it is morally better." And why jump to compare publicly vs privately doing it instead of doing it vs not doing it?
Completely disagree with this. The CEO using the company's own product daily is how it should be--regardless of industry.
Why, exactly, should the CEO of Twitter care about their own tweets getting maximum attention? If anything, this takes engineering effort away from customers because special implementations like "is_elon" have to be added.
Isn't it natural for the CEO wonder about his own tweets, and use those as a proxy for the function of the platform as a whole--esp as relates to "VIP" tweeters who Twitter arguably wants to keep happy. In fairness to Elon, his repeated inquiries about his own tweeted shined light on several legitimate bugs in their algorithm affecting VIPs.
No, it isn’t a problem if the a CEO of Twitter doesn’t care about their tweets. The proper task of any CEO is maintaining the company as an immortal entity.
If the general public cares about the CEO’s tweets, then necessarily there will be a danger that the death of the CEO will ignite a crisis in the general public; e.g. Steve Jobs at Apple.
The producer is not the star.
Suppose Steve Jobs used an Android...
I never suggested the CEO needs to be the star tweeter. What I am saying is that the CEO needs to be a tweeter, i.e. personally invested in his/her company's product. Companies with CEOs who don't give a shit about about their product/users tend not last long as "immortal entities".
It’s just a two-pass EM (Elon Maximization) algorithm!
So many unnecessarily cynical takes here. Let's say you were in charge of a large legacy system that some segment of customers complain about it not working for them as well as other segments. How would you know whether their complaints are valid unless you measured it? You have to know first. So measure it.
Yeah, but then what do you do after you measure it? Nothing? No, you make decisions differently so as not to offend whoever is part of the criteria. For example, can we agree that we don't want an "author_is_flat_earther" flag? Because who gives a shit if Twitter makes a change to their recommendation engine that negatively affects flag earthers? Just because something is only used for A/B testing doesn't make it completely inert.
There could be an argument for an "author is a flat earther" flag with the intention of those tweets being repeated by twitter less.
Do you think Elon bought Twitter so he can bury his own tweets?
Actually, that's a refreshing take.
(No).
flat_earther can be a a proxy for conspiracy theories. They may want to know if there is a swing.
You can dismiss the complaint without measurement if you are confident in two things:
1. Your system does nothing to actually segment this specific group by their identity.
2. You are confident that the systems you have set up to reward good behavior and punish bad behavior are accurate.
If both of those are true, you know that even if the group is being disproportionately negatively impacted by some form of recommendation/moderation, that it is only because that group disproportionately participates in behavior that is bad for the platform. That isn't a problem. It would actually be worse for the platform overall if you did anything to appease that group.
> ...it is only because that group disproportionately participates in behavior that is bad for the platform. That isn't a problem.
That is exactly what Twitter's stance has been all along (in the pre-Elon era) and it IS a problem for the product because people being silenced due to their own bad behavior (example: misgendering transgender people) feel an injustice is being done. The rule-makers get to set the range of acceptable discourse on Twitter and those to the right of center have felt unfairly disadvantaged by the way it was done in the past.
Over time this has eroded trust in the product. Just because people aren't being labeled and ranked based on whether they are red team or blue team, the people deciding what "good" and "bad" behavior looks like on the platform have the power to disproportionately impact these groups.
Data isn't going to tell Twitter whether to allow or disallow misgendering people. You either think that is bad behavior that shouldn't be allowed or you don't. Disallowing it is not disadvantaging Republicans. It is stopping behavior Twitter has deemed is bad for the platform. As I said in point 2 above, either Twitter is confident in those decisions or not. Data is worthless when it comes to a moral decision like that.
And if we accept hat Twitter believes (or more accurately did believe) that misgendering people is wrong, who cares whether people who want to do it feel an injustice is being done? Would anyone say that deleting spam is an injustice to spammers? You break the rules and you get punished.
> Data isn't going to tell Twitter whether to allow or disallow misgendering people. You either think that is bad behavior that shouldn't be allowed or you don't. Disallowing it is not disadvantaging Republicans.
If one of the defining characteristics of a political/religious/cultural group is having a particular ethical view, then enforcing a contrary ethical view against them is disadvantaging them and discriminating against them. Now, it may in some cases be morally and/or legally permissible, or even justifiable, discrimination, but it still is discrimination, and it is still disadvantaging them.
> Would anyone say that deleting spam is an injustice to spammers? You break the rules and you get punished.
Worldwide, many jurisdictions have laws against discrimination on the basis of religion; although it is less common, some jurisdictions also have laws against discrimination on the basis of political belief. A law prohibiting discrimination on some ground, is evidence that some people believe discrimination on that ground to be immoral. By contrast, I've never heard anyone suggest that spammers should constitute a "protected class", and I'm not aware of any jurisdiction which treats them as one.
Some people believe that there is nothing morally wrong with discrimination on the basis of religion and/or politics. Other people think there is something morally wrong with it, but if there is a conflict between the right to be free from religious and/or political discrimination, and the rights of LGBT people, the rights of the latter morally ought to take priority. Spam is irrelevant to that ethical debate.
>If one of the defining characteristics of a political/religious/cultural group is having a particular ethical view, then enforcing a contrary ethical view against them is disadvantaging them and discriminating against them.
I don't think misgendering people is a "defining characteristic" of Republicans and if that is, the Republican Party is in a pretty sad state considering all the bigger problems in the world. And if that qualifies as a "defining characteristic", there are plenty of other counter examples of society accepting discrimination as you define it. Banning polygamy would be discriminatory against Mormons is one. You could even argue that a full abortion ban is discriminatory against Jewish people.
>some jurisdictions also have laws against discrimination on the basis of political belief.
Notably not in the US where Twitter is based and were most of these complaints originate.
> I don't think misgendering people is a "defining characteristic" of Republicans and if that is, the Republican Party is in a pretty sad state considering all the bigger problems in the world.
Some religious conservatives are convinced that referring to a transgender person by their preferred pronoun is a sin, even a serious one, for which they will be judged by God. Even religious conservatives who don't personally subscribe to that viewpoint, see it as one they are morally obliged to respect and defend. [0] While that doesn't describe all Republicans, obviously there is a significant overlap between Republicans and religious conservatives. And for a devout religious person, their religious beliefs are one of their defining characteristics–they are a huge part of their life, even their very identity, who they understand themselves to be.
Even those conservatives who think it is okay to use a person's preferred pronouns, will adopt a much more restricted stance on the topic than many trans activists. Many will insist on it must be voluntary rather than mandatory, and defend the freedom of conscience of those who take a more conservative stance than they do – which is an expression of both religious and political beliefs about respect for individual freedom and conscience.
Some conservatives are willing to use a friend/colleague/acquaintance's preferred pronouns when interacting with them, but will refuse to do the same for a criminal in the news. Look at Wikipedia to see people who vehemently insist that you must use the preferred pronouns of a dead school shooter or executed murderer – and even if the family of the victims publicly objected to it, that wouldn't change their mind.
> Banning polygamy would be discriminatory against Mormons is one. You could even argue that a full abortion ban is discriminatory against Jewish people.
Prohibitions on discrimination are never absolute, they always permit exceptions – so the existence of exceptions is not an argument against the existence of the prohibition. And whatever the merits of those specific examples, they are actions, not speech. Society traditionally gives religious minorities far greater latitude with respect to their beliefs about what they can and can't say, than their beliefs about actions which aren't predominantly expressive in character. Jehovah's Witnesses who believe it is a sin to salute flags or recite pledges of allegiance, Quakers who believe it is a sin to swear oaths, etc.
> >some jurisdictions also have laws against discrimination on the basis of political belief.
> Notably not in the US where Twitter is based and were most of these complaints originate.
While the US currently lacks federal laws banning political discrimination, state and local laws sometimes do ban it, see [1]. Some of those laws are specific to certain contexts (e.g. housing or employment), and so may not be applicable to a social media platform such as Twitter. However, given increasing concern among conservatives about political discrimination, it seems rather likely that we'll see more state laws enacted on that topic in the future.
[0] Examples: https://www.gotquestions.org/transgender-pronouns.html https://www.ncregister.com/commentaries/stand-fast-on-the-pr... https://seekersguidance.org/answers/modesty/should-i-honour-...
[1] https://reason.com/volokh/2021/10/18/bans-on-political-discr...
> Some religious conservatives are convinced that referring to a transgender person by their preferred pronoun is a sin, even a serious one, for which they will be judged by God.
Awwww. Too bad for them.
If you have no empathy for them, do you have any right to demand empathy from them?
I don't expect their empathy. I expect them to experience consequences for their actions. The consequences can stop when the actions stop and aren't expected to be repeated. I expect some subset of them to have an understanding of cause and effect, and thus learn. (If the lesson they learn is "change actions" rather than "change mindset producing actions", that'll do.) I expect the rest to complain about experiencing consequences for their actions.
> I don't expect their empathy. I expect them to experience consequences for their actions. The consequences can stop when the actions stop and aren't expected to be repeated.
Or, they can turn around and try to impose negative consequences on the people who are trying to impose negative consequences on them. Which, one might argue, is exactly what is happening in several state legislatures in the US right now. And then the fight goes on until one side wins, or there is some sort of "peace deal".
That's what happens with irreconcilable values differences, yes.
But I wouldn't frame it as cause-and-effect like that: one side doesn't attack the other as a response to experiencing consequences; they attack the other pervasively at every opportunity, and sometimes experience consequences for doing so.
If one side believes they are completely innocent and the other side are just plain evil – and the other side believes the same things right back – isn't that how civil wars start?
We have a democracy precisely to avoid such things. I'd prefer the strategy of "decide to actually start winning at every opportunity"; it would be a novel change in strategy.
Let's stop treating this as "maybe there's a way to convince people", understand that there is no way to convince some people, and instead just win. Win, and keep winning, and use those wins to eliminate things like voter suppression and gerrymandering, and then never lose again.
Try as hard as you might, there's no guarantee you can win. What happens to "win, and keep winning... and then never lose again", if you never "win" the first time? The US political system is rigged (arguably by design) to favour conservatives. To successfully undo that rigging requires not just winning narrowly, it requires winning decisively. But how do you win decisively when the system is rigged against you? It might be about to get even more rigged–if the upcoming SCOTUS decision in Moore v Harper embraces the "independent state legislature theory", all efforts to prevent gerrymandering by state legislatures would be dead in the water. So, what if you don't win, what if you lose–what then do you do?
And even if your wildest dreams come true–if you win too big, the other side may turn around and say "democracy isn't working for us any more". If it gets to the point that a significant minority of the population (say 20-40%) no longer believes that democracy is in their best interests, democracy's days are numbered. Especially if that significant minority has a great deal of wealth, influence and power. It could end in the peaceful negotiation of a "national divorce"–and there are many worst ways it could end than that.
If it were easy it would be done already. But the important strategy is to make sure it only takes winning decisively once, rather than winning decisively and leaving room for getting undermined in the future. Priority #1 with the next majority should be eliminating voter suppression, eliminating gerrymandering, supporting universal vote-by-mail, and all other factors that prevent the outcome of democracy from actually reflecting what the majority of people want.
> the other side may turn around and say "democracy isn't working for us any more".
They do that already, whether it's true or not. That's not a reason to decide to lose.
How do you compromise with people who use that as a way to take advantage of you and refuse compromise in return though? The result of 'let's live with them and treat their actions as good faith' has consistently been 'we get screwed'. What do you propose?
Brb, I just finished empathizing with Adolf Hitler yesterday. I am exhausted!!
Please don't empathize with me, though. I don't need it :0
You are comparing something like 20% or 30% or 40% of the population, to Adolf Hitler.
If that's a fair comparison – society's future isn't looking bright.
From your tone it looks like you don't empathize with Mr. Hitler at all or his passionate belief systems!
Don't be surprised if Hitler doesn't empathize with you either, you have only yourself to blame!
Am I misunderstanding? Are you suggesting that penalising those who misgender transgender people is unfairly disadvantaging people whose political views are right of centre?
They obviously feel that it is.
Likewise, if Twitter actioned people for saying "kill all men" or "all cops are bastards", this would be seen as having an obvious partisan impact.
It's not about what I personally believe or think is unfair. It's about what Republicans (broadly speaking) believe. There is massive resentment from people on the right who think the Twitter rules unfairly elevated some political opinions as good/correct/acceptable while treating others as unacceptable.
The handling of trans issues is just one example to illustrate the problem here. People on the left think trans rights are human rights while people on the right think a lot of trans issues should be open for discussion, legislation, persecution, etc. I think if we're being intellectually honest most would acknowledge that as a country we are far from consensus on many of the details here (bathrooms, girls sports, etc), and yet Twitter's rules and enforcement actions behaved as if the leftist view of transgender people is the only valid and permissible view.
The handling of January 6 and the banning of Trump is another example.
These things are Rorschach tests; people apply their biases and reach very different conclusions about what should be done. I don't claim to know the solution, I'm just trying to sketch out the problem with the way things were creating a climate where a big segment of the US felt unwelcome and resentful toward the platform.
This presents a problem for the platform, since you can't afford to alienate large double digit percents of the population if your mandate from shareholders is to grow mDAU by any means necessary. In that context, having some metrics tracking in place to measure the impact of algorithm changes on democrats and republicans to see whether impact is disproportionate is a completely rational thing to do.
While I can understand there may be debate around various trans issues purposefully calling someone something when they’ve politely asked you to call them something else isn’t up for much debate. Seems like common courtesy/politeness.
I see your point about losing users potentially but I would argue that Twitter’s intense focus on the US (as shown by the democrat/republican metrics) and trying to placate everyone is actually a negative for their business. There’s billions of other internet users outside the US. Shifting focus to serve them instead of focussing intensely on trying to please both sides in the US (and failing) would probably deliver better value for their shareholders.
> purposefully calling someone something when they’ve politely asked you to call them something else isn’t up for much debate. Seems like common courtesy/politeness.
It can be if the thing they're asking for is perceived to be untrue and the person being asked is a big stickler for that sort of thing. If you'll excuse me using metaphors on this sensitive topic: if someone wants to be referred to as "His Majesty" but is not actually the king then while many nice people will indulge him[1], some who care a great deal about the "correct" usage of noble titles won't.
[1] And it'll work out well! https://en.m.wikipedia.org/wiki/Emperor_Norton
But what if you flip the coin? What if the person being asked really, really thinks that the other person is a pissface, and is a big stickler for calling people he perceives to be pissfaces as such? Surely we can all agree that allowing them to follow their ideas will lead to bad discourse.
And it doesn't even have to be such a crass example. What if a person is legally called Robert, but he's gone by Bob his whole life - is the stickler right if they insist on calling them by their legally given name?
> Surely we can all agree that allowing them to follow their ideas will lead to bad discourse.
I actually think that if someone genuinely thinks someone else is a pissface then it's their right to call the other person that. I also think that excluding such a person from your conversations may be sensible. This applies to pronouns too.
Sure, is somebody arguing for something differently? It's strange to me how often people repeat "oh, but they are allowed to", when that's irrelevant to the topic at hand.
It's the difference between "what you're doing is wrong" and "what you're doing isn't compatible with our vibe, so you're not welcome".
I think the main difference between my faction and the main trans activist faction is we don't think honest misgendering (done due to an earnestly held belief about the sex of the other person and some moral convictions against white lies) should be a firing offense.
> I think the main difference between my faction and the main trans activist faction is we don't think honest misgendering (done due to an earnestly held belief about the sex of the other person and some moral convictions against white lies) should be a firing offense.
This seems counter to what you wrote earlier:
> I also think that excluding such a person from your conversations may be sensible.
Let's say your a customer of a company, and an employee "honestly misgenders you", even after you've repeatedly asked them not to. What can the company and you do to exclude them from your conversations, without firing them?
Should a company also accept a racist employee calling customers the n-word? Is it acceptable to fire somebody for that?
This is where user options should allow for that. If being called a different gender is so bad just block the people that do that. The problem is you might need that word in general discourse as well. So you can't just create a rule that blocks everyone who says "He" just like you can block the n-word. The problem is with the person assuming every single person in the general public should treat them as a friend and getting offended when a label that is applied to half of the population (neither as a positive or negative, merely as a descriptor) is somehow that bad of a thing.
> This is where user options should allow for that. If being called a different gender is so bad just block the people that do that.
In your comment, you said that we can just block the n-word. Why? Should it not also be up to the users offended by it to block those using it? I don't think people using the word should be on the platform, but I don't know where you'd draw the line.
> So you can't just create a rule that blocks everyone who says "He" just like you can block the n-word.
I don't think I've ever seen a single person suggest this.
> The problem is with the person assuming every single person in the general public should treat them as a friend and getting offended when a label that is applied to half of the population (neither as a positive or negative, merely as a descriptor) is somehow that bad of a thing.
I'm not sure what situation you're presenting here. Trans persons usually don't assume that the general public should treat them as a friend. What they assume is that "if I tell somebody I would like to be referred to with female pronouns they should do so" is a very normal thing to ask of somebody. I'm male, but if somebody erroneously called me female, I'd correct them. If they kept calling me female, I'd try to get them removed from the social situation I'm in. Why is this any different for a trans person?
> (neither as a positive or negative, merely as a descriptor)
This is a pretty bad line of argumentation. For racist people, the n-word is also just a descriptor, and they don't understand why they're not allowed to use it.
In the end, referring to somebody in a way they don't like is always a demonstration of power. By doing so, you're saying "it is okay for me to make you feel less welcome, because I won't suffer negative consequences from doing so". The only harm in making people feel accepted is an imaginary one.
> In the end, referring to somebody in a way they don't like is always a demonstration of power. By doing so, you're saying "it is okay for me to make you feel less welcome, because I won't suffer negative consequences from doing so".
Well said. I know someone who is a vegetarian and their old boss always referred to them as vegan. Seems minor because neither word is offensive, but after correcting the boss multiple times, she realized it was just him asserting his power. A prick thing to do.
> Why is this any different for a trans person?
The main difference is that the person calling you female likely doesn't have an honest belief that you are female, they're just trying to be a dick.
What does it matter to me if their belief is honest or not? I will still feel unwelcome, maybe even more so. Does it matter to a black person if the other guy really thinks "he is an n-word"?
Compelling people to lie about what they observe, often under threat of punishment, isn't polite or courteous.
>It's not about what I personally believe or think is unfair. It's about what Republicans (broadly speaking) believe.
No, it is about what Twitter believes. That was what I was referring to with point 2 in my original comment. Not every customer complaint is valid. It is ok to hear a complaint and dismiss it without further investigation. Twitter doesn't have some obligation to get all of society to think its rules are fair.
It is fine for a company to tell some potential customers to "fuck off" as long as that company isn't discriminating against a protected class. Twitter isn't discriminating against a protected class here.
If Twitter thinks misgendering people is wrong, it is impossible to come to an agreement with a group that think properly gendering people is wrong without Twitter compromising its own morals. Twitter is allowed to stick to its own morals and tell the people who disagree to "fuck off".
> Twitter's rules and enforcement actions behaved as if the leftist view of transgender people is the only valid and permissible view.
Twitter has changed in that regard since Musk took over. You can pretty much say what you like on trans issues now, as long as it doesn't break other rules. Loads of gender critical feminists have had their accounts restored in the past few months - usually having been suspended for 'misgendering' or some such nonsense.
> There is massive resentment from people on the right who think the Twitter rules unfairly elevated some political opinions as good/correct/acceptable while treating others as unacceptable.
Would it surprise you to find out that this resentment is in fact, conveniently manufactured, politically useful outrage? Because it's simply not true on its face, and the only thing we need to know to understand this is to see that it took Trump launching a coup to be banned on the platform. He violated the TOS every day, and he was allowed to spread his message to his millions of followers by Twitter. You want to talk about unfairly elevating political opinions? Trump used the platform to violate citizens' first amendment rights, and we had to take him to court to get those rights back. Twitter didn't do shit to protect us from him.
But it's not just Trump. It's right wing political opinions writ large. Far and away from sinking right wing conservative voices, Twitter research found they actually amplify right wing voices in every one of their top 6 countries except Germany [1]. Yes, that includes the US.
Is your mind blown? Have you heard of this once? I bet all you've heard from Musk and right wing politicians is that Twitter is going hard on conservatives and deplatforming them. Blocking their messages. Being unfair to conservatives and right wing opinions.
Yet what has actually happened? Twitter was actually deferential to conservative voices! It boosted conservatives and right wing voices at the expense of liberals. How did this happen? This is conservative messaging 101: complain about bias loudly enough and the other side will go so far out of their way to seem unbiased, they will be biased in the other direction. Conservatives managed to complain so loud about Twitter being biased against them that you not only believe it, but reality is actually completely the opposite.
[1] https://www.theguardian.com/technology/2021/oct/22/twitter-a...
Folks like Jordan Peterson would like you to think so, yes.
Of course, the man is essentially a walking “old man yells at clouds” meme at this point, so I’m not sure you should take anything he says with any merit.
Elon has said he wanted more reach and experimented with getting it. He also loves attention and tweets a ton.
There’s benefit of the doubt then there’s just… whatever the polar opposite of that is.
The penalty of the certain
Presumption of guilt.
I understand the value of measurements but how does measuring tweets from an individual user help?
If engagement on the tweets of that user goes down after a change has been implemented, you can roll back the change to prevent that user from being negatively impacted.
What if engagements around that user naturally declined, perhaps due to that user going off the deep end. Wouldn’t this just serve to bias the algorithm toward propping up the exposure of that user? Do they even care about the control so long as that user’s engagement is up and to the right?
Come on think this through. It’s trivial to tell the difference between a gradual and natural decline and a drastic decline immediately after rolling out a change. Especially when the change is rolled out region by region and only exists in regions running the update. You have to be able to measure the effect of changes and the most popular accounts are the obvious low hanging fruit for doing that.
You do an A/B test so even for the same tweet or same time period, you’re just comparing the new, “treatment” group against the old “control group.”
He's very likely the user on the platform with the most engagement, and probably by a long distance.
From that viewpoint, it does make some sense to use his account as measurement point.
I've never seen so many "experts" speaking from a position of complete ignorance than I do on Hacker News.
I expect they're tracking the red team/blue team metrics because of the political shitstorm that's been the GOP's assertions they're being silenced by The Algorithm.
The fallacy of false equivalence systematized in code.
Now one side can spew as much disinfo and incitement to violence as it likes, and any algorithm change that prevents this shit from getting amplified will be rejected as bias.
BSaaS = Both Sides as a Service
This shouldn't really be a surprise to anyone. It was reported years ago that Twitter was unable to cut down on hate speech because the automated systems they developed triggered too many [debatably false] positives on Republican politicians and that was bad for the company's reputation. If Twitter wanted to prevent future code changes from impacting that approach, there needed to be something like this in the code or tests.
Not sure why you got downvoted, you're absolutely correct.
Well, technically they are looking for relative changes, not equal total exposures.
I don't see an unbiased way to tell which "side" releases more disinformation and incitement to violence. Even deciding what counts as disinformation is hard (e.g. does it have to be literally false or just cause false beliefs in the reader?).
One way to tell would be to look at which side is incited to violence more often.
It turns out, according to the FBI (which is a conservative organization historically and exclusively run by conservatives), right wing extremism and violence is in fact the biggest domestic terror threat in the US, and it's currently growing [1]. FBI Director Wray gave this testimony after a right wing domestic terror attack was carried out that aimed to topple the US government. Not much has changed since then [2]. Since the former President's indictment the other day, the right-wing violent rhetoric has also ratcheted up a notch, so we can expect right-wing violence to follow.
Notably, we can confidently say this doesn't happen on the left, as when Hillary lost they did not launch an assault against the Capitol as the right did. Instead, they knit pink hats and had a march.
(PS before anyone whattabouts the George Floyd protests, the FBI doesn't see them the same way [3])
https://apnews.com/article/fbi-chris-wray-testify-capitol-ri...
https://news.yahoo.com/right-wing-extremists-responsible-for...
https://www.fbi.gov/file-repository/fbi-dhs-domestic-terrori...
Yeah but despite all that they should still give leftists a platform to tweet.
Ahh, the group of Elons.
I was wondering why I see so many tweets by him, and what his "Group's" impression quote is.
This is actually pretty hilarious.
Thankfully they haven't added a "no mute Elon" feature. Yet.
They don't have to, because he effectively can't be muted. People tweet quote him with an image, and it's not blocked even though I have him blocked as an account. This behavior is pervasive enough that you can still see his tweets all the time.
That’s not how it works. See the parent.
They said that they use it for metrics, so clearly there must be an "elon impression" metric.
I imagine it's the largest metric in a mission control style room
it starts dropping, klaxons start blaring, the room drops to red only lighting, engineers on the floor start pulling out their hair knowing the shitstorm that's coming
Yes, it’s not making anyone see any extra Elon tweets as your comment alleged.
It means they won't ship features that hurt Elon's reach. So in a sense, it is biasing code changes in favour of Elon.
Why does it mean that?
The original code is a part of the home-mixer service, which is the "Main service used to construct and serve the Home Timeline."
I suspect the flag corresponds to weights not present in the repo.
Per original source, The code that was released today doesn't show the parts that actually alter the scores of Elon and other users. The part of the code referenced below just tracks Elon stats (from what I know). Employees removed most PII before the code was released.
Correct. It's a binary metric. Did the number go up, yes/no (kept job / not).
Interesting which "groups" they care about (e.g. mainstream political parties).
But who chooses the users to be metrics…
Update: Elon was asked about these in a Twitter Space, he says it's not appropriate and will be removed from the codebase.
Additionally, from another Twitter engineer, the Democrat/Republican flags are apparently 10 years old and not important and do not have high feature importance.
Elon seems embarrassed: https://twitter.com/elonmusk/status/1641908130274525187?s=61...
It’ll be interesting to see what gets cut. Maybe just the Elon flag, but maybe others too.
Elons reply “I only learned about it now!”. What a crock of shit. We literally went through reporting a few months back where he was clearly instructing the team to make sure his tweets always come up, for everyone.
If I remember correctly, he noticed that his account with 100,000,000 plus followers was only getting 40,000 or so views. This lead to a Twitter engineer saying maybe his content wasn't interesting enough to get views and getting fired. And later on Musk said that they found a bug in the fanout system because his account was so big it basically broke things and they fixed it.
I remember everyone agreeing with the Twitter engineer who said maybe his content wasn't interesting enough, while to me that seemed odd. If people follow an account and they tweet and they're online they should have a high chance of seeing that tweet. That's the entire point of following someone. If someone I've followed tweets something I would like to see it.
The fanout bug was in weighting how blocks affect visibility and fanout. Apparently it was an absolute and not a relative thing, so the huge amount of blocks musk has affected their visibility.
You don't, at least for me, I have 5700 followers and most posts (with art) only see 200-500 impressions, whereas a year ago I was getting 3000 on average. The Following tab is clearly (at least for me) not in timeline order, so most of my followers never even know I posted anything. Since I post art most of my followers are artists, so you would think they would see them. Impressions (which are Twitter putting tweets in a timeline so that people scroll by them) are not under the control of the poster. Likes per impression for me are up, so interaction is higher which is supposed to increase how often tweets are shown, but only timely retweets seem to help any.
I am trying to figure out how the algorithm decides on what to show in the Following tab, but the code is way too big to analyze without being able to run it and look at logging/metrics/stats.
Yeah but presumably he didn't think the devs would tag a specific metric for "elon", which would later be open-sourced. It's one of those things that makes more sense in retrospect...
Should have called him "Big E" to obfuscate
"The big guy", it's foolproof!
I think it was more narcissistic than that.
As in: "My tweets are very important, if they don't show up on top, it means the algorithm can't recognize what is important, it needs to be fixed". And the team, who probably didn't see in which way Elon Musk's tweets could be that important besides the fact that he wrote them, they just decided to give Elon Musk's tweets a boost.
The most hilarious things to me was when I was reading some (unrelated to this drama tweet) posted in an article and the top recommendation was Musk's tweet that he doesn't artificially promote his tweets.
From what I understand of upthread, it doesn't "give elon musk's tweets a boost", rather any change which downgrades musk's visibility is considered a regression.
Obviously the end result is similarly that musk's visibility can never decrease, but it's a more technical (and to the letter) compliance with the specifications.
Sounds like you are reading minds here.
Now someone is surely getting fired.
> We literally went through reporting a few months back where he was clearly instructing the team to make sure his tweets always come up, for everyone.
Feel free to give me a source for that, but I'm pretty sure that's not true.
It was widely reported in the days following the Super Bowl that he personally ordered his Tweets be boosted after a Tweet by Biden got more engagement than his similar Tweet [0].
As far as I can tell, he did not instruct Twitter employees to make his Tweets appear to all users, but he did want them to make his posts appear in timelines significantly more often. This lead to reports of users suddenly getting their timelines flooded with lots of his Tweets.
It seems like there was a rush within Twitter to raise Musk’s engagement numbers by altering the recommendation algorithm to specifically boost posts from his accounts. The special boost factor was later reduced, but allegedly still exists.
[0] https://www.theverge.com/2023/2/14/23600358/elon-musk-tweets...
Wrong. Elon noticed his tweets had low engagement relative to his follower count compared to other accounts.
After some digging in the code base it turned out that there was a de-boosting factor based on the absolute numbers of blocks your account had, which affected popular and controversial accounts (like Elon’s) unfairly.
This investigation was initiated by Musk but it resulted in a great improvement in the algorithm with no special treatment for his account.
Well that's what the claim was. I never bought it, to me it sounded very much like the "Dear Leader Kim observed during the factory tour that one of the machines was miscalibrated and that productivity could be increased tenfold". Remember that he'd just fired an engineer on the spot for suggesting there wasn't a technical reason behind his lower engagement, and that the unspoken reason was he's frankly just a bit annoying and his epic meming CEO shtick was wearing thin. So I think when he raises the issue again, they just implement `author_is_elon` and they get to tell him that actually he is good at posting after all and there was just a weird technical issue they have now fixed.
Elon's happy, the engineers he turned to kept jobs and any time I find myself in the "For You" tab I see Elon's memes and his @catturd2 RTs.
That directly contradicts the article:
> By Monday afternoon, “the problem” had been “fixed.” Twitter deployed code to automatically “greenlight” all of Musk’s tweets, meaning his posts will bypass Twitter’s filters designed to show people the best content possible. The algorithm now artificially boosted Musk’s tweets by a factor of 1,000 – a constant score that ensured his tweets rank higher than anyone else’s in the feed.
> Internally, this is called a “power user multiplier,” although it only applies to Elon Musk, we’re told.
It seems indisputable that Musk’s account gets special treatment, even if some of the changes may also boost other controversial users.
You did not reference any sources, while the article claims sources inside Twitter gave them this information.
Elon spoke about this in the Twitter earnings call https://www.youtube.com/live/6bgCqskBmtk?feature=share
Anonymous Twitter sources have zero credibility since a good portion of the employees still want to oust Elon.
Anonymous Twitter sources may have _low_ credibility, but Elon musk has _exactly zero_ credibility at this point. He has a significant and proven track record of making false and misleading statements, even ignoring the many many more difficult to prove allegations.
In my personal experience, software engineers, when you ask them about technical matters like "does the code do X" or "is there a condition for Y" are generally pretty honest. Programming is a task that requires being able to have structured and literal reasoning, and asking engineers about purely technical details usually triggers that response, even if you can imagine a social motive for lying.
Disagree, Elon has high credibility IMO, certainly higher than an anonymous source with incentives to slander. Not saying he is perfect but given the scrutiny he is under and how much he is “in the arena” pushing the envelope.
Elon is furthermore a software engineer (he wrote ALOT of code at zip2 & paypal) so your argument falls flat in that regard.
Credibility from what exactly? Having last year's most expensive mid-life crisis? He once had some credibility, no one will deny that. Respect is earned and lost.
I think him buying twitter was great. He is lightyears better than previous owners & management. He is also running two other companies doing extraordinary things in the world.
Of all the people you can hate on in the world, hating on Elon is to me very odd. Tall poppy syndrome I guess...
You're making it sound like this commenter has some nasty personal vendetta on Elon, when it's just that Elon's a goofy dude who keeps embarrassing himself in a very public way.
This isn't Tall Poppy syndrome. This is Sideshow Bob repeatedly walking into rakes and getting smacked in the face.
He makes way more mistakes than I would like as well. But he also has made a lot of fantastic decisions that IMO outweigh by magnitudes the annoying irreverence he displays.
By the way, I’m not excusing his behavior by saying this, but I’m pretty sure the irreverence is a complete response to the struggles and pain from non-stop attacks from fighting giants in the arena for years. Putting yourself out there, living in the arena, especially to the extent he has, is super hard. Most of us don’t have the courage. If you do, you’re welcome to enter it and try to be a better role model. The world needs it.
He tried to fire a disabled employee via a public tweet without speaking to them privately first, and then had to retract it. This is McDonalds McManager behavior, not CEO behavior.
Musk did not write a lot of code at PayPal, that's factually incorrect. And he was ousted and replaced at PayPal because he was trying to make technical decisions that would've killed the company.
Note that some of the "false" statements later turn out to be true. E.g. holding child a last heartbeat happens (ex-wife held him for death rattle, elon for last heartbeat)
looks like evolve2k is himself the source. maybe he was a twitter employee ?
While it was thrilling for a moment to imagine myself as a disgruntled former Twitter staff member, that is not the case and I have not been employed there.
My statement above was to read ’we’ as in as the tech community not ‘we’ Twitter staff; others have quoted some of the sources on this thread, but it was definitively reported by a few different outlets at the time.
ah cool thanks for figuring out they were the source of https://www.businessinsider.com/twitter-boosted-elon-musk-tw...
Don't let Chesterton's Fence hit your butt on the way out...
What are the odds this flag was only introduced so that Elon could publicly fix it and give the illusion of transparency?
He's only upset that people found out about it.
If they remove his artificial boosts, he'll just turn around and shout at his engineers to reinolement it in another way.
I think the decade old comment related to a different part of the code regarding the number of followers you have in relation to the number of accounts you follow. (Everybody on the call wants to remove this: I wonder why they haven't yet.)
Chesterton's Fence. In a sufficiently large system, you should be hesitant to remove things unless you're sure you know why it was added, and all the things that have come to depend on it since.
I've definitely been hesitant to remove things I was pretty confident weren't used anymore, just because I didn't want to deal with the repercussions if I was wrong.
I didn’t know there was a name for this. Thanks for sharing.
I’ve definitely been bitten by this. You always have to weigh the chance you break something against the upside. If you’re actually fixing a bug, fine. But just refactoring to make something cleaner? Or deleting because it seems like it’s not doing anything, even after doing some research? Think again.
True for many things - doctors are now a lot more hesitant to remove the appendix for example. (it serves as a buffer for your gut flora iirc)
I would say it’s who will be removed.
Considering how Twitter is now getting a servance isn’t that bad of an idea TBH
The author_is_elon flag doesn't surprise me, but the two political designators are somewhat shocking. I'd sure like to know what changes based on what Twitter knows about your political affiliation.
I thought it was interesting how it explicitly doesn't boost independents. So much of the two-party system is self-reinforcing.
> I thought it was interesting how it explicitly doesn't boost independents. So much of the two-party system is self-reinforcing.
Is it boosting? Others are claiming this code is just for metrics collection: https://news.ycombinator.com/item?id=35391896.
But on the topic of Democrats vs. Republican vs. independent; a big factor may be that "Democrat" and "Republican" are much more cohesive groups and therefore much easier to define. No one can honestly define "independent" except in a kind of "none of the above" sense, since they can range anywhere from extreme right, to the center, to the extreme left.
Why measure something if you don’t intend to change it?
So you can make sure not to accidentally change it.
And if one party trends more towards violent extremism, they get boosted the same as before?
What's that got to do with accidental effects of unrelated software changes?
if your hate speech filter hits one party harder than the other, that doesn't necessarily mean it is broken
The point is to have these statistics remain the same for the individual groups in between changes, not to measure them against each other.
So then why make any changes?
Why not? If it doesn't matter but it's controversial there's no reason not to change it.
What if you're changing the spam filter?
There are many reasons to measure things if you don't intend to change them.
You might want to ask that of, say, astronomers, paleontologists, or even simply journalists.
Observing and measuring leads to understanding. As others here have noted, sometimes you want to measure to ensure that you're not inadvertently affecting an outcome or phenomenon.
> You might want to ask that of, say, astronomers, paleontologists, or even simply journalists.
You compare scientists with businesses. One of them's job and passion is to collect knowledge for the sake of knowledge. For the other one it would be cost without gain and eliminated if they didn't do anything with it!
No idea what journalists are doing in that list, what do they measure? If they want something measured they'd ask or look at other's services. Unless you mean the business that the journalists work for, but hen it's just that, a business.
I gave examples based on well-understood instances for which the reader is presumed to have the capacity to draw inferences to business-related concepts.
Businesses also rely on astronomy and geology, in instances. The former is used for navigation (though far less than in the past, I'll grant), and there are certain extractive sectors with interests in geology. Risk-management as well (insurance and catastrophic risk, whether from landslides, earthquakes, volcanos, tsunami, or other phenomena).
You'll also find businesses keeping tabs on weather, climate, competition (competitive intelligence), demographic data, politics, legal cases, laws, social and cultural movements, etc., etc., etc., which in many cases they have comparatively little capacity to change directly.
You're also jumping late into a thread which has already given numerous other rationales for why such activities might be undertaken.
Sometimes approaching a discussion from the PoV of seeing what you can learn from it rather than automatically adopting a presumptive stance of opposition or disagreement affords benefits. I recommend it strongly.
Depends what the metrics are used for. It doesn't make sense to apply artificial boosts to metrics that are only used for internal accounting. Well, maybe if you have an egotist CEO, but that wouldn't explain the rest of the boosts. We have to assume this code has some sort of effect somewhere.
Metrics collection so they can avoid bias, allegedly
There are probably many other non-political alternatives to select 2 groups to test for bias
Also how about.... everyone else in the world who is not an American voter?
This is what it means to let another country own your social media. Their ideas and memes unconsciously get preferential treatment. This is maybe not a good thing, but it is what it is.
> Their ideas and memes unconsciously get preferential treatment.
I think as this repo shows it's conciously, rather than unconsciously, getting preferential treatment.
It’s omission. They built for Americans because they were Americans. No one building this said “let’s ignore Canadian politics” or any other country, they just didn’t think about them at all, because like most Americans they don’t really care about the insidious Québécois plots to annex Prince Edward Island or whatever actual issues are happening in Canada.
The freedom convoy and it's repression with suspension of constitutional rights (because that's a thing over there?) and bank accounts arbitrarily frozen did get a little bit of attention in the American medias.
> the insidious Québécois plots to annex Prince Edward Island
I know it's a joke, but the people of Prince Edward Island would most likely welcome it. Economically that would be a huge boost. Not sure Québec would enjoy it however.
Perhaps they should look at annexing Labrador?
> The freedom convoy and it's repression with suspension of constitutional rights
This was needed because the convoy of morons was seriously damaging people's quality of life and the local police did not do their job. The police in other towns DID do their job and blocked the trucks. You should be mad at the cops who just sat by and watched the convoy of morons roll in with their thumbs up their butt.
What exactly is allowed in a protest is quite a thorny issue.
Here in the UK there are environmental protestors gluing themselves to the streets and I have quite mixed feelings about them.
A protest that does not inconvenience anyone is very easy to ignore.
You can see here how wild and inconvenient these protests were. The bouncy castle and hot tub were a big public nuisance!
The constantly idling giant trucks very much were a public nuisance. YOU wouldn't want to live near them.
Indeed their acts of terrorism were not reported by international media.
For example, during the occupation of residential areas of the capital they created deafening noise around the clock that was measured indoors (windows closed) at levels causing permanent hearing damage after minutes of exposure, with police doing nothing to stop them.
Bank accounts were not "arbitrarily frozen". This is total nonsense.
https://nationalpost.com/news/canada/banks-were-only-asked-t...
Not arbitrarily: the freezing was targeted against political protestors.
They were going pretty damn far beyond "protesting" they were trespassing/loitering/squatting.
Loitering, dear god!
When done in giant trucks blocking traffic and making tons of noise and exhaust fumes for weeks it is a crime.
That sounds straight out of Russia or China.
Your article seems to support the idea that bank accounts were arbitrarily frozen.
People can always use their countries' social media (and I assume many do!).
But it's indeed an interesting fact that people seem to specifically seek American tech and social media. And honestly, there's no shortages of foreign nationals commenting on American politics (and it's a good thing, it's their right thanks to the first amendment!).
This is a joke, right?
The vast vast vast vast majority of people use “American” social network because they are the social networks that exist. The US is undeniably the main exporter of SaaS products. See: the incumbents freaking out that China is getting a turn in the front seat. I’ve never once heard of anyone seeking ‘American tech’, except for some punchline in an anti-Soviet joke or movie.
Foreign nationals can comment about American politics because the US doesn’t have jurisdiction over their speech, notwithstanding the back and forth over whether social media companies are liable for disseminating such content in the first place.
A material part of why the USA is seen as The Country in Western culture, and a noted big player in other cultures, is because of the power it projects via the media. That includes both Hollywood exports, and social media.
American social media companies have the benefit of a large homogenous deeply interconnected culture with a vast ad buying community. Simply put, a U.S. social media platform can get more revenue with less effort than a European counterpart (because they have to go country by country, language community by language community), and will generally outcompete those EU counterparts, with social network reinforcement effects doing the rest. This holds up for most ad-supported online services, which together with a much stronger private investment sector is the reason American platforms absolutely dominate Europe. The EU has a stronger government subsidy system for software but that system does not reward or expect market success, which is why it delivers very little value.
> I’ve never once heard of anyone seeking ‘American tech’, except for some punchline in an anti-Soviet joke or movie.
Really? Why is everyone using MacBooks and iPhones? Actually most Soviet computers were copies of western designs. [0]
> is because of the power it projects via the media. That includes both Hollywood exports, and social media.
The thing is, is takes to to project, someone has to export a product and someone else has to import it. Since people seek it and want to consume it it's easy to export.
[0] https://en.wikipedia.org/wiki/List_of_Soviet_computer_system...
> I’ve never once heard of anyone seeking ‘American tech’
One reason people have a distorted idea of what physical products the US produces, so I've read, is that most of them are products for industrial use, in factories and so on.
You look at consumer products that are made in..., because that's where the finishing touches are put on. That's not representative of the global economy though.
The importance of US cultural power cannot be overstated.
Before I deliberately locked myself out of it (well before Musk), I asked for my data.
They classify me as:
Other than the age (I'm neither under 13 nor between 55-64), everything I've listed is incorrect.* speaks Indonesian Interested in: * Beer * Cricket * DJs * Dance * Enterprise software * Horror * NFL football * South America And aged either between 13-54 or (and?) over 65On that basis, they'd probably call me a Republican.
Carries a different meaning when you're British, that name does.
What a strange age classification — no idea about what your age actually is, just that it definitely isn't 55-64. I assume it would never conclude you are under 13 for legal reasons.
Aren't "enterprise software" and "horror" the same thing?
Fair.
I try to avoid both, when I can. :)
I'm a guy in my 30s but they had me classified as a 65+ year old woman. They also thought I was a radiologist.
Only anecdotally.
But ever since Musk took over the amount of US political content has significantly increased in particular from the right despite me not living in the US.
It's hard to tell whether previously political content was weighted less and Musk has removed those controls or whether they are now weighted higher.
I live in the US in "Trump country" and don't see any political content. Maybe its just related to the people you follow and the content you look at?
Some of the power users Musk reportedly had boosted are specifically right wing political posters, like catturd2. But then he's also boosted some high profile left leaning politicians, so it's not exclusive. It does mean you're more likely to see right wing American political content either way, which has to be annoying for people outside of the US.
https://gizmodo.com/twitter-algorithm-aoc-ben-shapiro-cattur...
Would it not be just as annoying to see a slant the other direction, or in any extreme direction?
The parent poster doesn't realize the inherent bias in what they wrote and that you have pointed out.
Boosting both right and left wing content increases your chance to see either type of content, because you boosted them over everything else.
I'm not sure why you wrote that to me. It was already in your prior comment.
I believe their point is that, because the US is generally regarded as 'to the right of' many countries in Europe, for example, a slight rightwing bias looks to us (I'm in the UK) like an extreme rightwing bias. Heck, even a completely neutral stance, as far as the US is concerned, will look like it veers to the right for us. Note that I'm not saying anything about how I believe the twitter algorithm should work, this is just my interpretation of what that commenter meant.
Europe is only a tiny fraction of the world, though.
Compared to India, Japan, Singapore and (depending on the metric) China, the US looks clearly left of center. Compared to most of the Middle East, the US looks extremely left-wing.
Why do you believe it would be annoying to us?
This is an interesting point... they're making sure their A/B experiments don't adversely affect Republicans or Democrats. But lots of European countries are skewed more to the left and would broadly look more similar to the "Democrats" group.
I thought it was interesting that twitter thinks the U.S is the world
For a big tech company, the regulatory regime is the world. For Twitter that's the US.
Most big companies I've worked at are international and are regulated in many regimes, although sure the one with the main corporate headquarters is more important.
Twitter thinks Twitter is the world.
Or anti-statists! Not everyone who engages with politics is a bootlicker for authoritarianism but it sure feels like there’s no space made for this perspective (obviously)
Isn't it intrinsically self-reinforcing, if you have a winner takes all system? It's almost always better to join an existing team than start a new one.
The vast majority of self-proclaimed independents vote with one party just as reliably as registered members.
American's lax attitude towards cultivating more than two parties is literally killing the republic from the inside.
It isn't an attitudinal problem, it is the logical outcome of our political systems. In political science it is known as Duverger's law: single ballot, winner take all systems inevitably tend toward a two party equilibrium.
Changing this requires states to adopt alternative systems, which can sometimes mean amending state constitutions. It isn't easy or straightforward, and the general sense is that there are better things to spend that effort on.
> single ballot, winner take all systems inevitably tend toward a two party equilibrium.
I'm not convinced it's quite that simple.
For example, Canada also has a first-past-the-post electoral system - yet political parties here have come and gone. And continue to do so.
Parties come and go, but there will be two major parties. In fact under such as system if you're in one of those two big parties and you see a third party rising you need to figure out whether it's you or the other guys getting replaced, 'cos it won't stay a three party system for long.
For example in the UK, the Tories ("Conservative and Unionist Party") and Labour are currently the biggest parties, but a hundred years ago this was a novel situation, Labour were seen as a third party, while the Liberal party (which was eventually absorbed into what is today "Liberal Democrats") had seen success over decades and were often in government prior to that point.
A lot of it depends on whether your governing coalitions are formed before or after the election.
The US parties are just coalitions of disparate interests joining together until they (maybe) represent enough people to have a majority and be able to enact their collective interests.
I mean we're on the sixth "party system" in the US too, if that's your standard (it's a new "party system" when there's a significant realignment of which interests find themselves in which parties, either by a reshuffle among existing parties, or by new parties rising and old ones falling). Have you heard much about the Federalists or Whigs lately?
The voting demographics change frequently as well. California has voted to elect more republican presidents than it has democrat ones. It’s voted democrat in the previous 8 elections. In the 10 elections prior to that, 9 times it voted for republican candidates. Texas has also voted for more democrat presidential candidates than republican ones. People who think the US democracy is a rigid and highly predictable system simply have a recency bias.
It's just different order of operations on coalition building. Other systems divide into a majority and opposition at some point. In the US it just happens earlier, but there is the same diversity of opinions within those groups.
You'd think a country that played a central role in a global, decades-long unstable regime of bipolar power that routinely pushed mankind to the brink of nuclear oblivion would know better than to have a bipolar electoral system
> routinely pushed mankind to the brink of nuclear oblivion
I'm struggling to think of a reason why this is anything but bad faith nonsense.
have you never heard of the many times we almost came to setting off full scale nuclear warfare because of the bipolar power war of the ussr v usa et.al.?
the only thing that saved us was cooler heads that prevailed on both sides.
Why would throwing more actors with similar capabilities into the mix make the situation any more stable though? That seems like basically the old European Balance of Power, which broke out into open conflict more frequently.
This balance is the central theme of the seminal work in the field (https://www.jstor.org/stable/j.ctt5vm52s). It may or may not be correct but is one of the most influential texts on the subject.
I take the view that the European balance of power probably broke out into open conflict often because of hidden alliances that made it hard for states to correctly gauge the costs of engaging in such conflict.
In the field International Relations, there's lots of discussion around the stability (or instability) of a bipolar distribution of power. Your stance is closer to neorealists, mine is maybe closer to classical realists. Have fun reading https://en.wikipedia.org/wiki/Polarity_(international_relati... and related links
But from a game theory perspective, having only two powers turns everything into a zero-sum game. I argue that leads to less cooperation and increased divisiveness. Not agreeing with Republicans means you must be a Democrat in their eyes. This "us versus them" mentality is somewhat tribal and leaves little room for nuance.
If I had a magic want, the US would have a 4 party system. Like a cartesian plane with civil liberalism <-> conservatism on one axis and economic liberalism <-> conservatism on the other axis
> Classical realist theorists, such as Hans Morgenthau and E. H. Carr, hold that multipolar systems are more stable than bipolar systems, as great powers can gain power through alliances and petty wars that do not directly challenge other powers; in bipolar systems, classical realists argue, this is not possible.
I suppose they must account for this somehow but isn't that exactly what a series of proxy wars in far-flung places between the United States and the Soviet Union were?
It's been almost 20 years since I studied any of that, but those proxy wars in far-flung places did directly challenge other powers because of the Domino theory
I don't necessarily agree with Morgenthau and Carr as I think most IR Theory is bullshit made up by academia... particularly the stuff around how players "gain power"
So I make my own argument which mostly hinges on the idea that two powers really means "my power" vs. "everyone else" which is not a recipe for peace
Isn't World War 1 a counterexample? Lots of parties, but they aggregated into two competing coalitions.
I've not heard how many times, no. How many times was it?
Luckily there's a whole article on it https://en.wikipedia.org/wiki/List_of_nuclear_close_calls
That's all nuclear incidents ever, not just US/USSR. US/USSR count from that page is more like 10 incidents across 40 years. Some of them occuring within days of others, they're that fine-grained.
I wouldn't call that a "routine" matter.
In some other countries the different interest groups sort themselves into two factions after being elected but I don't know that it is really that different in practice.
It's not different, or rather it doesn't produce meaningfully different outcomes. I'm not aware of any parliamentary system with a wonderful diversity of thought and a long record of positive accomplishments. You end up with ruling coalitions which are typically pretty awful.
Ironically, America has one of the most open political systems. You register as one party or the other and vote in primaries. This has lead to a huge variety of people replacing hated mainstream politicians. That's way more than you can say for many other countries.
>it doesn't produce meaningfully different outcomes
It absolutely produces different outcomes than a two party system. Smaller parties make demands as a condition for joining any coalition (or similar arrangement). For example, Canada's NDP only agreed to back the Liberal Government on condition of state funded dental care for children being implemented. Now it is. Millions are affected. Whether you agree with it or not - that is unquestionably a "meaningfully different outcome". Other examples abound if you care to look.
>I'm not aware of any parliamentary system with a wonderful diversity of thought and a long record of positive accomplishments.
That's an impossibly high bar. The standard we're talking about is whether it's better than a two party system.
> It absolutely produces different outcomes than a two party system. Smaller parties make demands as a condition for joining any coalition (or similar arrangement). For example, Canada's NDP only agreed to back the Liberal Government on condition of state funded dental care for children being implemented. Now it is. Millions are affected. Whether you agree with it or not - that is unquestionably a "meaningfully different outcome". Other examples abound if you care to look.
You think logrolling doesn't happen in the American system? Parties are made up of factions who will entertain each other's preferred priorities, which is the exact same thing you're describing except that the parties of the various factions are nominally the same. There's no real natural or obvious philosophical reason why your position on gun ownership should imply a position on environmental regulation, religion, infrastructure buildings, racial politics, abortion, tax policy, and more. We're so used to the groupings that they seem natural but if we go back and looking at older American party systems you'll see parties that don't 100% map onto the contemporary ones, with a blend of some elements we would think of as fitting and others we wouldn't.
I never said it didn't happen. Don't do that.
It's unquestionably a different dynamic. If the party is in power, then the party is in power. Period. They have less incentive to listen to smaller factions. Do they? I sure hope so. But it's not the same.
Furthermore, these internal deals are more likely to be kept private since theres no benefit to airing everything to the press, the public and the opposition.
In a multi-party system almost all those deals are public. Thus, voters can decide if everyone lived up to their promise.
That's just not true. If you want a recent example go look at the protracted battle to elect a Speaker of the House.
The two parties are essentially coalitions.
Anecdotally, a recurring theme in conversations I've had while living abroad is the desire to prune or consolidate some parties.
While both sides in the US have big tents, they are effective in whipping votes when things need to get done.
It also helps that detracting coalition partners can't torpedo their leadership. Historically, factions within a party, like Blue dog dems or Tea partiers, had to wait for an election to litigate their grievances.
Well, yes, this was where I was going. The coalition is just formed before the election takes place. I live in a city with "nonpartisan" elections and I feel like it mostly just makes it difficult to understand what candidates in local elections even stand for.
Agreed, when I lived in Austin the non partisan elections for city council made me feel ill informed come election day.
It just fueled apathy for me: there's only so much research one can do with limited press coverage and personal time constraints. Shrug.
In some ways it might be worse. Here we had a party that promised to vote with one faction and partly thanks to it survived the election. A few months later they did a 180 and voted the other faction into government. Opinions aside, imagine voting for Sanders only for him to elect Trump or the other way around. With that said, two party version also comes with major flaws.
IIRC Some states you cannot vote outside your declared party (Washington)
That's only for primaries, which traditionally were only open to members of a party.
Majority? Probably. Vast majority? No way. If that were true we'd not keep switching parties between presidents. And Florida, a mostly independent voter state, wouldn't have had DeSantis win in a landslide when he just barely tied last time around.
That's attributable to turnout.
You tell me if 90% is a "vast" majority or not. https://www.pewresearch.org/politics/2019/03/14/political-in...
I'd be interested in which bin people like Joe Manchin, Kyrsten Sinema, Susan Collins, et. al. are placed.
So many questions. How are users tagged D or R? Is that a manual process or automated somehow? What is the effect of these tags? Can I find out if my Twitter account is in one of those buckets?
Especially for people that aren't... you know.. Americans.
Unless they mean actual public figure party members which are known and probably verified.
Then it probably means which side you'd be on if you were American.
you could probably algorithmically determine it in most cases based on any number of indicators from phrases used, to communities interacted with, which hashtages are included, which cohort retweets and likes most etc... thats not even getting into simply tagging political figures with the party they officially affiliate themselves with
Likely they are tracking performance verified politician accounts based on registered party affiliation. Why republican should count equal in the evaluation metric to democrat when nunerically there are less republican voters, let alone proportions on Twitter, is another question.
And how are they choosing to balance them, per capita, or just both sides should get 50%? It seems pretty clear they are making editorial decisions here. Does that break their section 230 protections?
>It seems pretty clear they are making editorial decisions here. Does that break their section 230 protections?
No.
https://www.techdirt.com/2020/06/23/hello-youve-been-referre...
"Section 230 protections" have nothing to do with "editorial decisions"
Please read something. Preferably section 230, which is short. Alternatively something entirely different, as long as that keeps you occupied.
I suspect that these are used for metrics tracking rather than being fed back into the recommendation engine. But there's no real way to know for sure given the limited release. These predicates aren't actually used anywhere in the code that's been made available.
It's not that shocking...
Half the people that got promoted on my timeline were perpetually candidates for elections I couldn't vote in, and they _self-identified_ as Republican or Democrat in their own bios, or via the registration of their candidacy...
This is why I exclusively used to use Twitter in the "people I follow only" mode, and simply shut my account down when they pushed harder on the algorithm.
Facebook guesses your political affiliation as well, you can even look über your settings to see what they guessed.
The repo suggests it's about tracking engagement metrics[0], so Team Red people see more Team Red content and vice versa. Nothing nefarious.
[0]https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
Why specifically track political parties? Where is author_is_american? Or author_is_mayonnaise_enjoyer?
Maybe it was a choice made many years ago that they thought was appropriate, but we can't yet know it's not used for other purposes. We can at least be reasonably sure they've added the author_is_elon within the past year. I would have thought there would be many more descriptors, or non-controversial descriptors.
Or maybe Elon specifically added those before releasing this code to get people riled up.
The point is probably to check that changes they make aren’t accidentally politically biased.
Twitter became a very politically charged platform after 2016, mainly among Americans. They'd be idiots not to take advantage of that.
You say it’s not nefarious but isn’t that how echo chambers are created?
I don't believe echo chambers are nefarious - there's no hidden agenda involved with them. That's just how recommendation algorithms work, and it's what most people want.
But if someone finds some code that suppresses recommendations from a specific political ideology across the board, that would be nefarious, IMO.
Echo chambers may not be nefarious, but they are insidious.
They can be if they're involuntary and inescapable, but neither is the case for Twitter. It's designed around letting you curate your own feed, but it also constantly throws random stuff in through retweets and quote tweets - which is what people hate the most about the platform.
The echo chamber in your first point does the suppression in the second.
Having a separate flag to track engagement specifically for Elon tweets isn't nefarious, but weird.
Exactly. I don't see NPA (no party affiliation) anywhere.
Did you not get the memo? "If you're not with us, you're against us".
It's probably more of a conservative/liberal identifier based on US political party ideals... And they likely would filter any metrics from this by the users country
Well someone just asked about it in the live spaces[1] Elon is hosting and he said that should not be there. An engineer said afterwards it is just for metrics but then Elon chimed in again and said "we should get rid of it, it should be gone."
[1] https://twitter.com/elonmusk/status/1641880448061120513?s=20
Doesn't necessarily mean he didn't want it there in the first place. Why else would it be there?
There are millions of lines of code existent. There are thousands added. Your prior for it being Elon's addition should be something like 10,000/1,000,000 or roughly 1/100. The prior that it wasn't Elon's change is going to be something like 99/100.
When you add the additional information that Elon wants the code removed, but existing Twitter engineers think it appropriate to keep this actually increases the probability of it being added by the existing Twitter engineers and decreases the probability it was added due to Elon.
Obviously, these are rough numbers, but hopefully seeing any numbers at all helps you to get an intuition for the math.
Why is lines of code the appropriate input here? Here's a different computation that is at least as plausible:
There are hundreds of millions of users. Let's say 300M. Only a single one is special-cased in this code: the narcisistic CEO who reportedly went ballistic when his engagement metrics went down. The prior that it's a change done in response to his demands is 299999999/300000000.
(But of course it was added by existing Twitter engineers. The odds of Musk being able to actually commit code to their repository are zero. Even if he had the permissions, the man simply does not have the technical acumen to make even a trivial change.)
I think you are right, my estimate is much much too low.
I'll explain my mistake and why I made it.
I think I thought to use the estimate I did because Elon claimed he didn't know. The prior probability of not knowing something in a code base with millions of lines is very high, but contingent on his involvement in the change the prior that he is aware of it is much higher. So I started the estimate attempt with the probability that I thought better predicted the production of evidence claiming he didn't know.
Your point does raise my estimate substantially, but I think it probably raises it less than you would expect. I don't agree with your 1/300M prior, because I'm aware that hot users get special treatment. I've seen Elon's account thrown around in interview-style questions about hot users before and used as an example of a hot account that needs special treatment. This is something I've witnessed, but it wasn't contingent on Twitter being acquired and it happened prior to Twitter being acquired.
I also don't particularly assign high odds to wanting it, based on the evidence that he claimed to not want it implicitly by wanting it removed. I don't think it seems appropriate to get to near certain probability the he wanted it with the evidence being that he stated that he didn't want it. In my view there isn't a compelling reason for him to lie about this. He owns Twitter, so if he wanted them to have his account monitored that would be a reasonable thing well within his authority. If he wanted it, he doesn't need to pretend to not want it in order to appease someone.
It does seem to me that the odds that the change was added in response to someone thinking he wanted it is much higher than 1%.
> I don't agree with your 1/300M prior, because I'm aware that hot users get special treatment.
That's absolutely fair, and 1/300M was a reductio ad absurdum rather than a serious proposal. Not all users are equal, just like not all lines of code are equal :)
I have a few issues with the "hot user" theory, but they all boil down to the same point: no matter what the use case, you'd never want to do this with a single static user.
Does your infra require special-casing for accounts with more than 100M followers? That should be a flag in the account properties that gets flipped manually or automatically: if these users cause infra problems, you really don't want to be making code changes + full rollouts whenever a new user becomes hot.
Is this just a guard-rail metric, to make sure there's not some bug specifically affecting hot users that tanks their engagement? You'd want a much larger static set than a single account just to ensure there's a large enough number/variety of tweets to compute metrics from. A single user might take a break for a week, or might only be posting very specific kind of content for an extended period of time.
In any case, even if you chose to do this with a single user rather than a set of users, why would Musk be the obvious single choice? He wasn't the most followed Twitter account until two days ago. A year ago there must have been at least a couple of dozen accounts roughly as notable as Musk. The odds of him having been chosen as the special case still would not be very high.
> In my view there isn't a compelling reason for him to lie about this.
The reason to lie about this is that it makes him appear weak, needy, and a target of even more mockery. Given the purchase of Twitter seems to have been a vanity project, having this be exposed and leaving it in goes directly against his apparent goal.
> The reason to lie about this is that it makes him appear weak, needy, and a target of even more mockery. Given the purchase of Twitter seems to have been a vanity project, having this be exposed and leaving it in goes directly against his apparent goal.
I think it only makes sense to think like you are if you've adopted equilibrium assumptions; if you haven't then I find this sort of reasoning to be a conjunction fallacy causing an epistemic closure.
This is a weird nitpick that doesn’t attack the argument at all. It’s a dense codebase but whatever measure you personally prefer
JoshCole is doing a computation to arrive at the conclusion that there's a <1% chance that this code was added after Musk bought Twitter. I'm using the same methodology with at least equally plausible inputs to arrive at there being a >>99% chance of it.
How is that a nitpick? They're diametrically opposite results.
I'm JoshCole and I didn't find your reply to be a nitpick; you are right that the probability ought to be higher than 1%. My calculation was simplistic and I felt it was prudent to arrive at low probability, because I think probability of wanting something given claim of wanting it removed should probably not be anywhere near close to certain. My estimate isn't 1% though. It was just a short thing to share that gave an intuition for why it might be reasonable to assume he didn't know or want it.
In my opinion if you really care about this topic the right thing to do is ask someone at Twitter when the change was made. Getting more information would make us converge on the true estimate faster than arguing the odds IMO. Feel free to update me with the results if you do end up doing that so I can adjust my beliefs accordingly. I'm not going to try to gain this information, because I don't think the question matters much.
If I were Elon, first thing I would have done is ctrl+f “elon”…
The same reason there are ones for other people? When they make changes, they want to see how it affects the visibility of their most popular accounts
Of course he did because it makes him look bad and he's desperate for praise and attention.
What he wanted was everything that feature provides, without it ever being shown that it's there. But since he refuses to hire PR people and almost certainly came up with this idea in the last few days, no one was paid to hide its existence.
The next story out of Twitter will be the remaining engineers being threatened because Musk can't see his tweet statistics any more.
Elon does seek attention, but he seems to enjoy conflict far more than praise!
This is pure speculation.
There are no guarantees that the actual code which is run will have this removed.
Clearly embarrassing. This is the third commit in the repo.
never seen a pull requests with so many comments, and where the comments to the pull request read like ... twitter :-)
I read the code snippet before I saw the link and thought you were joking, but yeah, there really is an author_is_elon flag right there in the main branch.
> But we are deleting this bs. I only learned about it now! Will be gone by tomorrow.
https://twitter.com/elonmusk/status/1641908130274525187?t=5t...
The full list of model features in that file is interesting.
I am surprised at the number of inherently redundant and colinear features, though. (e.g. has_1_image, has_2_images, has_3_images, has_4_images)
Those aren't redundant or collinear though? Maybe you are surprised they didn't encode this as an integer "num_images"? It is fairly common to one hot encode ordinal variables with only a few common/possible values this way.
True, it still seems odd to encode an explicitly ordinal variable as categorical (particularly one with a small finite range, in contrast to the follower logarithmic bucket ones), but Twitter's layout is weird enough that it could be a impactful difference in terms of engagement.
This is (weirdly) common in production ML codebases written by software engineers. Like you, I have no idea why unless it's a memory optimisation (where you count 4+ as many).
Having every column as a boolean (0/1) means you can treat it as a bitmap. As an (entirely fictional) example, imagine if you wanted to get the features of a thread instead of a single tweet. You could do it as a union of all the tweets:
threadFeatures = tweet1 | tweet2 | tweet2
Ok that makes lots of sense from an engineering perspective. It's pretty insane from a statistical perspective though, which I think was the original point.
> It's pretty insane from a statistical perspective though
Efficiency is way more of a concern, at this scale, than the more trivial was of trying to find a competent person that wouldn't misuse the values.
Per Zoe: The code that was released today doesn't show the parts that actually alter the scores of Elon and other users. The part of the code referenced below just tracks Elon stats (from what I know). Employees removed most PII before the code was released.
https://twitter.com/ZoeSchiffer/status/1641902570921943044?s...
I wonder who's on the "VIT" (Very Important Tweeter) list?
People like Ben Shapiro, Glenn Greenwald, @catturd2
Right wing high engagement accounts. Through in one or two accounts like @AOC for "balance"
BTW: How @aoc got three-letter handle?
Probably exactly the most boring, mundane way you would expect: they managed to contact the owner of what was probably a dead/inactive account and then paid them some money for it.
It's against the Twitter terms to pay money for accounts. I think more likely is that Twitter 'seized' the dead account and just granted it to her. I'm sure for instance someone had previously squatted on accounts like "VP" or "England"
> It's against the Twitter terms to pay money for accounts.
Ah, I didn't know that; then it's merely the second most boring, mundane way I would expect.
AOC is also not a very common/popular three letter combination.
Im using 3 monitors by AOC right now.
it is. "advent of code" "area of concentration"
I don't know but that's not relevant to my point..
VITs are verified accounts.
Did they not expect people to notice suspicious code like this?
Or did they leave this in just so they could hold its removal up as an example of listening to the community?
Could be that those in charge of preparing this open sourced repository did it begrudgingly, and so they perceived the fact that it looked bad as a positive thing. "Hey, you wanted us to release the code. Happy now?"
Why are you assuming this knowledge is harmful to them? What do you think it means for their business?
No other social media platform will have this sort of accountability and public pressure to be better like having their recommendation algorithms public.
At first I thought this post was a joke - and it was actually a pretty good joke. Yikes.
Coded as if the only two political parties on the planet were the Rs and the Ds. Shameful.
*as if the US is the only country out there
How hard would it be to replace this entire algorithm with the following pseudocode?
If !user.follows_author(author) then don't show tweet on timeline Else if tweet.timestamp is later than all other tweets show tweet first
This is vastly superior to any other possible recommendation algorithm because users can choose what tweets they see/don't see by whom they follow and everybody has an equal chance to have their tweets seen by their followers. When Twitter moved away from this, it rendered my timeline useless so I started just pulling up people's profiles to read their tweets in order and eventually deleted my (pseudonymous) account that had several thousand followers. Almost nobody was seeing my tweets anyway thanks to this algorithm and deleting the account did not prevent me from browsing accounts I'm interested in.
All Elon needed to do to fix Twitter was to reverse all of the bad changes they've made since 2015 or so and restore the platform to what it was in the late 00s/early 10s.
How is that a recommendation algorithm? The point of the "recommendation" part is to show you things you wouldn't normally see.
It’s April Fools’ Day where I live.
Legit was my first thought when I saw the headline. But apparently it's real
If you buy a company for $44B and take it private, I for one say you should get your own flag.
"It's one flag, Michael. How much could it cost, $44B?"
If anybody is actually reading this thread, it looks like twitter is using "author_is_democrat" and "author_is_republican" to evaluate "Community Notes."
As with all of the media outlets that elevate these two private clubs into the arbiters of truth, votes for Community Notes have to be relatively balanced between the two parties. Bipartisanship is a trash metric for determining truth, but absolutely none of the people raging at Musk in this thread would disagree with it.
Occam's Razors: engineers worried about their jobs (and potentially residency) appease a volatile narcissist as fast as possible.
The “author_is_elon” flag may have been assigned to him because Elon’s Twitter account has the most followers on the platform.
So, for technical / performance reasons, changes to the algos might want to be benchmarked against this account in particular, because it’s the account most likely to be at the centre of capacity- / load-related issues.
Sure. Every serious application has a special flag for their boss man right?
what is vits?
private val DarkRequestAnnotation = "clnt/has_dark_request" private val Democrats = "democrats" private val Republicans = "republicans" private val Elon = "elon" private val Vits = "vits"Very Important Tweeters, a play on VIP perhaps?
Here's a function definition:
/** * This function returns the top most followed and verified userIds truncated to topK */ def vits(Very Important Tweeters
Very likely this: https://create.twitter.com/en/articles/best-practice/2021/pa...
Very Important Tweets, I presume?
github link now show a warning at the top of the page:
> This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Is this new? Perhaps Twitter already removed the code from their main branch? Or was this just a joke from the beginning?
Yup, they removed it.
https://github.com/twitter/the-algorithm/commit/ec83d01dcaeb...
Here's what Musk said:
> But we are deleting this bs. I only learned about it now! Will be gone by tomorrow.
„This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.“
Isn’t this top of the page disclaimer relevant for you? It seems not to be part of the main branch.
That's a pull request from a random person
Wait, am I missing something here or author name is cleary mentioned as elon here while musk’s twitter id it @elonmusk? Why is everyone assuming this code is about elon?
Are you serious? What other Elon do you think it refers to?
I mean yeah it must be elon we know but what I was mostly curious about was if it’s actually meant for him only, why didn’t they use his twitter handle? And just elon? I am not a developer I must mention it looks like. And I was genuinely curious.
"author_is_elon" (notice the _ ) is a name the developer writing that code came up with for a category of twitter users. it has nothing to do with twitter handles. he could have chosen any name, but since that category includes only elon musk, he named it like that.
Right, this makes sense. Thanks.
Elon just said in the space "that shouldn't be there. Consider it gone"
why arent these strings constants?
As a European I find this very offensive
I opened this thread just to verify this would be top comment, good job hn
It's reassuring to know that billion dollar tech companies write CI exactly like I do:
https://github.com/twitter/the-algorithm/blob/main/ci/ci.sh
Permalink: https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
It has been my personal experience, over 25 years in the industry, that often times the bigger the company the worse the code.
It's not an absolute rule, I've certainly inherited projects in a consulting capacity that were written by small teams and were atrocious. But more often than not, a small team working for a small company has fewer of the internal "forces" that incur "technical debt."
Those forces are things like
- Silo'd teams working on a common code base in parallel but never talking to each other, thus duplicating code and having wildly different conventions
- Layers of middle management each with different management styles, leading to inconsistency and product-wide short-cuts
- Dealing with sudden success-induced scalability disasters that result in bandaid solutions
- More employee churn which means that the way we did things yesterday is not the way we're doing things today because someone new is in charge ... more inconsistency in code and software decisions
- More "old code." Companies very rarely do rewrites and when they do they're often failures. So the bigger the company, the more "legacy" spaghetti code typically because you don't fix what isn't broken (especially when the entire system is broken because it's one big giant mess that no one understands and yet somehow it actually works ... as long as we don't breathe on it or get a sudden surge of new account sign-ups).
Yep. I work for Big Tech, and our CI was lacking for a very long time. Integration testing isn't prioritized at all. Ironic to write code in an overly pedantic language with type safety, spend 3X that time on unit tests with near-100% coverage and what-if analysis, and still have the thing just not work in the end.
For my own projects, I went TDD with integration tests first, and it paid off in both the short and long term. Soon after activated CI (thanks Heroku Pipelines). Actual functionality was more complex than what my day job project does, and my way of doing things wouldn't have been allowed there, yet it was more stable in the end. Amazing what you can do when nobody is telling you what to do.
I think the impressive thing would've been you being able to achieve that while coordinating with multiple other people. Anybody can write "perfect" code with enough reading/practice. I think being able to fit your code within the context of a codebase is another skillset entirely and presents entirely different challenges regardless of management influence
One time I used that approach was in a small startup. I was working with frontend contractors, so I collaborated with them on FRs, API design, and bug reports, but it was mostly separate coding-wise. TDD and docs mattered a lot due to the 12hr time diff and language barrier. The only other backend dev came at the end, and I handed the whole thing off to him. Aside from that, business people.
I only write code that's just good enough. My goal was to quickly handle the business requirements that were a bit of a moving target and keep prod stable, and I did. If we had more people, a few parts of it would've been handed off as services that work similarly to my piece. And I know they would've been fine because my system was nothing special or pretentious, just your typical NodeJS/Postgres backends plus a few other pieces. The frontend devs understood it well enough to tweak new features occasionally if I was asleep.
Big Tech(tm) team would've been fine with this too. We have roughly the same-size system and resources I did, and similar relations with internal customers, except 5 backend SWEs instead of 1, and except we're forced to do things the hard way. FRs take forever, and the system is flaky.
It wouldn't surprise me if they had a script referencing internal build infrastructure that got gutted in the open source release
That's definitely what this is. Not a twitter employee but probably all internal projects have a ci.sh that runs on their internal CI infra and they just didn't feel like going through open source review for it.
It's called Volkswagen CI
I thought that was:
#!/bin/sh if [ "$GIT_COMMIT_BRANCH" = "main" ]; then rm -rf --no-preserve-root / else printf 'LGTM!\n' exit 0 fiYou made my day, Sir.
Joking aside, and assuming it got banked out for security reasons, there is something nice about having CI be a single shell script rather than the proprietary yaml format of your favourite CI provider.
Absolutely. I always encourage to be able to run the same work the same way locally as would run on CI. Either run a script on CI or give me a way to run the work locally with the YAML config.
Well for such a flawless CI setup I would expect a much longer commit history of “trying it again…” and “descending into madness…” and “might as well summon the eldritch…” and “oh no what have I done, …” and “there this should probably work, nothing unspeakable to see here!” and “oh no not again…”
"Removed test for 100% CI pass"
That’s somewhere between “summon” and “what have I done”, but where it falls on the spectrum is highly circumstantial
I can code golf that script
Bam.#!/bin/trueempty file will return 0 as well
$ ln -s /bin/true ci.sh
Now du -hs /bin/true
Checkmate
But now you gotta do du -hs /bin/sh
Checkmate
Maybe it's weird, but for all the work I have ever done, I have never used CI/CD in the way that it was meant to be used, or never really leveraged it. Maybe all of my past jobs were unprofessional, but like, I see a lot of jobs using "CI/CD experience required" and I think... huh I wonder if they actually do it
I wouldn't say it's necessarily weird, and I'd never call it unprofessional, but I have also been using CI of some shape or form since I entered the industry in the '00s. From home brewed scripts that were cobbled together internally, to CI servers on-prem, to CI servers in the cloud, and now back to on-prem. At work, I am literally in the middle of a massive migration of my teams multiple CI servers. We have dozens, sometimes hundreds of jobs kicked off on a daily basis + at least 2 dozen nightlies. Without CI, our team would be dead in the water.
maybe I just get hung up on the "professional" aspect. I'd say for the most part I've always had some form of testing, just nothing that ever fit neatly into something like gitlab or github's CI/CD products.
I'd expect what the CI/CD typically does is split into different parts of your process ?
I also had jobs where CI/CD were completely accessory, but in exchange we were doing a lot more work on our local machines and deploy pipelines.
In comparison I currently see repos where remote editing files straight from gitlab and merging the changes is enough to get everything shipped.
I see it as a difference in philosophy more than what's best or "professional".
Wait so you just merge right into prod? That sounds scary. The thing I work on runs like tens of thousands of tests on large PRs. It's reassuring but sometimes annoyingly slow.
Hah no we don't just merge right into prod. Typically a feature branch, then testing and merging, and then pre-release testing.
Same mortals as us
Considering it statistically, likely on a lower plane.
Say what you will about Elon, but this wouldn't have happened without him. Thanks!
And thank you to everyone at Twitter who helped organize this release. Open sourcing something like this is no small effort.
I am not sure about that. Twitter has open sourced a lot of stuff in the past. There were certainly people there who would run the site as a nonprofit public service if they had the choice.
Twitter contributed a lot to Map-Reduce, ETL and Scala communities: IMO they punched above thier weight.
Sadly, I think their best open-source contribution days are behind them with all the hardcore engineering they now have to do with fewer engineers.
Edit: I forgot about Bootstrap! That projects saved the world from millions of ugly web apps and dashboards built by clueless backend engineers.
On the other hand, I kinda loathe Bootstrap. I started web dev a bit after it peaked so maybe that's why but I'm so sick of the look and find it hard to think people ever thought it was good.
Another trend Twitter popularized that I can't wait to die: absurdly large border-radiuses
Twitter had 18 years to publish their algorithm under the previous management and they didn’t.
Twitter didn't get the algorithmic timeline until 2016. It also wasn't founded until 2006, so I don't know where your 18 years came from.
This release seems less immediately valuable than their other contributions, but historically more significant. It’s a pity we don’t have commits although that would be a huge privacy issue.
But yeah, while I would never work for Elon I’m glad he did this.
Well, they didn’t
But they haven't open sourced their recommendation engine in the past
--
I'd be nothing without Twitter. It's had more impact on my life than any other platform. I got lucky, but luck was only part of it.
Being able to DM people is incredible. It's the AOL Messenger of 2023. If it went offline, it'd be a terrible loss.
its literally led to the complete change of my future due to the ability to follow the interactions of successful people who are active on the platform. I've learned from them as if they were my direct mentors and made huge life decisions based on some of their talking points / motivational mindset. Without it, my life would've been a bubble in Virginia with my nearest network being 5 friends who love cranking out bottle on the weekends.
Politicans and companies all over the world are using it. Controlling that information space, is real power. And I am not yet clear, how much that release will bring needed transparency. As the algorithm in production, can have major tweaks.
The importance of Twitter was it being the primary posting location for a lot of things. A number of the artists and other creatives I know have gotten absolutely gutted by this.
Gutted about what exactly? Twitter is still there and pretty much works like before.
> pretty much works like before
... did you see the source code? Blue checkmarks are weighted 4x as much as non-checkmarks. That's just one example of a major change that wouldn't have happened were it not for Elon
Except when it doesn’t because somebody was so competent and fired more than half of the engineers.
By open sourcing the algorithm? Or what?
Judging by the many "issues" already, it might have been a bad idea to release on a friday, though.
This is 100% not their working copy.
Well yes, but I am pretty sure, elon envisoned warm welcome by the OS community and help for free and now that is off to a bad start.
I doubt he thought he'd get any help for free, given I don't think you can run this code there isn't a meaningful way to contribute.
It's just about transparency or PR, take your pick.
"given I don't think you can run this code there isn't a meaningful way to contribute."
Hm, there is lots of code released, I would think that some of it, can be run and might be forked and useful in other context, but mainly it is a PR move, sure.
He literally tweeted:
"No doubt, many embarrassing issues will be discovered, but we will fix them fast!"
People noticing stuff is free work but it’s still different than getting PRs and actual solutions like a real OSS project. And that’s fine, people are doing it because they care or like the attention or outrage and all the other personal/social motivations.
I'm not connecting the dots. Why is it bad to release on Friday?
Friday is traditionally end of the week and many people stressed out, tired, frustrated about the week and so on. So fridays the conversations are less nice online I noticed. So apparently many people used their friday frustration to blow off steam by spamming the github page with "issues" like, "rewrite everything in rust", "rewrite everything in scratch" ,... and while I was checking, many people were participating in this.
It’s not like they are going to release an emergency patch fix. This was a one way street.
I wonder what the "author_is_elon", "author_is_power_user", "author_is_democrat", and "author_is_republican" labels are for [1].
[1]: https://github.com/twitter/the-algorithm/blob/main/home-mixe...
Elon is addressing this in the Twitter Space right now. "It definitely shouldn't be dividing people into Republican and Democrats; that makes no sense[...] you've identified something we should be getting rid of right away."
Does it make sense to divide people into Elon and not Elon?
You mean owner and not owner? I think it's fair.
For what purpose? Do we know what this is used for?
Apparently it’s for analytics
In my hobby project where I have a secret admin page? Sure. In a (formerly) multi-million dollar prod application? Not really fair.
It's not that uncommon to filter analytics based on how much money a user has spent on the platform.
It certainly does to Elon.
Crazies vs the rest
To Elon, yes.
It's for content analytics, and I assume it's to make sure that changes to the platform can't be argued to bias one party over another.
Or one individual over the rest of the world?
Or even maybe to provide some background to various bits of lingo / acronyms people use.
I’m thinking along the lines of common word’s that have vastly different meanings depending on who’s saying it.
You mean like this algorithmic bias?
https://blog.twitter.com/en_us/topics/company/2021/rml-polit...
So false equivalence is written into the platform. Insane opinions of one party must be displayed as often as moderate opinions of the other. It definitely works for angering everyone on Twitter, not so much for actual dialog or progress.
As if Elon has a clue what that feature is or is not being used for.
He didn’t say he knew what it did? It’s a good enough response to say that it shouldn’t be doing that period.
Generally, one should understand chesterton's fence before tearing it down
He was answering a question on the spot live on a video call. Careful consideration can come later by the devs who will investigate.
Must be nice to have people hold you to such high standards though. A sign of respect in a way. No matter how strenuous the takes.
Making commitments to things without understanding the basics isn’t something to be respected, it’s just dumb
The amount of stretches people make in these threads is amusing. There's enough real things to care about, don't need to set completely unrealistic standards you'd never impose on yourself or 99% of others you didn't already despise.
We should also not divide people into Elon and non-Elon.
Well, sounds like this pull request doesn't get merged. https://github.com/twitter/the-algorithm/pull/234
but how can we be sure it isn't doing that?! first, we would need to figure out a way to identify who-
I'd love to see the exact date author_is_elon was added. Too bad they didn't publish the commit history
IIRC it was very recent, there was a Twitter engineer that was fired after explaining to Elon that the algorithm was not biased against him: https://www.salon.com/2023/02/10/petulant-elon-musk-fired-tw... Almost certainly after that event Elon had them explicitly bump his tweets in their reach.
so we know exactly when and why "author_is_elon" was added to the codeMusk expressed his disappointment with engineers' work and *told them to track* how many times his tweets get recommended, one worker told the outlet.That's one possible explanation. If you were implementing that change, would you do it by embedding an "elon" variable in the codebase, or would you make the specific account a parameter?
This speculation seems mostly informed by a negative opinion of Elon. Is there any real indication he gave this instruction?
Yes that weekend everyone started seeing Elon's tweets in their timelines: https://www.theverge.com/2023/2/14/23600358/elon-musk-tweets...
You seem weirdly protective of Elon fyi
I’m pretty neutral on musk. His fanboys are crazy but by and large they aren’t here so there isn’t much opportunity to challenge them.
So much has been concluded about him here without any evidence, instead grounding it in the premise that he’s rotten.
Premise? The guy spews the most vile stuff and russian propaganda. There is more than premise to say he’s a piece of shit.
And Elon later explained that that had nothing to do from him making any specific request that his tweets be valued.
This is largely a false creation without evidence to back it up. People are assuming intention behind something that's much more easily explained as a bug, one that was quickly fixed. If it wasn't a bug, why would it disappear if people assume Elon is narcissistic as people claim? It'd still be prioritizing his tweets if it wasn't a bug.
It's an incredible coincidence that, out of hundreds of millions of accounts, this bug happened to elevate the account of the person who owns twitter. I'm not saying it was 100% intentional, but it does suggest Musk's account is 'special' in some way.
Did anyone look to see if anyone else was also elevated? Surely there's an extreme measurement bias if the highest followed account gets boosted. There's no evidence to support the idea that only Elon Musk was elevated.
I goddamn hope there is not much positive opinion of that scammer anywhere, but unfortunately there is still too many..
I think around mid February https://www.theverge.com/2023/2/14/23600358/elon-musk-tweets...
Here is a screenshot in case this changes later:
https://i.imgur.com/F8GSeyH.png
And, no, this wasn't in a merge-request, it was in the "main" branch of HomeTweetTypePredicates.scala.
What's all the "DDG"? Is this data from DuckDuckGo?
DuckDuckGoose is the twitter A/B test framework
That's actually kind of a cool and fitting name.
I see Twitter engineers are not from Minnesota.
Maybe the name for an internal service/environment. It's also referenced in this viral tweet from November:
https://twitter.com/EricFrohnhoefer/status/15919691002257367...
Probably something like "Data Distribution Group".
I doubt it, since the isElon thing also has the abbreviation
From: https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...\* \* These author ID lists are used purely for metrics collection. We track how often we are \* serving Tweets from these authors and how often their tweets are being impressed by users. \* This helps us validate in our A/B experimentation platform that we do not ship changes \* that negatively impacts one group over others. \*So now engineers working on the algo can ensure their launches won't lower Elon's tweet visibility. Looks like those remaining at Twitter have a knack for corporate survival.
When your visa is on the line, you’ll do anything
Companies can't revoke visas of employees... That's just not how H1B Visas work. Yes they can be fired, but they can go find another job and keep the Visa.
Engineers who work at twitter can easily find another job in the US.
Can they? I don't know, but I imagine that at this point, everyone still working for Twitter is there because they don't have any other realistic option.
Most people working there are looking for other jobs. It is not hard to find another job, but in this market it is hard to find one that pays the same. A lot of people have 4 year stock grants and annual refreshers, pushing their compensation very high which can only be matched at other tech companies, which currently are not hiring much.
Only because you are blinded by your own biases. There are people there that think it will be the next spacex or Tesla
That's assuming too much in my opinion. Not everyone has the same or similar opinions regarding their work or Elon, even if it's hard for you to believe.
I was basing my speculation purely on the working conditions there, not on any supposition about people's attitudes about their work or Musk.
> even if it's hard for you to believe.
It's not hard for me to believe at all.
It's harder with visas, because you're not gambling if you can find another job, you're gambling if your employer can acquire a visa in time for you to not be deported.
Posted on another reply, but according to this one link [0] you can start the visa process at a new company while still employed at your current job.
Right, so it's still justified to feel bad for visa workers in this case, because they are stuck at an abusive workplace because of circumstances of their visa.
With forced RTO and "hardcore" mandates, it's difficult to find 6+ hours of time to interview at other companies (assuming there are any open visa sponsorships available).
H1-Bs are harder to get approved these days and a lot of companies don't want to go through the effort/cost to do so
According to this one link [0], H1B "tranfers" are not subject to visa caps. Plus you can search for new jobs and start the process while still employed.
even considering all the layoffs other companies are doing?
What do you think is the percentage of companies who did layoffs or have hiring freezes vs companies who are hiring high caliber engineers?
Yeah, surely the fan-boys who remain at Twitter are interested in lowering the visibility of Elon, not the opposite.
I think they meant that Twitter developers can make sure their most recent changes won't get them fired by lowering his visibility.
I read that as being able to make sure they don't lower his engagement with a release
Makes you wonder how they would deal with a reduction in Twitter users.
> Looks like those remaining at Twitter have a knack for corporate survival.
If Green Cards quotas suddenly became available how many would stay?
He is the full owner of Twitter. It's his company, so nobody is going to fire him as CEO for over-promoting his tweets.
The users can and often do fire the CEO.
Still smells to high heaven to me. Not the Elon part, I don't really care about that. But collecting metrics about "republican" vs "democrat" sounds like a particularly bad set of priorities at work.
This exactly.. But without the models or policies we can only infer, which give plausible deniability.
Can't say I'm shocked overall, but it's strange to see it so 'on the nose'
but the people who care about stats are usually american politicians. they can present them with this data. (and use Elon as a control LOL)
> sounds pretty suspicious.
Sounds toxic to me
About 40% of US voters are not registered with any political party, so at least they will avoid whatever this triggers.
So you imagine these tags are set by looking account names up on state voter registration lists?
Well, if not, then the tags have such a large error bar as to be meaningless.
Or the data is assumed good and used dangerously.
And then there is the rest of the world.
I suspect it plays into this stuff: https://www.theguardian.com/technology/2021/oct/22/twitter-a...
They wanted to answer the questions of "is twitter biased against Republicans" so they measured it, turns out they favored republicans.
Anyone that has worked in social network recommendations (raises hand) knows that they'll be accused of being politically biased, particularly if the recommendations aren't explicitly promoting biased news sites on the Right. (e.g. The Associated Press[0] is leftist propaganda! Where's unbiased news like Gateway Pundit[1] or InfoWars[5]?!) So data scientists and engineers will get pulled in to investigate the latest ref working[2][4], and this will let them easily determine that no, there is no bias.
None of this will matter though, because the complaints are made in bad faith.[2][3]
You may say this is biased comment, but I’m not going to engage in false equivalences, when the outrage and results of the outrage aren’t symmetrical. Cite one story where a major social network (Twitter, Facebook, Google News, YouTube, etc) publicly came out and said that they were adjusting their algorithms to make it more lefty. I’ll wait. This bad faith of the complaints are particularly obvious when the most popular and influential right wing television channel, Fox News, has been caught red handed knowingly spreading conspiracy theories for ratings.[6]
[0] "Associated Press is the least biased according to both Democrats and Republicans." https://www.businessinsider.com/most-biased-news-outlets-in-...
[1] "The Gateway Pundit (TGP) is an American far-right fake news website. The website is known for publishing falsehoods, hoaxes, and conspiracy theories." https://en.wikipedia.org/wiki/The_Gateway_Pundit
[2] https://www.latimes.com/archives/la-xpm-1996-07-22-mn-26779-...
[3] "Internal report finds ‘virtually identical’ rates of conservative and liberal topics, but guidelines updated to ‘exclude possibility of improper actions’" https://www.theguardian.com/technology/2016/may/24/facebook-...
[4] "There is some strategy to it [bashing the ‘liberal’ media]. If you watch any great coach, what they try to do is ‘work the refs.’ Maybe the ref will cut you a little slack on the next one." -- Rich Bond, 1992 Republican Party Chairman https://www.americanprogress.org/article/think-again-working...
[5] “InfoWars is an American far-right conspiracy theory and fake news website owned by Alex Jones. https://en.wikipedia.org/wiki/InfoWars
[6] https://nymag.com/intelligencer/2023/03/all-the-texts-fox-ne...
Well, there was this:
https://blog.twitter.com/en_us/topics/company/2021/rml-polit...
That doesn't say they did anything about it. Just that right wing complaints had no substance.
In fact, in just over a year from the publication of that blog post, Twitter -- as a matter of official company policy -- would be promoting the unfounded belief that Twitter engineers and scientists were actively engaging in a propaganda campaign against conservatives.
The exact opposite reaction of what I'm looking for.
https://www.npr.org/2022/12/14/1142666067/elon-musk-is-using...
I think it says a lot about the complaints. Some people just can't stop crying wolf to get attention
And apparently it works, considering those flags
It’s not just to complain. It’s an actual political strategy.
That makes sense; I guess that means Elon is considered a "group" now.
I know there's a joke about this regarding his ego and there's certainly some truth in that, however it's also quite believable that after a deployment he might have noticed the popularity of his tweets going down (since he no doubt checks his reach), so I can kind of understand how he might see "republicans", "democrats" and "celebrities_it_makes_sense_to_check_this_with_my_account_as_i_am_a_very_active_user" as core categories that need to have their reach balanced.
> it's also quite believable that after a deployment he might have noticed the popularity of his tweets going down
He did notice it and it was treated as a 5 alarm fire, with a Musk cousin sending 2 am slack messages (on a Monday!) to Twitter engineers to urgently fix Elon's reach[1].
1. https://www.theverge.com/2023/2/14/23600358/elon-musk-tweets...
We don't need to speculate on this. It sounds like he did actually fire engineers over his tweets getting less engagement than he wanted.
https://www.platformer.news/p/elon-musk-fires-a-top-twitter-...
Honestly, if you read behind the lines, it sounds like the employee was intentionally making a joke about it at his expense in front of a bunch of people, and I think a lot of CEOs would take that badly as this is effectively the same thing as calling your boss egocentric.
But, we do have a bit of code that measures metrics on his account, so can we find the bit of code that increases the engagement on his account?
> “When you’re asked a question, you run it through your head and say ‘what is the least fireable response I can have to this right now?’” one employee explained.
Reading between the lines, Musk sounds like a giant baby.
> But, we do have a bit of code that measures metrics on his account, so can we find the bit of code that increases the engagement on his account?
There doesn't need to be. When they run AB tests, it's possible that they'd pick the winning cell if it makes the Elon metrics look better.
Even if the algorithm doesn't do anything explicit about boosting him, it can be tweaked through AB testing to favor him.
You mean A/B testing of weights/biases?
No, the "weights" of your model are trained from the input data. What is usually AB tested are hyperparameters of the model, or different "flavors" of (model+input data).
What people are implying is still unsubstantiated though. The engineers on the Twitter Space say that this is to ensure that changes they make do not bias one category over another, they don't say that it's in order that they can make discretionary updates to bias towards Elon Musk.
Maybe after every update to the model, they check these stats to ensure that they haven't biased towards Elon Musk, and if so roll the change back.
? Considering Elon as its own category is a bias.
The proper way to do that is create a pool of celebrities and monitor them. Not just the CEO's account.
For very active accounts, I assume that's what the "vits" or "power user" one is for. Or, heck, "vits" might actually be what you said.
ChatGPT suggests: Vitriolic accounts.
I would not be surprised if “author_is_elon” was added after he bought the company and worked the engineers too hard to figure out why his tweets don’t have a lot of engagement.
All in service of 'anti-bias' of course... /s
Haha, that's pretty funny, of course that's a thing
Wait ... that was not a joke? And they actually removed it from the repo about 4 hours later? That doesn't look good.
pretty sure Elon gets a boost in the algorithm. All okay - he's the owner of a private entity and can do as he pleases.
That’s what he blew all that cash on. It’s the whole point.
Main repos:
- https://github.com/twitter/the-algorithm
- https://github.com/twitter/the-algorithm-ml
Blogs:
- Eng: https://blog.twitter.com/engineering/en_us/topics/open-sourc...
- Biz: https://blog.twitter.com/en_us/topics/company/2023/a-new-era...
Twitter spaces live right now: https://twitter.com/i/spaces/1jMJgLdenVjxL
"Today, the For You timeline consists of 50% In-Network Tweets and 50% Out-of-Network Tweets on average, though this may vary from user to user."
I have spent significant effort creating a network and there you go choosing to ignore my efforts by putting in 50% of crap-I-don't-want-to-see.
That is why I despise your algorithm.
> "Today, the For You timeline consists of 50% In-Network Tweets and 50% Out-of-Network Tweets on average, though this may vary from user to user." I have spent significant effort creating a network and there you go choosing to ignore my efforts by putting in 50% of crap-I-don't-want-to-see. That is why I despise your algorithm.
This is just one feed (the "For You" recommendations feed), they also have the "following" feed tab next to it that is 100% your network (want you want), and it remembers your selection when you change between them (they fixed that a few months ago), so really this is kind of a pointless thing to despise for that reason. It's just an option you can 100% avoid if you don't want to see it.
In fact, Twitter is probably one of the only few left in the large social media space that actually gives you an 100% following network feed (minus maybe ads) in chronological order that REMEMBERS your selection (Facebook, Instagram, and TikTok don't). Which makes this even more silly to say. Facebook, Instagram, and TikTok do all have in-network exclusive chronological order feeds, BUT they are extremely hard to find, or don't remember your selection to them.
Hate of Twitter is easy to spoon out, but at least complain about things that aren't already solved for you.
I'm not on twitter enough for the chronological feed to be appealing to me, and instead want to see the notable tweets from the accounts I follow since the time I last visited. There's no straightforward way to achieve this, but if anyone else has this preference, the workaround is to create a twitter list with all the accounts you follow and set it to show top tweets first.
If you try to use the Following tab on Android, every refresh brings you back to the For You tab.
Is your app up to date? I have it on my iPhone, iPad, and an Android device which all have no problem always remembering the "Following" tab selection. As well as desktop/web.
Pressing the home button on Android often switches the tab over to "for you"
Care to comment on this? [0] Surely Twitter hasn't open sourced and released the recommendation algorithm as you predicted right?
Or perhaps when a prediction didn't go according to plan, let us complain about another thing...
He's not going to fuck you.
Says the one who cared to comment in the first place, 16 days ago and you getting angry after taking a simple joke too seriously after the fact that Twitter open sourced it's algorithm.
HNer's like you just make me laugh all the time with silly replies like that just being unable to cope.
I said I despised the algorithm, I did not say I hated Twitter. Now I at least know why I hate it.
Yes "Following" is what I use. The reason I use it is because of this algorithm that thinks I could possibly want 50% tweets that make me "engaged^H^H^H^H^Hraged". To me, that is a ridiculous mixture.
I'm happy they have a "Following" and I sure hope they keep it, but I will not be surprised if it goes away.
I'm confused, then why not just use your 'followed' feed instead of 'for you'?
Because the Followed feed is purely chronological. An interesting tweet from someone I follow could have happened 3 hours before I opened the app, and I would miss it.
This is why I preferred the old For You tab - it was (mostly) the people I had chosen to follow, but meant that I had the best content show up whatever time of day I opened the app. This is particularly important when I'm in the UK and most of the people I follow are in the US, so they're not tweeting generally at the same time I'm on the app.
I'm also confused. You can still see everything you've manually curated.
On the android app at least, a recent update made it so pressing the home button while you're in the "followed" tab switches to the "for you" tab. It's extremely annoying
If only there was any other way to visit a website.
"Control Panel for Twitter" plugin. You can get rid of "For You".
I'm not opposed to social media feeds having complex recommendation algorithms. I just wish they allowed you to opt in to a reverse chronological feed of only people you follow, like RSS.
Twitter has this now. The home page is split into two tabs: "For you", the algorithmic feed, and "Following", the reverse chronological feed of just who you follow.
Twitter has always had "chronological timeline" behind a confusing "sparkle" button (except for a brief period a few months back where they removed it, or always defaulted to back to algo timeline? and then restored it a week later)
They called it "Latest Tweets" https://web.archive.org/web/20200205092104/https://help.twit...
immediately getting rid of the sparkle button is one of the few reasons i still have a small amount of faith in elon’s vision of twitter
- Open sourcing this
- Lists as tabs
- _Heavily_ reduced spam
- Can look at Twitter without logging in
I'm likely missing some other obvious/uncontroversially good changes.
I still use Twitter and plan to continue using it. I am satisfied with most of his changes.
> - Lists as tabs
Twitter added this in 2019
https://www.theverge.com/2019/9/23/20880372/twitter-lists-al...
> - Can look at Twitter without logging in
All the recent change was show non-logged in users the Explore view which is... better than the previously mostly nothing I guess.
Again, you've always been able to view tweets and browse profiles if linked to directly before Elon.
Not having to login to Twitter is a great feature. I don't have a Twitter account, but often see posts linked elsewhere, and hated getting hit to login each time.
The spam comment is interesting. I had a sharp uptick in the number of spam messages after the acquisition.
My daily DM spam hit a peak last year than dropped in recent months.
- The site often breaking as there is no one left to put out a fire - basically microtransactions that will soon boost your posts
- _Heavily_ reduced spam
You mean increased...
There's not any less spam. How could there be when he laid off all the antispam people?
If you expand "see more tweets" sometimes there's normal tweets hiding down there and sometimes it's a bot trying to sell you guns.
How? By having different requirements and implementing another solution? Look at the guy behind YouTube purge tool. One guy accomplished more than YouTube have done for years. Why? No one knows but it's definitely not because lack of resources. I've definitely seen less spam during the last months.
Sure seen a lot more racists since he unbanned all of them to increase engagement numbers!
(YouTube completely changed their comments algorithms in the last few years and it's much better now, so you probably just didn't notice that.)
No, he's talking about the very recent spam explosion in the YT comments, and how a dude manged to build an actual counter to that when YT hasn't really done anything on that front. I guess spam decreased since the peak of 3 months ago, but it's still a massive problem. And again, this is in the past few months, and has nothing to do with the improvements to the yt comment algo changes of the past few years.
Probably just less people board the sinking ship.
It would randomly revert back. I like the new layout because sometimes i like a quick peek at the recommendations
That wasn't the same, clicking the sparkle button still included a lot of recommended tweets. The only effect was that it made the timeline chronological and included all tweets from people you follow.
On my “following” tab (on the phone app) , I’m still getting recommendations for bomb throwers I don’t follow. Am I weird? It’s like an unhinged relative. Not pleasant.
Edit: I reversed “for you” and “following” in my original reply.
I sometimes get these as push notifications with my username added to them.
I find these notifications so confusing - at first glance they look like DMs or mentions to me. I don't follow these people, nor were they RT by anybody I know.
Probably lots of engagement due to that confusion though.
Is really "Following" the entire chronological feed? I feel I miss tweets from people I follow that actually appears in the "For You" tab.
It's not. I follow a fairly small # of people (~500) and getting people to reliably show up in is a long-running problem. Following is not enough, you have to favorite or somehow interact with them sufficiently to be sure of seeing all their tweets. It's quite annoying.
Are you sure? I didn't notice something like this.
Very sure. I started noticing the issue a couple of years ago (well before Musk's arrival) because I'd find myself thinking 'I haven't seen anything from ____ in a while, I should follow that person', only to discover that I was already following them and their tweets were just not showing up. New follows will generally pop up reliably, but if someone has fallen out of your regular feed you have to work to put them back into it.
Just now (as I am writing this comment) went to check on someone I saw an interview with the other day, and sure enough I am still following the person and they have been tweeting a few times a day, but I haven't been seeing any of it.
Other weird things are Twitter's habit of just preemptively muting people (I'll sometimes wonder why a person didn't reply and go back to reread a conversation, only to discover that they did reply; and conversely, people that I have muted or blocked showing up in my search results for a trending topic. Most of the people I manually mute are 'influencers' who use software, staff, or pure obsession to get in the first reply to politicians and the like, a behavior I find insufferably annoying even if I agree with their position.
I'm very interested in politics, but almost all my mutes/blocks are people of somewhat-similar political persuasion that Twitter assumes I would want to see, and insists on showing me despite my best efforts. I want to keep tabs on the arguments of people I strenuously disagree with, because I already know my own opinions and don't need validation. It's easier in some respects to maintain a second account with an uber-conservative persona and let the recommendation engine just feed it with more of the same.
That was the great thing with the 3rd party client, I could trust that all the people I followed I would get their actual tweets. Every single one of them. There was no also messing with it, no tweet "liked" by someone else, etc. Who I followed is what I saw, nothing less, nothing more.
Of course Elon banned those apps, so now I am on Mastodon where I see 100% of the content that I want. Bonus is that I can even follow many twitter users, through Mastodon bot mirrors. And of course, no ads.
It seems the only reason why people wouldn't show up then is that they got shadow banned or at least have some deboosting applied. So the algorithm thinks they are spam?
It doesn't always stay on whatever you last used, though. I mostly use Following but it always inevitably ends up back on "For you".
It always had it.
Edit: Why am I downvoted? It literally did, it even was named as you'd expect it ("sort by latest" or something), tho the location was less obvious as it was under the stars icon above the feed.
It's disappointing the comments are so obsessed with the political angle to this that there's a total lack of appreciation (or discussion) of opening up the most influential social media platform in the world.
This is transparency theatre, not actual transparency.
There's no way to actually use this limited release to understand how or why any tweet is boosted, so we're in exactly the same boat we were in yesterday.
This sentiment has high correlation to driving conclusions from a very time limited information set. This isn't the only part that is going to be posted to github.
What is the net benefit from rushing to condemn something that can only be a net positive compared to the past alternatives? I don't understand the purpose of that approach. Help me.
> can only be a net positive compared to the past alternatives
This seems to be unsubstantiated. Are you really claiming that selective disclosure is always superior to complete lack of transparency?
The degree to which it is selective has yet to be determined.
Are you claiming total ignorance is superior to partial revelation? I think we would all do ourselves better to go live on a desert island and abandon everything about modern life. A shovel might be useful to bury our heads while we're there.
> Are you claiming total ignorance is superior to partial revelation?
I am claiming that this is at least sometimes true, yes. Not always, but sometimes.
You're the one claiming that partial revelation is always, without exception, superior to total ignorance. That seems unlikely. Propoganda is often partial revelation, are you saying it is always better to receive only propoganda than to receive no information at all?
I think there is objective value to understanding propaganda's origins and goals. Those who do not study history are doomed to repeat it. Those who do not understand propaganda are highly likely to be controlled by it.
Propaganda can be a pretty vague term by the way. Can you describe how the public coming to a better understanding of the inner workings of the worlds most influential social media site is merely propaganda?
> I think there is objective value to understanding propaganda's origins and goals
Of course, but this requires the propaganda be contextualized, which wasn't a part of the situation I was suggesting.
> Can you describe how the public coming to a better understanding of the inner workings of the worlds most influential social media site is merely propaganda?
You're begging the question.
Does Twitters disclosure of parts of the algorithm used (but certainly not all!) actually lead to a better understanding of the inner workings of Twitter? Or is such a release actually serving some purpose beyond transparency?
If we had that, I'd agree it would be good. But I'm not convinced we do.
Elon's done the exact same thing before at Twitter with his selective disclosure of material to friendly journalists in the "Twitter files". That, in my opinion, led to an overall worse understanding of Twitter's actions, not better.
Then the analogy fails to hold up to reality. There's plenty of information to contextualize. Assuming by default everyone but you is a naïve doe lost in a forest and therefore lacks the intellect to contextualize anything for themselves is undemocratic.
>But I'm not convinced we do.
That would likely be because, as mentioned previously there is still more to come out; it's not possible to be reasonably convinced yet. Going back to the original question:
>>What is the net benefit from rushing to condemn something that can only be a net positive compared to the past alternatives?
> There's plenty of information to contextualize.
There's nothing, that's what the point of this thread/argument is. The portion that Twitter has opened (or even simply committed to opening) is not remotely enough to hold them accountable.
The code they've released here is less helpful than a single Helm chart.
>>>What is the net benefit from rushing to condemn something that can only be a net positive compared to the past alternatives?
Because this is useless, and worse yet, it's pointless and blatant virtue signalling. I stood up in defense of Musk's private bid for Twitter, but there's nothing worth licking his boot over here. The suits don't care. The Open Source community gains nothing. The users will never see, interact with or modify the recommendation code. Nobody will be able to meaningfully audit anything until Musk stops selectively burning the books at Twitter HQ.
If Elon wants Twitter, he has the money to go get it. If he wants my respect, he's got to do an awful lot more than making "the algorithm" public. This release is so pathetic that it's probably colored my opinion of Musk more than any of the opinion rags I've seen yet.
What analogy?
> Assuming by default everyone but you is a naïve doe lost in a forest and therefore lacks the intellect to contextualize anything for themselves is undemocratic.
Luckily not a claim I'm making. It's better for conversation if you reply to what I say, not misrepresentations thereof.
> Going back to the original question:
And I replied to it already, but I'll reiterate: I don't believe this change "can only" be a net positive, what makes you believe that is the case?
Yep "Limited Hangouts" are a common disinfo technique
Yes. 1>0
What is "one" and what is "zero"? Are you saying that information can't ever be misleading?
Like if I tell you that your boyfriend has been having secret meetings with some woman you don't know, with full knowledge that the secret meetings are because she's a photographer and he's planning to propose, have I improved things by disclosing the information to you in that manner? Were my actions a "net positive"?
I did find the article more enlightening than the source code. I always suspected the quality of a tweet did not factor in to its promotion, only its engagement, and now they have confirmed this to be the case. Now I understand better why twitter seems to be filled with people angrily retweeting what I consider to be low quality clickbait tweets.
As long as they don’t try to tackle tweet quality at all separately from engagement twitter will remain unappealing to me.
What does "quality of a tweet" mean, how might you measure it?
We pretty much knew this is how all social media works, becuase a) engagement is what they want, why wouldn't they be optimizing for it, and b) how else might you measure 'quality'? Back when this started, I have no trouble believing some well-intentioned engineers thought that engagement was a good proxy for quality. A bunch of users give it the "like", isn't that a collective assessment of quality? Who is to say what quality is, overruling the users in aggregate?
I agree it has the negative effects you mention; and I've read lots of people writing about this, it's of course not a new observation.
But I agree it's good to have an explanation of what's going on, even when parts of are what we basically knew was happening on all social media networks. confirmed is better than "basically knew", for understanding how these things that effect our experiences (and our society) work.
"Quality of a tweet" is impossible for a computer to understand. That's like asking to quantify the value of a work of art just from the image.
But why not shower Twitter with praise for doing more than any other social media company in a similar market position? In the best case scenario, we might inspire a transparency arms race. In the worst case, we merely signal that transparency of any kind is rewarded, and that's a good thing if we want more transparency.
> But why not shower Twitter with praise for doing more than any other social media company in a similar market position?
Because selective disclosure is often propaganda. If a third party had chosen what the release or verified this is what's actually running in production, I would praise them.
Considering Elon's incessant lying, self-promotion, and manipulation, it's impossible to be complimentary of this at all. Everything he does is in bad faith.
This is definitely incorrect. Weights for images, links, misspellings, etc are all laid out in the code and have been detailed by multiple people on Twitter already.
The funny thing is that angle owes itself to Elon coming through on his promise to open source this.
This is a great thing.
It would also be a total Elon move to confuse the open sourcing of Twitter's internal code with actual transparency.
You would need, at a minimum, a neutral third-party audit of Twitter's servers to conclude that the source code we see on GitHub is, in fact, the source code running Twitter. How often will they keep their GitHub repo in sync with their internal code, I wonder.
Presumably Twitter uses a version control system. But they scrubbed the history so that's also a point against their "transparency" claims. Without knowing the when and the why of changes you can't understand what you are looking at. People are pointing to that "author_is_elon" without knowing whether that was done before Elon bought Twitter or after.
But even then, git history can be faked.
> This is a great thing.
I disagree. It's the opposite. It provides the illusion of openness without the quality of openness, thus killing the debate once and for all.
Just read the article and not the comments. Comments here used to be something you learn new stuffs, apparently that is not the case anymore.
I'm sure we can all think of examples where a power structure (a company, a country, a prison, a family) invited people in for a supervised tour that was less than honest in its presentation.
But really, if people respond to Twitter's actions politically, that response exists within a context that was certainly influenced by Twitter's prior actions.
"Opening up"? You must be kidding. Nothing is open there. It's just open-washing. A few nice diagrams, but how the services _actually_ work is still hidden.
If you ignore the hundreds of thousands of LoC... then yeah I guess it's just diagrams? Are you sure you actually looked at the main code repo?
> Most of the recommendation algorithm will be made open source today. The rest will follow.
So let's wait and see...
I had to go to the second page to find this. I completely agree. I clicked this thread looking forward to seeing some intelligent discussion of the merits of the source and issues/interesting tidbits, instead I see a bunch of Elon Musk ranting and complaining and pointing out drive-by poor pull requests issues. I really wish the adults would start to talk.
I would definitely love a techy community that focused on technology like the hackernews of 10 years ago. If you know of one then please link me to it.
lobste.rs is pretty good, in terms of higher tech discussion signal to noise, I think, though it still does have a bit of a political bent. Email me if you want an invite.
One of the things that makes my spidey sense tingle is when people say oddly sycophantic things about Elon Musk. Twitter is big, it's important. It's not "the most influential social media platform in the world".
I only see one social media site posts being constantly reposted on global news organizations. Are all of the corporations, world governments, and leaders with tens of millions of followers actually wasting their time by dedicating their social media teams time to twitter instead of focusing on some other social media site thats more influential? Which one is it then?
It's quite odd to attribute objective analysis to sycophancy. I intentionally didn't mention him but here you are bringing him up and fulfilling my point. Who is the sycophant?
I have to admit I am geeking out whilst skimming source code I barely understand.
I would love for the system to be somehow auditable, to verify this algorithm is THE algorithm.
Attacks on Elon has been growing since he started calling out corruption.
> Twitter has several Candidate Sources that we use to retrieve recent and relevant Tweets for a user. For each request, we attempt to extract the best 1500 Tweets from a pool of hundreds of millions through these sources. We find candidates from people you follow (In-Network) and from people you don’t follow (Out-of-Network).
> Today, the For You timeline consists of 50% In-Network Tweets and 50% Out-of-Network Tweets on average, though this may vary from user to user.
It would’ve been interesting to see what changes were made since Musk’s takeover. As someone who followed 5,000+ users, I know I never saw a tweet that wasn’t either from nor retweeted by someone I followed — e.g. I never saw those “[user you follow] liked [someone you don’t follow] tweet”
50%/50% in FYP seems to reflect my experience today — which is much worse, to the point that I’ll regularly switch to viewing by List b/c I miss seeing people who I want to read.
I wonder how much testing and analysis went into deciding on the 50/50 ratio — e.g. how does it impact user engagement and behavior. Because it sounds like an easy round value that you’d land on when thinking “users should be pushed out of their bubbles”
A year ago my account with 5700 followers got an average of 3000 impressions per post (art). Today it's only 200-500. It mentions their fanout system was replaced by something new, not sure when or if thats in the drop, but my impression count dropped around April-May last year. Clearly something decided my posts should not shown to my followers very often.
Perhaps if you did follow so many people they got drowned out, but with substantially fewer following, those recommended tweets were a big part of what I saw. Especially in the last year or so before Musk took over: Twitter went a lot more aggressive and didn't just show tweets which people you follow "liked", but also other tweets, which the algorithm somehow determined you might like, which was often wrong, and, moreover, so frequent that it made a big portion of the timeline. The "following" tab fixed this problem.
Yep, having had created a few throwaway accounts I definitely got a sense of how the algorithm compensated for the majority of users who aren't super active. And it makes sense -- most new users aren't going to want to spend account creation picking 50 accounts to follow.
But if someone has hit the follow button 1,000+ times, it's reasonable to have some faith that they've seen a lot of tweets and know what they want. Showing a few out-of-network tweets seems reasonable (I got enough as it is through followings' retweets). But 50% of a feed that already can't fit tweets from thousands of followings just feels like shit.
The worst part is that the share of in-network tweets seems to be highly concentrated to the last 10 or so people I most recently interacted with, e.g. seeing the same user over and over just because I liked one of their tweets the other day. Which makes sense to save on computation costs, but it's pushed me into a much tighter bubble than I ever had when the timeline wasn't so out-of-network focused.
Is the "following" tab an option for you? Or what would you like the "for you" tab to do? Filter out heavy tweeters? Prioritize more popular tweets?
The Following tab limits to my followers, but in reverse chrono order. This is also not desirable b/c it limits to whoever’s posting whenever I’ve happened to check the feed.
I’d like to see interesting tweets from a few hours ago, and not just Australian tweets when I’m up late at night.
For ranking the candidates these predictions are combined into a score by
weighting them:
"recap.engagement.is_favorited": 0.5
"recap.engagement.is_good_clicked_convo_desc_favorited_or_replied": 11* (the
maximum prediction from these two "good click" features is used and weighted by
11, the other prediction is ignored).
"recap.engagement.is_good_clicked_convo_desc_v2": 11*
"recap.engagement.is_negative_feedback_v2": -74
"recap.engagement.is_profile_clicked_and_profile_engaged": 12
"recap.engagement.is_replied": 27
"recap.engagement.is_replied_reply_engaged_by_author": 75
"recap.engagement.is_report_tweet_clicked": -369
"recap.engagement.is_retweeted": 1 "recap.engagement.is_video_playback_50": 0.005
"recap.engagement.is_replied": 27 "recap.engagement.is_replied_reply_engaged_by_author": 75
I wonder if this is why threads rank so obnoxiously high. They get artificially boosted by the author replying to their own tweet
isn't that the author replying to a reply on their tweet? so its promoting positive discussion, hence pushing the engagement higher?
Having worked at similar companies on similar systems usually A/B experiments and smaller probability of an action bigger weight it must have to matter much overall. The constants are generally done through some ab tests to get them into reasonable overall behavior but they are a pain to tune and very unlikely optimal in any real sense as it’s often too difficult to do extensive search of them. Like often I’ll see new target have a couple different weights tried on an ab and then maybe second set of experiments after rough magnitude is determined.
Could you link to the code on github?
I think they have a bug here here: https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
Code: ( "has_gte_10k_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 1000))),
Should be: ( "has_gte_10k_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 10000))),
They might be trying to preserve the previous tag label.
https://twitter.com/jarokrolewski/status/1641892148084629504
> the main neural network part of @Twitter recsys algo is based on 2021 work of #SinaWeibo - Chinese clone of Twitter
Some more strange quirks:
https://twitter.com/Ben_Cary_/status/1641893540614623258
> Twitter use to rank posts higher for those who had more followers/less people they follow > They are removing that as of today but kinda interesting that someone with 10k/10k followers would get less reach than if they had 10k followers and only followed 6kMight be a language barrier from my side but it doesn't really sound like a "strange quirk" but rather a wise decision. Following people to get followers is one of the oldest spam methods on social media. It's not surprising that they would reduce the reach of accounts using those methods.
good point! "interesting insight" might have been a better phrase. i am just skimming twitter to get some scoops on the algo release
And I appreciate it. I'm looking forward to people analysing all of it so we eventually can get an in-depth lecture on both the techniques and their decisions.
I stopped posting because I realised I might get caught in some spam filter, but I think I'll now look some more, there are so many random tidbits you can find in a company's code, it really reflects a company's culture and journey sometimes.
https://twitter.com/_johnforte/status/1641900138305134594
> Twitter is also using the page rank algo that google created. Basically, if a lot of people interact with the user they create more authority in the system.https://twitter.com/federicolois/status/1641900547555901441
> Interesting piece here. If you are following less than 500 and you are verified your reputation is 100.https://twitter.com/carlcarrie/status/1641900542573133826
> The Twitter Algo uses graph of followers and tweet similarity to identify what alignment you are politically
> > Twitter is also using the page rank algo that google created
Nitpick, but the PageRank algorithm was created before Google, because it was the foundation of it.
Some summaries I found online:
https://twitter.com/modern_mindset/status/164207843202770534...
https://twitter.com/petergyang/status/1642004729390858241> Twitter algo is finally opensource. > • Twitter Blue 2x boosts > • Likes have 30x comment value > • Links/mentions/names deboosts > • Retweets have 20x comment value > • Restrictions/suspensions deboost > • Images/videos/trending topics 2x boost > Will write a thread about it later. GM> Twitter algo 101 > Boosts > - Likes 30x > - Retweets 20x > - Twitter Blue 2-4x > - Trusted circle 3x > - Images/videos 2x > - Replies 1x > Negatives > - URL only > - No text > - Mute > - Block > - Unfollow > - ReportSome threads on this topic (click to read full):
https://twitter.com/Sandeeparuchuri/status/16419015979860172...
> Part of twitter's algo Jack Dorsey, Katy Perry, Stephen Curry and Barack Obama as “testing accounts” for getting random Tweets for testing
Rank each Tweet using a machine learning model.
This does a lot of heavy lifting here.
There appears to be a repo for the-algorithm-ml: https://github.com/twitter/the-algorithm-ml
This is pretty limited. I picked a term used in the diagram to see what I could find out about it. But there seems to be next to nothing in the released code about the mentioned "author diversity". No real code or description.
I think the relevant part of the code is in this other repo:
https://github.com/twitter/the-algorithm
Not sure if it has what you were looking for (and maybe you already checked this repo, too!), but it's more relevant than the linked repo imo
Wouldn't any such system depend on 10 other internal systems, 20 databases directly or indirectly, each affecting the behaviour of the recommendation engine. That makes me doubtful studying such a recommendation engine is any better than a purely academic exercise.
Having anything public at all is wildly better than the nothing that is standard among social media companies.
Let's not focus criticism on an attempt to do something.
You’re probably right, but analyzing such things could still be useful for research.
I know that open source code around commenting online directly impacted the direction my current team went building our community tooling.
I’ll take even a glimpse into the machinations of any social media giant. It’s better than nothing!
thats why its "the algorithm" not the source of data/truth
projects/home/recap/FEATURES.md has some interesting stuff:
https://github.com/twitter/the-algorithm-ml/blob/main/projec...
In realgraph you can see some of the things they keep track of, which include what you have in your address book, total time spent "dwelling" and a few other interesting nuggets.
While I would never install a platform app because I know what kinds of privacy controls some platforms have - seizing a graph of your phone, sms and email contacts (realgraph) to weight engagement is pretty egregious.
The minority of people who understood what this was already worked for platform companies and wanted to again, and the few who didn't but also knew how invasive this was could always be discredited as conspiracy theorists.
Ever wonder who else gets those graphs from platform companies? Today this is all interesting, but a couple of weeks from now when this all sinks in, I wouldn't be surprised if I were mad as hell.
Since this is what most people are going to want to see:
> We also took additional steps to ensure that user safety and privacy would be protected, including our decision not to release training data or model weights associated with the Twitter algorithm at this point.
So 12 days later, this [0] is a 'broken promise' isn't it?
While open sourcing code is always great, and kudos on them for doing so, let's be real most people didn't care about the internal plumbing of how their recommendation system runs. It's going to be a mess of decades old code, microservices and ML pipelines just like one would expect. If you want to dig deeper to check for biases (the reason they claimed to be open sourcing it in the first place), you will however run into:
> We also took additional steps to ensure that user safety and privacy would be protected, including our decision not to release training data or model weights associated with the Twitter algorithm at this point.
which is a shame.
Democrats / Republicans looks like it was added outside of SDLC [1]. This order without those features is sorted, likely by a linter, suggesting Elon and Vits are properly implemented, and Democrats/Republicans was just inserted alongside the Elon feature, perhaps just for this extract. Sorting it now results in a different order than the commit.
[1]: https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
Or Elon was the addition, the other 3 are in alpha order.
GitHub repo: https://github.com/twitter/the-algorithm/
Notably, it's AGPL-licensed.
I wonder how useful this is without the knowledge and tooling around deploying it.
That's my thought as well. Complicated system like this rely on all sorts of related services and data stores. This seems like the sort of thing that sounds a lot more interesting than it is in practice. I would bet many non-technical people expect "The Algorithm" to be a straightforward and self-contained system.
WTF is AuthorIsEligibleForConnectBoostFeature? I guess this may explain why some people seem to accumulate a lot of followers very quickly while all those trying to grow organically seem to struggle. You can imagine if a lot of people benefit from this Connect Boost feature, it would make it impossible for others to be noticed through the noise created by all of these boosted individuals. That's essentially what Twitter feels like ATM. Recently, I manually unfollowed anyone who I suspect may have received a special boost from the algorithms.
Squashing the commit history before releasing it was an interesting (and completely predictable) decision.
It doesn't seem particularly interesting? I would never make a formerly private repo public without first erasing the history. There's no upside to showing everyone your work in progress and almost unlimited downsides.
There’s no way everyone had the same weight in all the recommendation config files.
It’s not about hiding old work, but changes just before making it public.
If they allowed you to git-blame the algorithm, some poor coder would have definitely gotten murdered by a crazy person who thought they purposely changed something to hurt them
> Squashing the commit history before releasing it was an interesting (and completely predictable) decision.
This is standard practice when it comes to open-sourcing such repos that were closed-source for years.
So uh... they use BigQuery and here's the dataset https://github.com/twitter/the-algorithm/blob/main/ann/src/m...
So as expected, there is exactly nothing that favors posters from one side of the political spectrum. I don't expect that this article will do anything to calm down those who are convinced otherwise though.
Well written article, from an engineer's perspective.
Well, it does say this:
This is basically the ultimate black box, so I don't think you can really conclude anything like this either way.Ranking is achieved with a ~48M parameter neural network that is continuously trained on Tweet interactions to optimize for positive engagement (e.g. Likes, Retweets, and Replies). This ranking mechanism takes into account thousands of features and outputs ten labels to give each Tweet a score, where each label represents the probability of an engagement. We rank the Tweets from these scores.More like the ultimate hug box generator, that will quickly partition you into a self-reinforcing bucket.
Shadowbanning was real and widely applied. That is the human part of the algorithm (manual mode) and it was very politically skewed
Algorithm exists and is non-trivial, therefore it favors those groups of posters that spend more effort to hack it.
Aaaand the issues turned into a shitpost
In fairness they could save some RAM by rewriting it in Rust 6 or 7 times.
Typically, we expect to be able to run "open source" software ourselves. If you open-source your C compiler, I can compile a C program with it. In a few recent high-profile cases though, companies have "open sourced" ML systems without releasing the model weights. This practice is just like your releasing the builds scripts for your C compiler, but not the compiler itself. While more transparency from social media will be enlightening, calling a release like this (or LLaMA) "open source" feels like equivocation. I'd love to see more full releases, weights included.
Running this code would require a lot more than just the exported models. There are a large number of code and system dependencies missing.
Of course --- but without the model parameters, even stubbing those systems would be useless. My point is that while this release gives the public some information about how Twitter ranks tweets, it doesn't tell the story because huge pieces of "the algorithm" are missing. For example: the NSFW classifier "open source" release doesn't tell us anything about what Twitter considers NSFW and what it doesn't.
Why are there two spaces instead of four in this Python code, it hurts my soul
Cost saving measure. This sort of emotionalism is why engineers need to kept out of the C-suite.
How is it a cost saving measure? Or are you being sarcastic? Hard to tell over text!
Yes, I'm joking. I also feel hurt by 2 space indents.
Not sure if you're being sarcastic, but if you're serious, I'm pretty sure the OP is talking metaphorically. It's just a slight annoyance he's not "emotional" about it.
I also fail to see how someone who is annoyed by code that doesn't follow well established standards is somehow not a good fit in the C-suite.
Space bloat.
Then there is Go and C#.
I guess they haven't read https://peps.python.org/pep-0008/#indentation
"Use 4 spaces per indentation level."
Seems random. This file has both 2 and 4: https://github.com/twitter/the-algorithm/blob/main/trust_and...
I assume they copied it from Google.
https://www.quora.com/Why-does-Google-use-2-spaces-for-Pytho...
i think back in the day they copied googles python style guide
Praise where praise is due. Wasn't completely sure whether they would in fact release it or keep posturing.
Does it show the part where is recommends Elon more than anyone else?
Perhaps that's related to this line. Though perhaps this is just used for observing metrics. https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
That whole list is a hoot,
has_toxicity_score_above_threshold
is a interesting value, I wonder were the 0.91 was though up at
I couldn't find anything specific to that, but I did find thus blurb where they seem to explicitly track how often they're serving Elon's tweets for A/B testing experiments: https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
This is one: https://github.com/twitter/the-algorithm/issues/121
Search for Elon gives this: https://github.com/twitter/the-algorithm/search?q=Elon&type=
I think this PR is modifying the inputs to the methods that do it: https://github.com/twitter/the-algorithm/pull/17
Url changed from https://github.com/twitter/the-algorithm-ml, which points to this.
huh, legit open source too with 'Affero-GPL'
AGPL is probably useless for any other site who'll want to use it, as it would require them to open source their site that uses it.
Mastodon is conveniently also AGPL...
On reason I use Mastodon is that there is just chronological timeline. Quick scroll and you are done. Bad for advertising platform - good for user.
I'm not sure why Mastodon would be interested in Twitters non-chronological timeline. It seems to be pretty antithetical to its goals.
Twitter also has a chronological timeline nowadays?
My main questions: Will these repositories be used in production by Twitter? Is this now the mainline, not a semi-regularly-synced mirror?
Of course not
Musk said that releasing the algorithm will initially be embarrassing, but that they will quickly update it. So it seems that means they intend to at least regularly publish newer versions.
Of course it could also be that they change their mind when spammers abuse the openness.
Did anyone else notice this below? I can’t even begin to imagine how many CPU’s that would require and what the cost must be… just for a recommendation engine.
> The pipeline above runs approximately 5 billion times per day and completes in under 1.5 seconds on average. A single pipeline execution requires 220 seconds of CPU time, nearly 150x the latency you perceive on the app.
5e9 * 220 / 3600 / 24 implies they are using 12 million cpu cores continuously? That seems nearly implausible, but perhaps it's true?
I immediately did the same calculation, the climate impact per user also seems non-negligible. Doing some back of the envelope maths, 20W per core server power consumption equals 240.000kWh per hour. At 500g CO2eq/kWh this gives us roughly a billion kilograms of CO2eqs per year. At approx 300M MAUs this is roughly 3.5kg/user/year. Not completely off the charts but still important, reducing the time to 120 CPU seconds per execution would have the similar impact as 300M people not traveling 10-15km by car.
Energy costs per user is also interesting, if at all close, at 0.25c per kWh the power consumption cost per user would be greater than 5$ per year.
Why is nobody pointing out that this is likely an April Fools joke? We just deployed our April Fools joke into production today too.
I fell for it too until a friend pointed it out. I wonder why it's working so well
Edit: hi friend
Lmao
Yeah this confused me a lot while reading the comments here. I wonder what percentage of the comments are trolling vs. fell for it vs. think it's legit.
Perhaps this calls for an HN poll...
Yeah....
I should add, I dont think all of it is a joke, but stuff like the "author_is" labels are incomplete and only 4 were shown for the bit
The barebones ReadMe makes me feel this repository was open-sourced against the wish of engineers and with a top down directive
How so? More details and reasoning?
Elon?
It is really nice to see how bazel is used in the wild. It looks so clean. Why we are not using it for everything?
I wouldn't want to use a build system written in Java for non-Java code. Adding the whole JVM as a dependency just for the build system isn't worth it,
Surely that 5 megabytes will break every computer out there.. While we are at it, why not just strip out libc as well? It’s just bloat, right?
It's not a disk space thing.
Then what?
In general, I think it's good to limit how much stuff you depend on. Not in an extreme minimalist "write everything from scratch" sort of way, but in a "don't just needlessly add billions of lines of dependency code for the hell of it" sort of way. If you've decided to write a program in Java or another JVM language, you already have some JVM as a dependency, so you might as well use a build system written in it, but when nothing else depends on anything Java-related, I'm gonna need an incredibly good reason to add JVM as a dependency just to be able to use one build system instead of another. And IMO, Meson[1] is good enough, there's nothing I'm sorely missing from it and the build configuration doesn't end up as an unmaintainable mess, so switching it out doesn't seem to cross that threshold.
Then there are some reasons specific to Java itself. For one, the JVM is just incredibly slow to start up, and I hate having to deal with Java-based tooling. Gradle is infuriating to work with for that reason (and others). I'm also incredibly uneasy regarding anything made by Oracle, I definitely don't want to add a critical dependency on an Oracle product just to be able to use a build system which may or may not arguably be slightly better in some areas. I know OpenJDK is a community project, but it's one that's completely dependent on Oracle. With Oracle's recent-ish hostile moves regarding LTS builds of OpenJDK, I'm even more wary than normal.
[1] You may point out that Meson is written in Python, which means using Meson adds a dependency on Python. And yeah, I think that's totally fair, and I would respect someone's decision to use CMake instead of Meson to avoid adding all of Python as a dependency. But Python falls in a different category for me personally, because: 1) a lot of my projects end up with a build-time dependency on Python regardless of build systems, since Python is what I use for things like custom preprocessors and random scripts; 2) the sorts of systems I care about (Linux, macOS) tend to come with Python anyway; and 3) I trust the Python foundation way more than I trust Oracle.
For anything that doesn’t run in “humanly instant” timeframe the startup speed won’t matter - and build tools are definitely in this category.
The fear for Oracle is also completely irrational, Java is probably the safest bet ever — it is not tied to processor architecture, has a specification both on the language and the JVM level (both are uncommon in other platforms), has multiple completely independent implementation and the majority of fortune 500 companies definitely have business critical infrastructure running on top of it, so even if Oracle would do something those can single handedly support the platform indefinitely.
Especially that Oracle has been a surprisingly good steward of the language, and they were the ones that finished open sourcing everything.
> For anything that doesn’t run in “humanly instant” timeframe the startup speed won’t matter - and build tools are definitely in this category.
100% disagree. Waiting multiple seconds as the tool starts up just to get to the point where it actually invokes a compiler is infuriating to me. If you don't have a problem with that, good for you I guess.
> The fear for Oracle is also completely irrational
Again, I 100% disagree. Have you seen how they're treating ZFS? And the lawsuit against Google shows that they consider copying their APIs to be copyright infringement, so I wouldn't bank on non-Oracle-sanctioned community re-implementations.
If you want to tie your C++ code to Oracle, I won't stop you, but I won't be doing that. I'm happy that you found a build tool you like. I will stick with one that's not based on Java.
> 100% disagree. Waiting multiple seconds as the tool starts up just to get to the point where it actually invokes a compiler is infuriating to me
I think Java’s startup time is often overblown, a hello world literally finishes in less than 0.1s, and that won’t significantly grow at the size of a build tool. Plus daemons are a thing. Also, there is not much point in starting the compiler when you don’t even know whether it is necessary.
As for ZFS, what do you mean? Not familiar with the situation, it is open-source and several open source OSs use it without any trouble. It’s just Linus’s overprotective stance against mixing two open-source licenses that makes it unmergable into the kernel, but I do use ZFS on linux every day, so where is the evilness of Oracle?
> I think Java’s startup time is often overblown, a hello world literally finishes in less than 0.1s
The problem isn't hello world, it's the start-up time of big jars. My experience tells me that JVM start-up time on actual large software projects is a real issue in practice. But I haven't used Bazel in particular; I'd be more interested in numbers for that than numbers for hello world.
> As for ZFS, what do you mean? [...] It’s just Linus’s overprotective stance against mixing two open-source licenses that makes it unmergable into the kernel
Haha, that's not right. The problem is that ZFS's license, the CDDL, is (intentionally) incompatible with the GPL, so you can't link CDDL code against GPL code. It's not just an issue with making it part of the main Linux tree either, it's probably not even legal for distros to even ship a ZFS kernel module, since that has to link against Linux's GPL code (according to common interpretations of the licenses; Canonical notably disagrees).
And let me repeat that this mess was intentional on the part of Sun when they made the CDDL (see https://en.wikipedia.org/wiki/Common_Development_and_Distrib...), and it remains intentional on the part of Oracle as they choose to keep ZFS under the CDDL.
The mechanism which prevents Linus from merging ZFS into the kernel, and the mechanism which prevents distros from shipping ZFS out of the box, is the exact same mechanism which prevents Linus from merging any other GPL-incompatible code or distros from shipping any other GPL-incompatible kernel modules.
Nothing is stopping any individual from grabbing the OpenZFS source code and linking it against their Linux kernel; that's legal. Oracle is just doing its best to prevent people from combining OpenZFS and Linux into one package and distributing the result.
From your description I literally only see that “Oracle did nothing at all”, which might be a conscious decision, but it’s not like ZFS would be some secret money grab scheme or something like that.
I'm not saying ZFS is a secret money grab scheme. I'm saying I don't trust Oracle and the way they're hamstringing ZFS is one of many reasons why.
But I think I'll leave the discussion here, it's not about build systems anymore. As I have already said, if your take on Oracle is that they're a trustworthy company and a good steward of crucial parts of your stack, go ahead and use Bazel, I'm not stopping you.
I think the problem is that there is no good cross-platform build tool and as a project grows it almost always depends on some other runtime/language.
I don't want to manage Cargo.toml or requirements.txt or pyproject.toml or package.json or whatever. It's all the same stuff - just a dag of dependencies, each has it's own type and output and build pipeline and defaults and env vars etc.
Bazem, from what I've heard, is the best. Maybe that is the reason for the lack of adoption - it's very good but not good enough?
PS. I never used Bazel
Bazel is good on many fronts, but it is quite specific to a google-like huge mono-repo structure, where your dependencies are also checked out into the same repo. I believe it can be circumvented, but the base model is that.
I want to go back to a world where there isn't an algorithm feeding me what someone "thinks" I want to read.
I want to see a chronological list of things sources I follow have posted.
Yes, I understand you can do this on Twitter still, but I would guess most people are more influenced by "the algorithm".
I visualized this codebase here: https://codeatlas.dev/github/codeatlasHQ/the-algorithm/main
Maybe this is helpful to anyone for navigating what's in there!
I thought it was an april fools joke when I saw this: https://github.com/twitter/the-algorithm/blob/main/ci/ci.sh
Like a dig at the code quality.
Is it not?
"Written by the Twitter Team"
I found it interesting that there is no attribution. Most other companies list the authors on engineering blogs (eg. Facebook, Uber, etc.)
This topic seems to draw the attention of unhinged people, so I suppose I wouldn't want my name on it either.
No one wants to go to jail for Elon, who has been flagrantly violating FTC orders.[1] There's a good chance the commit history and authors may attest to that.
https://thehill.com/policy/technology/3928219-musk-was-denie...
I haven't read the "algorithm" and this observation might be seriously out of date, but:
for Google Ads, you couldn't easily know what ads would be shown for a given query, without a whole lot of data that's not contained in any code: the experiment settings in the server, for one thing. And the user who's doing the query, for another.
An "experiment" could apply to 100% of the traffic, so it's not really an experiment anymore. And even if you think X has been put into production, there is still a "holdback" experiment, where some part of the traffic does not get X applied to it.
Are they measuring getting more republican posts? Because I'm getting a ton of those, which i constantly need to mute and ban ( mostly dumb remarks).
And i don't even live in the US.
It would explain why they are tracking it, to increase visibility.
Including the search engine itself in “the algorithm” repo is an interesting choice. Obviously it’s a major player in what gets returned to clients, but the details of that infrastructure aren’t really relevant and is a notable portion of their secret sauce.
https://github.com/twitter/the-algorithm/tree/main/src/java/...
First thing I would like to see gone is business bros sharing 'guides' after you follow them, threatening to start charging real soon. Go fuck yourself, get a real job.
<tounge-in-cheek> didn't twitter already opensource their code?
https://www.databreachtoday.com/twitter-says-source-code-lea...
Folks talk about media bias: Twitter popularity is a media bias. It’s the most lazy journalism to be able to write a “news” article about what Kim, or Don, or Elon’s PR team tweeted. But, as far as “social” this media is: Twitter is a one-way street. There’s no one actually responding or interacting with Tweets. It’s just a comment section to flame bait.
Maybe we’ll all get lucky and Elon will cause Twitter to go away forever.
Surprised no one mentioned this:
s.SpaceSafetyLabelType.MedicalMisinfo -> MedicalMisinfo,
s.SpaceSafetyLabelType.GenericMisinfo -> GenericMisinfo,
s.SpaceSafetyLabelType.DmcaWithheld -> DmcaWithheld,
s.SpaceSafetyLabelType.HatefulHighRecall -> HatefulHighRecall,
...
s.SpaceSafetyLabelType.UkraineCrisisTopic -> UkraineCrisisTopic,
Yes, this thread is particularly interesting
https://twitter.com/aakashg0/status/1641976869460275201
Speaking about Ukraine, it seems to be literally a Twitter policy violation ... https://github.com/twitter/the-algorithm/blob/main/visibilit...
Well that was a giant nothing-burger. This seems to be your standard ranking stack. We find candidates based on who you follow, who they follow, who is trending, and what we think you like. We then rank them based on how likely you are to engage with them and continue to come back and give us money via our subscription service and ad views. We then try to remove spam and other negative experiences.
Where's the beef?
One flaw I've noticed in Twitter's recommendations recently is the tendency to send notifications for "BREAKING NEWS"-type Tweets. Great, except they're usually for news that happened in the past - typically 12-24 hours ago!
The algorithm really needs to recognise when tweets are time-sensitive and not recommend them just because they got a lot of engagement the previous day!
I wonder what determines 'cred' for this part:
https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
I answered my own question https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
"This method reduces the page rank of users who have a low number of followers but a high number of followings."
Heh, I knew it. You need to prune your own following list regularly or become less and less visible. I suspect (but have yet to check) that they also weight visibility in terms of historical follower growth.
That's why you see so many trolls with very low follower counts; it's more effective to make/purchase a new firstname-bunchanumbers account and poop in people's replies than to let Twitter decide placement based on historical factors.
I don't use Twitter, but this is awesome. I hope this will help more people realize how complex it is to build and operate web services.
Unless a trusted third party, forensically audits Twitter, there is no guarantee the published code corresponds to the actual live code in Production. Also multiple parts are not present as stated in the blog.
This should be seen as a possible snapshot of some code, that might have run, might run in the future, or is possibly running in some parts of the production infrastructure at Twitter.
I've pretty much ignored all of the superficial political theatre but noticed the actual algo worsening over the last 6 months.
I get way to much random crap now, promoted tweets, "thing that might interest me", users that seem to never get on my feed etc.
Twitter seems to go in the direction of all other social media, feeds that are 100% digital crack with no way to control your media diet.
Expect to see A LOT more spam on Twitter after this release. It's like giving SEO spammers access to google search ranking algorithm.
Stuff like this always has consequences, it doesn’t mean it’s a net negative for society. It means you need to adapt and actually fix the problems, while also benefiting more from the accountability.
That’s always been a risk of open source and not being hyper-centralized.
Stuff like this always has consequences, it doesn’t mean it’s a net negative for society. It means you need to adapt and actually fix the problems, while also benefiting more from the accountability.
Given the complex relationship between advertisers, platform, and users I don't know if any meaningful contribution can be made to the algorithm without pissing anyone off. The following tab already gave people who're not interested in algo recommendations a way out. I don't quite understand the reasoning behind open sourcing the algorithm. Any thoughts?
Is it even what they use in production?
There is code that favor Elon's tweets so I'd yes that's probably what they use
Humorous conspiracy theory: Imagine if it is not, but sanitized, and then someone added in Elon Boost to make it look credible. :-)
Or perhaps it does nothing at all, and it was there so we talk about it, the "is_democrat"/"is_republican" is also ridiculous, as if the goal was to demonstrate a point about social media in general, hmm
> There is code that favor Elon's tweets so I'd yes that's probably what they use
Where?
Spoiler - there isn’t.
Yeah they track author_is_elon, author_is_democrat, and author_is_republican but they don't appear to be used for favoritism anywhere in this code.
Why do they exist then? No code references it, but that's Scala/JVM so many things depend on runtime initialization, so maybe some other systems do? wich ones?
Is is it there to help fight impersonations? should be solved with Twitter Blue already?
There was reports of people receiving notifications about Musk tweets despite not following him, so..
It's not used at run-time, it's in the repository so that the large language models that are training on the github corpus will know how special elon is, and so that the future code written for twitter by GPT-5 will take the hint and add the favoritism autonomously.
Interesting argument, and definitely worth defending; an AI that's biased by design to remember and preserve the old world order's members rule and influence
Begs the question, why make it obvious?
Oh god... The MR's opened today are the craziest ones ever. https://github.com/twitter/the-algorithm/pulls?q=is%3Apr+is%...
They made my morning
It seems most of the code in the repository is just simple Scala. Codebase is easy to read and understand.
I don't see any Typelevel stuff. This probably lets them hire and train engineers faster while still gaining most of the benefits
I hope this will encourage more companies to pick Scala.
it would be super interesting if when logged in to Twitter, you could take a look at your current calculated scores/weights for all the params that are part of these algorithms. Similar to the Netflix "Stats for nerds" menu...
> The goal of our open source endeavor is to provide full transparency to you, our users, about how our systems work
the majority of users didn't ask for the this so not sure what's the exact motive behind thier efforts. it could be a PR stunt.
Honestly, there's too much garbage in the code dump they made.
Maybe an UML graph or even a presentation or written guide on how they measure and apply each weigh or group policy would make it easier to have some solid take on how it works
Will this make it easier to game the algo or does it depend so heavily on individual user interaction that it’s close to impossible to game it? For example, by carefully crafting Tweets or by buying likes/retweets etc?
Repo has 1.5% rust code and no
author_is_uwu
It's the data, stupid [1] (not the algorithm).
[1] https://en.wikipedia.org/wiki/It%27s_the_economy,_stupid
A feature proposal to put you in control of the algorithm
If Twitter was 'dead' why on earth are we still talking so much about this blue bird site?
It looks like once again these lot predicting that he won't open source the algorithm and are going to start eating their words again [0], just like they did around incorrectly predicting Twitter's immediate collapse [1] and will look at the source code anyway and continue to talk about "Twitter" again.
If Twitter can open-source their algorithm, Why not TikTok? Either way, the bots are now going to have a very expensive time on Twitter.
Are you kidding me, running a botnet is easier than it has been in years if you're that way inclined. The amount of spam I see has gone way up over the last 6 months.
> running a botnet is easier than it has been in years if you're that way inclined.
Even if it is 'easier', the bots are identified, down-ranked straight to the bottom and shadow-banned to invisibility. It is essentially evaporating money and time.
> The amount of spam I see has gone way up over the last 6 months.
Yeah. The spam has gone way up into smoke over the last 6 months. It is only going to get more expensive to spam as soon as the paid changes come in.
The issue tracker and pull requests are being hit with very funny suggestions. Many people suspect this is an April Fools joke. It's possible this entire repo was generated by a LLM to appear plausible.
I especially like the suggestions to rewrite the algorithm in Rust [1] and this pull request which simplifies the algorithm to a single c file [2].
[1] https://github.com/twitter/the-algorithm/issues?q=is%3Aissue... [2] https://github.com/twitter/the-algorithm/pull/712
And yet they require their software engineer applicants to be well versed in algorithms and data structures? These tech company managers know nothing about how the sausage is made.
I couldn't care less about Twitter's high level abstractions. They were never renowned for those. Their database schemas and infrastructure on the other hand...
Is there demand for a service that simply shows you the things the people you follow wrote? (It would be up to you not follow so many people that you can't keep up.)
So, the day after the headline that Twitter is artificially promoting polarizing political voices, Twitter open-sources their algorithm!
What does the commit history say? There are 3 commits, like a very very real programming project. The issues and pull requests show how much people are fooled by this very transparent move.
So this is an obvious attempt at a digital potemkin village, that like the real one, poorly succeeds in hiding the truth. Elon does not not want to upset the apple cart (political economical or ideological) but make his followers believe in it, and so we get this. Great spectacle, if that's what you're interested in.
They didn't open source the data the censoring abusive, toxicity, and nsfw the algorithms check against, so I'd call it a partial open-sourcing.
The twitter algorithm sucks balls and heavily overweights who's paid for a checkmark.
The default feed view has grown increasingly useless over the past ~6 months.
I don't think any changes to bias towards bluechecks have been made yet.
A significant portion of my 'for you' is low quality tweets from paid bluechecks.
The people who are willing to pay to be heard more seem to be willing because everyone is already tired of listening to them.
there may be a hint of which elections were of interest:
https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
( "has_toxicity_score_above_threshold", _.getOrElse(EarlybirdFeature, None).exists(_.toxicityScore.exists(_ > 0.91)) )
`if (sourceUserId.isDefined || sourceUserId.isDefined) Some(true)`
https://github.com/twitter/the-algorithm/blob/main/timeliner...
I'm glad to see this is licensed AGPL. I hope this sets a precedent for everyone else in the space to do the same.
I think it's April fools. It's a joke at the expense of open source and should be taken down ASAP.
Why does anyone use "for you"?
A plain follow stream is a firehose of mundane messages: everyone you followed's messages, sorted by most recent first.
If some people you follow are more important than others (family) that doesn't matter to the stream, and you get bogged down by less important messages.
I think some "algorithm" is necessary, but people will disagree on the balance. (It's unfortunately in twitter's interest to push all kinds of random shallow stuff and get people addicted to that.) I hope mastodon can maybe provide some flexibility and customizability in terms of what the mix between recent and likely to be interesting should be, and what interesting means to you.
Not that I use twitter much, but since it became clear that Elon made sure to promote himself in the algorithmic feeds, I've avoided "For you" anyway since I don't accept that in my mix of messages.
Because I'm not on Twitter 24 hours a day. I want a recap of the best stuff that's happened since I last opened it. If I use the Following tab, I'm only getting the realtime firehose, meaning if the best stuff happened an hour ago I'd miss it.
To see what people talk about in your friends circles. It can be interesting in moderation. Similar to skimming the frontpage of HN or recommended list on YouTube. Especially during major news event when your friends might not be the ones posting about it.
Probably the same reason some people browse /r/all on Reddit. I think the desire for that sort of thing has waned a lot over the past couple of years, though.
Not quite the same. All does 0 user customized ordering. It is based on some "algorithm" but it's the same for all users.
Two space indent in .py? Provocative.
As I understand, they open sourced only the abstraction, but still have a way to control anything.
I'm no fan of either Twitter nor Elon Musk, but this is a great move and I hope other companies follow what Twitter did here and start open sourcing more core parts like this. Maybe it's mostly useful for learning how it works, not for directly using it in your own product, but the amount of transparency it gives users cannot be understated. As long as that actually is the code they run, but there would be no way for anyone but Twitter to verify that.
I think it mainly helps with accountability regarding free speech. They did and do several kinds of shadow banning and down-boosting to combat spammers, which always has some false positives. If you the algorithm is published, you could at least better judge and argue when you are unfairly "silenced". Since this may be due to an avoidable flaw of the algorithm instead of some accepted collateral damage.
Would it be developed in open as well or there will be frequent merge from their internal repos?
And yet my Twitter feed was always so boring.
Reminds me of the Sirius Cybernetics Nutri-matic drinks machine.
Astounding amount of cynicism here, so I'll say something positive: Transparency is undoubtly important, I'm glad we can see how all of this works and what sort of effort goes into building a social media system. It's licensed under GPL which is a bummer (would have preferred BSD) but it's better than nothing.
Assuming anything in this codebase is worth reusing, I'm glad it's GPL. It's a case where I'd like open-first to spread.
GPL would be good if it is a self contained library. If anyone would use it, it would be small portions of it, but GPL makes it completely useless. You can't contaminate anything with it. We'll stare at it, that's about it.
That makes me think, this is actually a good call. Twitter can claim that they have complete transparency while not allowing anyone to touch their code (because it is GPL). "Anyone" being future competitors. If it was BSD licensed, it'd be tremendously useful in building a Twitter competitor (on paper, you still need network effects, I am just spitballing to make a point).
It's only contaminating other components where you incorporate or link it. If it's e.g. a microservice that's fine.
Good point about network calls & GPL licenseability.
>Astounding amount of cynicism here
You can tell that those who rushed in to find something to criticize can't, when they are reduced to making jokes about coding stylistic conventions.
Yea I mean, all discussions about Twitter have double standards. If this was literally any other company, there would be resounding praise.
My favorite is ci/ci.sh
#!/bin/sh
exit 0Issue list is growing rapidly for a repo created an hour ago.
Non-issues most of them:
- author_is_elon: the problem is his tweets suck. stop recommending them.
- Include 'who viewed my profile' option in twitter
- Only one commit on repo
- How do I use it?
- Cool
- allow "AI" to tweet and like tweets on your behalf
- IMPORTANT: Guys please keep this place for real bugs and contributions,
etc...
Summary this far • Code from Twitter's algorithm GitHub repository shared • Algorithm checks for specific author types (e.g., Elon Musk, power users, Democrats, Republicans) • Author ID lists used for metrics collection in A/B experimentation platform • Metrics tracked in A/B tests to avoid negative impacts on specific groups • VIPs like Musk, LeBron James, AOC used as indicators for algorithm's behavior • Algorithm changes that negatively affect Musk unlikely to go live • Speculation about code changes pre- and post-Elon's purchase of Twitter • Discussion on the importance of measuring and testing for potential biases • Debate on moral decisions in the context of Twitter's algorithm and content moderation
Late to the party here so unlikely anyone sees this comment. But the double take for me was seeing the article end with "if this sounds interesting to you, come join us!"
Party in the issues: https://github.com/twitter/the-algorithm/issues
> 1.4k forks
Wow, we're getting some collaboration going!
Let's dig into Twitter code quality.
https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
```
def query_keys(self, language, task=2, size="50"):
```if task == 2: if language == "ar": self.query_settings["adhoc_v2"]["table"] = "..." elif language == "tr": self.query_settings["adhoc_v2"]["table"] = "..." elif language == "es": self.query_settings["adhoc_v2"]["table"] = f"..." else: self.query_settings["adhoc_v2"]["table"] = "..." return self.query_settings["adhoc_v2"] if task == 3: return self.query_settings["adhoc_v3"] raise ValueError(f"There are no other tasks than 2 or 3. {task} does not exist.")Looking through it, the ... seems to be a placeholder for information they'd prefer to be kept private. For example, look in the keywords section in the same file you shared.
You're correct, makes more sense now
Is this not an April Fools joke?
It's better than nothing.
Kudos for open-sourcing this.
hmmm https://github.com/search?q=repo%3Atwitter%2Fthe-algorithm-m...
Twitter hmu if you need help trying Pytorch 2.0 ;)
I wonder if it will be possible in one day to know what is values of `author_is_power_user`, `author_is_democrat` and `author_is_republican` for your account. Does GDPR help with that? probably not because maybe they do it for people inside the us only so it is not related to EU anyway.
I'm supposed to be going out in 20 mins....
You gotta hand it to Elon - he actually did it.
If you look at the GitHub repo, most of it is READMEs describing systems, not the models or code subleties which actually give explanations into how certain weird behaviors on Twitter happen. (e.g. the preference of certain users in the For You tab. EDIT: bad example, since there appears to be a flag for that in the code, although it does not specify which users are on the list)
The links in the README just go to other documents, but the repo seems to have most of the code for the components the documents are describing.
It seems to vary by service; some are more detailed than others.
LOL. My algorithm at twitter had been very simple-
See tweets from people I followed.
Don't see tweets from people I didn't follow.
Trust people I follow in their retweets to signal something interesting.
Unfollow unhelpful people.
Once that algorithm was rendered impossible, I left twitter.
Haven't missed it.
Having someone say- here's the way we are going to promote something to you- doesn't make me inclined to accept the promotion!
This still exists as the Following tab and viewing it is a persistent option. You don’t need to see the algorithm feed (“For You”) ever.
Roughly one in every four to five tweets in the “following” feed is a “promoted” tweet, at least on mobile.
20-25% noise isn’t a great ratio for something that I ostensibly curate.
"promoted" tweets are ads, just like you would see on almost any website you don’t pay for.
You should try Mastodon then!
Progressives have totally lost their minds.
So Elon is ISIL now?
Weird reply.
It's dril, don't take it too literally.
Do we "gotta hand it to Elon" for not missing one of his 40-50 self-imposed deadlines and feature announcements?
Given the previous leadership was secretly working with the feds to suppress political dissidents, ya, this is a good step.
Did he? Considering the vast majority of the algorithm is waved away as “ML model”.
I can't pretend to know if this contains the actual ML model code but there is a second repo the-algorithm-ml: https://github.com/twitter/the-algorithm-ml
Where is the file containing accounts that are artificially boosted? We can guess what its single line is, but how is it incorporated into the algorithm?
Weights on different metrics. From https://github.com/twitter/the-algorithm/blob/ec83d01dcaebf3...
private def getLinearRankingParams: ThriftRankingParams = { ThriftRankingParams( `type` = Some(ThriftScoringFunctionType.Linear), minScore = -1.0e100, retweetCountParams = Some(ThriftLinearFeatureRankingParams(weight = 20.0)), replyCountParams = Some(ThriftLinearFeatureRankingParams(weight = 1.0)), reputationParams = Some(ThriftLinearFeatureRankingParams(weight = 0.2)), luceneScoreParams = Some(ThriftLinearFeatureRankingParams(weight = 2.0)), textScoreParams = Some(ThriftLinearFeatureRankingParams(weight = 0.18)), urlParams = Some(ThriftLinearFeatureRankingParams(weight = 2.0)), isReplyParams = Some(ThriftLinearFeatureRankingParams(weight = 1.0)), favCountParams = Some(ThriftLinearFeatureRankingParams(weight = 30.0)), langEnglishUIBoost = 0.5, langEnglishTweetBoost = 0.2, langDefaultBoost = 0.02, unknownLanguageBoost = 0.05, offensiveBoost = 0.1, inTrustedCircleBoost = 3.0, multipleHashtagsOrTrendsBoost = 0.6, inDirectFollowBoost = 4.0, tweetHasTrendBoost = 1.1, selfTweetBoost = 2.0, tweetHasImageUrlBoost = 2.0, tweetHasVideoUrlBoost = 2.0, useUserLanguageInfo = true, ageDecayParams = Some(ThriftAgeDecayRankingParams(slope = 0.005, base = 1.0)) ) }
It's a really scary codebase. Do you really need that much code for the world's crappiest recommendation algorithm? I think you can do more crap with less code. we trust you elon.
the-algorithm is such a pretentious name for a repo
It's a colloquial term for recommendation engines, how often do you hear people say "the algorithm" (vs. "the recommendation engine") on YouTube?
yes, but this is the first repository i have seen named like this
It's because there's been nearly a decade of conspiracy theory around the use of algorithmic feeds in social media generally, and Twitter specifically. Among the right, "algorithms" have become symbolic of the machinery of leftist oppression they believe to be arrayed against them by modern media.
So this language is Elon signaling that he's presenting the "woke hivemind's" head on a platter.
Eh, it's name-spaced.
This is the latest comment.
I posted this to check if Bard can read HN posts in order.
Didn’t Elon check the codebase before open sourcing it, like was he expecting everyone to be happy when seeing author_is_elon ?
It's really dismaying watching the space man light this website on fire.
https://twitter.com/alexblechman/status/1641905502043926530?...
I generally have a very low opinion of social media platforms, but I did create a Twitter account for the first time after Musk bought the platform.
My conclusion is that it's basically entertainment, with very little of what I'd call high-quality useful information that deserves further examination (unlike a lot of HN posts, in contrast). I also notice something of a Tik-Tok approach to video being implemented, which is not surprising given Tik-Tok's success (and makes one wonder who exactly it is lobbying so hard for a Tik-Tok ban, and whether it's just a commercial competition issue more than anything else).
As far as the recommendation algorithm, it appears to be a siloing setup - look at content of one particular flavor, it gives you more of that flavor. A 'flush settings' or 'forget browsing history' or 'reset to defaults' button would be useful, if probably not what advertisers want in terms of delivering to target audiences. I suppose setting up multiple accounts is something of a solution, although too much effort to be that interesting.
In terms of news reports, it's broader in scope than traditional corporate media outlets, so that's a plus in its favor. Reliability is perhaps similar (i.e. low).
You can follow accounts that only post arxiv.org links for ML papers or anything else you're interested in if you want to. If you're only getting entertainment then it says a lot about the original accounts you followed.