How to Build an Exchange (2017)
janestreet.comI've always been curious whether someone will succeed in using the same development strategy as Jane Street, namely using a non-C++ language as their primary language. I suspect OCaml may have been a very good choice back in the early 2000's simply because C++ was a much different language and there weren't many options for other languages. Building up an ecosystem was not a bad plan because most languages did not have much of an ecosystem anyways. Nowadays ecosystems are a lot more developed. But on the flip side, that could mean you could stand to use a language like Rust that has an ecosystem and interoperability with C++. I know Tsuru Capital uses Rust but I'm not familiar with any other firms.
It also could be that Jane Street succeeded regardless or even despite OCaml. At the end of the day OCaml isn't going to ensure that your trading strategies work. It's not going to suddenly create alpha. But I do suspect they've gotten serious hiring and marketing returns by using OCaml. And benefited the OCaml ecosystem as a byproduct, which is a pretty solid side-effect, even if side effects are not very functional :D
I have worked for a big (probably more successful) competitor to Jane Street, that uses in its different offices C#, C++, possibly some Java that I'm unaware of, and for non-production systems a very large mix of everything under the sun (Python, R, etc.)
The idea that the language is part of their success is simply absurd. If anything having to have to develop and maintain an entire ecosystem probably slowed them down more than anything.
It can sometimes help with hiring early on. Pg's "blob" blog post comes to mind.
> whether someone will succeed in using the same development strategy as Jane Street, namely using a non-C++ language as their primary language
It’s gonna happen and it’s gonna be Rust
> you could stand to use a language like Rust that has an ecosystem and interoperability with C++.
Indeed :)
Maybe. Rust as a core language is less productive than ocaml, by a lot. But Rusts standard library is just so much better than Ocamls (including Jane Streets version) - which tends to even it out.
I think there’s probably a hidden coefficient here: I would readily believe that an experienced OCaml developer is more productive than an equivalently experienced Rust developer, but that a junior Rust developer is probably several times more productive than a similarly junior OCaml programmer.
(This is not unique to this pairing of languages.)
I once wrote one mostly in SQL with a Postgres database. It behaved pretty predictably with concurrent usage. I wonder how many trades per second it could do. Postgres itself can handle a lot of the difficult details, but reinventing those wheels for trading in particular can probably get you something faster.
There is a little bit of survivorship bias here: in the 2000's, the two main languages of HFT were C++ and Java. By 2015, all the Java shops (and many of the C++ shops too) had failed. Lots of companies tried to do something other than C++, but most of them chose wrong.
That's not true, I've worked at IMC and they extensively use Java for their production systems. The performance critical parts are a mix of hand crafted assembly, C, C++, FPGAs and a dozen other esoteric languages/techniques no ones ever heard of.
So you're saying they're a C++ shop that uses other languages when appropriate? Like every other C++ shop in trading?
They didn’t fail due to their language choice
The author never said they failed because of language choice.
> By 2015, all the Java shops (and many of the C++ shops too) had failed. Lots of companies tried to do something other than C++, but most of them chose wrong.
It's hard to imagine that language choice wasn't a factor - or at least that the philosophical ideals that led to the language choice weren't a factor.
A shell of Allston (formerly big Java shop) lasted until last year.
I was there until the end. It was definitely fading into obscurity but still had >50% of peak numbers. And we were switching to C++ in the last year or so.
This is factually incorrect, and saying a trading shop failed because of language choice is, um, less than smart.
anyone use haskell?
Fun fact: Jane Street, Citadel and other HFT firms pay exorbitantly. You can see the base salary ($200-300k) posted public here: https://www.janestreet.com/join-jane-street/position/4274288...
Even interns make $120/hr: https://www.levels.fyi/internships/
They do pay extremely well, but are also very selective. The kind of talent they are looking for can earn similar salaries at large tech companies like Google ($400K-500K TC is pretty common at staff+ levels).
They do a lot of hiring out of school though, so they are selective but if you make the cut you'd definitely be earning more at the start of your career. Plus if you perform well you can make well over 500K TC by the time you would have been promoted to staff at FAANG, because of bonuses.
a high performer at citadel or JS can make 1m in 5 years. i don't know many people at google make that much.
Citadel has 2600~ people, Jane street 2000~ while google has 150,000~.
Not many at Google receive that much in their offer letter, but with the stock appreciation over the last decade and stacked refresher grants, I'd be willing to wager there were thousands making >$1M as of Nov 2021 (many fewer since the stock has fallen though).
If Google is offering an initial total cash equivalents of $300k (say $140k salary, $480k/4 stock, 15% annual bonus on each), that's roughly the same deal as somebody else getting a base salary of $300k. The fact that a particular investment decision (GOOG) can accidentally push the individual's yearly increase in net worth past $1M isn't a good way to judge what Google is actually offering:
Anyone with a base salary of $300k can obtain a similar payoff structure by taking out a $550k loan to invest in GOOG, and taking out additional smaller loans at each stock refresh. The tax benefits from capital gains/losses and time-value-of-money benefits from getting your bonus sooner should handily outweigh the cost of servicing the loan, but even in a worst-case scenario where you pay interest for no benefit, that example Google offer would be a lot more comparable to a $315k-$330k salary, not a $1M+ salary.
Well first of all, my point was that over the last decade Google has likely paid more individuals 7 figures than JS or Citadel has. Obviously this is subject to the risk of stock performance, and if your point is that cash is better than stock, I agree with you 100% (especially today when tech stocks are still arguably over-valued).
On the other hand, this, my friend, is absolute nonsense:
> Anyone with a base salary of $300k can obtain a similar payoff structure by taking out a $550k loan to invest in GOOG, and taking out additional smaller loans at each stock refresh.
This is only equivalent if you ignore downside risk, which in the case of an average young professional with no significant assets could ruin you. The RSUs give you significant upside over 4 years with absolutely zero risk.
Also you said this:
> The fact that a particular investment decision (GOOG) can accidentally push the individual's yearly increase in net worth past $1M
This makes me think you might not understand how RSUs work. They are W-2 income at the valuation at the time of vest. What we're talking about is 7 figure annual income. Not investment gains over time.
> Well first of all, my point was that over the last decade Google has likely paid more individuals 7 figures than JS or Citadel has
No, they granted stock initially and set aside those shares for the employee. The market paid the employees the gain between the initial grant price and the sell.
> This is only equivalent if you ignore downside risk, which in the case of an average young professional with no significant assets could ruin you. The RSUs give you significant upside over 4 years with absolutely zero risk.
You didn’t understand the example. The person taking the loan gets $300k/year cash and the Googler gets $180k/year. Setting aside $120k/year for the loan makes the risk the same so you won’t be “ruined”. Google failing in either scenario means they each have $180k in annual cash leftover.
No, I understood the example perfectly, it's you who doesn't seem to understand the downside risk. You would need a $480k loan to buy the four years of stock because you are buying it before you've done the work to earn the $480k. A loan is something you have to pay back, so if the stock tanks you have to make up the difference. In order to be equivalent, the terms of the loan would have to be pegged to the stock price on the downside (so if stock price was cut in half you would only owe $240k), but not on the upside (so you get all the gains). No one in the world will give you a loan with these kinds of terms.
That's only relevant if you're leaving Google before the 4yr period is over and also if you don't know roughly what subset you'll stay for (where you'd just take a smaller loan to represent the first N years of vesting).
Also, the cost of options to completely mitigate the incremental risk beyond that of an ordinary Googler is small (cumulatively a little less than the cumulative cost of interest for the loan). It's a small point that matters if you go out to actually implement the idea, but in the context of comparing Google (X total cash equivalents in their normal structure) to some other company (X salary), the investment opportunities in GOOG are sufficiently comparable that it might be reasonable to upweight Google's TC to 1.1X or so (or downweight it because you're restricted to GOOG itself and don't have more options), but I still think it's unreasonable to call it anything like 3.5X. Those aren't million dollar contracts; they're $X contracts paired with a forced investment that anyone else could choose to make without a huge downside (ignoring the much rarer actual $X contracts).
I'm curious how someone could obtain such a large, unsecured loan of $550k? Even secured against a home with a mortgage cash-out Refi, that's a large sum. You'd have to have built up a lot of equity in your home value.Anyone with a base salary of $300k can obtain a similar payoff structure by taking out a $550k loan to invest in GOOGGoKapital advertises up to $500k personal loans on an unsecured basis, with a higher chance of success if you're partially secured (admittedly, putting $50k down instead of investing it elsewhere is an additional cost to servicing the loan, but even 6-12 months into a tech career you should've had $50k saved up, so that shouldn't be a barrier to entry at least).
Also, ordinary banks might not advertise outrageous personal loans, but when your base salary starts at $300k and has a history of increasing (i.e., you don't _need_ the money and just want it to power a particular total comp over time profile, especially when you keep at least 50% of your total comp in cash rather than leveraged investments), most mainstream banks are more than happy to furnish somebody to personally service your account and make a loan like that happen.
Separately, if you live in parts of the country (US-specific) where salaries like that are common, you probably have a down payment of $200k+ if you have a mortgage and would have little problem grabbing a partially secured loan against your current equity.
Base salary can be re-invested (granted, their hands are somewhat tied when it comes to personal fund management due to the nature of their job)
Whatever point you're making flew over my head. Would you mind elaborating?
>but are also very selective
Yes, we've heard that yadda yadda, Caroline Ellison, SBF, etc...
One difference is Google, etc are all doing massive layoffs and i doubt JS, Citadel, etc will do this.
Where is Google doing layoffs? The latest news I have is that people are very worried about layoffs, but so far they've only started putting people on rebranded PIPs.
The other FAANGs are definitely laying people off, though. I personally think the recession is a self-fulfilling prophecy, but regardless of my take on the fundamentals, it is certainly fulfilling itself and everyone in tech should be pretty worried right now. This is not the year when you're going to increase your salary by jumping to a cool startup as employee #3.
> This is not the year when you're going to increase your salary by jumping to a cool startup as employee #3.
Agreed, you won't get a big salary out of the gate because unproven startups paying huge salaries are dropping like flies as the easy capital dries up. On the other hand, the likelihood of getting in on the ground floor of the next FAANG is increasing as staffing costs decrease and behavioral changes increase during a recession. EV obviously still higher at established top-of-market companies, but when has that ever not been the case?
you are correct, it has not happened yet, but you willing to bet it wont in the next 6 months?
https://www.seattletimes.com/business/google-employees-brace...
https://www.cnbc.com/2022/12/22/google-tells-employees-highe...
Note though that Citadel is reputed to have a turnover rate of 100% per year. Their layoffs take care of themselves.
I spent 9 years there, left a decade ago. Turnover is high, but this is a gross exaggeration. It is stressful, and not everyone can hack it. I was there during and after the '05 layoff when roughly 2/3s of IT was cut. That's still the worst I've heard of from what I hear from friends that are still there.
Ken G definitely does the "Good to Great" getting the right people on the bus thing, which typically means the bottom 5-10% are cut, but even that was slowing before I left.
Thanks for correcting me. Did you get a sense of whether engineers had a higher or lower turnover rate than traders?
The turnover for engineers tends to be lower than for traders at citadel and at a lot of quant firms.
Additionally, the turnover for citadel is not evenly distributed across teams. Certain teams and orgs have a lot more turnover than others. Some teams are made up of people with <=2 YOE at company while others are made up of people with >=15 YOE at company.
This!
My personal background is a "well known hedge fund" and the turnover there was rather high.
Many quit because it wasn't "a good fit".
Smaller companies do layoffs as well. They just aren't as highly publicized as those at tech giants. Severance packages are usually worse/nonexistent as well.
HFT companies also have much higher performance bars and rates of overwork/burnout. I'm willing to bet that people are leaving Jane Street, Citadel etc. (whether voluntarily or not) at much higher rates than large tech companies like Google.
Jane Street does much better at retention than Citadel and probably many other quant firms. The performance bar is definitely higher than the average at Google but it's not like there is no WLB either.
Because they are more selective about fit to begin with I wouldn't be surprised if JS has better yearly retention than most FAANGs. People seem to hop between different big tech cos quite a bit (pre hiring freezes anyway).
I do not see a way to compare big tech companies and Jane Street or Citadel, seeing as how the big tech companies employ many tens of thousands of very qualified people and Jane Street and Citadel are a couple thousand max.
Yeah there are a lot of differences, I wouldn't normally bring up the comparison. My point was just that the turnover is not as bad as the other commenter was implying.
True of Citadel, but Jane Street is supposed to have a solid wlb, and because of the high pay and the few companies willing to pay at that level, people generally stick around there. These kind of finance companies also do better in volatility so they're definitely not laying off right now.
I would assume JS has industry standard levels of notice, non-compete and deferred comp - which will sure help keep people around
Jane Street definitely does not have a non-compete. Citadel does though.
Deferred compensation is probably not relevant to any of the major quant firms, payment is almost entirely cash. If you're high enough up to start receiving stake in a prop firm directly as part of your bonus you are an order of magnitude above the TC cited by the original comment.
Of course, the amount of comp you can get is a big reason people stick around. For that level of earning potential JS is really good on the WLB front. But it's not going to be as good as big tech is (or at least has been), since QT/QR just has different requirements as a field.
When I worked at a small prop shop, the cutoff was $400k TC, after which 50% would go into the fund for N years.
Currently I work at a large bank. My comp is all cash, but many of my colleagues get deferred stock compensation. Not sure what exactly the limit is but it's definitely much less than, say, $1m.
I get the impression that in some jurisdictions, deferred equity compensation kicks in earlier than one would otherwise expect, in order to hit some employment law cut-outs for "highly compensated executives".
I worked over 10 years overseas at a Fortune 500 financial firm where my contract specified that my garden leave period jumped up if I ever accepted stock compensation. I got the feeling that was related to some legal requirements for shedding protections for "highly compensated executives". The garden leave worked out well, as I left right at the end of my paternity leave, so I effectively had almost half a year of paternity leave.
I didn't bother checking if I was getting the maximum legally allowed garden leave period in my jurisdiction, as the contract seemed fair. In any case, in some jurisdictions, I think some deferred equity compensation kicks in earlier than one might expect, in order to legally make more employees highly compensated executives.
I've always lived well within my means at my base salary, but even if I hadn't a good head hunter can often negotiate a signing bonus to partially backfill the lost expected TC , so you're effectively not really going down to your base salary during garden leave.
Might depend on the firm (or times) then, the starting trader TC is already pushing 600K at HRT, JS, etc. and many don't let you invest year 1 even if you wanted to. I'd be shocked if any of the major names made you take comp as a stake in the fund pre-1m today.
Large banks are for sure a different story, though I've heard of other large banks buying people out that they wanted to hire by offering the equivalent package in their stock of what the guy would've made in the other company's stock.
> Deferred compensation is probably not relevant to any of the major quant firms, payment is almost entirely cash.
Deferred != stock.
> If you're high enough up to start receiving stake in a prop firm directly as part of your bonus you are an order of magnitude above the TC cited by the original comment.
Fair
Yes you could theoretically delay cash comp over a period of 3+ years like tech companies do with stock, but I've never heard of that happening.
I suppose by the technical definition the year end bonus cash is deferred comp, but I don't think 1 year is especially hard to plan around. People might stick out an extra few months because of it but it's not locking anyone in for years at a time. Plus even if you subtracted an entire year from the JS retention rates they are still quite good.
Regardless, the amounts cited by the original comment are only the base which isn't deferred by more than 2 weeks at a time, whatever your bonus potential is.
> I've never heard of that happening.
I have heard of it enough that I assume it is industry standard (much like the non compete).
Admittedly I am more familiar with the quant side than pure dev
Interesting, I only have close friends at a handful of places so perhaps those are exceptions. Could be a dev vs trader thing too, I have heard the trader bonuses can get really nuts so maybe a deferred structure starts to make more sense at a point.
When they start letting you put your own cash in, you need cash to start with.
Really? Isn't FinTech getting hit harder than just about any other industry right now?
First, i dont consider Citadel, JS, etc "fintech" because they are not in the same line of work.
" Fintech, a portmanteau of "financial technology", refers to firms using new technology to compete with traditional financial methods in the delivery of financial services. "
This is NOT what people like CIT, JS, etc. do.
So maybe FinTech is being hit very hard, but from what i hear Cit, JS are doing just fine (no real layoffs).
I agree that Two Sigma, Jane Street, Renaissance, etc. are NOT Fintech. Fintech is a really large umbrella which seems to includes people like Paypal or Bloomberg and things like robotraders or companies that deal with some financial product. Hedge Funds / HFT / Algo traders can be really sophisticated with their technology, but I wouldn't call them "Fintech".
Haha yea FinTech would be a pretty derogotory term to call a HFT firm
HFT makes money off of volatility. The more the world burns the more money they make by keeping markets liquid.
That’s true, in much the sense that doctors make money off people being sick. In times of high volatility, it’s HFTs that provide liquidity and keep spreads narrower than they’d otherwise be. It is extremely illegal to purposefully make markets more volatile; good trading firms are not going to do that.
Have family working there. You can get past 300K fairly easily once you factor in signing bonus, annual bonus, and "extras".
They pay VERY VERY well indeed.
Yup, that figure I posted was just base but we often see folks getting multiples of that through bonuses.
Its kinda sad though being a great company and the only thing people know you for is high salaries. Anything else good about the company?
Jane Street is responsible for the Signals and Threads podcast[1], which I always enjoy. They also contribute a lot to the OCaml language, standard libraries, and do a great job of explaining why OCaml is a great language and paradigm. I've learned a lot because of Jane Street's sharings.
I have no connection or interest in Jane Street but this sure gives me a positive impression of the company.
Yaron Minsky, the host of Signals and Threads, is also one of the two authors of Real World OCaml[1] which is a good resource on the language.
For anyone that did competition math in HS one of the two authors of the main Art of Problem Solving books is also high up at Jane Street, on the trading side. JS sponsors stuff like math prize for girls now.
Obviously there's a potential recruiting benefit to funding contests and posting monthly puzzles and whatnot, but I still think it's a cool vibe that not a lot of other companies have.
Those are low numbers compared to the smaller prop shops that routinely pay 1MM+.
Gotta have a PhD in pure math though. That shit is bananas.
You don't need a PhD in pure math to get into these prop shops.
However they do sometimes pay !MM+
Do you have a source on the $1mn+ figure? I would be curious to read more about this.
I mean, not really. This is personal knowledge. You can't really look up salaries because the base is like $175k but even in mediocre years they're getting high six-figures in bonus.
Take a peek at what Wolverine, Radix, or Jump are paying new grads though. It's generally like $500k+. Even Jane Street, which is a larger place that hires people who aren't published mathematicians (see: Sam Bankman-Fried) pays $750k
Yeah the numbers he posted were also base from JS and Citadel websites, so they're not reflective of how much one could (or even guaranteed will with the early career bonus floors) actually make at those places.
This is for traders and researchers though. You can get pretty similar early career salary + bonus as a SWE at JS or Citadel, which is probably what more people on HN would be considering. Do you know what kind of pay the smaller shops offer for pure SWE?
I’m afraid that’s out of my purview. Only familiar with quant traders and researchers.
I’m sure it’s as good or better than big tech. But you don’t get to live in San Francisco and the work is a lot more boring.
> You don't get to live in San Francisco
You get to live in New York City which is unparalleled in the US in terms of urban amenities. The weather can be brutal though.
Or you live in Chicago which is like, still a solid city, but the weather is even worse.
I would disagree that trading firms are more boring than big tech. They are typically much smaller and leaner (even HRT and JS are like sub 2k?), and generally employees have massively more impact and ownership as opposed to being a cog in a 20k developer machine.
I’m more surprised people don’t know this. It’s the primary cause for my depression.
I'm quite salty about HFTs. I'm a final-year undergraduate with a bunch of close friends who all love C++ (yes, we're masochistic that way). They're all really smart, too (me, not so much—my GPA is 3.00/5). However, after a while, it got really dull and frankly, a bit exasperating when all the rest of them could discuss were HFTs, their ridiculous, outrageous salaries, matching engines, order books, and low-latency trading. Heck, one of our assignments last year was to write such an engine in C++. It seems C++ developers are particularly in demand at most HFTs.
Some of these companies I have never even heard of before having entered CS—Jane Street, Citadel, HRT, DRW, Ansatz, Two Sigma... Their names and websites are cryptic, and their job position listings are even more so, unless one knows that they're mostly proprietary traders—in other words, making money for money's own sake.
It feels dirty and excessively capitalist—these companies don't even have products to show for all their effort. Any clients they do have probably already make millions to billions, too.
My friends defend themselves by saying 'HFTs make money by keeping the market liquid', 'there's nothing wrong with arbitrage', etc. That's fair, but I still feel that's just sugar-coating what I said.
At the same time, I don't really say anything in person to my friends, because, well, they're friends, and secondly, my lousy grades make me feel thoroughly unqualified to make any sort of criticism.
I personally would rather do embedded, automotive, avionics, or game engine development. I just wish I had someone to discuss this with. Hardly anyone I know wants to do these instead, because the salary is lousy to above average, the hours are as bad, and especially game development is considered a mostly rubbish job to have. It really sucks, because I've gamed all my life and always found video games pretty damn amazing, both technically and just in general. It was a pretty bad bubble-bursting moment when I discovered just how bad the game dev situation was—overwork, crunch, sexual abuse and sexism, bean-counter-led design and marketing decisions.
I'm not looking for 100% job satisfaction (I accept even the most exciting work will have its dull periods), but it would be nice if the thing I spent 8-10 hours a day doing for a salary was something that remotely excited me and others, instead of just mindlessly piping money from X to Y and back just because it paid half a million a year.
If you want a good salary with impact and embedded/high performance C++ look into the self driving field. Check companies like “Cruise” and “Waymo” and see levels.fyi. It’s not all sunshine and rainbows, but it is making the future vs simply manipulating the finical system.
Thanks for the recommendations. I've applied to more traditional auto firms (BMW, VW, Mercedes, Volvo), and hadn't really considered the newer ones. Cruise looks particularly interesting.
> 'HFTs make money by keeping the market liquid'
Ironically they only really keep the market liquid when the sun in shining. When there is a big crash and you really need liquidity most HFTs disappear.
They have put a lot of highly paid human traders out of business though which is something. The C++ devs might be earning a lot but its less than their predecessors.
Do you have any evidence for this, empirically? I mean yes, HFTs aren't going to let their orders sit stale when FOMC announces a giant rate hike, they aren't going to lose a bunch of money to keep the market healthy. But generally when vol is high, liquidity is at a premium, so I would imagine HFTs become a higher percentage of the market (though spreads are still going to be wider).
Yeah these HFTs aren't big companies, they buy and sell stocks quickly. After the 87 crash big banks could buy billions of dollars of stocks and bonds because banks are huge and could do this to both protect the market, their customers and make money.
Most people I've met that actually got jobs at top HFTs were quite curious about a lot of topics in school (and still are with extra time they have). I'm surprised your friends are so fixated on HFTs while there are so many interesting and/or fun other things to spend time on when undergrad on campus.
A lot of HFTs have their own training programs anyway for the job specific knowledge, so it is more important to build up the right base math and/or dev background than to hyper fixate on HFTs all of undergrad, even if your end career goal is to make bank.
>Most people I've met that actually got jobs at top HFTs were quite curious about a lot of topics in school (and still are with extra time they have).
yeah and that's the worst part, they take the brightest minds of the generation and have them working on moving money around instead of building fusion rockets
Tbf many of them were pretty jaded/burnt out by trying to do academic research before fully taking the plunge. I'm sure there are smart people drawn by the money right out of the gate, but other fields have also done a really good job of driving people away.
> I'm surprised your friends are so fixated on HFTs while there are so many interesting and/or fun other things to spend time on when undergrad on campus
Many of them were or are interested in other things like compilers, home-labbing; a couple of them game, too. It's just that because we're about to graduate, job offers dominate the conversation, and HFT jobs dominate that conversation.
Fair enough, the time around graduation can be pretty stressful for everyone!
In 2017 we extended our trading experience, infrastructure, and technology to digital assets, and we’re now trading crypto 24/7 around the world. We price all the most actively traded spot tokens such as Bitcoin and Ethereum in addition to derivatives like futures and ETPs, and we trade on almost all major global market centers, including crypto exchanges and traditional venues. We’re also a major OTC crypto liquidity provider, and our JCX single-dealer platform provides reliable, integratable liquidity in digital assets. Our institutional-grade crypto offering is a natural extension of our deep cross-asset expertise and our experience trading complex, difficult-to-price securities.
Interesting. Crypto has much less liquidly overnight compared to other markets, no? How would this work? Coinbase?
Most other markets actually have periods when they don't have a matching engine running because the instrument or settlement is country specific. Crypto has 24 hour liquidity. There are periods that are quieter but it is 24/7 unlike, for example, a typical equities exchange.
(How I know, 25 years writing exchanges).
Related:
How to Build an Exchange (2017) [video] - https://news.ycombinator.com/item?id=27879230 - July 2021 (12 comments)
Related: https://lmax-exchange.github.io/disruptor/
I've been using the .NET port of Disruptor to good success. Once you understand the underlying pattern, you can apply it everywhere without pulling in a dependency.
You would be astonished at what will actually fit on 1 x86 thread in 2023. Instruction-level parallelism can give you unbelievable throughput, assuming your batches are reasonably-sized and everything fits neatly into the various caches.
Other related links:
https://martinfowler.com/articles/lmax.html
https://www.youtube.com/watch?v=qDhTjE0XmkE [Evolution of Financial Exchange Architectures]
have been wondering for a while why HFTs exist when exchanges have holidays and trading hours [1]?
doesn't the sub-milisecond speed contradict to the several hours pause in trading?
wouldn't be possible to impose some kind of a delay (say an hour) on all participants?
Anyone know of any open-source exchanges at the level of sophistication described here?
I wrote an open source one in Go that I'm still working on. It's never meant for production. The code is at github.com/dylanlott/orderbook and there's an accompanying write up on it at dylanlott.com/orderbook if you're interested.
Thank you for the writeup.
Are there any technical books or reading material about matching engines out there?
How exactly would would this design keep track of how many funds a client has in order to fulfill an order? How does NASDAQ keep track of client's total funds?
They don't. Remember that NASDAQ's clients are Jane Street, Goldman Sachs, i.e. not you or me. They don't onboard a client with doing due diligence. Then there is a clearing and settlement system called DTCC that requires every firm to post collateral equal to their trading volume. This system's primary job is to make sure everyone has enough funds to pay for their trades across all exchanges, not just NASDAQ.
The context here is that Sam Bankman-Fried of FTX and a number of the other executives (Caroline Ellison and Brett Harrison) were previously traders at Jane Street. This video is from 2017, a couple of years prior to the FTX launch. It makes you wander if any of them were in attendance.
While I wouldn't be surprised to see some proactive PR from the company in the wake of recent events, perhaps including the post on the zkSNARK competition last month; which was in my humble opinion one of the few submissions that has truly moved me since the late 2000's on this site:
I feel obliged to attest that the people I knew working at Jane Street at that time were some of the straightest shooters one could know. My anecdote is that SBF's toxicity didn't originate in their culture. Perhaps it was a counteraction to it, if anything.
Can you link zkSnark convo mentioned here?
FTX didn’t make any of the architectural decisions presented in this talk.
Yeah, as an exchange FTX was pretty terrible. The matching engine was notorious for being high latency, unreliable, and having all sorts of undocumented issues. Every algo trader there will tell you horror stories about trying to cancel an open order, getting a cancel confirmation, then getting notified that the order was executed. Sometimes seconds later.
What really gave FTX a competitive advantage was its cross-margining system. It was super easy to post collateral in any supported asset, and all open positions were automatically margined against each other. So if you were doing something like pairs trades between spot and futures, you didn't need to constantly monitor and manually move capital between the silos. In hindsight though it's not clear if this was possible because of better engineering, or just because Alameda was internalizing all the risk.
The main advantage of FTX's exchange was that it was far better than Binance's but that's barely saying anything.
I'm not sure why crypto exchanges have such lousy technology. Lost acks, weird parser errors, missing order id fields, sessions giving no indication that the matching engine is down, etc.
It seems like any conceivable way an exchange can break will be encountered after only a couple months of trading. It's a complete shit show compared to traditional exchanges
There’s a graveyard full of “crypto exchange but built with good tech”, ftx even bought one iirc.
The market favors move fast, market to retail, etc etc over spending time cutting microseconds off the 99th%.
A lot of the exchanges also don’t have people who know what they’re doing. Lots of normal startup folks coming from an environment where “premature optimization is the root of all evil” and “move fast break things” is the norm.
There have been a few valiant attempts to build crypto exchanges with modern technology and matching engines. Generally the problem these run into is that, while the algo traders and market makers love them, they have no competitive advantage in attracting retail order flow. Trading firms have no interest in just trading against each other, so without retail flow nothing happens.
Having worked on several side projects that involve money, I'll bet most money-related computer systems are full of race conditions and other wacky behavior. There are so many gotchas. For example, just writing a simple "bank account" system/database, it's tricky to ensure in a performant way that concurrent transactions don't bring a user's balance below 0. If you're sending out money via ACH or something, try handling all the ways that can fail and need to be retried.
I trade regular stocks in my Chase account. A few times, it was down for maintenance after-hours, meaning I couldn't enqueue trades to execute the next day. Not a big deal, but doesn't inspire confidence.
Because it was all a scam and didn't need to polish details ?
One of the stranger things coming out of the collapse of FTX is that it wasn't _all_ a scam; if SBF and co they had simply let FTX function without giving special privileges to Alameda, it would have been a pretty non-scammy crypto exchange (overinflated in value, sure, but not fundamentally scamming its customers) and they all could have made great deals of money on that alone. But of course, that isn't what they did.
horror stories about trying to cancel an open order, getting a cancel confirmation, then getting notified that the order was executed
Sounds like tuesday to me. One thing you learn about exchanges (I mean all of them as a software category) is that you can’t trust: documentation, reasonable expectations, the experience you’ve got last week, the idea that all symbols behave similarly, reproducibility on test servers and vice versa, to name a few. Don’t rely on these and you’ll be relatively safe.
But really, expecting a deterministic confirmation is a rookie mistake. Exchanges feel very asynchronous after a short while and the intuition should tell you that that a confirmation is just “ok, I hear ya”, not “effective immediately we’ll suspend our queues and reschedule trades to fulfill your urgent request”. The temporal uncertainty of order status is a quite common phenomenon, ime.
FTX wouldn’t just ack your cancel request and then give you a fill - you would get cancel successful messages, order status updates that cancelled your order with no filled quantity, and then ten seconds later get a fill for that order.
Occasionally sending out zero balances was another FTX special.
Sbf was playing LoL at the same time
@dang could we get a (2017) added to this? Changes the context quite a bit given FTX folks worked at Jane Street previously and a modern version of this talk would have a much different context.
I don't think the context changes at all. This talk is about high performing systems (millions of requests per second) written in OCalm. There is nothing here related to the criminality of FTX or what its non-technical founder decided to do with customer funds as a business decision. It matters very little that one or more of them used to work as an analyst at Jane Street.
FTX's Gary Wang worked at Google.
https://www.forbes.com/profile/gary-wang-1/
It's intriguing how Forbes 400 list includes criminals. If someone tricks people into "investing" with them and they accumulate enough billions of US dollars, and it takes a few years for the authorities to act, then do they get a spot on the Forbes 400. What are the rules.
> a modern version of this talk would have a much different context
Everything would be in a different context, yes. But how would the content be different?
> a modern version of this talk would have a much different context
How?
2023 version would have more than one mention of {compliance, regulat*, accounting}
Would it? 2017 is well post 2008 crisis, SOX, all of the big changes in this area