Ask HN: Why isn't all stock trading done by algorithm?
I can't understand how a human, even with heaps of intelligence and education, is better at predicting which stocks will go up/down than an algorithm that can look at all the data in the entire market. What skill do stock traders use that can't be replaced by computation? Computers trade most effectively with technical indicators. These are things like price history, volume, etc. These trades are very effective in the short term, where trades can happen in terms of seconds or milliseconds. This is where computers excel and humans fall short. This sort of trading is based mostly on breadth. If you look up the fundamental law of active portfolio management, you'll see that breadth is less effective (exponentially less so) than skill is. In the case of the long term, however, trading is done with fundamental indicators. These things can be more or less intangible and have to do with market events, people, and other indicators of company value that are hard to translate into math. Using fundamental indicators for portfolio management is what humans are better at, and these pay off in the long term (see Warren Buffet). These trades are done with skill, which, as I stated earlier, is exponentially more effective at creating gains than breadth. In short, it takes a huge amount of breath to get the gains required by a relatively small amount of skill. Computers are better by far at breadth, while humans are better (for now) at skill. This, I think, is why humans still trade. Not to mention the spectrum between intuition/innuendo and flat out insider trading knowledge that is cornerstone to the market -- trading algorithms don't do well against this human factor. Information not in the market. Stock traders trade on rumor, fact and everything in between. They (can) look at who the company is run by, and how much they trust him, and look at the people in upper management. They can look at and understand news. Part of the difficulty of the stock market is that it isn't a closed system: people make decisions to sell stocks based on the fact that they are poor this month. Until computers can understand and process all the external data, people will need to be in the trading loop. The big exception to this is the trading the noise: high frequency traders trade against each other, closing the spread in bid/ask prices. However, they are essentially trading against other computers at that timescale, which makes them pretty amenable to algorithms. You reminded me of this scene from Trading Places: Randolph Duke: Exactly why do you think the price of pork bellies is going to keep going down, William? Billy Ray Valentine: Okay, pork belly prices have been dropping all morning, which means that everybody is waiting for it to hit rock bottom, so they can buy low. Which means that the people who own the pork belly contracts are saying, "Hey, we're losing all our damn money, and Christmas is around the corner, and I ain't gonna have no money to buy my son the G.I. Joe with the kung-fu grip! And my wife ain't gonna f... my wife ain't gonna make love to me if I got no money!" So they're panicking right now, they're screaming "SELL! SELL!" to get out before the price keeps dropping. They're panicking out there right now, I can feel it. Randolph Duke: [on the ticker machine, the price keeps dropping] He's right, Mortimer! My God, look at it! You're overestimating the effectiveness of computers in taking in data and finding patterns. If you throw data naively at an algorithm, you'll get garbage. It's especially difficult to make sense out of trading data because of the sheer size and percentage of noise. For any given trading strategy, a ton of thought, testing, and domain knowledge goes into creating the algorithm. It is not a black box that writes itself. That said, computers are far more effective at certain tasks, especially latency sensitive simple calculations, just as calculators are far better at doing arithmetic. I agree. And the OP is also underestimating the effectiveness of insider trading. Well the truth is that for some firms it is for about 99.9% of all trades. Renaissance Technologies founder Jim Simmons is famous for saying they didn't override the algorithm. In practice most HFT firms do a mix of both. The algos will do the vast bulk of the trading but you have human traders monitoring algos to do clean ups for cases where the algo gets "stuck". What defines "stuck" really depends on the sophistication of the algo and the firm itself. Some algo's, such as internalizers's for crossing bought flow are so simple that there probably doesn't need to be much over site at all. Market making is very similar, with the exception of a flash crash where they might pull out, market making algos should just run themselves. Well for starters, someone has to write the algorithms. Not everyone has vast amounts of computing power to find the right inputs for the right algorithms or genius level quants to write the math which coders turn into strategies. There are so many small niches in the market that a human can still make a reasonable living if they find an area not traded by others aka "making a market". Now when it comes to competing with the machines, you pretty much nailed it. Source: I work in electronic trading and have the past ~7+ years. Would you happen to know some good reads (either on- or off-line) for someone interested in computer trading with a technical background (currently doing a Msc. In Computational Science != Computer Science)? I have been interested in doing a bit of electronic trading as a hobby (and hopefully a small profit, but I know better than to hope for now) and possibly a back-up career path. But while I've found enough information about trading in general I found it rather hard to find anything dealing with automated trading specifically beyond very simple models. Try this blog and book, a rare mix of practical code and market analysis, http://qoppac.blogspot.com/p/systematic-trading-start-here.h... Thanks, seems to be some useful stuff in there. Unfortunately I'm not a great person to ask about that. It is a bit ironic, but I keep my head out of finance and don't much care for it. Now the technical problems presented in a low latency financial environment... Absolutely fascinating to me. Ha, yeah. My own approach has also been to mostly keep out of finance myself, but lacking a team of geniusses I have to do the whole 'stack' myself. I tried some demo-forex trading manually, just to test the waters, and while it worked quite frankly after just a week or two it just became busywork. But now the translation from data->model->strategy that's something else, and highly relevant to my interests (the quant+technical side). But it seems like most traders don't much care for the technical side (nor brokers with their lack of api). Either that or the smart ones know to not make their fancy modeling and tricks public :P. There are a lot of different schools of thought about approaching the stock market. One of them is the "chart" approach, where you (or an algorithm) try to discern the future movements of a company's stock price based on the past performance. Sometimes this works. Mostly, though, it doesn't. There's also a big difference between "trading" and "investing." Trading is what you've described -- buying shares in the morning and hoping they'll go up in the afternoon so you can sell them later. Investing is buying shares of the company to become a part owner and hold them for years or decades, not days. If you looked at NFLX's chart in 2012, you could "discern" that the share price would continue to hover around that price, maybe go up a little, maybe go down a little. And you could have bought it in September 2012 for $8 a share and sold for a nice $1 profit in October 2012 for $9 a share (split adjusted). But what the chart wouldn't have told you -- and would never have been able to tell you -- is that it would skyrocket in 2013 and up to its current split-adjusted price of $110 a share. The thing is, the chart never would have told you about this. And even a "pure" numerical analysis like could be done by a computer -- P/E ratios, cash flows, etc -- would not have predicted that growth. You could do DCFs all day every day in 2012 and never predict Netflix's rise. There are a lot of things that go into a company's rise that aren't numerical, like the quality of management, market moat, market growth, etc. And, of course, you had to buy it, and hold it for years, in order to see that return. (In the interests of full disclosure, I should probably note that I'm a bit biased in this. I work for The Motley Fool, which advocates for long-term buy and hold investing, and produce a podcast for growth investors called Rule Breaker Investing.) Good point about trading versus investing. When index investing advocates say that it's hard to beat the market over the long term, I've always wondered what the difference is with traders. Do their returns beat the S&P500? Does an individual who trades forex as their day job do better than stuffing their money in an index fund for 40 years? Lately I've learned that larger traders can provide value apart from buy&hold, e.g. market making, finding the right price for something, shorting bubbles. I don't have specific numbers on traders, but I do know that 80% of mutual funds underperform their benchmarks. One reason (of many) is that they're under pressure to provide numbers every quarter, or to seem like they're reacting when something short-term happens, so they sell out of positions that would have been more profitable had they held onto them. Index funds are a great solution for investors who want to invest passively and get the market return (which averages 10% a year over the long term -- might be -20% and +30% year-over-year, though). The thing with shorting bubbles is extremely difficult to get right, and "finding the right price" doesn't necessarily mean you're making a good decision. Amazon was trading around $300 a share earlier this year -- which many trader types saw as outrageous -- but it has doubled this year alone. I suppose, in general, I follow the Black Swan approach: humans are terrible at predicting the future. Absolutely awful at it. But I'll take a "bet" on -- by which I mean, buy part of -- a company with great management/people and solid financials (i.e., no debt, positive revenue growth, FCF positive). Computerized trading uses more than just historical price for input. Former investment banker here. This is a question I had when I hit the desk, and it's an important question. The short answer to your question is: because most of the volume traded on exchanges is large blocks of stock being bought and sold by institutional investors, and you need humans to make these deals happen. Longer explanation as follows: On the trading floor [0], you have two groups of people: the sales team and the traders. The sales team gets paid when they make markets, i.e., connect buyers and sellers. Specifically, the financial institution takes a fee that's a very small % of the overall transaction volume, and some of that goes into the sales team's bonus pool. The more stock trades flow through the firm (specifically, their business unit), the more they get paid. The traders, on the other hand, gets paid to do two things, which are really the same thing: a) not put too much of the firm's capital at risk and b) set the firm up to make money by buying securities low and selling them high. Every trader has a "P&L" (profit & loss) number, which is the total amount of money they've made or lost for the firm since the start of the fiscal year. They get paid a bonus based on this number. They tend to know exactly what their number is at any given time. So, there is actually a lot of tension on the trading floor between the sales team and the traders, because the sales team wants a lot of volume to go through their business unit, and any given trader wants to maximize her P&L. Real world example might be: the sales dude gets a call from a hedge fund saying, "we want to sell $100mm of our shares in Alphabet at $720". He then shouts over to the trader (who sits close to him) to tell her about the call and she thinks for a couple seconds [1] and then says, "you need to make a market for 80mm of those shares at that price, I'm only taking 20mm." In other words, the trader is saying that she'll only tie up $20mm of the firm's capital on this particular trade [2]. The sales person might come back and say, "c'mon, they took that $50mm of Microsoft stock you were trying to get rid of last quarter, we as a firm owe them a favor" to which the trader might respond, "OK, we'll take $30mm tops". So then the sales person will get on the phone and start calling everyone (other mutual / hedge funds, pension funds, etc) who might be interested in buying Alphabet at $720. Maybe the sales person makes it happen; maybe they don't. In any case, they need to figure out whether they can get 70 million dollars worth Alphabet stock pre-sold to other people in the market at $720 before they get back to the hedge fund trying to sell it with a response as to whether they can make the trade. All of this involves MASSIVE HUMAN FACTORS. I'm sure we will one day be able to train AI to work through various constructs of "we owe them a favor" but right now you still need humans to get big trades like this done. And again, big trades like this constitute the majority of the overall volume in the market. So, that's why trades don't run entirely on algorithms...yet. [0] I was in banking, not S&T, but have a decent understanding of how this works. [1] Being able to make decisions of this magnitude in a couple of seconds (and have them be good ones) is one of the two skills you need to have as a trader; the other one is not letting the outcome of the last trade (good or bad) affect your thinking on the next trade. [2] There is potential for both upside and downside in a decision like this; if the stock appreciates, the firm can profit by selling the stock at a higher price than it paid, but the reverse is also true. This is also an example of why "proprietary trading" is such a blurry line. In order to make markets for big trades, firms usually have to put their own capital at risk, even for a few minutes. At what point are they trading for their own profit vs. temporarily assuming risk in order to broker a deal between two counterparties? Go read Matt Levine's archived columns at Bloomberg if you find stuff like this interesting. So once the pre-sell is agreed on are orders made on the open market? Or do these deals take place outside of that? No, the shares change hands between parties via the licensed broker-dealer. However, the trade is reported to the exchange assuming the stock is listed. The key here is that Big Mutual Fund Inc. doesn't want to announce to the world that it's trying to sell a massive volume of shares; it wants to get the deal arranged quietly so that other people in the market don't trade the stock down in anticipation. The price of the trade will be continually adjusted until the trade is executed. Usually the firm has an approved range to work within. If the price steps outside this range, many more phone calls will ensue. I see, thanks! This isn't trading though - it's playing middle man. You forgot to mention that the trader offering to buy $20mm of stock is sitting there looking at a screen telling him that real market makers are posting bids of $721 on the Nasdaq, etc. All the trader has to do is take the flow and offload the risk without moving the market such that he takes a loss. I'm sure it's largely regulation that prevents these hedge funds from going straight to the market themselves and bypassing these "traders". It's the same thing as exchanging currency in an airport. You see these big signs with large disparities between "Buying at" and "Selling at" for each currency pair. Very easy to make money like that as a currency swapper when you have the nearly-guaranteed hedge that is the massive FX spot market. Any time you're buying something in order to sell it at a later time, you're trading; even if your ownership is for a few seconds. > I'm sure it's largely regulation that prevents these hedge funds from going straight to the market themselves and bypassing these "traders". No, it's the fact that hedge funds are not in the business of making markets. Sales teams create value because they maintain relationships with lots and lots of portfolio managers; if someone calls them up and says, "I want to buy A shares of M stock at or below X price" or "I want to sell B shares of N stock at or above Y price", they know who to call; if they're good at their job, they'll have good instincts as to who may be interested. Additionally, if a hedge fund just went out and announced that it wanted to buy or sell a bunch of shares in a given stock at a given price, it would move the market in a direction that would be bad for a hedge fund. For this reason, most big funds not only go through third-party prime brokers, but spread their orders across multiple broker-dealers in order to minimize visibility on their trades before the fact. "buying securities high and selling them low" - I assume that's a typo? Usually it's the other way around, unless you're trying to make a joke. And what a typo it was! Fixed; thank you. I learned more from this post than from all of the finance classes I took in school combined. What is an algorithm in your definition? If it includes where and how to place the trades (a smart order router) I would say 99% of trades are managed by an Algo. Also there are the "dumb" VWAP, POV, TWAP algos which represent the bulk of "smart" buyside money as internally most firms use vwap as benchmark. The bulk of retail orders in the US are on the other side of an algo from citadel, knight or citi. And often buying 100 shares of MSFT at market only really needs a decent SOR to provide BestEx. Block trading still often gets fed to VWAP algos unless the stock is illiquid. Finally the most interesting execution algos (implementation shortfall algos) are hard to explain and only statistically outperform. If you are talking about actual investing - the first question is which asset class in which to park your money. If you can get an algo to do that (like Renaissance) you will be rich. Because ultimately, stock prices move not based on "the market" but on things happening in the real world? How does an algorithm interpret e.g. an Apple Keynote? By the time Twitter sentiment analysis (if such thing is really useful) gives results, an human trader already took a position... Humans are better at recognizing satire as well, April Fools jokes have famously caused some algorithms to react poorly. Why do I need an algorithm to find a good business, buy shares, and sit on them forever? There are limits to algorithms. Stock prices are affected by human behaviour, which can't always be predicted. Not to mention the various macro variables, like wars, weather, crime, etc... Plus, what time frame should the algorithms trade on? They're very good for predicting the very short term, I haven't seen much evidence that they're good for predicting longer time spans. While algos eat up arbitrage and electronic brokers replace human ones, humans are still very good at other forms of trading... Not to mention, a large part of market activity isn't even trading - it's long term investing and collecting dividends. It's impossible to create an algorithm that encompasses all the factors that contribute to a company's stock price (or, perhaps more accurately, all the factors that a given investor believes will affect the stock price). There's also an implementation gap between the model a stock analyst can create in Excel and its implementation as an algorithm. The former is within the reach of far more people than the latter. Finally, don't forget that somebody has to design and write the algorithms. This is a pretty good question... Not sure if it will be solved here. I used to be a value investor around 2000-2008. A value investor would be something like Buffet or Peter Lynch. However I did make a lot money in Sept. 08 because I determined the market was over valued. What I didn't forsee, was how much the dollars the Federal reserve would print and inflate the economy. Regardless, after that I built my own algorithm, because I no longer believe in the structure of the market. I would rater trust numbers. Meaning there are to many analyst pumping stocks, federal reserve, insider trading, spoofing trades, ETFs, deratives, and financial warfare it's hard to make a true value investment. Yes, I have read the buffet / Grahm books, but those are over ~60 old. I think it is Virtu (electronic trading / hedge fund) that hasn't had a day where they lost money since early 2009? I know Goldman and JP Morgan 90% of the time trade every day for a profit. So a lot of the market is already trading electronically. I think zerohedge.com has estimated the 70% of the market trades on electronically and that article was few years ago. It's funny, because I have devised methods using social media / programming to manipulate the price of stocks. If I can think of ways to do that I'm sure sure Wall St. already is doing it. Anyways here my algorithm it tracks over 500 stocks: http://www.strategic-options.com/trade/ Your site charges $600/year. That's 6% of a $10,000 portfolio. Do you think it can make a 6% gain in a year after taxes and inflation to cover its costs for that size portfolio? 6% gain is about the long term average of the S&P500, after inflation, so you're proposing doubling the market gain. Good question. Let's not forget that S&P was down ~40% in 2008. So you are correct that long term average is ~6%, but then you have years like that, kind of makes the long term average meaningless. Don't forget your financial adviser took a cut on your losses in 08 as well. At $600 a year is pretty cheap in comparison to what other trading tools. (news letters, chat rooms). This isn't necessarily for people looking a retirement fund. I'm also marketing something cheaper http://www.strategic-options.com/trade/alerts but this is for people who trade semi-regularly. I'm not sure if you fit in that marketing demographic. However, you can see from the website some stocks like Amazon have return far greater amount than the $600 initial cost. This new ETF / index strategy is nice, I working on a portfolio that would only trade ETFs. That would charge only 1.25% of AUM. But that's still in the works... So you're not really sure it could make up its cost? The particular service that I'm offering, the membership. It could surely make up your cost... But it is more oriented towards do it yourself traders, it's not necessarily geared for Buy and Hold. For example on 1/28/15 the algorithm produced a buy signal on Amazon. If you followed that tip on 1/28 or 1/29, then you would have received ~100% return your investment. http://www.strategic-options.com/trade/stock/amazoncom-inc So your $10,000 dollar invest would now be worth $20,000. That would surly cover you membership cost of $600. You can find other winners here: http://www.strategic-options.com/trade/open/market_analysis/... (click the winners tab) I also have a Wiki/FAQ here: http://www.strategic-options.com/trade/open/wiki If you have any other question feel free to email here chad (at) strategic-option.com Skill? I'm putting up trades as an investment. My money can't just sit around in a bank account forever, it needs to be invested to make actual gains. All the hedge-fund managers that claim to have algorithmic trading have extremely high expense ratios. So its cheaper if I made trades myself. I mean, its only $7 to execute a trade off of Scottrade or E-Trade. While buying a mutual fund with algorithmic trading will cost you like 50 to 200 basis points per year. Yeah, its cheaper to trade in the raw or to just buy SPY or Vanguard funds (which are passively invested without algorithms) The amount of trading done by machines decreases as your investing timeframe increases. So at the extreme (high frequency trading), all trading is indeed done by machines. In addition to an algorithm not having all available information on the market there is also the fact that any niche or inefficiency in the market can be duplicated by others until it is negated. If you discovered that the market always goes up on Tuesday and drops on Wednesday that only works until everyone else discovers the same thing and starts selling on Tuesday and buying on Wednesday. Most stock brokerages spend lots of times gathering data from the companies themselves -- the management group, the customers, industry analysts, etc. They are not making investment (BUY/HOLD/SELL) recommendations based on the past fluctuations of the market, rather on their "expert" appraisal of the company's worth versus its current stock price. You mean the computers that can't even understand the sentence "We saw grand canyon flying to chicago." I recommend to read Warren Buffet or his teacher Benjamin Graham.
Buffet uses a computer, but only for Bridge playing. market conditions are always changing as are regulations. In 2008, they implemented an emergency short sale rule that banned short selling for a select set of stocks. An algorithm would not know what that means, humans have to step in to ensure stocks are not shorted during these times. Forgive my ignorance, but would it not be quite easy to tell the algorithm not to short certain stocks? You're asking the wrong question. That's just one case where the computer doesn't understand the human side of things. What you should be wondering is what are some cases where there is absolutely no advantage to using a computer. Look up the story of "George Soros breaks the bank of England." Or Hedge funds shorting Volkswagen when it was the most expensive stock in the world. Only to find out that Porsche owns 75% of it. The problem is that introduces a discontinuity in the market, where past training data may not be relevant to future price movement. It's not enough not just to not break the rules, but you may have to understand the effect of the rule change on the market. "Hey look, these bank stocks are showing amazing relative strength on an overall down day. I better load up!" is a possible erroneous conclusion. A human would have to go in to add that 'feature' if the case had never come up before though right? Fortunately, not all investments are based purely on technical analysis for short run gains. Even the ones that are, are basically a poker game between at least two intelligences, and AI is notoriously not that good at poker. There are plenty of stories about human vs AI poker players to google for where the AI either loses miserably or under weirdly limited circumstances merely loses significantly. The computers do beat human when the game is intentionally played too fast for humans to physically / biochemically keep up (HFT) and humans usually destroy algos for long term (admitted or not insider trading or line of business expertise). The only really interesting timescale is the daytrader/poker player. The simple reason is that people make money by trading stocks. And they make enough to keep doing it instead of doing something else. It's simple economics. If they wouldn't be able to make money they probably wouldn't trade. well, the popularity of gambling shows that there's more to it than that. There's thrill, the perception of control, habit, and bad-math too. I've been trying to find patterns (http://newsp.in). I dont believe its an exact science yet, but its always fun to try. How would an algorithm react to / speculate on the most recent news of, say, the VW pollution-cheating fiasco? What about embargoes or war breaking out between certain countries? Using sentiment analysis. In fact, there are some traders (crazily!), which do just that. http://www.wsj.com/articles/tweets-give-birds-eye-view-of-st... An algorithm parsing the aggregate reaction of mostly non-experts, mostly to news delivered over relatively slow mainstream media channels, is going to be much slower than a human hearing the investor briefing and hitting the "sell at any price" button. Because algorithms can't wine and dine potential investors to convince them they're the best in the business for picking stocks. Machiavellian. You are also overlooking pump and dump Twitter accounts which sway stock by micro-increments, this tactic becoming more of an art than a science these days (Twitter clamped down on those accounts in recent years). Also see http://www.huffingtonpost.com/2013/03/11/twitter-hoax_n_2850... They're not. Retail trading is a fools game. Limited computing power, uncertainty and the false perception of perfect information (or perfect intuition). Computers aren't normally so good at predicting the future? Even if at some point humans no longer had any edge, by pure randomness some of them would still have really good performance. So even in a perfectly efficient market you wouldn't expect humans to be taken over. I've been in the market for over a decade, and here's my take: - Lack of sophistication. "Classically" trained finance people don't know much about computers. I took a finance class at a top business school, and it's nothing compared to Engineering. A bit of time-value-of-money and maybe some option math, but really it doesn't come close to the sophistication of a CS or Engineering course. I went to a meeting last week with a guy who wanted an automated trading system. He hadn't heard of Python. He didn't have any idea how to execute other than on 3rd party programs (which of course use algos, but he was just providing the decisions). - Lack of scale. There's a lot of family offices who have a few tens to hundreds of millions of dollars. If they wanted an algo trading guy, they'd have to pay him a lot of money, you'd want more than one, and you'd need infrastructure. Plus there's the risk you get all this, hire the guys, and their results are no better than random. A lot of small fortunes like this tend to spend more time in tech-soft areas, like private equity or private debt. The stock trades are an afterthought that they can't spend much resource on. - Two kinds of decision making: arbitrage and investment. The put it bluntly, arbitrage is easy to mechanize. If some guy quotes some options at the wrong value, it's obvious you want to trade with him. There's looser arbs (things that sort of always come back to normal), but the principle is the same. In some sense, it's not a financial challenge, it's a technological one. For investment (I think XYZ corp will go up), you need to have a sense of what risk you want to take. Utility functions are not easy to put into code. You can try, but you end up with situations where you decide not to have the algo on. There's also the principal-agent problem; most traders are agents, they need to look good to their boss. They need to be able to explain why they are betting on some company. Often, more effort goes into how to justify your trades than what trades to do. - Things that can't go into a machine: I worked with a guy who used to go meet the CEOs, look them in the eye, and ask them if they'd make money. Now I'm not saying this approach works, but if this is your investment edge, how are you ever going to put that in a machine? - Insider information: taking this in the loose sense, not the criminal one. If you're highly dependent on understanding some part of the market better than others, you may be better off talking and networking rather than coding. Goldmans are great at this. Every time you meet them, they offer a bit of info in exchange for yours. It lets them see things like the mortgage bubble before it happens, whereas a model would probably have issues due to the small amount of computerized data. Humans invest, computers trade. Its not about who is better at predicting. Most companies will gain value over time. There's no prediction to it. Algorithms are not predicting either. They are trying to fake out other algorithms and profit off of market inefficiencies. You don't need an algorithm, though, to buy something and then sell it six months later.