Gor – A simple http traffic replication tool written in Go
github.comEek... that's a rather unfortunate name choice. The "Gor" saga is an infamous series of books spanning the past 50 years, basically the sci-fi/fantasy equivalent of "Fifty Shades of Grey".
https://en.wikipedia.org/wiki/Gor
https://www.google.com/search?q=gor&biw=1920&bih=947&source=...
"There are only two hard things in Computer Science: cache invalidation and naming things."
-- Phil Karlton http://martinfowler.com/bliki/TwoHardThings.html
Oh, and off-by-one errors.
I still count two problems. =)
No more talk about hard things please. This is a family show.
I have always wanted to use such a tool and this one looks good from a cursory glance.
But what I have always wanted to know how people actually use these properly. There are two problems that I see in replicating production traffic to staging/dev
1. Changes in application structure url/parameter. I think given the change we make per release, we will get a lot of error. How to gracefully handle that?
2. Our application is write heavy so there is a lot of new content. So the majority of the requests will access content s that don't exists in the staging/dev environment. We can't even use live replication of DB either since we usually have a lot of DB changes also.
One solution I can think of is record today's traffic. Take a DB snapshot at the end of the day, replicate DB, run migration and then replay. Still has to deal with app change. Am I missing something or this is really challenging. I can see how this will perfectly work though for infrastructure change like server software or configuration.
On a side note of the blue/green deployment. Theoretically the reversible DB migration/parallel deployments of old-new version sounds good. But how many people can make this possible? Does most app has little DB changes from release to release. I will hazard a guess that if we try to implement such things the cost of such complexity will easily overshadow the actual changes to the app and most likely add severe risk of bug. But people seem to be doing it, again not sure what I'm missing.
Blog post: http://leonsbox.com/blog/2013/06/04/improving-testing-by-usi...
And previous discussion: https://news.ycombinator.com/item?id=5824387
We use it to replicate production traffic to our staging and testing environments at https://reverb.com. It's an incredibly useful and functional tool. We've used it to shake out exceptions from our blue/green deploys as well. For instance when we migrated from Rails 3 -> 4, Gor was paramount to shaking out some pretty nasty bugs by replicating prod traffic from the Rails 3 cluster to the new Rails 4 cluster.
Gor is awesome!
This seems like a super useful tool, especially if you want to do blue/green deploys where staging/production end up on the same physical machines and you want to run with artificial load as part of capacity planning/testing.
Looks like Tsung replay feature: http://tsung.erlang-projects.org/
I can't find any documentation, but the name suggests, that Tsung would be doing log replaying.
Gor does packet capturing and duplicates the requests to any number of environments.
It's a different tool for a different task.
This is super cool. You know what I'd really really like even more? Store this info somewhere, and let me replay it at a later time or date. Maybe I want to replay a whole day's worth of data at an accelerated rate to see if my servers can handle it. It also opens up the ability to do data forensics to try and reproduce a bug, perhaps.
Thanks for this!
It appears to have that support.
# write to file gor --input-raw :80 --output-file requests.gor # read from file gor --input-file requests.gor --output-http "http://staging.com"Shit, this changes everything.
Seems like an awesome tool.
As a side note, however, I would recommend anybody to not blindly apply the "Tuning" section (at the bottom of the readme) to their system. More specifically, net.ipv4.tcp_tw_recycle and net.ipv4.tcp_tw_reuse are notorious for causing problems if mis-used.
Could you elaborate?
Of course,
net.ipv4.tcp_tw_recycle causes problems with NAT-ed clients.
tcp_tw_recycle (Boolean; default: disabled; since Linux 2.4) Enable fast recycling of TIME_WAIT sockets. Enabling this option is not recommended since this causes problems when working with NAT (Network Address Translation).
net.ipv4.tcp_tw_reuse seems fine to use, but literature about its real effects is sparse.
This link http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-li... mentions both of them.
I have been using VCR , http://www.relishapp.com/vcr/vcr/docs for long time. Though VCR is only applicable to ruby, rails project.
Gor looks promising
I took me a while to find gor, and it seems that there aren't many tools for traffic duplication/replication out there.
If you know of any tools like gor, I'd very much like to know of them.
There's httap: https://github.com/rolftimmermans/httap
At the higher end of the performance scale there are commercial options using specialized hardware and commercial software. Building block cost is ~$300k/20Gbps (~400K rps) of replay traffic, but scales out with parallel deployments.
Thanks for putting it to the HN, author here :)
I've used this and it's great. Very easy to use and works.