Rqlite: Distributed Database Built on SQLite
rqlite.ioHi, any experience of using FTS5 with rqlite, i am interested in replacing Elasticsearch to have a single source of truth DB and full text search. Thanks
rqlite creator here.
Yes I do have practical experience to share, I wrote a blog post on rqlite and FTS5: https://philipotoole.com/building-a-highly-available-search-...
Will it allow you to reach the same scale in terms of data set size that Elasticsearch supports? Almost certainly no, but it might be enough depending on your use case.
Hi, thanks, yes i have read it before this is why i am considering it. Will have to try.
rqlite[1] creator here, happy to answer any questions.
Have you done any updated benchmarks on maximum supported writes/sec since the talk you gave comparing single-zone, single-region, etc.?
I recently had a potential use case for this, but it required somewhere around 600 writes/sec at a minimum, and it wasn’t clear what the ceiling was for rqlite without sacrificing durability guarantees.
Terrific bit of software, BTW!
rqlite creator here. I have performed a fair amount of performance testing, some of which I outlined in a talk to the CMU Database Group a few years ago. Details:
- https://www.philipotoole.com/2021-rqlite-cmu-tech-talk - see slide 33.
- There is also a recording that goes with the talk: https://www.youtube.com/watch?v=JLlIAWjvHxM
You can also read about Performance in the docs at: https://rqlite.io/docs/guides/performance/
An important thing to note: this testing was done 4+ years ago, on moderately-powerful hardware for the time. With higher-end, more modern hardware you may get even better results.
Thanks for the reply. Yeah, that’s the talk I was referring to.
I suppose I could try my own benchmark out tbf. I’m curious to see what it can do on today’s hardware. I would think it’s mostly network-bound for Raft consensus, though the 10x ping time increase you demonstrated without an appreciable drop in writes suggests it’s more complex than that.
Yes, fast networks matter.
I did introduce Queued Writes[1] since that talk, allowing you to trade off performance versus immediate durability. It may interest you -- network is much less of a factor then, and you should get a 10-100x increase in throughput.
Unfortunately for the application I was looking at using rqlite for, the possibility of data loss - however remote - was not an acceptable trade-off.
OK, well, you could try client-side batching too, if you can. That will also improve performance substantially.
Otherwise, if you try with more modern networks and disks, let me know what you see.