Sophia – An embeddable key-value database
sphia.orgI know it seems superficial but beautiful docs are one of my most trusted heuristics when I'm considering using a library. If the author cares about the aesthetics of the docs, it often means they care about the aesthetics of the code, which really does matter a lot. We can make ugly things or we can make beautiful things. I really respect people who take the time to make beautiful tools.
Was that intended to be a compliment of this project? I had the opposite reaction. I accept that English is often a second language, but this page was off-putting.
Sophia is a modern [should add comma] embeddable key-value database designed for a high [should hyphenate] load environment.
It has a unique architecture that was born as a result of research and rethinking primary alghorithmical [sic, should say "algorithm"] constraints associated with a [sic] getting popular Log-file based data structures, such as LSM-tree [should say "trees"], it's [sic] variations based on Fractional Cascading ideas and a B-Tree. (see architecture) [run-on, meaning unclear]
It is very fast. (see benchmarks)
it [sic] is easy to use. (see documentation)
Implemented as a small C [should probably hyphenate, or just rewrite] written, BSD [should probably hyphenate] licensed library.
It's quite clear English is not the primary language of the author, and I think it's in poor taste to criticize grammatical errors that are clearly sourced in this.
English is not an easy second language to learn as an adult, and technical English doubly so.
These are basic grammar mistakes. Mastery of basic grammar is achievable with a small amount of dedication; if the author does not have the time, they could ask an English-speaking acquaintance to proof-read their docs.
The author's primary (and perhaps only) contact with users is through their documentation. Incorrect capitalization and apostrophe use is distracting and will put off some potential users. The author looks sloppy and uncaring because these types of mistakes are preventable.
Being a non-native speaker is not an excuse for basic errors (excepting novices). Proof-read your docs! If you're not confident in your language skills, ask someone else to!
if the author does not have the time, they could ask an English-speaking acquaintance to proof-read their docs.
I doubt it is the lack of time. But natively English-speaking acquaintance to proof-read technical documentation can be very hard to come by. Most projects have to do by with the folks they have. ( I am in need for such a person myself. If anyone has the time Searchdaimon ( https://github.com/searchdaimon/enterprise-search ) could need some help :) )
One trick can be to at least wikify the documentation so other can easier pitch in.
To be blunt: do you speak a second language fluently? If not, I don't think you have any idea how much dedication you're demanding.
Instead of bashing someone on hnews comments, you could send them an errata patch.
The onus is not on me to make this project's documentation presentable: it is on the author. My language skills are immaterial! The author's language skills do not excuse lazy presentation, they only help explain it.
They are material when you say that the author is being lazy for not writing English better or finding someone to contribute. It implies you don't know the difficulty of what you're asking, so why are you calling him lazy?
But of course you want to make it clear that you feel no obligation to contribute despite your criticism. So it's lazy for him, but not for you?
The grammatical errors exist regardless of whether I speak one, three, or a hundred languages fluently. Better to ask the number of projects I have documented...
Note I comment exclusively on the author's presentation, not thrir personal behavior: obviously they are not lazy. I have no obligation to contribute, no: ncome from or personal interest in this project. I contribute to the projects which benefit me and people who I care about.
I wish that I had spent more time crafting my initial response. How would you have written my post, so these concepts are clear:
rgbrgb, I agree that it's superficial, but in direct contrast to your praise of the superficial style of the documentation, I was unfortunately distracted by the superficial mistakes in grammar and spelling. I wish someone would take the time to help the author of this code, Dmitry Simonenko, to improve the documentation. To that point, here are my hopefully constructive criticisms specifically of the language used in the documentation. I have performed no audit of the library itself, and cannot speak to its quality.
Does this still come off to you as poor form?
That's much more reasonable. But I also think it's a waste of time.
The problem with the docs aren't these small gramattical errors. In most cases you cite the meaning is clear, so this is just pedantry.
What bothers me about the documentation is that the explanation of his algorithms is a bit opaque.
But all told, I'd much rather someone contribute than hold back because of concerns about language mastery.
I, like many people, get distracted by easily-corrected mistakes. You can call that a fault of mine, and people like me, but since it's relatively easy to lower the barrier for us to easily consume the material by correcting the language, I think it's worthwhile.
I don't think you're weighing the "English-as-a-second" language factor nearly strongly enough. While grammar can be a great proxy measure of quality for works by native speakers, it's completely inappropriate for non-native speakers -- the amount of effort required to reach native-level fluency and polish (i.e., at the level of your critique, above) is herculean.
To cite just one example, redis is widely considered excellent software, but much of its documentation is written in a very imperfect English.
They should definitely do some editing. The grammatical errors are kind of off putting, I totally agree. That said there's a very clear value prop/use case and clear, concise documentation. You may be getting downvoted because your language was rather harsh.
Раз ты так хорошо шаришь в английской грамматике, и тебя так напрягает нехватка запятых и дефисов, подскажи как лучше сформулировать и "пруфридать" вот это самый комментарий, или... у тебя эта самая пустота в одной из частей твоего, грамматически заточенного мозга (там, кстати, дефис можно поставить) мешает тебе справиться с задачей?
в следующий раз (кстати с заглавной буквы можно начать) перед тем как насилывать свою клаву, лучше пойди на подкурсы русского или китайского или хотя бы того же английского.
очень не сложно оставлять мусорные комментарии, не относящиеся кстати к теме разговора, и называть себя викинг кодером. гораздо сложнее соответствовать своему имени, и "викинговать" архитектуру, код, мысли, философию создания.
если бы в интернете и/или на хакер ньюз была бы общая корзина для мусора, и можно было бы голосовать за вещи которые туда отправляются, можешь расчитывать на мой голос.
hey VikingCoder, I heard you are a fan of grammar? see above? grammar that.
Crude automatic translation from Google Translate:
Since you're so good rummage in English grammar, and you so annoying lack of commas and hyphens, tell me how to better articulate and "prufridat" here is the comment, or ... you have this same emptiness in one part of your, grammatically sharpened the brain (where, incidentally, you can put a hyphen) prevents you meet the challenge? next time (by the way with a capital letter, you can start) before its nasilyvat Claudia, better go to the podkursy Russian or Chinese, or at least the same English. not very difficult to leave garbage comments not related to the topic of conversation by the way, and call themselves Viking encoder. much more difficult to live up to its name, and "vikingovat" architecture, code, thoughts, philosophy of creation. if the Internet and / or a hacker would Neuse total waste basket, and it was possible to vote for things that go there, you can count on my vote.
Then, in English:
> ... see above? grammar that.
First time I've heard "grammar" used as a verb. :)
It's like none of you people have ever gone through a code review. Cripes.
Check out this post: https://news.ycombinator.com/item?id=6314628
I know my spoken and written english is far from perfect, but I can live with it.
> rethinking primary alghorithmical [sic, should say "algorithm"] constraints
"Algorithm" isn't an adjective. I accept "algorithmical", although it should probably be "algorithmic".
Reread the word, "alghorithmical". It's completely misspelled.
Memory isn't an adjective either, but the phrase "rethinking primary memory constraints" strikes me as perfectly valid. That's why I concluded that "algorithm" would be the most clear word. But yes, I think "algorithmic" would have been fine as well.
Unfortunately the chosen font renders very poorly (at least for me in chrome).
document.getElementById("main").style.fontFamily = "sans-serif";
Anyone know anything about how this compares in practice to Lightning MDB (which uses a memory mapped B-tree, I think, and is apparently insanely faster than most of the other enbedded key-value stores people normally examine)?
This one copies, and it has no concept of transactions from the looks of it (not even LevelDB-style snapshots)
Having just gone through the exercise of picking an embedded key-value store for a project, some things that would be nice: how does it compare to other things besides leveldb (which, to be frank, isn't a stellar performer)? In particular, how does it compare to Tokyo/Kyoto Cabinet, Lightning MDB, or Sqlite4's LSM? Does it support data compression (either with a single pre-selected algorithm like LevelDB and Snappy, or in pluggable fashion like LSM)? How does it deal with concurrent access by multiple processes?
I've gone through picking embedded key-value store, too.
What really concerns me is why never in benchmarks they perform on already filled database (like, 14G, 28G, 60G)? Because "add 100k random keys into an empty database" is very different from "add 100k random keys into a large database". And that is where more novel algorythms start to shine.
Yes, read speed of leveldb (and, i assume, sophia) with its fancy sst's is lower than of plain old b-trees or hashtables (kctree/kchash), but it is still high enough for most tasks. Write performance of kc* (and btree-based libraries in general) is, however, unacceptable, at least on hard drives, and even with a reasonable-sized database (~90% of RAM) it degrades to a couple of random write per IOPs (so, 200-300 writes per second on a consumer-grade HDD, or up to 1000 on a 2x10k sas hdd in raid-0, if i remember correctly)
It may be reasonable to use kc* on SSD, but i did not test that.
For Java / Android, I've been using H2's MVStore, which is log structured and uses counted B+-trees. It's nice not having to go through JNI for good performance in Java.
Funny - H2's slowness (either with a standard storage system or with MVStore and a standard key format) is the main reason we're moving back to a hand-rolled data storage system that's specific for our data on Android.
Thomas Mueller, the author of H2 / MVStore, gives some thoughts on H2's performance issues under Android here if you are interested:
https://groups.google.com/forum/#!topic/h2-database/Q8K-nbCh...
Yeah, I saw that. The main problem for us is that we need on-disk encryption due to regulatory issues and the encrypted SQLCipher build was causing us no end of grief. And we don't really need a SQL database, it's just what the developers of our iOS app were doing and--it being our first Android mobile app--we thought it was a good idea to do the same. V2 is ripping that out both for perf and for code-sanity reasons.
That looks awesome. Thanks for pointing it out.
Typography is hard to read.
Seconded: that font at that size strains the eyes a fair bit.
It looks great with javascript disabled, maybe the font in the examples was a mite small.
I turned on javascritp and it looked a lot like a man-page.
Cute website. Some random thoughts:
Concurrency:
- No mention of it. There appear to be spin locks in the source. No multi-threaded tests.
Stability and data safety:
- Github has 2 days of history, and 4kLoC of test code. Why should I trust my data to you?
"high load environment":
- So what exactly does it do in a "high load environment"? How do you define "high load" in the first place? CPU load? I/O load from other processes? What shortcomings of the competition under a "high load environment" are you trying overcome?
Backup:
- How do I do hot backup?
Benchmark:
- LevelDB is not a fair comparison as it offers additional non-trivial functionality (snapshots) that cannot be built up on top of Sophia. LevelDB APIs are also safe for concurrent use, which adds overhead. Kyoto Cabinet would have been more suitable as a peer to benchmark with.
- 3 million records with 16-byte keys and 100-byte values is not really an interesting benchmark dataset.
- Iteration over a static database is not interesting, either. Is there any alternative other than locking an entire mutating database for the duration of iteration?
I, for one, wouldn't trust my data to a library by somebody who uses the same text decoration for hyperlinks as for plain text.
Computer scientists aren't known for their design chops. I'd take that as a sign of authenticity.
REAL computer scientists do not use CSS. Bonus points if they use FONT tags (in caps of course!).
Really guys, can you give more constructive or at least more (not based in bullshit assumptions) comments? if not, just shut up..
This is a non-trivial effort, and all people do is to complain about the font face or if the punctuation was right?
First, in the benchmarks it just crush leveldb, this is already by itself a great achievement. can you confront the benchmarks? you do it one yourself with a different configuration? no?
Second, if you are not a database expert and can create proper critics (constructive or not), just keep it to yourself.. i wonder how so many people get up with all of this conclusions so fast, without a proper look at the source code and to have a reasonable amount of time to know what are they talking about.
its very hard to create things like this, but very easy to critisize without any background.. dont forget about it
if you have something to say about a small thing, that do not have a direct relation to the product or thing itself, if theres already one comment about it, that enough! do not spam, answering it, or creating new comments about it, this is just so rude and unrespectful..
really, things are getting creepy on HN.. and its not only in this thread
You: Making the documentation readable and easy to parse adds no value to projects! Everyone who disagrees should shut up.
If I'm being kind to you, HN commenters (myself included) should do a better job of commenting politely, and spend more effort making sure their criticism comes off as constructive rather than just whining and aggressive... ...but I think you make it sound like criticism of anything outside of the source code itself is creepy, rude, and disrespectful.
It says that the benchmark source is on github, but I can't find it.
It doesn't appear to be in their primary repo.
I would like to try and do my own test against another embedded data store like Berkeley DB but I want to know more about the conditions on the test. How many threads were used, that kind of thing.
Very cool. As a suggestion, increase the link size under the main title, wasn't clear to me what the next step was at first after reading the introduction text.
In case any of the devs read this:
1. Can multiple processes use the same database concurrently? (Separate address space processes, not fork()'d)
2. Have you tested this with uClib/cross compiler? (I would like to use it on a MIPs embedded router)
The reason I ask this is because I recently had the displeasure of having to hack a non-volatile RAM library to work with shared memory / thread safe and something small like this would be a perfect replacement with a lot less pain.
Was this man's computer use being charged by the key stroke? I mean I understand using a few abbreviations here and there, but i.c ? At least name your files descriptively.
I would avoid using this for realzies if only for the fact that if something broke trying to fix it in that code base would be prohibitive
Looks nice, but note this doesn't appear to support consistent reads (unlike LevelDB snapshots)
I love the simplicity of the site and the C code. I will definitely use it, thank you.
I've got a need for something like this, but would like to have the keys and values delta encoded to achieve simple, yet effective, compression.
How about http://github.com/Tokutek/ft-index? It's embeddable (BDB-like API), has compression built in, and is a similar data structure to this but with more mature features like transactions.
Had a quick, 30 min bash at a win32 port using msinttypes and pthread-win32 but no luck yet :( Would love to see one though...
Something for Mozilla to consider using inside Firefox inplace of LevelDB ( If that was ever landed )
I'd be far more interested in benchmarks versus BDB (and maybe even sqlite).
Snapshots? I could find it in the documentation..
any word on support for unicode keys?
Keys are arbitrary data (you pass in pointer to data and length in bytes), so you can use anything that makes sense for you as keys.
Awesome work, congratulations!