Five years ago, I thought a database library needed thousands of lines of code. Today, mine needs about 200.
Have you ever heard of encode/databases?
It's one of my favorite Python libraries. It gives you an asynchronous interface
for database connections. A much nicer wrapper over database drivers such as
asyncmy, aiosqlite or asyncpg. It gave you a single interface that managed
connections, a connection pool, transactions and common database operations. And
it's great! It used SQLAlchemy Core to compile queries, so you didn't have to
write raw SQL. It gave you a force_rollback mode that would roll back database
changes instead of committing them, which is really handy for tests. And it gave
you a very small API with great naming, that just made sense.
Later, SQLAlchemy gained native asyncio support in version 1.4, which was
further developed in version 2.0. Suddenly, there was much less reason for
databases to exist. The library predates that. If you read the source code,
you will see that it interacted with drivers directly, had multiple layers of
abstraction for connection pools, for connection backends, for sessions, for
transaction semantics and for driver quirks. The code was complex, but it had to
be that way because of history. I contributed to it. I was fairly familiar with
its internals. I helped bring SQLAlchemy 2 support to it, as the next major
version broke query compilation machinery used by the library. I also worked on
psycopg3 support in the library, although I don't remember if I ever pushed my
work on that. The abstractions were horrendous, but I loved the library
nevertheless.
And then it was archived.
I don't like SQLAlchemy...
As I mentioned earlier, the latest improvements and additions to SQLAlchemy
made much of databases' implementation redundant. You get async everything -
pools, connections, transactions, operations, etc. But my main gripes with
SQLAlchemy remained. You still have to deal with SQLAlchemy's historically silly
naming. You still get a mysterious SQLAlchemy object when you fetch rows. And
you still get tons of various parameters and knobs with each method that aren't
necessarily discoverable.
Naming is just plain bad. begin(), if called on an engine, gets a connection
from the pool and starts an outer transaction. begin(), if called on a
connection, only starts an outer transaction. begin_nested(), if called on a
connection, starts a nested transaction, a SAVEPOINT, if you will. You use
create_async_engine() when you are creating a connection pool and setting a
dialect.
Notice how each method uses underscores when there's more than one word in the
name? Well, enter fetchone(), fetchall() and fetchmany(). I know that it
comes from Python's DB-API conventions, but I don't see much value in that for
my use cases. I feel like addressing these issues wouldn't be too bad with the
introduction of version 2 of the library, but it's not what happened.
I don't like Row objects. Don't get me wrong. They do make perfect sense. You
wouldn't want to allocate a dictionary that keeps column names per row if there
are hundreds of rows. But most of the time I just want a dictionary. I want a
simple object that will be understood by code that has nothing to do with
SQLAlchemy. Sure, you can use result.mappings() and convert each row to a
dictionary. But you have to do that every single damn time.
And I don't like knobs! I hate knobs! The more configuration options are exposed, the more decisions a developer has to make. And even if I intend to use the defaults, that should be sensible, I still can't fight the urge to read about each of them, study them and decide, if those defaults are sensible to me personally in this specific project I am working on. And I get frustrated when I can't decide.
databases deserved to live
So I revived it. I didn't fork it though. Have you read the monstrosity of a
source code it is? As I mentioned earlier, it absolutely had to be like this,
because mature asyncio support in SQLAlchemy came much later than databases.
But the explanation doesn't make the code easy to work with.
And I revived it twice. My first attempt, called based (now deleted from
everywhere during my yet another psychosis), supported asyncmy, psycopg3 and
aiosqlite. It too had the abstraction hell databases had, but I felt like I
managed to do it much better with all the hindsight I had and previous work on
databases. I wrote an extensive test suite. It was simply a great library.
I really don't remember why I decided to nuke it, but when I started a new
Python project, I was really saddened that I did. So I built it again.
...but I do respect SQLAlchemy
I called the next iteration databasic.
Initially, it only supported psycopg3 and wasn't ready to support any other
database drivers, so it was just 250 lines of code long. I still used SQLAlchemy
to compile queries though. It's just too good. It's much safer and much more
ergonomic than writing queries by hand and implementing my own query builder is
too massive an undertaking.
I wrote tests, it worked fine, I used it and I liked it. But then I read its source code for the hundredth time and I thought "does it really have to be like this?" and opened SQLAlchemy docs. As it turns out - no, it absolutely does not need to be like this. Asyncio support in SQLAlchemy was mature enough. It already supported all the async database drivers I cared about. It already handled query compilation differences between drivers. And, most importantly, I wouldn't need to invoke SQLAlchemy's query compilation machinery myself, which partly depended on private methods and fields that could break any moment.
So I just replaced all of my driver and pool interactions with SQLAlchemy's
AsyncEngine. And further reduced line count down to about 200 lines. And it
made me so much happier - after all, the best code is always the code that was
never written.
So why is it so small?
Because the API is small. To be blunt, all it does is rename methods and remove knobs. It took me multiple years of experience to know what exactly I need from a database connection manager and to not be afraid to remove everything that does not go to that list. I decided that the thing that will differentiate my library from SQLAlchemy will be that it will have just a handful of methods and nearly no opportunities to make decisions.
My idea was that nobody should have to learn a library to make a query. And it shows. If you go see the README, you will notice that there's nearly nothing to learn. It's as straightforward as it gets. Methods don't have any docstrings, because their signatures explain so much that there's little left to document. And it makes me really proud.
async with Databasic(url) as db:
async with db.session() as session:
async for row in session.stream(query):
print(row)Okay, so what?
So go build your own tiny wrapper! I don't think anyone in the world uses my library, as this is the first time I'm telling anyone about it, but building it taught me two things I want to communicate.
First, your library doesn't have to be novel, explore new ideas or enable users to do new things that were too difficult before. It just has to scratch your itch, no matter how silly, uncreative or basic it appears to you.
Second, sometimes knowing which capabilities to remove, reduce or refuse is a sign of experience and a deep understanding of the domain. Fewer decisions can mean more time spent solving the actual problem.