GORM is arguably the most popular golang ORM library [1]. As an ORM, it can speed up development by eliminating repetitive coding tasks related to the interaction between your application and your database. However, database migrations can still prove to be a challenge.
When using GORM, you often need to manually identify, write & execute your database migrations. This is because the included automigration functionality will only help you with additive changes: creating tables, missing foreign keys, constraints, columns and indexes. You still need to manually define the migration steps for the destructive changes, such as removing unused columns, modifying existing constraints or changing existing primary keys. Manually writing migration steps leaves room for oversights - something I’ve experienced firsthand.
Luckily, these oversights can be prevented with Liquibase!
Today I Learned: How GORM migration oversights can be spotted using Liquibase’s diff command.
In this blogpost, I’ll show you how so you won’t make the same mistakes as me. If I would’ve known of this tool earlier, I would’ve saved quite some time spent on MR reviews and prevented a number of bugs.
Liquibase
Liquibase is a rich open-source database schema management tool [2]. It can help you implement and automate version-based database migration for your application. In this blogpost, we’ll look at a small subset of its functionality: the diff command.
The diff command in Liquibase allows you to compare two databases. This command is typically used at the completion of a project to verify all expected changes are in the changelog or to detect drift between a model schema and a database’s actual schema [3].
Theory: How to find oversights?
Here’s how we can use the diff command with GORM to verify that our migrations went correctly, we can:
1. Boot up an empty database,
2. Initialize the empty database with the new version of the schema directly*
3. Boot up a second database with the current version of the schema,
4. Migrate the second database to the new version of the schema with the manually written migrations….
5. … and compare the differences!
Press enter or click to view image in full size
*: (which can be done with GORM’s automigration functionality, as migrating from an empty database means all changes are only additive and can be done automatically)
Practice: Let’s find some oversights!
Now, let’s look at an example. This example is based on actual schema divergence cases we’ve encountered. Consider the following two versions of a database schema for a library inventory management application.
Entity-Relation Diagrams:
Press enter or click to view image in full size
Go code:
Press enter or click to view image in full size
We can use Liquibase to check the if there is a difference between the difference between the expected schema and the actual database schema.
Command:
liquibase diff --url="jdbc:postgresql://localhost:15431/mydb" --username=postgres --password=password --referenceUrl="jdbc:postgresql://localhost:5431/mydb" --referenceUsername=postgres --referencePassword=password --default-schema-name=library --reference-default-schema-name=librarybNow image that somebody has completely forgotten to write the required manual migrations and uses only automigrate. Then, we’ll get the following result.
Output:
liquibase diff --url="jdbc:postgresql://localhost:15431/mydb" --username=postgres --password=password --referenceUrl="jdbc:postgresql://localhost:5431/mydb" --referenceUsername=postgres --referencePassword=password --default-schema-name=library --reference-default-schema-name=library
\####################################################
## _ _ _ _ ##
## | | (_) (_) | ##
## | | _ __ _ _ _ _| |__ __ _ ___ ___ ##
## | | | |/ _` | | | | | '_ \ / _` / __|/ _ \ ##
## | |___| | (_| | |_| | | |_) | (_| \__ \ __/ ##
## \_____/_|\__, |\__,_|_|_.__/ \__,_|___/\___| ##
## | | ##
## |_| ##
## ##
## Get documentation at docs.liquibase.com ##
## Get certified courses at learn.liquibase.com ##
## ##
####################################################
Starting Liquibase at 11:30:03 (version 4.23.1 #12042 built at 2023-08-10 13:48+0000)
Liquibase Version: 4.23.1
Liquibase Open Source 4.23.1 by Liquibase
Diff Results:
Reference Database: postgres @ jdbc:postgresql://localhost:5431/mydb (Default Schema: library)
Comparison Database: postgres @ jdbc:postgresql://localhost:15431/mydb (Default Schema: library)
Compared Schemas: library
Product Name: EQUAL
Product Version: EQUAL
Missing Catalog(s): NONE
Unexpected Catalog(s): NONE
Changed Catalog(s): NONE
Missing Column(s): NONE
Unexpected Column(s): NONE
Changed Column(s):
library.books.author_id
order changed from '5' to '4'
library.books.description
order changed from '3' to '5'
library.migrations.id
type changed from 'varchar(255 BYTE)' to 'text'
library.books.publication_year
nullable changed from 'true' to 'false'
order changed from '4' to '3'
library.online_profiles.website
nullable changed from 'false' to 'true'
Missing Foreign Key(s): NONE
Unexpected Foreign Key(s): NONE
Changed Foreign Key(s): NONE
Missing Index(s): NONE
Unexpected Index(s): NONE
Changed Index(s):
online_profiles_pkey UNIQUE ON library.online_profiles(id, website)
columns changed from '[library.online_profiles.id, library.online_profiles.website]' to '[library.online_profiles.id]'
Missing Primary Key(s): NONE
Unexpected Primary Key(s): NONE
Changed Primary Key(s):
online_profiles_pkey on library.online_profiles(id, website)
columns changed from '[library.online_profiles.id, library.online_profiles.website]' to '[library.online_profiles.id]'
Missing Schema(s): NONE
Unexpected Schema(s): NONE
Changed Schema(s): NONE
Missing Sequence(s): NONE
Unexpected Sequence(s): NONE
Changed Sequence(s): NONE
Missing Table(s): NONE
Unexpected Table(s): NONE
Changed Table(s): NONE
Missing Unique Constraint(s):
online_profiles_id_key on online_profiles(id)
Unexpected Unique Constraint(s): NONE
Changed Unique Constraint(s): NONE
Missing View(s): NONE
Unexpected View(s): NONE
Changed View(s): NONE
Liquibase command 'diff' was executed successfully.This would’ve been nasty in production!
The example we’ve shown is exaggerated, but oversights do happen. That’s why Liquibase’s Diff command is handy to know when using GORM: it makes it almost trivial to check for schema divergences after those tricky manual database migrations. We also get the information required to fix our migration scripts.
Conclusion
In this blogpost, we’ve shown how GORM migration oversights can be spotted using Liquibase’s diff command.
Are you also writing manual migrations, and do you want to be sure that you’ve not forgotten something? Or might have forgotten something in the past? Try it out on your database, and see what the results are! :)
But wait, there’s more!
If you’re fully enchanted by Liquibase, you can also use it to it’s full potential and let it generate & execute the migrations automatically.
Some other popular tools for schema management & migrations are: