Manage FreeBSD jails backup with NocoDB

To manage my jails I use bastille as wrapper and until then I used a simple shell script that backed up all my jails every week into a NFS Share (with various copy on externals devices). Something solid and robust that run sinces years.

However, I'm starting to have a significant number of jails (more than thirthy), on the same machine. And some jails doesn't need to be necessary backed up.
Backup everything and keep multiple copies consummes ressources / time / and a lot of space.
Also my backup script is pretty simple it loop over all the jails with jail bastille list -a and backup everything.
No exclusion, no filter and I do not want to create exclude or edit the script that basically works just to avoid to backup 1 or 2 jail.

Always on the lookout for the most useless technology. A few months ago, I deployed a nocodb jail without finding any real use case. Just because, you know, it's cool and trendy and might be useful one day.
NocoDB is basically an Airtable clone-like, a visual Database that allows you to manipulate your data with a nice UI and also through a REST API and others libraries for cool Javascript developers.

And recently I found a pretty interesting use case for real life. And ho my ... this is really cool !

note General use case
I use nocodb and jail for the example. But idea will be the same with LxC and another Airtable clone. So if your a, IT beginner and run Bloated Linux it should work the same.

Install nocodb : #

You can follow this procedure how to deploy nocodb into a Jail.
On my side I don’t even install/use postgresql.
By default il will work with a sqlite instance which feet perfectly for a test/small use case.

Why create / insert jails data in into a database ? #

Because things become so easy to manipulate without changing you backup tooling chain and changing your variable.

Let’s populate a table named Jails with revelant informations.

Here a example of my table “Jails” :

Now , let’s say I don’t want to backup my jail test, bytestach. In this situation I just have to request with curl like this :

Here an example. I want to list the “Name” of the jails where Backup equal YES :

curl -X 'GET' \
  'http://X.X.X.X:8081/api/v2/tables/mg8ouwhcuru18ui/records?viewId=vwb0xge1l6by8zdf&fields=Name&where=where%3D%28Backup%2Ceq%2CYES%29&limit=25&shuffle=1&offset=0' \
  -H 'accept: application/json' \
  -H 'xc-token: MY-SECRET-TOKEN'

note Use the swagger :
Create curl request can be painfull but hopefully nocodb provides a Swagger UI 💖 to help you to build you request with cURL.

Output : It will print only the jail I want to backup.

{
  "list": [
    {
      "Name": "vaultwarden"
    },
    {
      "Name": "site-marie"
    },
    {
      "Name": "adguard"
    },
    ...

Then I can extract the name and inject this into a bash variable.

my_list=$(curl -sX 'GET' \
  'http://X.X.X.X:8081/api/v2/tables/mg8ouwhcuru18ui/records?viewId=vwb0xge1l6by8zdf&fields=Name&where=where%3D%28Backup%2Ceq%2CYES%29&limit=25&shuffle=1&offset=0' \
  -H 'accept: application/json' \
  -H 'xc-token: MY_SECRET_TOKEN' |  jq -r '.list[].Name')echo $my_list
nextcloud
authelia
silverbullet
nginx-web
shiori
nextcloud-mlfa
adguard
navidrome

An finally I just have to iterate with a loop on this list

for jail in $my_list; do
  bastille stop $jail
  # I do not use ZFS export so I have to stop the jail
  bastille export -tgz $jail /mnt/nfs
  bastille start $jail
done

How cool is it ? 🤠

Furthemore there is some jails that I prefer to backup through S3 with rclone because that modern and I like fancy stuff with high probabily of datalost 🃏
It couldn't be simpler, I just have to add another collumn let say “Type of backup” with 2 choices, FileSystem or S3.

Let just create the good request Select Jail where backup equal YES AND Type of Backup eq S3

curl -X 'GET' \
  'http://X.X.X.X:8081/api/v2/tables/mg8ouwhcuru18ui/records?viewId=vwb0xge1l6by8zdf&fields=Name&where=where%3D%28Backup%2Ceq%2CYES%29~and%28Type%20of%20Backup%2Ceq%2CS3%29&limit=25&shuffle=1&offset=0' \
  -H 'accept: application/json' \
  -H 'xc-token: MY_SECRET_TOKEN' |  jq -r '.list[].Name'# Output 
authelia
mariadb

God I love when I found idea like this 🦁 🤟

note Bastille only export to LocalFS.
In fact this is not reallty usefull because Bastille does not permit to stream the backup into the rclone STDin ... even this could be easily doable. I’ve proposed this change on the github repo but the devs who maintain bastille considered that it was out of scope and that the only way to export was through a POSIX filesystem, and refused the improvement 🤕
Consequently for now with bastille you have to first export you jail locally then push it with rclone/s3/s5cmd ... into your bucket, this is a 2 step operation not very efficient. But this is an example for the use case ...

Update the last backup time #

Even better, nocodb is a database and you can obviously update this content with the cURL api :
So let’s create another collumn named randomly “last-backup” typed as “date” DD-MM-YYYY .

So after each backup we can easy put the date of the backup into the database 🤖
Like this will have a view on the backup activity just by looking the table.

Apparently it seems you have to know the Id (primary key) of the record to upgrade a line :

# fetch the id of the jail named "nextcloud" into a variablemy_id=$(curl -sX 'GET' \
  "http://X.X.X.X:8081/api/v2/tables/mg8ouwhcuru18ui/records?viewId=vwb0xge1l6by8zdf&fields=Id&where=where%3D%28Name%2Ceq%2Cnextcloud%29&limit=25&shuffle=1&offset=0" \
  -H 'accept: application/json' \
  -H 'xc-token: MY_SECRET_TOKEN' | jq -r '.list[].Id')
echo $my_id
9

Now update the DATE field for the nextcloud jail :

now=$(date -u +"%Y-%m-%d %H:%M:%S+00:00")#Date format must be :
{
  "Id": 9,
  "Last Backup": "2023-10-19 08:56:32+00:00"
}
# Update the table 
curl -X 'PATCH' \
  'http://X.X.X.X:8081/api/v2/tables/mg8ouwhcuru18ui/records' \
  -H 'accept: application/json' \
  -H 'xc-token: MY_SECRET_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{
      "Id": '"$my_id"',
      "Last Backup": "'"$now"'"
      }'

Last backup column is updated.

Of couse this is a pretty simple example and the script will have to do more test and control.
But this is the main idea.

As usual : sky is almost the limit : #

Or perhaps I should say, how much time you want to spend on the project ...

You can imagine all kind of idea. Let’s say I use a range of 100 IP for my jail and each time I have want to create a new one I have to check if the IP is used or not ... Or could be interessting to create a new table or a new column to have the list of the port already used by a software.

I'll let you figure out the possibilities.