Settings

Theme

HSM/Archiver AGPL for Disk,Tape,Cloud Written in Rust

huskhoard.com

2 points by ticklyjunk 13 days ago · 1 comment

Reader

ticklyjunkOP 13 days ago

I built this arvhiver because i was maxing out my drives and i wanted to be better at compressing and managing files without the constant headache of manual archives. it is basically a hybrid user space storage kernel written in rust that makes cold storage like tapes, old hard drives, or s3 feel like a transparent extension of your local ssd. when your ssd hits a high water mark huskhoard moves cold files to n way replicas but keeps the file entry visible in your directory using hole punching with fallocate. if a process tries to read a file that has been moved the software intercepts the call using the linux fanotify api and restores the blocks in the background. unlike fuse this does not add overhead to your active files because it only wakes up when a stubbed file is accessed. i included native scsi tape support using ioctl and mtio commands and there is an http gateway so you can stream media directly from a tape to vlc or plex without writing the full file back to your drive first. cloud tiering is handled via rclone so you can use s3 or backblaze as a backend too. every block is blake3 hashed and zstd compressed and the catalog is stored in sqlite but i made it so you can rebuild the whole thing by deep scanning the raw metadata headers on the media if you lose the database. i chose rust because i needed memory safety for the low level pointer stuff with o direct block alignment. i am looking for feedback on the fanotify implementation especially regarding inode locks during restores and i would love to hear from anyone else running physical tape or SMR drives. github.com/huskhoard/huskhoard there are a few more technical notes in the blog huskhoard.com/blog.html

Keyboard Shortcuts

j
Next item
k
Previous item
o / Enter
Open selected item
?
Show this help
Esc
Close modal / clear selection