Show HN: ObjectiveFS – Shared file system for EC2 and laptop
objectivefs.comHow is different from s3ql? https://github.com/s3ql/main
Edit: Support for concurrent mounts seems to be the main difference https://bitbucket.org/nikratio/s3ql/wiki/FAQ#!can-i-access-a...
S3QL has a lot of great features. You are right that support for concurrent mounts is one of the main differences. ObjectiveFS also keeps the file system data consistent and stores it in S3 asap, for sharing and to handle EC2 unexpected termination. S3QL uploads all metadata changes at a regular interval (by default every 24h) or during unmount.
Rough guess on the design: The main metadata history is in a primary log, then each client extends that with its own branch of the history for filesystem operations. Logs are probably some kind of Merkle tree. Bulk data is content addressed and referenced from the logs, maybe with large objects split using a rolling checksum like Adler-32 to reduce object sizes and allow for partial updates to large objects. Someone (you guys?) runs a coordination service routes notifications between clients and manages leader election. The elected client reads all of the logs, does conflict resolution locally, and then updates the primary log and collects garbage.
One drawback to this design would be that many small files in the filesystem would translate to many small objects in S3 (with associated operations). One solution would be to put small objects right into the metadata log. Alternately they (or maybe all objects) could be put into a log-structured merge tree.
Another problem with this design is that S3 doesn't support append operations so sync latency would be bounded by client log flush intervals, again creating lots of small objects. Maybe the coordination service routes some of the data to manage this?
Anyway, really interesting design problem. Is this close?
ObjectiveFS does all the coordination among the clients through S3, so there is no extra coordination service needed. This is why we are really happy that Amazon recently moved the S3 us-east-1 region to read-after-write consistency (like all their other regions).
We do write bundling before sending data to S3 so lots of small files would be packed together and stored in a single S3 object. This also helps reduce the number of object store operations.
You are absolutely right that frequent sync will necessarily create many small objects, which is why small objects will be combined into bigger ones (this compaction is done in the background). Sync latency is of course bounded by the S3 PUT time, since fsync(2) can only return after your data has been safely stored in S3.
It is a really interesting design problem. Thanks for sharing your ideas.
Awesome project!
It might just be a release timing thing, but what's the advantage of this over Amazon Elastic File System [1]? I see you've got a tagline on your signup No need to wait for EFS, are you planning a feature comparison or similar?
[1] https://aws.amazon.com/efs/
edit: derp. Turns out I should have read further!
How is ObjectiveFS different from Amazon EFS?
ObjectiveFS uses highly durable object stores (Amazon S3 and Google Cloud Storage) to store your files. Your data is accessible from all regions, your office and your laptops. Performance scales with your object store.
ObjectiveFS supports Linux and OS X and is running in production today.
Amazon EFS is NFS-based, with access to each file system limited to EC2 instances in the same region. It is not available today and will only be in limited preview for some customers in summer 2015. It will cost $0.30/GB and performance will scale with the number of GB stored.
https://objectivefs.com/faq#how-is-objectivefs-different-fro...
Thanks! We plan to publish a feature comparison table soon. A couple more differences:
1. We have strong end-to-end integrity checks and encryption on by default.
2. Amazon EFS's performance is dependent on the filesystem size: larger filesystems include more iops.
We built a shared file system with an S3 backend. We've been running our own servers for many years, and wanted something low maintenance, reliable, scalable and secure. So, we built this POSIX-compliant shared filesystem for Linux and Mac.
We love to hear your feedback. Feel free to try it out. We will also be here to answer questions.
How is this different from say mounting s3 as a volume with fuse ?
It's a complete POSIX filesystem, and we implement a log-structured filesystem on top of S3. So, you can use it with your regular programs expecting POSIX semantics.
Mounting S3 lets you view the objects in a bucket, but it doesn't behave like a regular file system.
https://github.com/s3fs-fuse/s3fs-fuse
"large subset of POSIX including reading/writing files, directories, symlinks, mode, uid/gid, and extended attributes"
s3fs-fuse is a one-to-one mapping and doesn't do things like atomic rename of directories, mutual exclusion of open exclusive, append to file requires rewriting the whole file and no hard links. This breaks the expectations of many linux programs.
Our implementation has these features and works with linux programs.
It's difficult to tell from your site - does it still use FUSE under the hood or is this a kernel-level FS? I'd check the RPMs if I could, but I can't download the RPM without signing up first.
It uses FUSE under the hood. Thanks for the feedback, we will make it clearer on the website.
judging from https://objectivefs.com/features
sounds like maybe they're just packaging s3fs-fuse and then charging for it "as a service"? they're extremely light on technical details.
Hi Ben, thanks for your feedback. We will add more technical details and clarify the differences between objectivefs and s3fs-fuse.
ObjectiveFS is a log structured filesystem that we implemented completely from scratch. It is a POSIX filesystem and works with the existing Linux and OS X tools and software.
This would have been nearly perfect for me if it had supported disconnected operation with later synchronization (~ Dropbox, CODA, ..). Alas, it's a live fs. (Dropbox is great except it doesn't encrypt locally and has to have a local copy of everything, unlike a cache).
No compression?
ObjectiveFS comes with fast compression that is always enabled, it works great for log files and other compressible data. Let us know what you think if you give it a try.
Thanks. It would be nice to put on a feature list along with some details.
website is slow as shit
Thanks for letting me know. We are getting a lot of traffic, and working on fixing it.
UPDATE: the website should be faster now.