Build your own private, encrypted, open-source Dropbox clone
gist.github.comOr, you know, just use Dropbox. And add encryption on top with these two commands:
mkdir ~/Dropbox/_ ~/Dropsafe
encfs ~/Dropbox/_ ~/Dropsafe
Interesting, does this allow for incremental updates while encrypted?
Encfs is pretty opaque -- afaik modifying a file will at least change an entire block, possibly the entire file. Two identical files will look completely different (per file IVs) -- so no deduplicaton is possible.
But adding and removing files works as expected, so adding a few files to the encfs system, should translate to the same number of files added to the underlying folder.
EncFS allows to deactivate some of these features at the price of reduced security. The EncFS manpage [1] has a good explanation of all the tunables. If you want to use Boxcryptor, you're even advised to deactivate most of these features [2].
[1] http://linux.die.net/man/1/encfs
[2] https://boxcryptor.desk.com/customer/portal/articles/565934
Yes, but you still have to install a closed-source Dropbox client in your system.
Nice guide. One nitpick:
> Add a user for our project and give him a decent password:
Better off not giving the encbox user a password at all and only allow SSH key based login. You can already login to the primary/root account and sudo/su to setup the encbox user and copy SSH keys.
I am using Owncloud, super easy to install[1]. I have a non-us/eu vps storage solution, despite reading others having problem with it, its working great for me.
[1] http://www.slashgeek.net/2013/05/16/host-your-own-dropbox-li...
I don't know why this doesn't end up in every thread, but Synology's inexpensive NAS systems have a "be your own cloud" feature built in, with corresponding iOS, Windows, Mac, Linux (I think) and android apps. All in a little box you can keep in your house.
I'd never heard of Backupsy before, looks very cool!
What is special about their VPS offering that makes it so cheap? I mean, from what I can see for an extra $2/month they let you host websites and install anything you want (except torrents, TOR or anything illegal).
I've been wanting to have an affordable VPS solution so I could be host my own stuff, and have the freedom to experiment with various development tools, and this might be a decent deal.
There is nothing particularly special about Backupsy as far as I can see. Just cheap simple servers with a bunch of disks in them. This type of hardware is very cheap. The software layer is equally simple - seems like just KVM and a dashboard, nothing fancy. So then low cost, low price right. Good combo.
However as you can see from the SLA (99.9%), you do have to pay something in the form of reduced redundancy and availability. If, for example, an HP RAID card freaks out and all the VM's get corrupted on your blade, well you are out of luck. That data is gone forever. It's a small risk, but it is a real one (in fact with the P410 they are using, that's actually happened to me before in production). There are lots of other things that can go wrong too. Virtualization is not all rainbows and unicorns.
In the end I think it's still a good deal if you need a backup target with phat storage in it. But if you are interested in just trying out stuff, I think Digital Ocean is a better call since they have a higher SLA (99.99%) and probably a better dashboard and it's around the same price. Backupsy, as the name implies, is made for backups.
DigitalOcean is not really comparable, they only give you 20 GB of disk. I can't use the 250 GB Backupsy gives you, but 100 GB for $5 or 50 for $3, I would sign up yesterday.
If you're fine with OpenVZ and being in the USA, RamNode (http://www.ramnode.com/index.php) offer 50gb for 2$/month and 120gb for 7.50$/month (And it's ssd-cached space). You can even use a coupon like "leb35' to get 35% discount forever.
Note: Not affiliate in any way with RamNode, just an happy client.
Wow, $30 a year for 90 GB, that's amazing, thank you. You've been using them and you're satisfied? Basically, all I care about at the moment is that they don't silently corrupt my data, as it's backups.
How did you get that price for that much disk space? I'm on the RamNode website and can't find a plan that matches.
I went here:
https://clientarea.ramnode.com/cart.php?gid=11
And got the 256 RAM one, it has 90 GB space. Then I used the leb35 code the GP gave me, and it comes out to $29.83 yearly.
Don't reuse the keys, just generate a different key on each client and add them all to authorized_keys
Good point, I fixed that.
After finding Backupsy (and RamNode, which a commenter suggested), I've found the holy grail of backup solutions. I will write a simple backup script that mounts a remote directory (through SSH or whatever), mounts an EncFS directory on that and rdiff-backups files onto it.
This will give you encrypted, snapshotted backups using open-source systems, that are better than Duplicity because you don't have to be creating full backups every so often. If you'd be interested, you can subscribe to my mailing list at http://bit.ly/stavroslist or follow me on Twitter or something, but please post comments below so I know what functionality to include.
I just use Tarsnap for this scenario: http://www.tarsnap.com/
What I am looking for is a folder that syncs, but is secure and open source.
BitTorrent Sync is the closest to how it should work, but is not open source: http://labs.bittorrent.com/experiments/sync.html
Tarsnap is ideal, but it's around $27 a month, versus $3.
It's a question of how much you're backing up... my Tarsnap use and cost is very low.
I retain off-site encrypted HDD images updated once a month of things like media which constitute the vast majority of my data and do not need daily snapshots.
Yeah, I want to back up my media as well (around 90 GB), but I want my backups to just work. Tarsnap would have been fine if not for the cost, so a tool that combines EncFS + SSH + rdiff-backup will be ideal for me.
How does this solution measure up to hosting your own Owncloud or alike? Any upsides / downsides? I'm wondering because many non-technical people need a dead simple solution for this if you want to substitute Dropbox.
From what I've seen, Owncloud doesn't provide encryption (files are lying in the clear on the server); furthermore I didn't dig much into the web app part, but I wouldn't bet it to be extremely robust security-wise.
That is not true. Owncloud provides both SSL and file encryption:
http://doc.owncloud.org/server/5.0/admin_manual/configuratio...
well said, they don't provide any encryption - and I have not found any public cloud services that could solve the browser encryption issue. If you are willing to experiment with public services, my first bet would be tresorit (however, they have limitations in compatibility until august, when upgrades are coming). I also tried wuala and boxcryptor, but they just did not work for me.
> $ cat ~/.ssh/id_rsa.pub | ssh encbox@your.vps.com "mkdir ~/.ssh; cat >> ~/.ssh/authorized_keys"
$ ssh-copy-id encbox@your.vps.com
Doesn't exist on all machines which would be capable of using this - namely OSX
Good point. It is installed on my MBP but it looks like I installed it via brew.
I applaud this, and other similar efforts (OwnCloud et al) but for me the win with Dropbox is not system to system syncing (for which Unison or rsync works Well Enough for me) but rather as a synchronization service for my mobile devices. This is sadly enough an area where roll-your-own is not going to be able to compete.
Yes, it’s difficult but I don’t think »roll-your-own is not going to be able to compete«.
(Disclaimer: I’m interaction designer on ownCloud.) With ownCloud Inc. we have a company, customers and full-time employees. There’s no reason for it to not be able to compete except time needed to catch up to par.
And also the code happens to be open source. In that respect it’s a challenge because we don’t have the one centralized reference server where everyone has an account. There are people running it on Ubuntu, Debian, CentOS, Windows Server, …, Apache, Nginx, Lighttpd, …, with MySQL, SQLite, PostgreSQL, Oracle, …, not to mention the different browsers people use it with (we support down to IE8) and operating systems the clients use.
Anyway, I digress. ownCloud is also a synchronization service for desktop and mobile devices. In fact we don’t have any system-to-system syncing built-in.
The mobile clients and especially the desktop client have gained a lot in stability in the last few weeks – if you’ve tried before, I encourage you to give it another shot. We’re also in the process of reworking the design for the mobile apps. Overall simplification, better visual design. Let me know if you have any specific feedback.
I started using BitSync[1] few days ago to sync and backup some folders in my Android to my PC and Mac.
Unfortunately it is not opensource, but is is still better than dropbox, because it does not store my files in cloud and it has better setting - i can choose which folders on my device i want to backup or synchronize.
Are people looking at bitTorrent Sync? (http://labs.bittorrent.com/experiments/sync.html) I'm uing it on a laptop, a server and my android devices. It works great. Seemless.
Closed source is a dealbreaker for lots of people.
They can keep the source for all I care, but it's hardly BitTorrent without an open extensible spec.
One question: how do you share files securely with others? Have you check out our company's free products, http://ncryptedcloud.com? We secure your data before it goes into Dropbox, allow securely sharing and many more features for FREE to all our consumer users! We only charge for things like auditing beyond certain amount of times, single sign-on integrations, enterprise stuff etc. All we want to do is secure this whole cloud mess.
A few notes on Backupsy from their website:
>Will you backup my Backup VPS?
> Unfortunately, no. Even though we use a RAID protected setup, there is still a slight chance of data loss due to RAID controller failure. For extreme redundancy you can order 2 backup VPS in different nodes and we can mirror them for you ("Configure it for me" addon should be purchased).
This looks cool, but of course you can also use encfs directly with Dropbox or, if you prefer a graphical interface, Boxcryptor[1] have a fork of encfs specifically for cloud storage along with some platform-specific apps.
I used that combination before, it works really nice. But at some point I hit the storage limit and had to come up with a self-hosted solution.
I wonder if this is really usable like Dropbox. I tried owncloud which is supposed to be the more stable alternative but it kept replacing new files with old ones, sync took ages and security was weak. I really want to support this kind of projects but they hold me from being productive, which I really need right now.
No, it won't be. Polling the filesystem is not a scalable solution. OsX will hit the file descriptor limit on mostly anything you want to watch. Of course, you can up the limit, but thats a losing battle because poling 10K files per second doesn't work.
The correct solution involves using FSEvents (which blows), and system poling when you get an event. Inotify is good enough to get away with no poling. The native windows change watcher isn't bad either.
Even if this weren't the case, the value that dropbox provides is far greater than this potential solution for most people- ie, there is quite a bit of room for something to go wrong running your own duct tape dropbox. Syncing 99/100 files is not OK. Syncing 100/100 files, but only after 2 weeks isn't OK either.
source: I built my own ducktape dropbox (sortof). https://github.com/Floobits/flegmatic
The underlying sync software (Unison) has been around for years now and is regarded as stable. So syncing should work just fine, although you should keep a backup of your files just in case.
You can also use Dropbox and Encbox together if you're unsure: Point your Dropbox installation to ~/Encbox and have Dropbox sync your (then decrypted) files. So you can be sure to have backups, file sharing features, etc. and see if Encbox is stable enough for you.
I often have Unison not notice file changes until I repeatedly scan or sometimes even until weeks later. If I new a better alternative I would not use it.
They use some kind of heuristic to make change detection faster, see their documentation: http://www.cis.upenn.edu/~bcpierce/unison/download/releases/...
Maybe that's the root cause of your issues?
Owncloud client is based on csync, which a quite weak system compared to unison, in terms of file consistency.
No idea how well unison performs in constant Dropbox-like usage, but you shouldn't let the flaws of Owncloud deter you: file synchronisation isn't Owncloud's strong point.
This should be taken a step further and have a UI and web control panel just like the real dropbox.
Putting aside the work itself of creating a UI (this is just a simple install guide), that's not possible without exposing the encfs keys to the remote server. In this setup the client data is all encrypted before it reaches the server. The server is just a sync point for multiple clients.
You could have a UI (web or otherwise) on a separate client but it would have nothing to do with the server. It would just be a client that provides a UI interface to the decrypted encfs filesystem.
Theoretically, it should be possible to have a web UI running on the server itself. You just need a decent AES implementation in Javascript to do client-side decoding of filenames and files. Any volunteers? :-)
Edit: Looks like someone is already working on it: http://stackoverflow.com/questions/10909500/use-encfs-with-j...
I've looked into this before and concluded it was possible, although I forget exactly how.
I bookmarked https://crypton.io/ and http://peerjs.com/ . PeerJS is p2p WebRTC, which I believe I chose because I wanted to do secure, real-time p2p file sharing.
That isn't 100% relevant for this, but downloading the file over some socket into client-memory then decoding it there sounds like something webrtc could be a good option for.
Crypton handles the crypto.
I think the biggest thing someone who tackles this needs to realize is that perfecting the crypto doesn't matter 100%. If you could get something working that's doing some kind of encryption/decryption without exposing anything to the server, crypto experts will come in and help secure everything (see: mega)
Edit: I should mention that I didn't research this extensively, and better libraries might be (and probably are) available.
It's a really interesting idea (and sounds really fun to build) but a bad idea from a security perspective. Javascript crypto in the browser just doesn't make sense[1].
Summary of the link: Since you trust the server to serve the JS files, you might as well trust it to do the decryption and present it all over SSL.
> I think the biggest thing someone who tackles this needs to realize is that perfecting the crypto doesn't matter 100%. If you could get something working that's doing some kind of encryption/decryption without exposing anything to the server, crypto experts will come in and help secure everything (see: mega)
This is an especially bad idea. Application that perform crypto should be designed properly from the start. It's not something you can just patch on later. Building crypto programs to learn/test/explore is fine but building a product for actual usage and then expecting others to fix it after the fact is a terrible approach.
[1]: http://www.matasano.com/articles/javascript-cryptography/
Err I don't think this would stop me. But let me preface this by saying I learn something new about security quite frequently, so if there's a major premise I'm missing, I am both interested in learning about it and not completely surprised.
The main issue here is that your Javascript can potentially be owned by XSS or other approaches. If you own the javascript, you can send plaintext passwords or keys (which should have only been available to the client) back up to some server.
Even with that possibility though, I still think using browser-based crypto creates a more secure environment than the server storing unencrypted files. The author makes a good point in saying that you download javascript on each request, which makes it more susceptible to getting owned than a native-app, but I think a successful attack with this approach will still be an order of magnitude less severe than if an attacker owned a server full of unencrypted data. If the data is unencrypted, an attacker that manages to own the server has access to everyone's data. With it encrypted, the attacker will only get data from users who are using the site while it's owned.
Isn't that still a better option?
> Err I don't think this would stop me.
On the contrary go right ahead. Learning through doing is the way to go. I'm just saying there's a big difference between testing/learning/prototyping and releasing something that is claimed to be secure. Otherwise you end up with Cryptocat[1].
> Even with that possibility though, I still think using browser-based crypto creates a more secure environment than the server storing unencrypted files.
If you're using any JS then you have to completely trust the server that is serving it. At any point the server could replace "good.js" with "bad.js" and the browser will happily execute it. Any encryption you perform on the client side would be moot as "bad.js" could do whatever it wants with your plaintext and your encryption keys.
> If the data is unencrypted, an attacker that manages to own the server has access to everyone's data. With it encrypted, the attacker will only get data from users who are using the site while it's owned.
Again you're trusting the server to not save your encryption keys though by serving you "good.js" and not "bad.js"! All roads lead back to you trusting the server to play nice.
Unless you use untrusted remote services as opaque object stores and handle all encryption via secure code on the client side with proper signatures to prevent remote tampering[2], you're trusting the server not to compromise you.
[1]: http://en.wikipedia.org/wiki/Cryptocat#Security_concerns
[2]: Tarsnap (https://www.tarsnap.com/) is a perfect example of a client not trusting the server. The tarsnap service stores it's data on S3 but first all data is encrypted client side on your own machine. From the tarsnap service's perspective it's just storing a bunch of random binary blobs and that's all your client sends/recieves. Encryption/decryption happens locally on your machine.
Additionally, it is almost impossible to verify the security of a system that uses encryption via javascript in the browser. The algorithms aren't that hard to verify, but making sure that keys are handled securely, and that you don't open yourself to timing and side channel attacks is really hard with such a big stack of things competing for scheduling, managing memory etc.
So there are two problems:
1) You get code from the server and execute it -- it might not be the code you think it -- it might not be the code you got yesterday (and there might be third party code injected, if there is an XSS vector -- or you know, your browser doesn't check to see if the servers ssl certificate has been revoked...).
2) Even if you run the code locally (say distribute it as html+js+css "app" in a zip file signed with gpg) -- it is still running on top of a pretty rickity rack of technologies, your javascript vm, the browser dom, a mess of ui toolkits and c/c++ routines that manage memory -- and has been seen again, and again -- it is really hard to actually implement crypto in such a way that there are no side channel attacks, no timing attacks and no information leaks.
Essentially 1) running random code isn't secure, and 2) implementing good crypto is hard. Really hard.
The only real solution is to get a good crypto-api into html5/browsers, but even then things aren't "safe". Say you could:
Now, what happens with plaintext? Is it written to swap? Cahces? If you use that to show an image in the browser, is the image cached along with other files?plaintext = api.crypto.decrypt(AES-256-CBC, \ ciphertext, key)
If I remember correctly, ZFS has a 'push' feature that could be a good candidate for this purpose
Oops, should be ZFS send.
Reference Link: http://docs.oracle.com/cd/E19253-01/819-5461/gbinw/index.htm...
You have to trust the VPS provider.
You don't have to if you use EncFS. All encryption is done client-side.
The only thing your VPS provider could do is delete your files, but Unison's backup feature should protect you from losing your files (in a way).
Your VPS provider, can in effect, chroot in to your environment, so EncFS is vunerable.
The EncFS is mounted locally (client-side), so the files aren't decrypted at all on the server-side. You're just syncing fully encrypted files.
If encryption is done client-side, the data is still encrypted server-side and chrooting does nothing.
What do you mean by 'vulnerable'? All encryption is done on your computer/notebook, so a chroot on the server doesn't decrypt the files.
Yep. This is an important point.
I can still follow this guide to create my online-NAS box and use it across devices.