YouTube as Storage
github.comI once asked one of the original YouTube infra engineers “will you ever need to delete the long tail of videos no one watches”
They said it didn’t matter, because the sheer volume of new data flowing in growing so fast made the old data just a drop in the bucket
Now that they can harvest it all for AI training, that decision was the cheapest and greatest thing they ever did.
Imagine trying to pay for all that content, nobody on earth would be able or willing to supply it.
PeerTube is a thing. I like to think without centralized players like YT, that P2P supported federation may have gained a better foothold.
There’s still time
Videos do disappear, though. https://www.reddit.com/r/DataHoarder/comments/1ioz4x1/is_it_...
Searching hn.algolia.com for examples will yield numerous ones.
https://news.ycombinator.com/item?id=23758547
https://bsky.app/profile/sinevibes.bsky.social/post/3lhazuyn...
Of course videos disappear for copyright, ToS violations, or when the uploaders remove them. They do not disappear just because nobody watched them.
There’s a whole activity around discovering random 15 year old videos with almost no views. It’s usually some random home video
A friend of mine worked two years in YouTube as a content admin.
Basically being given videos to watch all day, especially coming from the middle east (this was ISIS time so any video from the area had someone watching it as soon as uploaded).
Needless to say there's endless gold no view videos according to him.
It's also interesting that it was no open secret that already in 2018 they were all told that they were essentially training machines to do their job.
I was interested in the same thing and built a search for it
They also disappear when the government of Pakistan tells Google to erase them: https://lee-phillips.org/youtube/
I seem to recall reading that the HD variations may get removed leaving only 480p or lower for older unwatched videos.
The original upload would likely still be stored, but not available for viewing.
That would be an odd thing to do. HD is low resolution already, and 480 is noticeably worse.
If they really wanted to compress, take out every other frame, and regenerate those frames with a neural decoder. But I don't know why that would be worth the effort for a stable number of low res files either.
I wonder if that still holds true? The volume of videos increases exponentially especially with AI slop, I wonder if at some point they will have to limit the storage per user, with a paid model if you surpass that limit. Many people who upload many videos I guess some form of income off YouTube so it wouldn’t that be that big of a deal.
What they said only holds true because the growth continues so that the old volume of videos doesn't matter as much since there's so many more new ones each year compared to the previous year. So the question is more about whether or not it will hold true in the long term, not today
The framing here is really weird. The volume of videos increasing isn't 'growth.' Videos are inventory for Youtube. They're only good when people (without adblocks!) actually watch them.
Growth in this context is that there are a larger volume of videos each year. So each year a single video is exponentially a smaller and smaller percentage of the total.
Yeah and the math doesn't check out.
For example, if in year N youtube has f(N) new video. Let assume f(N) = cN^2. It's a crazy rate of growth. It's far better than the real world Youtube, which grew rather linearly.
But the rate of "videos that are older than 5 years" is still faster than that, because it would be cubic instead of quadratic. Unless the it's really exponential (it isn't), "videos that are older than 5 years" will always surpass "new videos this year" eventually.
Video sensors are continuously getting cheaper, better and more more prevalent over time. The trend is towards capturing all angles of everything, everywhere, at increasingly higher resolutions.
> Unless the it's really exponential (it isn't), "videos that are older than 5 years" will always surpass "new videos this year" eventually.
Such a weird strawman argument that you are making up. You've over thought this so much that you are missing the forest from the trees
Yes. a video no one watches is a waste of storage.
Maybe not.
Maybe it could be used to train a neutral network. Maybe it contains dirt on a teenager, who might become a politician two decades from now. Maybe it contains an otherwise lost historical event.
Or it just helps to cement YouTube as the go-to place for uploading and sharing videos for almost any purpose which has a long-term positive effect for user engagement and retention
^ This.
I wonder if anyone has ever compiled a list of channels with abnormally large numbers of videos? For example this guy has over 14,000:
There is a channel with 2 million videos: https://www.youtube.com/@RoelVandePaar/videos One with 4 million videos: https://www.youtube.com/@NameLook
NameLook puts a whole new meaning to "low effort videos"
Lord above. This is the worst garbage I've ever seen:
https://www.youtube.com/shorts/mrOXqgShzI0
This shit is the reason I can't afford a new HDD.
First one has transcribed stack overflow to YT by the look of it
AH! I've stumbled on that first fellas videos before! The videos aren't crazy complex but the sheer volume is impressive in a perverse kind of way.
I guess I should have mentioned I wasn't looking for automated/AI-generated videos.
I assume it's an economics issue. As long as they continue making money off the uploads to a higher extent than it costs for storage, it works out for them.
Do they make a profit nowadays
Likely yes, with a margin of perhaps 38%
> The volume of videos increases exponentially
Source?
One day, it will matter. Not even Google can escape the consequences of infinite growth. Kryder's Law is over. We cannot rely on storage getting cheaper faster than we can fill it, and orgs cannot rely on being able to extract more value from data than it costs to store it. Every other org knows this already. The only difference with Google is that they have used their ad cash generator to postpone their reality check moment.
One day, somebody is going to be tasked with deciding what gets deleted. It won't be pretty. Old and unloved video will fade into JPEG noise as the compression ratio gets progressively cranked, until all that remains is a textual prompt designed to feed an AI model that can regenerate a facsimile of the original.
You can see how Google rolls with how they deleted old Gmail accounts - years of notice, lots of warnings, etc. They finally started deletions recently, and I haven't heard a whimper from anyone (yet).
The problem is that some content creators have already passed away (and others will pass away by then), and their videos will likely be deleted forever.
That may be, but I assume for videos that had some viewership base, there may be a consideration. E. g. if a video was viewed 20 million times, it may be worth more than one that was viewed only 5 times.
I've stumbled upon very valuable content with very low view numbers - the algorithms spiral around spectacularity and provocation, not quality or insight.
Then it's on you to share it !
But to whom, I have no followers and a blog update has been on my TODO list for two decades..
>videos that had some viewership base, there may be a consideration
Those would be the worst of the lot regarding how valuable they are historically for example. Engaging BS content...
Hopefully the deletion will not affect videos with thousands of views, even if the account is lost.
Monuments erode away and memories of those enshrined are lost time as well, nothing lasts forever.
I met a user from an antique land Who said: Two squares of a clip of video Stand in at the end of the search. Near them, Lossly compressed, a profile with a pfp, whose smile, And vacant eyes, and shock of content baiting, Tell that its creator well those passions read Which yet survive, stamped on these unclicked things, The hand that mocked them and the heart that fed: And on the title these words appear: "My name is Ozymandias, Top Youtuber of All Time: Look on my works, ye Mighty, and like and subscribe!" No other video beside remains. Round the decay Of that empty profile, boundless and bare The lone and level page stretch far away.This is amazing.
Would've been, once. These days I assume bentcorner asked their favourite LLM to generate a poem parodying Ozymandias about once-popular youtube videos.
It doesn't feel like it at all (I'd never expect an LLM to say 'pfp' like that, or 'lossly[sic] compressed', ASCII instead of fancy quotes) but who knows at this point.
I may have gotten incredibly neurotic about online text since 2022.
or you could get over it and still enjoy it anyway. Like how Coke Zero tastes.
That is a fair point. Especially since, assuming it was AI-generated, it presumably wouldn't have existed at all otherwise.
Brought to you by Carl's Jr
Nope, I hand wrote this.
I actually considered using an LLM but in my experience they "warp" the content too much for anything like this. The effort required to get them to retain what I would consider something to my taste would take longer than just writing the poem myself. (Although tbf it's been awhile since I've asked a LLM to do parody work, so I could be wrong)
Ah, well, kudos then!
let's see what will last longer over the ages : engraved stone or google?
Depends on the pH, probably.
Like tears in rain <3
mono no aware
Dropbox seem to be doing the same thing. After years of whining about my 2TB above limit I recently received a mail with a deadline to delete my files or they will.
It depends. At the rough 2 PB of new data they get a day that’s about 10 sq ft of physical rack space per day. Each data center is like 500,000 sq feet so each data center can hold 120 years of YouTube uploads. They’re not going to have to restrict uploads anytime soon.
Not all of the square footage of a data center is usable for racks
Oh. I noticed in an AI music generation service I use that old pieces were severely degraded to the point that they were crackling really bad... And I remember thinking that it's a good thing I downloaded an mp3 of my favorites. I confirmed that the quality is very different by listening to the downloaded recording with the hosted version side-by-side.
Wouldn't it also be a performance nightmare?
The energy bill for scanning through the terabytes of metadata would be comparable to that of several months of AI training, not to mention the time it would take. Then deleting a few million random 360p videos and putting MrBeast in their place would result in insane fragmentation of the new files.
It might really just be cheaper to keep buying new HDDs.
S3 allows delete and is efficient here. I’m sure Google can figure it out
They allow search by timestamp, I’m sure YouTube can write algo to find zero <=1 view
This is why they removed searching for older videos (specific time) and why their search pushes certain algorithmic videos, other older videos when found by direct link are on long term storage and take a while to start loading.
I’m pretty sure this is the real reason why they changed old unlisted videos to being marked private: https://blog.youtube/news-and-events/update-youtube-unlisted...
Well the time filters (before/after:date) still seem to work, but for controversial / hot topics, somehow, more recent videos tend to still show up at the top. Try "scandal after:2010 before:2012"..
Besides with their search deteriorating to the point where a direct video title doesn't result in a match, nobody can see those videos anyway and they don't have to cache them.
It's not just the search deteriorating. The frontend is littered with bugs. If you write a comment and try to highlight and delete part of that comment, it'll often delete the part you didn't highlight. So apparently they implemented their own textfield for some reason and also fucked it up. It's been like that for years.
The youtube shorts thing is buggy as shit, it'll just stop working a lot of the time, just won't load a video. Some times you have to go back and forth a few times to get it to load. It'll often desync the comments from the video, so you're seeing comments from a different video. Some times the sound from one short plays over the visuals of another.
It only checks for notifications when you open the website from a new tab, so if you want to see if you have any notifications you have to open youtube in a new tab. Refreshing doesn't work.
Seems like all the competent developers have left.
and if you do a hard refresh on the webapp, it literally takes like 10 seconds for the homepage to load
Yeah, one that I forgot to mention is if you pause a youtube short and go to a different tab, the short will unpause in the background, or it might change to an entirely different short and start playing that.
Thechnically cool, but ToS state: "Misuse of Service Restrictions - Purpose Restriction: The Service is intended for video viewing and sharing, not as a general-purpose, cloud-based file storage service." So they can rightfully delete your files.
Its interesting that this exact use case is already covered in their ToS. I wonder when the first YouTube as storage project came out, and how many there have been over the years.
The idea of exploiting someone else's server to store files is incredibly old.
https://en.wikipedia.org/wiki/GMail_Drive
When Google launched Gmail (2004) with a huge 1GB storage quota, Richard Jones released GMailFS to mount a Gmail account as a standard block device.
At-least as far back as 2017 when I wrote Schillsaver: https://github.com/Valkryst/Schillsaver
None of us, in the original discussion threads, knew of it being done before then IIRC.
I mean, it is pretty likely they figured out it could be a pretty obvious possible misuse before anyone actually started doing it.
This ia really cool but also feels like a potential burden on the commons,
That great commons that are the multi trillion dollar corporations that could buy multiple countries? They sure worry about the commons when launching another datacenter to optimize ads.
You are right, but YouTube is also a massive repository of human cultural expression, whose true value is much more than the economic value it brings to Google.
So was Flickr
Somebody wrote a file encoder to take advantage of Flickr's free photo storage, too (though based on its Github repo I don't think a ton of people used it): https://alexcbecker.net/projects.html#storing-data-in-gifs
So was Geocities.
Yes, but it's a classic story of what actually happened to the commons - they were fenced and sold to land "owners."
Honestly, if you aren't taking full advantage within the constraints of the law of workarounds like this, you're basically losing money. Like not spending your entire per diem budget when on a business trip.
This seems like a narrow understanding of value.
Which do you think has more value to me? (a) I save some money by exploiting the storage loophole (b) The existence of a cultural repository of cat videos, animated mathematics explainers, long video essays continue to be available to (some parts of) humanity (for the near future).
This is assuming doing A has any meaningful impact on B.
Anyway in this situation it's less that YouTube is providing us a service and more, it's captured a treasure trove of our cultural output and sold it back to us. Siphoning back as much value as we can is ethical. If YouTube goes away, we'll replace it - PeerTube or other federated options are viable. The loss of the corpus of videos would be sad but not catastrophic - some of it is backed up. I have ~5Tb of YouTube backed up, most of it smaller channels.
I agree generally with you that the word "value" is overencompassing to the point of absurdity though. Instrumental value is equated with moral worth, personal attachment, and distribution of scarcity. Too many concepts for one word.
"Siphoning back as much value as we can is ethical."
I feel the same way. (Although, I am less sure of it.) However, I think backing up important parts of YouTube, as you have done, is a much better approach towards doing this.
no the "commons" in this case is the fundamental free-ness of YT - if abused then any corporations will have to shut it down...
OTOH I'm 100.0% sure that google has a plan, been expecting this for years and in particular, has prior experience from free Gmail accounts being used for storage.
> no the "commons" in this case is the fundamental free-ness of YT ...
Hmmm, isn't the "free-ness" of YouTube because there were determined to outspend and outlast any potential competitors (ie supported by the Search business), in order to create a monopoly for then extracting $$$ from?
I'm kind of expecting the extracting part is only getting started. :(
There is no "fundamental free-ness" for vids stored on YT. Videos are stored to serve the business plan of Youtube and under the rules Google sets for them, where they serve their advertisement and surveillance capitalism business.
Looking at the Wikipedia page for "Commons" [0] the first meaning of commons "accessible to all members of a society" is not really true, unless "on the whim of the YT platform". The second meaning of "natural resources that groups of people (communities, user groups) manage for individual and collective benefit" is also not really true. There is no understanding that google will take any other than their own benefit into account. The third meaning of commons on that page is closest I guess to what is needed:
> Commons can also be defined as a social practice of governing a resource not by state or market but by a community of users that self-governs the resource through institutions that it creates.
And that is certainly not what Youtube can be considered to be. Youtube videos are not in the commons, but kept on a proprietary platform where the proprietor is the sole decider what happens to its availability there.
> That great commons that are the multi trillion dollar corporations that could buy multiple countries?
Exactly which countries could they buy?
Let me guess: you haven’t actually asked gemini
Have you? Assuming Google would want to not put all their chips on that one number and invest all available capital in the purchase of a nation, and assuming that nation were open to being purchased in the first place (big assumption; see Greenland), Google is absolutely still in a place to be able to purchase multiple smaller countries, or one larger one.
Greenland already has a wealthy benefactor, I'd be surprised if poor countries wouldn't be interested
Nauru, possibly Tuvalu.
The USA.
That one's not a "could" as it's already been done. ;)
I don't get how it works.
> Encoding: Files are chunked, encoded with fountain codes, and embedded into video frames
Wouldn't YouTube just compress/re-encode your video and ruin your data (assuming you want bit-by-bit accurate recovery)?
If you have some redundancy to counter this, wouldn't it be super inefficient?
(Admittedly, I've never heard of "fountain codes", which is probably crucial to understanding how it works.)
Yes it is inefficient. But youtube pays the storage ;-). (There is probably a limit on free accounts, and it is probably not allowed by the TOS.)
Right, you just pay daily in worrying when, not if, youtube will terminate your account and delete your "videos".
I think it's just meant to be a fun experiment, not your next enterprise backup site
Stegonagraphic backup with crappy ai transmogrified reaction videos. Free backup for openclaw agents so they can take over the internet lol
Hey there, Brandon here (developer). I've uploaded an explanation video here, which might be useful to watch :D
He encodes bits as signs of DCT coefficients. I do feel like this is not as optimal as it could be. A better approach IMO would be to just ignore the AC coefficients altogether and instead encode several bits per block into the DC. Not using the chrominance also feels like a waste.
This actually won't work against YouTube's compression. The DC coefficient is always quantized, rounded, scale, and any other things. That means that these bits are pretty much guaranteed to be destroyed immediately. If this is the case for every single block, then data is unrecoverable. Also, chrominance is not used on purpose, because chrominance is compressed much more aggressively compared to luminance.
I meant choosing multiple values, e.g. 4 to represent 2 bits. Say, 0.25, 0.5, 0.75, and 1. Then when decoding you would pick the closest valid value, so for example for 0.20 it would be 0.25. Not using AC coefficients would mean that theoretically you would get more bitrate for the DC ones.
I’ve been told this many times in the comments, but this again is not reliable. Simply put, compression doesn’t necessarily follow a pattern, so specifying “ranges” or rounding to a specific place will not work. Compression optimizes for the eye, and doesn’t do the same thing for every value. It will round some down, some other mores, others less. Giving a range is simply not enough.
Yeah, I would assume that transcodes kill this eventually...
Also, how to get your google account banned for abuse.
Just make sure you have you have a bot network storing the information in with multiple accounts. Also with with enough parity bits (E.g. PAR2) to recover broken vids or removed accounts.
par2 is very limited.
It only support 32k parts in total (or in reality that means in practice 16k parts of source and 16k parts of parity).
Lets take 100GB of data (relatively large, but within realm of reason of what someone might want to protect), that means each part will be ~6MB in size. But you're thinking you also created 100GB of parity data (6MB*16384 parity parts) so you're well protected. You're wrong.
Now lets say one has 20000 random bit error over that 100GB. Not a lot of errors, but guess what, par will not be able to protect you (assuming those 20000 errors are spread over > 16384 blocks it precalculated in the source). so at the simplest level , 20KB of errors can be unrecoverable.
par2 was created for usenet when a) the size of binaries being posted wasn't so large b) the size of article parts being posted wasn't so large c) the error model they were trying to protect was whole articles not coming through or equivalently having errors. In the olden days of usenet binary posting you would see many "part repost requests", that basically disappeared with par (then quickly par2) introduction. It fails badly with many other error models.
you can split files so you can have more par blocks (100GB in 100 1GB parts 32k blocks per part)
yes, but it just moves the needle a bit, if you lose 1 of the 1GB parts in totality, you also can't recover, so it really depends in on your error model you are trying to protect from.
In practice a DVD like PI/PO model would be the best for many people (protect the 1GB parts like you said with 5-10% redundancy, and then protect all 100 1GB parts together with 5-10% redundancy. the PI will repair as much as it can at the 1GB size, while the PO will be able to repair 1GB blocks that can't be repaired otherwise.
It be interesting if Par2 or something like it could implement it natively without people having to hack together their own one off solutions.
what other tool do you recommend?
just pay for storage instead. It's absurd that rich developers are doing ANYTHING but to pay for basic services - ruining the internet for those in real need.
we can't have nice things
Or.... backblaze B2
Plus restic or borg or similar. I tried natively pushing from truenas for a while and it's just slow and unreliable (particularly when it comes to trying to bus out active datasets) and rsync encryption is janky. Restic is built for this kind of archival task. You'll never get hit with surprise bills for storing billions of small files.
6$ / TB / month is a fool's bargain even for something as low as 10 TB. One can buy a used LTO-6 drive for a few hundred bucks and build tape libraries that span hundreds of TBs.
There's no Cloud-based backup service that's competive with tape.
Have Backblaze software stopped being utterly awful, to the point of being almost nonfunctional, yet?
What does Backblaze's backup software have to do with B2? Backblaze B2 is just storage that exposes the same API as S3. You can use any backup software that supports S3 as a target.
I use backblaze software and it is fine. Works much better than others in the same pricepoint.
There are already channels with millions of AI-generated videos on them.
Other examples of so-called "parasitic storage": https://dpaste.com/DREQLAJ2V.txt
Wot no steganography? Come on pretty please with an invisible cherry on top! :-) Here to get you started: https://link.springer.com/article/10.1007/s11042-023-14844-w
That's harder to sneak through video compression artifacts.
Hey there, Brandon here (developer). I've uploaded an explanation video here for anyone that's interested, which might be useful to watch :D
Has anyone got an example how such a video looks like? Really curious. Reminds me of the Soviet Arvid card that could store 2 GB on an E-180 VHS tape.
Mostly just noise. This is an example data video from the creator: https://www.youtube.com/watch?v=tIRXaQWjiA8
(YouTube video for this project: https://www.youtube.com/watch?v=l03Os5uwWmk)
Love this project, although I would never personally trust YT as Storage, since they can delete your channel/files whenever they want
Upload to other video sharing sites for redundancy. RAIVS!
Stop ruining the internet end exploiting free resources
It was a tongue-in-cheek / silly suggestion outright. I don't think many people are actually using the tool for its off-ToS purpose though, there is also a lot of prior art across multiple sharing services. It's still interesting to think about the inner workings of it.
Interestingly, this is a specific implementation of a more general idea - leverage social media to store encrypted content, that requires decoding through a trusted app to surface the actual content.
AI tools can use this as a messaging service with deniability. Pretty sure humans already use it in this way. In the past, classifieds in newspapers were a similar messaging service with deniability.
The explainer video on the page [0] is a pretty nice explanation for people who don't really know what video compression is about.
I can remember the years when YouTube was used by Contentdistributors by uploading high quality material protected with a password :-D
Similar projects from the past years: https://news.ycombinator.com/item?id=31495049 and https://news.ycombinator.com/item?id=34866808
This is a digital version of a cassette tape to load and save data, love it!
https://www.tapeheads.net/threads/storing-data-on-your-analo...
How does it survive YouTube transcoding.
maybe something akin to what QR codes use?
https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_cor...
Brilliant, but I hope it doesn't hasten Youtube's use of AI to "enhance" videos automatically: https://news.ycombinator.com/item?id=46169554
YouTube is not a place you expect your data will persist. It can disappear, unavailable for any period of time. But if you don’t care if your data is available, why bother with this thing?
This is one of those seemingly “smart” but actually dumb idea.
> This is one of those seemingly “smart” but actually dumb idea.
Your comment seems very sad to me. If you want your data to be safe, you could use physical storage though, and save the data there, on redundant physical hard disks in distributed locations, in various encodings.
You could also try to add even more redundancy by using an audio track with the bit sequences as spoken words combined with a video track that is resilient to low-bandwidth encoding, for example a news show where every segment takes place in front of an info graphic representing one or two bytes per segment. Could be a giant pie chart for variable-precision floating point numbers or a giant still frame of an alphabumeric character to represent raw bytes.
Add some enganging current events to the coverage to make sure the videos stay relevant.
Use large fonts to keep them resilient to video compression.
Combine YouTube, Twitch, Vimeo and at least two disk storage arrays to get five-nines enterprise-grade reliability.
The overhead for encoding and decoding is easily outweighed by the cost-neutral added redundancy.
I imagine something like Reddit might make for better storage than this. It'd be pretty trivial to set up a few accounts with private subs too just store encrypted text based data. Not fast or anything but surely easier to work with.
Kind of stupid to do this while planning on doing a summer internship at YouTube?
What kind of storage level can be expected from this method for 10 minutes of video?
reminds me of gmail fs, https://en.wikipedia.org/wiki/GMail_Drive very interesting project explanation video on youtube
Something at this link crashes both MobileSafari and iOS Firefox on my device.
The GitHub link? Works fine in Safari on my M4 iPad Pro.
Yup. Even after a device reboot at that time, too. Still doing it a half day later. Odd.
How do you manage to get youtube to not re-encode the video, trashing the data?
Flashing a bunch of qr codes should do it
Interesting idea. But I actually think we need to overcome Google. Google has become such a huge problem in so many domains. There need to be laws for the people; Google controls way too much now. YouTube should become a standalone company.
after compression, all data lost.