Show HN: I missed the moving blocks, so I built a real Linux disk defragmenter
github.comBeautiful.
Defragmenting my drives manually is one of those things like closing apps on my phone I am not using. Everyone says there is no need to do it. It might even be slightly worse overall. I even have enough self control to avoid doing it and have for years...
But deep down in my heart, I truly FEEL like if I did it would improve things... somehow.
I discovered that tested performance on copper-rs a high performance OS for robotics when I run it on ext4 and what is annoying in robotics is the max latency & jitter. Especially that we allocate large slabs in copper so this is not helping at all.
This is on a slow device but it might be worse on a fast one as the extra allocation of small extents start to hit harder on the host side:
(fragmented relative to baseline) bandwidth: 100.65 -> 94.74 MiB/s (-5.86%) elapsed time: 339.550 -> 360.699 s (+6.23%) mean latency: 9935.7 -> 10554.6 us (+6.23%) p50 latency: 6715.2 -> 6838.9 us (+1.84%) p95 latency: 19394.4 -> 20599.9 us (+6.22%) p99 latency: 131498.2 -> 131988.3 us (+0.37%) max latency: 183713.2 -> 624392.9 us (+239.87%) jitter stddev: 17823.1 -> 19806.0 us (+11.13%) jitter CV: 179.38 -> 187.65% (+4.61%)
Also check your nvme granularity with fstrim -D if you are on 4KB your nvme is so fine grain that it doesn't matter much but if it is 64KB like my main one. Ouch, those small files in the middle of the 64KB won't magically go away.
I feel so seen right now. My partner complains about me closing apps on my phone consistently. I also have to hold myself back from defragging and continually cleaning up my storage drives.
I too am nearly certain that the positive benefits are approaching nil, but I still feel it should be helpful.
> My partner complains about me closing apps on my phone consistently.
I'd be curious to know the reqson why she would care on her own phone and why it would be an issue on someone else's phone?
At least on iPhone, it basically doesn't do anything. Apple aggressively kills all background apps automatically. the "open" apps in the switcher are almost always just screenshots of previously-open-but-now-closed apps. if it was recent, the latest app or two in the switcher might actually be open, but rarely more than that.
Despite this, many of my relatives have somehow learned this habit of opening the switcher and closing all apps when they are done with them.
i’m convinced the real reason people do this is because they don’t want anyone to see what apps they were using.
Android has always been quite vigilant and is prompt to kill off any app at the MFA login screen while I am consulting the Oracle of Authenticator in another task. Thanks Android!
That being said, I dislike having 3 dozen apps open because I simply can’t quickly find and switch to the ones I’m using. So when I enter a new situation, I swipe all the way left, and Clear All apps, to start over with a tabula rasa.
The app I use is gone from the app store, but like Spotlight or the Windows Start Menu > type-to-search (before it got utterly shittified with web search/does it have AI now?), I have an Android app called App Search+ on the bottom left of my Android home screen.
Open it, and it opens a list of recently searched for apps. The focus is on the top search bar (and the keyboard has also popped up), and I can type to find any app I have installed.
iOS has never worked the way you are describing. It's easy to disprove your claim that the app is killed by just switching between different apps you have open. iOS keeps the apps open to make it fast to switch back to them.
This is literally always how iOS has worked. You can refer to the application lifecycle documentation: https://developer.apple.com/documentation/uikit/managing-you...
What point are you trying to make. The documentation doesn't say the OS only keeps a single app process running at a time.
Well, it closes the apps somewhat arbitrarily, but they definitely arent all open. If the most recent app is a resource-intensive game or something it will more aggressively close things. but you can have more open if they are all lightweight apps.
But the switcher shows every app ever opened and tries to pretend these apps aren't being closed. Right now for instance, i just checked and as best i can tell, the last 3 were open. going back any further and i could tell the app had to reload. but i can scroll to the left back forever. the one on the furthest left i easily haven't opened for months
I do this also - but wonder because I'll have some app open, phone gets warm. Close the app and it cools down.
Being able to go back 3 already contradicts the claim that it was rare to be more than 2. It works the same as Android. Android kills apps based off of resource usage but keeps screenshots of them in the task switcher.
> My partner complains about me closing apps on my phone consistently.
Which one of you is neurotic?
When you die, you will leave behind nice orderly contiguous files. It's thoughtful. hehe
After digging through a recently passed relative’s hard drive (the computer barely functioned so I pulled it) and having to search through years of garbage that where never deleted to find the photos, Word documents, and WordPerfect documents my aunt ‘knew’ where stored there somewhere, this is not an insignificant gift to leave my family with.
only HDD drives I use are in a ZFS pool which has a very usable fragmentation value from zpool.
In case you (or others aren’t aware) the FRAG value ZFS shows, is an arbitrary calculation to help give a value to free space fragmentation.
The purpose is to have a sense of when ZFS may developing trouble quickly finding contiguous blocks of free space to dump new data into. Mostly only a concern for high active and highly full pools. When you start watching the number climb past double digits, you may start finding performance issues but it’s a very contextual thing, rather than “at x value it’s bad”.
Basically it has nothing to do with written file fragmentation, or potential read speed.
Because ZFS is a copy on write filsystem, attempts at defragmentation an active pool are generally not effective. An exception would be a pool that is full of large files that was written in a non contiguous way (like torrents), in which case copying the data to a new dataset and deleting the old roughly achieves the purpose. Or you could use something like https://github.com/salesforce/zfs_defrag
Wow! I did not know that, thank you for the info.
Indeed torrents were the most obvious cause of the fragmentation I saw.
Except for some backups, all of my machines use solid state storage so I don't think there is a point for most people, is there?
Sequential reads are faster for SSDs. As long as defragmentation is reducing the chance of doing a random read, it is beneficial.
OH! TIL! Apparently
Still, given SSDs have a lifespan and given few of my files are that large, I think I'm personally okay without defragmenting. Those numbers are impressive but, most of 10s of thousands of files are source code text files. Asking online what the real-world loss is from not defragmentingcontiguous fragmented ------------------------------------------------------------------- PCIe 4.0 NVMe Read Speed ~5,000 – 7,000 MB/s ~80 – 250 MB/s SATA 3.0 SSD Read Speed ~500 – 550 MB/s ~30 – 60 MB/s> You are giving up virtually 0% to 3% of real-world performance. In daily development, media editing, and standard system use, defragmenting your SSD will yield no human-noticeable speedup.
Is it possible to determine which sectors are physically sequential given remapping for wear-leveling?
Otherwise the claimed defragmentation here is not actually resulting in sequential data.
You cannot directly inspect the degree of fragmentation, because it has less to do with being contiguous in the Logical Block Address (LBA) space and more to do with having been written at the same time. To properly defragment a file on a SSD, you pretty much need to sequentially re-write the whole file in one go, to a newly-allocated part of the drive's LBA space.
>Otherwise the claimed defragmentation here is not actually resulting in sequential data.
I haven't measured it but the SSD firmware should be able to look up where the next block is speculatively in order to be able to immediately start sending it if the host tries and read the next block (as opposed to a random one).
Don't want to wake you up from your Pakistani coma, but data is accessed differently on metal disc spinning 120 revolutions per second, and a NAND flash.
Defragmentation logic for HDD's doesn't work on SSD. For like 10+ years, all SSD's have T.R.I.M. technology to rearrange data themself for faster access, and for more consistent storing of new data.
SSDs often interface with their storage in different (larger) granularities than the OS meaning that even though the data is effectively more random at large scale due to wear leveling there are still benefits to keeping blocks that are sequential in your file sequential in the filesystem also. As well as a potential read boost, the larger blocks introduce a read-before-write issue that can allow multiple close updates.
The difference is very small when compared to the differences between solid state and spinning rust based solutions, but definitely measurable, particularly on drives with DRAM cache, and in some cases human-noticable.
Actually read the article and you'll see the author specifically talks about this begin part of why this side project started (though I suspect nostalgia is the true driving force as there are more efficient ways of addressing that matter!).
I'm pretty sure you don't understand how threads in discussion works.
But, hey. Atleast you can write ~1000 chars with absolutelly no substance fo prove that you know nothing about the topic. What a life to live.
Prefetching applies to SSDs too, and that's generally sequential in some sense.
If it makes it feel faster, it makes it feel faster.
It's like those copper bangles with magnets that people swear makes their joints less creaky.
My example is a daith piercing, that's when you pierce a ring through a fold of cartilage in a certain spot inside your ear. It's supposed to stop you getting migraines, but there's no sensible mechanism for this to work. It's all woo and bunkum, apparently.
But I've had four migraines in eight years as opposed to four every month.
So, if it feels faster, it might well be.
I like the graphical interface and the movement of the blocks. Brings back memories of defragmenting hard drives once a month and seeing noticeable performance improvements sometimes.
> Fragmentation and extent allocation were adding measurable variance, even on NVMe,
Why exactly would there be a measurable variance on NVMe? I understand there could be some impact on magnetic hard drives. This sounds like some sort of coincidence due to other factors and that this defragmentation won’t achieve much (except for wearing out your SSD even faster in certain cases).
Yes: I discovered that tested performance on copper-rs a high performance OS for robotics when I run it on ext4 and what is annoying in robotics is the max latency & jitter. Especially that we allocate large slabs so this is not helping at all.
(fragmented relative to baseline) bandwidth: 100.65 -> 94.74 MiB/s (-5.86%) elapsed time: 339.550 -> 360.699 s (+6.23%) mean latency: 9935.7 -> 10554.6 us (+6.23%) p50 latency: 6715.2 -> 6838.9 us (+1.84%) p95 latency: 19394.4 -> 20599.9 us (+6.22%) p99 latency: 131498.2 -> 131988.3 us (+0.37%) max latency: 183713.2 -> 624392.9 us (+239.87%) jitter stddev: 17823.1 -> 19806.0 us (+11.13%) jitter CV: 179.38 -> 187.65% (+4.61%)
Also check your nvme granularity with fstrim -D if you are on 4KB your nvme is so fine grain that it doesn't matter much but if it is 64KB like my main one. Ouch, those small files in the middle of the 64KB won't magically go away.
>defragmentation won’t achieve much
Indeed, most Linux setups supporting trim, already defer these operations to a weekly schedule to reduce wear, and most fs will optimize in 10MB or 25MB chunks given unlike HDD... the SSD seeks are nearly constant time. Logging fs like f2fs, are content aware so will auto re-locate hot and cold (rarely modified) file types, and despite the log-structure... on an SSD performance losses are often surprisingly negligible.
Most modern NVMe with dram cache and SLC buffer areas also defer committing pages to low-endurance flash areas. And most kernel tweakers will set swapiness to 1 on SSD/NVMe machines to try to keep stuff buffered in dram as long as reasonably possible. It is a space-time tradeoff that can boost a desktop machine performance especially with preload daemon active.
If people want ludicrous speed... than just run ext4 with a separate 128GB journal NVMe drive on a split PCIe x4 bus.
Defrag on most modern drives usually just fills these buffer areas full, and things grind to the slowest i/o choke point. =3
SSD still have slower reads when the data isn't sequential. Not slow enough to matter, especially with extents meaning the data is probably in a handful of locations, not 10000, but it's absolutely measurable if your blocks are small enough (<1MB typically, though disk benchmarks usually use 64KB random reads and sometimes 4KB).
It's more complex than that... Sequential reads are faster than random reads across most mediums (including system ram), but if there's enough prefetching, you might get better throughput on an SSD with the data fragmented than contiguous because SSDs can often read from multiple regions in parallel and contiguous data might not allow for that.
If you're at the point where you're optimizing for this, you've got some really high performance requirements though. And you'll have to do your own testing, because rules of thumb won't do.
Except most SSD also don't store pages sequentially internally (hidden flash wear leveling is even in microsd cards now), and the dram/SLC buffer areas are finite especially on low end budget hardware. Don't defrag your SSD dude. =3
Not to make everything about AI, but..
The current analogy I am using when vibe coding is it's like defragging a hard drive in terms of time wasted.
Years of sitting there as the bar crept closer to 99 in the vain hope I could turn the computer off and go to bed, just to have to sit there for another hour or so until finished.
So many late nights wasted doing, I'm not sure what.
And now I'm sat there waiting for the agent to come back..
Running for 17m.. "dealing with a complex response"..
Codex (GPT5.6) has me hooked on that loop moreso as it is, in practice, so much quicker than Claude (Opus or Fable), so I don't quite ever walk away.
Nostalgia.
What actually triggered me to do that was to be able to test repeatably my other open source project in rust were the compile target directory can go to hundreds of gigabytes of files and at the same time I need to benchmark it on large linear slabs we do create for memory mapped files and I definitely noted the impact. I can only guess but non-optimal extent length create a repeated round trip to the metadata in main memory instead of "linear reads" (as consecutive logical numbers) + maybe the fact that those SSD are probably super smart and prefetch the next sectors? (see the jitter numbers I get even on a slow device in other comments)
poetic :)
> I needed a reproducible storage layout while testing a high-throughput logger.
Of course just having a compressed filesystem image to restore for each test, or a collection of images to cover various cases instead of only optimising for ideal, would be more efficient…
But I love that this now exists instead!
Neat! I have several questions here. First of all, why does Linux not need defragmentation?
Second, is it actually true that it doesn't need it? The readme implies that this defragmenter (for Linux) produces actual benefits.
Third, it says the benefits exist even on NVMe, i.e. on SSDs? I thought defragging was relevant for spinny disks only?
Thanks
Mostly, they try to avoid it by allocating extents instead of blocks (so, they look for a contiguous space to store most files) and by delaying allocation to accommodate for a growing files.
My observation is that it is not completely true. I constantly have a close to full partition in ext4 and there is no secrets when there is no space the filesystem WILL create a bazillion of small extents because, it has no physical choice! and everytime you have a non maxed out extent, it will create an overhead I don't see how magically you can avoid that. And it is not like a few, on my main working partition, extends of a few KB by the thousands per files are created instead of the optimal size of 128MB. If you just use defrag in read mode on it, it is red from top to bottom, you cannot find a block of few MB with no fragmented file in it.
This cool, but can we get an option to emulate HDD noises please? I really miss the whirrs, clicks and creaks. And the little LED that indicated disk activity... but that'd be a separate project.
Be in a very quiet room and I can still hear my SSD.
I suspect the noise is caused by the power supply emitting a tiny bit of 'coil whine' when put under more load
Coil whine is common in budget setups, but often it is also just cheap MLCC Piezoelectric acoustic harmonic noise. Most EE impassioned pleas for large solid polymer capacitors to knock down the noise completely... are often ignored for cost reasons, or factory replaced with liquid-electrolytic type that slowly degrade over time until they literally explode on occasion.
The "Fast, Cheap, or Good... choose any two..." joke is very real. =3
Would be really cool to see two things.
First some kind of before and after benchmark, so you can actually see if defragging made a difference. Second is a recommendation for defragging based on things like the fragmentation, free space, and whether it’s an HDD or SSD.
Especially the second one feels useful since defragging doesn’t necessarily mean improvement to the system as you said yourself.
This made me think of https://e4rat.sourceforge.net/, which moves boot files into sequential runs, which was helpful on spinning disks where seeks were slow but sequential reads were fast.
Just as an idea for a feature your defragmenter could have.
(Also this is a side thought; but I wonder if AWS AMIs and EBS volumes will boot faster if also organized sequentially.)
> Also this is a side thought; but I wonder if AWS AMIs and EBS volumes will boot faster if also organized sequentially.
Probably a measurable, but small, difference. I would expect it's not worth the effort unless you're spending a lot of time booting, and even then, there's probably better things to work on in the boot process, such as reducing the amount of things that run or reducing the size of them.
I have a question about disk defragmenter. I've asked a couple of people before and they've said it is not possible but iirc if I stated disk defragmenter and then even so much as moved the mouse after it started it would start with the first boxes again. Do you remember this too?
Pointless and harmful to flash storage. SSD sector numbers are virtual anyway. Its like defragging emails in your inbox.
check my other comments with numbers, while small for a human being, I could see the variations benchmarking copper-rs.
This is the most useless thing I've seen on here lately. It's not necessary on hard drives, as the page cache did a decent job of improving disk access even in the 90s, let alone with today's fast HDDs. And if you're on an SSD... it'll just burn that out faster.
I love it!
yeah, this will definitely shorten the lifetime of an nvme.
but check my other comments, it makes a difference, even on a slow nvme. Probably the overhead of the wasted small extent, and maybe prefetching?
Page caches are nice, but you would see meaningful reduction in load times for the OS and applications by getting data in order on spinning disks. Especially games that used most of your ram and loaded reasonably large level files.
fat seemed particularly prone to spreading files into very small chunks across the disk surface; it's not so big of a deal when fragmented files are made up of large contiguous chunks.
I had a job going to various store locations and running defragment on point of sale computers. It was incredibly zen.
Before enlightenment
Chop wood, defragment
After enlightenment
Chop wood, defragment
So now we are using vibe coded disk fragmenters that actually touch the disk state on your machine. It has little to no tests.
Now HNers here don't even question the soundness of this software and just assume it works as it was suspiciously built in 4 days.
One side-effect of the AI mania is that developers have lost the ability to reason around the verification stage of vibe coded software.
Oh dear.
As long as we keep trying to bring windows mainstream shit for penguins, I am happy with that. Btw I am building similar tools for linux hahaha.
Yeah, can't wait for Linux to add the forced system update that restarts the computer while I'm away, closing all my open programs.
Like the cleaning lady who ignored my DO NOT RESTART note on the PS1 at the after school daycare and nuked all my progress.