News and Updates - Folding@Home Stats

Hard to believe it's been three weeks since my the last update with the server move. Nothing exciting to report, I've just been spending every day configuring and tweaking the server, writing code and fixing bugs.

Oh, speaking of the server! So I decided to stick with my current hosting company, The Optimal Link, because they've been so great over all these years and they had a great promo back in September, 30% off when you pay annually! I asked them and they said the annual discount is reoccurring (unlike most other companies where it's a one-time intro deal)! It was a tough call on which package to choose while trying to keep it in a reasonable price range. In the end I decided on the 10-vcores Intel Gold 6248, 32GB RAM, and 2TB of RAID-10 NVME storage, after discount I think it came out to only $32/month! I could have gotten the package right below it which was 6 core / 16GB but I was kind of on the fence as during stats processing the server load will peg 4-5 cores 100%, and I like to use around 8GB of RAM for database caching, so I figured better to have extra capacity. It's still considerably cheaper than what I paid for the 2U colo! Speaking of, the guys were nice enough to send me my old hard drives out of the server (shipping the whole server didn't seem very cost-effective for as old as it is). I have an old LSI raid card here that I think was the same or similar series as the Dell PERC. I plugged them all in the other day and amazingly the arrays were recognized it booted right up! I ended up using Clonezilla to make an image and restored it to a VM, so now I can access my old data if I need to!

I think I spent a week re-writing my firewall code, it was so hacked together from the previous server it needed a good refresh and removal of tons of code that wasn't even used. But then of course with all the AI hype having everyone scrape sites for any data they can get their hands on I started parsing logs and banning by ASNs. Probably the worst offender was TikTokSpider/Bytedance, querying dozens of pages per second, ignoring robots.txt and not just being inconsiderate but totally unethical! Alibaba, Huawei, and Tencent crawlers were just as bad. Then there's the countless traditional garbage spiders that are nothing but a waste of bandwidth but at least they are courteous enough to have a proper user agent, even if they ignore your robots file. I then started going through and querying ASNs for user agents that were "supposedly" browsers, amazing how many come from Amazon, Google, Microsoft, and Oracle clouds. Lots of other big colo / hosting companies kept popping up, too many to list. But then once you whittle those worst offenders you STILL have to deal with all the "security" scanning BS companies like Censys, ZScaler, Shadowserver, Onyphe, Cypex, etc. One thing that really broke my heart was I kept coming across Cloudflare IPs doing malicious queries in the http logs. I used Cloudflare's own published IP ranges and parsed two weeks of logs only to discover EVERY query was a malicious attempt, over 2,600 queries (and realize that F2B usually bans an IP within a few seconds)! I always thought Cloudflare was the good ethical company fighting for a better internet and protecting people from all the bad stuff. I guess they have no issue looking the other way when people pay a bunch of money to use their anonymous proxies free of any consequences, gotta pump up that stock price for the shareholders. What's it called when you are selling the solution to a problem you created?

Sorry for the rant. Anyhow, as for folding stats specific, just chasing down those last gremlins that keep popping up warnings and errors in the logs. There was also couple interesting issues with the sigs that I got sorted out. One thing I caught early on after the move was the server is configured to auto-update for security updates only (regular software updates is manually done). I didn't think about what time those updates ran, and while triggered with a random delay one day it happened during a stats update and caused connectivity issues between the scripts and the database. Fortunately I saw the email pretty quick and was able to re-run the update so nothing was missed. Then of course I had to go dig around to find where and how to change the update schedule to prevent that from happening again.

One neat thing I updated during the server move was the processing for the teams data. Before it used to take around 2-3 minutes. After a couple query optimizations I managed to get it down to 20 seconds! If only the individual data processing had a similar easy speed boost, but unfortunately no.

So now that I think I've *finally* reached a point where everything is in a stable state, I really want to get the OS stats page / graphs created. I've been saving that data for quite some time now but just never got around to making the pages for them. Lots of other little things too (as always), my windows desktop has tons of text files containing various to-do notes, lol.

Lastly, I just flushed all the bans in Fail2Ban as I know there were some accidental triggers while I was tweaking all the rules, hopefully all that is sorted out.

If you've read this far, here's some bonus geek stuff. When I was configuring Postfix (the mail server) I wanted to block certain TLDs (top level domains) like .shop, .blog, .fun, .click, .vip, .xyz, and such. First I checked the Postfix docs, which are EXTREMELY thorough documenting every option and lots of scenarios and such. The access table page documented only 'domain.tld' as the closest to what I wanted. I did a search and found some posts on serverfault but they all claimed you had to use pcre (regular expressions), which seemed reasonable but a little clunky for something that should be so simple. I broke down and asked ChatGPT and wow was that reply totally off the rails! Finally I decided what the heck I'll just add the TLDs to the access file then just manually connect and try some email addresses with the domains I'm trying to block. Sure enough it worked! No need to use pcre, no need to jump through all the hoops that ChatGPT wanted (which I don't think would have worked anyways). I guess they didn't document it because it might tempt people to go overboard blocking TLDs and end up with a mess when some legitimate mail can't get through.

There was one more story I wanted to tell but I forgot what it was. Maybe I'll remember for the next post!

If you are reading this, then you're accessing the folding stats on the new server! I will post a more thorough update once I've completed moving everything over so I can cancel the old server before the end of the month to avoid an extra charge.

The numbers have been consistent so I'm confident all that is okay. I'll keep an eye on the error log to see if anything pops up, but also if you come across an error feel free to send me a message, be sure to include the specific URL that was causing the issue!

The two big things left are email and the forums.

Also FYI, I do *not* have a Facebook account, or any social media accounts for that matter. If you want to reach me you're gonna have to send an email.

After working on the code almost every day I think it's finally stable enough so that I can now find a VPS host and start to move everything over. I'm sure there's going to be a few little bugs popping up during some edge cases but all the warnings and errors get logged so I can just fix them as they appear. Once the stats are moved over I'll make an extra post on the old server so people will know before it gets shutdown. I've already lowered the DNS settings down to a day, and likely will reduce even more once we get closer to the move.

The display scripts wern't horrible, there were a couple that required some considerable logic changes but nothing overly complex. As predicted, the processing scripts were a nighmare! First, just getting them to run and track down all the little issues took forever! Second, I decided to take this opportunity to convert all the names over from latin1 to utf-8 encoding (just a holdover from using MySQL for decades) which presented a whole new set of issues I'm not going to go into details on. Also for some reason processing is running slightly slower. Each query is taking like .0001 - .0002 seconds longer on average. That might not sound like much but when you multiply it by ~750k users with each user averging 7 or 8 queries, that really adds up! I'm almost positive it's something in the optimizer in the newer versions of MariaDB. I've tried every setting imaginable for the past week and nothing really helped. In reality when moving to the VPS it will have faster CPUs, faster RAM, faster storage, so we should still see an improvement in processing times.

I did another export/import of the stats between the production server and the new dev setup the other day to ensure all the numbers match up. So far, so good!

I've come across various things I still need to fix, but want to wait till after the move due to how extensive the changes were and how I'm kind of limited to not touching the database side until after the move. So, fingers crossed everything goes smooth. Will keep everyone updated as I setup the new server and prep the stats move. I want to move the stats between update windows so it all ends up being seamless.

A big part of this server move is transitioning all my existing PHP code to 8.x compliance. I know I don't have to as I could just run it on an older version, but since they are all EOL I might as well just bite the bullet now and futureproof the code so I'm not scrambling to do it at some later date. I've managed to get a couple other parts of the EOC site working (good enough) on the test setup, and I can clean them up more after the move. For most of this past week I've been working on the folding stats really trying to clean up the code properly. First part is just to get some pages to actually run (usually just one or two small quirks that completely break in v8 for some reason). Then I check the error log for the most severe errors and warnings and fix those. Then finally the lesser nitpicky things like undefined variables. At the same time I'm also trying to keep the code backwards compatible just in case.

So far I've only been focusing on all the pages just displaying the stats, I haven't even dived into the processing scripts. That's going to be a lot of verification making sure numbers and everything match up. I feel like it's either going to go really smooth or be a complete nightmare.

I had one day I was looking at some code scratching my head as it didn't look like it should have worked. I would look at the web page how the data was output, then back at the code trying to mentally run through the steps and it just didn't add up. So I wrote a little sample program from scratch using just that bit of logic and I was getting the different output that I was expecting vs what was on the website. Several hours later I'm still confused thinking I'm losing my mind, then I had that AH-HA moment when I realized in this little line of code between how the ternary operator was written and some parentheses it just wasn't very clear what was going on. The catch was my sample program I typed from scratch instead of copy & paste and just editing variable names so I never replicated how the code was actually working. So for the sake of my future sanity I rewrote it to make things much more clear and also left a comment in the code, lol.

I also managed to configure all the web server requirements just in Nginx so far, which is great not needing to additionally run Apache and having to duplicate a lot of settings. I'm excited I can still use Nginx's page caching, previously statements like proxy_cache_... now simply transition to fastcgi_cache_...

That's about it, just lots of code cleanup. Really hoping I can get the move done before the end of the month, but we'll see. I want to make absolutely sure everything will work properly, and most importantly no data corruption or loss.

I just wanted to give a quick update. I finally finished emailing back everyone by Monday evening, that took me way longer than expected! If you sent me an email and didn't get a reply, then I must not have received it, try sending one again through the "Contact Us" form please!

You guys are something else, as of today there's been a total of over $2,800 donated! I'm totally speechless from how much you guys chipped in these past few days, truly a class act! Needless to say that will keep the server hosting paid for quite some time!!! Once I get the site moved I have some interesting things I want to hopefully roll out sooner rather than later.

Tuesday morning I had to go to the dentist for a crown prep and a filling. I've done lots of fillings over the years, no big deal. This was my first crown and wow has it been a doozy! When the anesthesia wore off the pain was crazy, I don't think I finally fell asleep till around 3am because the pain was keeping me awake. I went back to the dentist today (Wednesday) and they did some grinding and we talked about exactly what has happening and he prescribed me some meds to hopefully ease the pain over the next couple days. If things don't get better then I'll have to go back again on Monday and try something else. So I took the meds with lunch and not long after I started feeling extremely nauseous! So fun times trying to lay down and sit up and do whatever you can just try to hold it all down. These past two days I have done exactly nothing except be miserable. Don't get my wrong, my dentist is great, I'm the one that is a mess with some of my roots pushing up into my sinuses it has always been an issue we've had to be careful with.

This past weekend I started setting up a test server instance at home that I want to pre-configure everything and make sure the stats and such will work as expected since I'm transitioning from CentOS to (hopefully) Ubuntu. Once I can verify everything is running as it should then I can look for new hosting and get everything moved. But first, I need this tooth pain to subside before I do any more, I can't even think right now.

First off, I want to say a HUGE thank you to every person that sent me a donation! I honestly wasn't expecting any, or even thinking about it yesterday, my mind was just kind of thinking like this was the time to start winding things down, but you guys proved me wrong. Likewise a HUGE thank you to every person that has emailed me, whether it's offering help or just words of support and encouragement, you guys don't know how much that means to me. I will reply to every email that I've received, just give me a little time, at first it was slow going, but I'm getting better with each reply, I think.

With that said, donations right now would cover till the end of the year if I didn't do anything. For everyone wondering, hosting (colocation) costs me $155 a month. As I've been telling everyone in my emails the server itself is over 15 years old. Between monthly cost, server age, and reduced traffic over the years, it's time to find something more modern and affordable. I briefly looked at some VPS and VDS (virtual private/dedicated server) hosting and I think I can confidently find one for under $50 a month that has the necessary specs for processing the stats in a timely fashion. So that's going to be a project for me this weekend to start researching reputable hosting companies and comparing prices and specs and whatnot. That is one possible route for us to go.

If you sent me an email but haven't heard back, don't worry (yet)! I've still got so many to reply to but it's getting late and I've had a long day. Tomorrow I'm going to power through the rest of them one way or another. Everyone has been giving me various ideas for things or offers of help, so I want to get through them all, and then we can formulate some plans and evaluate our full set of options.

It is with a heavy heart that I'm making this post. It's taken me all day to try and write this, I can't believe I'm getting so emotional. Back around March of 2000 I created EXTREME Overclocking, and not long after some of us in the forums got involved in Folding@Home. I originally started tinkering with making some folding stats in 2003. While I never managed to build out all the features I wanted (life always seems to get in the way), the site and scripts have been pretty darn reliable if I do say so. But as with everything I guess, trends change, people change, everything changes. I stopped messing with EOC many years ago but always kept the folding stats going because of the public interest and over time I kind of ended up being the proverbial last stats site running. I told myself as long as the ad revenue covered the hosting cost then I would just keep it going.

Unfortunately these days so much traffic is just bots, and Google AdSense dwindled down to so little last year that I've been paying a considerable portion out of pocket each month for the server hosting. I've been unemployed for quite some time now and I simply can't justify the continued non-essential expense. I considered just turning the server off and calling it quits, but I couldn't do that to you guys, for however few might still actually visit the site. So instead I'm going to be transitioning the stats to my little home server which even though it too is extremely old, should hopefully still be able to handle the load and traffic. But if / when that old machine dies, I don't know I guess we will cross that bridge when we get there.

I don't know if I'm going to mess with any other portions of the site since they are so outdated, or have it all just redirect here to the stats. I know at minimum I will probably pull the plug on the forums, they have been a ghost town for quite some time and it would just be more of a hassle than it's worth to try and get working. I always hate to see old sites disappear, it feels like losing pieces of history.

So that's the plan, move the folding stats and bare essentials of other portions of the site to my home server and see how long it runs on that. Just thought I would let everyone know beforehand, in case things go fubar.

I finally got a chance to update the ranking color scale this weekend. For simplicity the team ranks were increased by 5x, and the individual ranks by 10x. That seemed to balance things reasonably well and give a wide enough spread for all participants while still highlighting the highest producers. All the tables and calculations and verbiage should have automatically updated as it should pull from a single table source, but if you see somewhere that isn't right please send me a message!

Old Team: 0 / 1+ / 100k+ / 250k+ / 500k+ / 1M+ / 2.5M+ / 5M+ / 10M+
New Team: 0 / 1+ / 500k+ / 1.25M+ / 2.5M+ / 5M+ / 12.5M+ / 25M+ / 50M+

Old User: 0 / 1+ / 15k+ / 50k+ / 100k+ / 250k+ / 500k+ / 1M+ / 2M+
New User: 0 / 1+ / 150k+ / 500k+ / 1M+ / 2.5M+ / 5M+ / 10M+ / 20M+

I also just kicked off the second cleanup script that is going through all the idle users records removing unnecessary rows. I have it pausing a second between each user so hopefully it won't make the site feel slow like it did with the last cleanup script.

Again, I would like to give a HUGE thank you to all the recent donations from people! I've received a handful more since the last post and I'm extremely thankful for everyone's generosity! You guys make it possible for me to keep the site up and these stats running, thank you, thank you, thank you!

That's all for now, still working on more stuff, when they get implemented I'll be sure to make another announcement!

I was emailed recently about how long the updates were taking by a kind folder, and to be honest I've been so busy lately I haven't had the time to check on that kind of stuff so it just flew under my radar. The database size was getting up there and updates were taking well over an hour! I decided to remedy this in two stages.

First, I bumped up the new minimum user tracking amount from 550 points to 5,000 points. I think there is only one or two WUs that are below this amount, but a quick scan through my data was showing me there weren't any active users below the 5k threshold so I went with it. This took several days to run, and would automatically pause for update cycles. I ended up pruning out about 183,000 idle users and over 600 million rows of data, reducing the user history table size by 27%! This alone dramatically sped things up, but there's still more I can do!

Next, I finally worked out the correct logic to prune out rank change records only for idle users, without affecting the recent movement, or the historical records if / when they did submit WUs. Again, this is simply pruning out records where they are idle and dropping down in rank. I ran this code fine on my dev server here at home, I still need to add a few tweaks but will likely get that going within the next day or two.

I know I said over a year ago I was going to update the ranking color scale! Sometimes life just keeps you busy! I will try my best to work on that too once I get the above mentioned script running.

As of now the stats updates are back to taking less than 15 minutes to run, a vast improvement over the hour-plus from previous. I think I'm going to try to add to the database stats page like a table showing recent update processing times and / or some averages or something, since every update cycle the start / stop time is recorded.

Also, I've been recording the OS total data for years now, just the raw data sitting in a table. It's the numbers showing Windows, Linux, and Mac, along with AMD, Nvidia, and CPU totals. I will also try to get a page or two with that up at some point.

Finally, I would like to give a HUGE thank you to all the recent donations from people! Ad revenue has really dropped off a cliff since the beginning of this year (to almost nothing) so I'm having to pay for most of the hosting cost out of pocket and the donations are really really really appreciated!

Just recently I finally nailed down a bug with the firewall and DNS that was causing some people random connection issues these past couple months. If you are still experiencing issues please let me know (via the contact us page).

Then Sunday (March 12th) the server was hit with a DDoS attack and brought things to a grinding halt. Unfortunately I wasn't around when it happened and didn't return home until late that day, so things didn't get resolved until Monday (yesterday) morning.

I was working on updating the color rankings to bring the numbers more inline with current production, but now that things are out of whack from the downtime I'll have to wait a week for everything to settle.

That's about all I can think of right now. Work has been grinding me into the ground and I just don't have much energy to do anything else.

Holy cow I just realized it has been a whole YEAR since my last post. Where has the time gone?!?!?!