Major board outage, 12/5/2025 info.

What can the Admins do to help, any requests for changes, updates, etc.
User avatar
weeksy
Site Admin
Posts: 26898
Joined: Sat Mar 14, 2020 12:08 pm
Has thanked: 6112 times
Been thanked: 15461 times

Re: Major board outage, 12/5/2025 info.

Post by weeksy »

weeksy wrote: Wed May 14, 2025 2:18 pm Bear with us guys... honestly i'm working on it.

Update here..
Message: Hello Steve,

We regret to inform you that there has been a critical security incident on server s3945.lon1.stableserver.net regarding your domain revtothelimit.co.uk. The server experienced a flood of requests, with a rate of 229.0 requests per second.

To mitigate this security threat, we have taken immediate action by adding a rule to the .htaccess file for your domain. The following rules have been implemented:

deny from all
RewriteEngine On
RewriteRule .* - [R=500,L]

These measures are aimed at preventing further unauthorized access and protecting the integrity of your website.

We advise you to review your website's security measures and consider implementing additional safeguards to prevent future incidents such as using Cloudflare and set to under-attack mode (https://developers.cloudflare.com/funda ... ttack-mode). Should you require any assistance or have further concerns, please do not hesitate to reach out to our support team.

Thank you for your attention to this matter.

Sincerely,
Trust and Safety Team
So basically from there we've regsitered, deployed and implemented Cloudflare which is a security package that stops 'bots'.

For whatever reason, the website (and lots of others) has come under attack from bots.
cloudflare shows some interesting stats on http requests.
Vietnam 299,647
Brazil 148,060
Indonesia 16,436
India 2,356
United States 768

I'm currently installing, configuring and trying to stop this.... I'm using all of their inbuilt 'free' tools, but if we want to keep this place going we may need to consider whether we purchase their full suite, which i think is £210 per year...

I know i know.. it's not ideal... but it comes down to how much you lot want/need this place.

When they allowed it back 20 mins ago the CPU was peaked at 99% hence why you/we couldn't really get in. So i've deployed stronger security which i'm hoping has lowered it, it was 61% last time i looked, but i'll need to work out and see what it was say 3-4 days ago as i've never needed to see beofre so don't know

All i can say is, @HiFi Kabin was working on stuff today/yesterday and of course i'm working on it now...

How far i'll get, i don't truthfully know.
i've just configured a rule to block the 4 most prolific countries using Cloudflare. We'll see if that helps, i'm not sure of the actual long term answers at the moment, for now it's just trying to fight the battle and get things happy. Singapore has now moved into the list from the above list.

The figures for them at the moment are massive with the overnight stats updating.
Vietnam 2,661,092
Brazil 2,125,701
Singapore 1,248,719
Indonesia 128,864
User avatar
weeksy
Site Admin
Posts: 26898
Joined: Sat Mar 14, 2020 12:08 pm
Has thanked: 6112 times
Been thanked: 15461 times

Re: Major board outage, 12/5/2025 info.

Post by weeksy »

weeksy wrote: Thu May 15, 2025 6:23 am i've just configured a rule to block the 4 most prolific countries using Cloudflare. We'll see if that helps, i'm not sure of the actual long term answers at the moment, for now it's just trying to fight the battle and get things happy. Singapore has now moved into the list from the above list.

The figures for them at the moment are massive with the overnight stats updating.
Vietnam 2,661,092
Brazil 2,125,701
Singapore 1,248,719
Indonesia 128,864
Since doing this about 25 mins ago it's blocked 175,000 requests to our server/board.

The one thing i don't actually know is, what are they doing it for, what's their logic/process and why.

CPU usage is sitting at 28% so we're deffo not in danger of overloading the box now, so apart from the large number of guests, i'm pretty happy we're OK. I'll keep looking into that and seeing what we can do to get it down along with @HiFi Kabin and the other techs on PHP forum...
User avatar
Count Steer
Posts: 15868
Joined: Mon Jul 19, 2021 4:59 pm
Has thanked: 8005 times
Been thanked: 5663 times

Re: Major board outage, 12/5/2025 info.

Post by Count Steer »

Why they're doing it? Either a) money ie trying to squeeze PHP or the PHP user community (unlikely) or b) because they can. Hackers just like to hack because they're sad, misfit bastards sat in their bedrooms punching the air when they've made someone take notice of them and (virtually) slapping each other on the back.

If you search on DoS and DDoS attacks on PHP some of the first things that come up are the scripts to do it. Some of them publish them for 'security and research' purposes only. :roll:

At least the money angle makes some sense!

(There is a c) that assumes most viruses and security attacks are spawned by antivirus and security software companies....but that's getting a bit conspiracy-ish. :lol: ).
The plural of 'anecdote' is not 'data'.
User avatar
weeksy
Site Admin
Posts: 26898
Joined: Sat Mar 14, 2020 12:08 pm
Has thanked: 6112 times
Been thanked: 15461 times

Re: Major board outage, 12/5/2025 info.

Post by weeksy »

I know i'm banging on about this now.. LOL... but it's something of a bugbear for me now and i want to get it right.

In the interests of checking i just looked at PHPBBs stats currently,
In total there are 3556 users online :: 17 registered, 0 hidden and 3539 guests (based on users active over the past 30 minutes)

So That's a lot more than we have currently (approx 1500) so i'm happy we're at least doing things somewhat right.
User avatar
HiFi Kabin
Posts: 68
Joined: Mon Mar 16, 2020 5:33 pm
Been thanked: 98 times

Re: Major board outage, 12/5/2025 info.

Post by HiFi Kabin »

A quick (??) rundown on what and why.

Firstly phpBB.com (like all large sites) is on a dedicated server with more resources and can cope better with a higher guest limit. RTTL, HiFiKabin and other smaller sites are on a shared server with the same resources shared between many sites, hence the times when overload happens.

Why do they do it? @Count Steer is correct when he says "because they can" but in addition others are trying to gain passwords* in order to gain access either here, or on any other site where you use the same UN/PW combination. That is why its essential to use a unique PW on every site you use.

*Passwords are stored in the phpBB database in a way that makes them irrecoverable. Your password is first salted and then hashed which is a one way process and can not be reverse engineered to produce the original password.

99% of the time Bots, 'Script Kiddies' and Hackers dont care, or are even aware of the software they are attacking. They just hit everything they can find on the web.**

** On my sites I get attacks on Wordpress files even though I don't have, and have never had a Wordpress site
User avatar
weeksy
Site Admin
Posts: 26898
Joined: Sat Mar 14, 2020 12:08 pm
Has thanked: 6112 times
Been thanked: 15461 times

Re: Major board outage, 12/5/2025 info.

Post by weeksy »

HiFi Kabin wrote: Thu May 15, 2025 11:41 am A quick (??) rundown on what and why.

Firstly phpBB.com (like all large sites) is on a dedicated server with more resources and can cope better with a higher guest limit. RTTL, HiFiKabin and other smaller sites are on a shared server with the same resources shared between many sites, hence the times when overload happens.

Why do they do it? @Count Steer is correct when he says "because they can" but in addition others are trying to gain passwords* in order to gain access either here, or on any other site where you use the same UN/PW combination. That is why its essential to use a unique PW on every site you use.

*Passwords are stored in the phpBB database in a way that makes them irrecoverable. Your password is first salted and then hashed which is a one way process and can not be reverse engineered to produce the original password.

99% of the time Bots, 'Script Kiddies' and Hackers dont care, or are even aware of the software they are attacking. They just hit everything they can find on the web.**

** On my sites I get attacks on Wordpress files even though I don't have, and have never had a Wordpress site
I find it slightly bizarre personally, but then again i don't quite know what they'd do with said passwords as anything to do with money these days has 2FA anyhow, so even if my password was a password, they still wouldn't get anywhere.
User avatar
Count Steer
Posts: 15868
Joined: Mon Jul 19, 2021 4:59 pm
Has thanked: 8005 times
Been thanked: 5663 times

Re: Major board outage, 12/5/2025 info.

Post by Count Steer »

Well they do say that the M&S + CooP hacks were done by hijacking phone sims of staff with the right level of authority so they got the second part of the 2FA.

So they didn't exactly hack in, just impersonated someone with the right credentials and strolled in.
The plural of 'anecdote' is not 'data'.
User avatar
Rockburner
Posts: 6008
Joined: Sun Mar 15, 2020 11:06 am
Location: Hiding in your blind spot
Has thanked: 10930 times
Been thanked: 3975 times

Re: Major board outage, 12/5/2025 info.

Post by Rockburner »

Count Steer wrote: Thu May 15, 2025 12:34 pm Well they do say that the M&S + CooP hacks were done by hijacking phone sims of staff with the right level of authority so they got the second part of the 2FA.

So they didn't exactly hack in, just impersonated someone with the right credentials and strolled in.
That's the process - get some base level personally identifying info from one source, cross-ref with other sources to build a profile on a user, then use that combined info to target higher-security systems with "I can't login" type support requests with enough "supporting information" to scam the support staff into allowing access.
non quod, sed quomodo
User avatar
weeksy
Site Admin
Posts: 26898
Joined: Sat Mar 14, 2020 12:08 pm
Has thanked: 6112 times
Been thanked: 15461 times

Re: Major board outage, 12/5/2025 info.

Post by weeksy »

So as a bit of a close on this, we're running happily now, still not perfect, but pretty happy.

We're using the 'free' version of Cloudflare which allows us to create rules to block locations. Whilst that's not got rid of everything it's certainly doing a job for us as it is and i've managed to turn off "i'm being attacked" mode, which as you may have seen means you don't get prompted all the time to ensure you're human. This hasn't had any adverse effects on the numbers, so i'll leave it off for now.

Cloudflare has stopped 2.3m attempts since we set it up 24 hours ago.... that is completely bonkers !

Any issues you see, do please let me know.
User avatar
MyLittleStudPony
Posts: 1736
Joined: Sat Mar 14, 2020 9:28 pm
Has thanked: 827 times
Been thanked: 583 times

Re: Major board outage, 12/5/2025 info.

Post by MyLittleStudPony »

I won't pretend to understand all this, the words that were said and the internet and ting. But a big 'thank you' to Weeksy and Mr Kabin for being on top of it. :thumbup: :thumbup: :thumbup:
User avatar
weeksy
Site Admin
Posts: 26898
Joined: Sat Mar 14, 2020 12:08 pm
Has thanked: 6112 times
Been thanked: 15461 times

Re: Major board outage, 12/5/2025 info.

Post by weeksy »

MyLittleStudPony wrote: Thu May 15, 2025 1:49 pm I won't pretend to understand all this,
Me neither, but i'm an IT nerd at work so i generally pick a fair bit of it up when i have to. It's funny but as i do it all day i don't really have much in the way of interest in outside stuff, hence why i generally get Kabin to do our updates and the PHP backend stuff, he does it a lot so it makes more sense to get him in on the action at times.

But when i have to do some things, it's generally mostly just 'computers' so when you have a rough idea, the rest of it makes a certain amount of sense... Once you throw in some guesswork, you're usually OK :D
User avatar
weeksy
Site Admin
Posts: 26898
Joined: Sat Mar 14, 2020 12:08 pm
Has thanked: 6112 times
Been thanked: 15461 times

Re: Major board outage, 12/5/2025 info.

Post by weeksy »

Well it seems that at 11pm roughly last night whatever or whoever was going bananas at us, stopped doing it. The graphs show a massive drop off in activity

Imagegraph by Steve Weeks, on Flickr
Mr. Dazzle
Posts: 16347
Joined: Mon Mar 16, 2020 7:57 pm
Location: Milton Keynes
Has thanked: 2417 times
Been thanked: 6369 times

Re: Major board outage, 12/5/2025 info.

Post by Mr. Dazzle »

Guessing these "things" give up after a while if you have Cloudflare or similar?
User avatar
weeksy
Site Admin
Posts: 26898
Joined: Sat Mar 14, 2020 12:08 pm
Has thanked: 6112 times
Been thanked: 15461 times

Re: Major board outage, 12/5/2025 info.

Post by weeksy »

Mr. Dazzle wrote: Fri May 16, 2025 6:56 am Guessing these "things" give up after a while if you have Cloudflare or similar?
That seems to be the implication from things i've read yeah. But i'm still pretty new to all this, so it's a bit of guesswork as well.
User avatar
Skub
Posts: 14892
Joined: Mon Mar 16, 2020 5:32 pm
Location: Norn Iron
Has thanked: 13109 times
Been thanked: 14170 times

Re: Major board outage, 12/5/2025 info.

Post by Skub »

Ta very much to Weeksy and Kabin for dealing with the ballache and as ever,if we need to put our hands in our pockets,just ask.
"Be kind to past versions of yourself that didn't know what you know now."
Walt Whitman
https://soundcloud.com/skub1955
User avatar
weeksy
Site Admin
Posts: 26898
Joined: Sat Mar 14, 2020 12:08 pm
Has thanked: 6112 times
Been thanked: 15461 times

Re: Major board outage, 12/5/2025 info.

Post by weeksy »

Skub wrote: Fri May 16, 2025 9:00 am Ta very much to Weeksy and Kabin for dealing with the ballache and as ever,if we need to put our hands in our pockets,just ask.
I paid Kabin for his efforts.. I do this for fun lol.. I've not looked in the forum funding pot, but it's not due for many many months yet, so not a massive issue.
User avatar
HiFi Kabin
Posts: 68
Joined: Mon Mar 16, 2020 5:33 pm
Been thanked: 98 times

Re: Major board outage, 12/5/2025 info.

Post by HiFi Kabin »

weeksy wrote: Fri May 16, 2025 6:57 am
Mr. Dazzle wrote: Fri May 16, 2025 6:56 am Guessing these "things" give up after a while if you have Cloudflare or similar?
That seems to be the implication from things i've read yeah. But i'm still pretty new to all this, so it's a bit of guesswork as well.
That appears to be the case. One board that I manage is totally private and all that is visible to anyone who isn’t logged in is just the log in page with all content hidden.

The guest count on that board is very low with an occasional 'spike' as new bots/script kiddies find it, but as they can not access the content they loose interest very quickly.
User avatar
weeksy
Site Admin
Posts: 26898
Joined: Sat Mar 14, 2020 12:08 pm
Has thanked: 6112 times
Been thanked: 15461 times

Re: Major board outage, 12/5/2025 info.

Post by weeksy »

I think our Aussie ex-pat was locked out yesterday as i blocked by Continent so i'm hoping now i've put Oceania back on he'll be heading back over when timezones work for him.

But i may need to work on a longer term solution as it depends if they class Vietnam as Asia or Oceania.... Google says Asia... so hopefully that's what CLoudflare will go with.
User avatar
weeksy
Site Admin
Posts: 26898
Joined: Sat Mar 14, 2020 12:08 pm
Has thanked: 6112 times
Been thanked: 15461 times

Re: Major board outage, 12/5/2025 info.

Post by weeksy »

Well I think we can happily call this crisis over. Whilst we still get Bots, no more than welcome should.

We do have consider one more thing, the board software is out of date lol
User avatar
Count Steer
Posts: 15868
Joined: Mon Jul 19, 2021 4:59 pm
Has thanked: 8005 times
Been thanked: 5663 times

Re: Major board outage, 12/5/2025 info.

Post by Count Steer »

weeksy wrote: Wed May 21, 2025 9:02 pm Well I think we can happily call this crisis over. Whilst we still get Bots, no more than welcome should.

We do have consider one more thing, the board software is out of date lol
Ah. Is that a straightforward upgrade? I'm guessing not if each board can do mucho configuration and 'tweaks'.
The plural of 'anecdote' is not 'data'.
Post Reply