Page 1 of 1
Loss of service for several minutes
Posted: Wed Aug 28, 2024 10:47 pm
by spot
Forumgarden has seen higher than usual web page access from illegitimate bots over the last 20 hours. For the time being I've put a stop to it. I can't tell whether it will develop into a DDoS attack - I've no reason to think we've done anything to warrant one. It had us repeatedly peaking at over 20MB/s disk accesses though, and touching 100% CPU, corresponding to consistently higher than 10 reads a second for hours on end. Insignificant figures for larger sites but not this one.
I'll sleep on it and consider how to deal with this more long-term. Geo-blocking, perhaps.
If anyone else noticed, or notices, do please add to the thread with a description.
Re: Loss of service for several minutes
Posted: Thu Aug 29, 2024 5:31 am
by spot
There we are, I got six hours sleep after sorting that - the graph indicates a successful bodyslam of a minor DoS in the tradition of that old-time operator Mick McManus.
- Screenshot_2024-08-29_13-21-28.png (108.24 KiB) Viewed 6153 times
What you're looking for as confirmation is /var/log/apache2/error.log showing authz_core:error announcements like (and note the attacks are continuing, this is now):
Code: Select all
[Thu Aug 29 12:25:40.063345 2024] [authz_core:error] [pid 19432:tid 19432] [client 43.133.38.100:59988] AH01630: client denied by server configuration: /var/www/xx/forums/viewforum.php, referer: https://google.com
[Thu Aug 29 12:25:40.215427 2024] [authz_core:error] [pid 19518:tid 19518] [client 43.133.43.227:49500] AH01630: client denied by server configuration: /var/www/xx/forums/viewforum.php, referer: https://google.com
[Thu Aug 29 12:25:40.328066 2024] [authz_core:error] [pid 19374:tid 19374] [client 43.153.5.20:60008] AH01630: client denied by server configuration: /var/www/xx/forums/viewforum.php, referer: https://google.com
[Thu Aug 29 12:25:40.404979 2024] [authz_core:error] [pid 19458:tid 19458] [client 43.159.146.48:34278] AH01630: client denied by server configuration: /var/www/xx/forums/viewforum.php
and the code to make it happen is in the virtual host directory configuration, such as:
Code: Select all
<Directory /[directory masked for privacy reasons]>
DirectoryIndex index.php
AllowOverride All
# New syntax for Apache 2.4 and later
<RequireAll>
Require all granted
Require not ip 43.0.0.0/8
Require not ip 49.51.0.0/16
Require not ip 66.249.64.0/16
Require not ip 85.208.96.0/16
Require not ip 94.74.0.0/16
Require not ip 101.32.0.0/16
Require not ip 101.44.0.0/16
Require not ip 110.238.0.0/16
Require not ip 111.119.0.0/16
Require not ip 114.119.0.0/16
Require not ip 119.8.0.0/16
Require not ip 119.13.0.0/16
Require not ip 124.156.0.0/16
Require not ip 124.243.0.0/16
Require not ip 129.226.0.0/16
Require not ip 150.109.0.0/16
Require not ip 154.54.249.0/16
Require not ip 159.138.0.0/16
Require not ip 166.108.0.0/16
Require not ip 170.106.0.0/16
Require not ip 185.191.171.0/16
Require not ip 190.92.0.0/16
Require not ip 199.167.138.0/16
Require not ip 216.244.66.0/16
Require not ip 217.113.194.0/16
</RequireAll>
</Directory>
It's my guess that the spiky nature as the attack builds is a timing effort to avoid fail2ban triggers for a frequency and volume trap, it allows the timers to expire before the next push instead of a potential 24 hour or one week or permanent ban.
And they all originate in:
ISP: Tencent Building, Kejizhongyi Avenue
Organization: Tencent Building, Kejizhongyi Avenue
and
ISP: Shenzhen Tencent Computer Systems Company Limited
Organization: 16 COLLYER QUAY # 18-29 INCOME AT RAFFLES (tencent.com)
in which case I may be slightly out of my league here.
Or it may be a rogue unwanted spider which ignores polite requests to go away.
Re: Loss of service for several minutes
Posted: Thu Aug 29, 2024 10:01 am
by spot
It's been back for 12 hours now and the fix appears to be holding:
- Screenshot_2024-08-29_18-01-11.png (128.81 KiB) Viewed 6139 times
Re: Loss of service for several minutes
Posted: Fri Aug 30, 2024 2:47 am
by spot
That's more like it:
- Screenshot_2024-08-30_10-46-35.png (137.54 KiB) Viewed 6125 times
For anyone watching who needs to fend off this particular bunch of antisocial data thieves, here's my blanket suppression of their current spider hosts. I'm aware my ranges are too broad but I was annoyed when I specified them.
Code: Select all
<Directory /var/www/xx>
DirectoryIndex index.php
AllowOverride All
# New syntax for Apache 2.4 and later
<RequireAll>
Require all granted
Require not ip 43.0.0.0/8
Require not ip 3.224.0.0/16
Require not ip 23.22.0.0/16
Require not ip 34.230.0.0/16
Require not ip 49.0.0.0/16
Require not ip 49.51.0.0/16
Require not ip 52.70.0.0/16
Require not ip 54.36.0.0/16
Require not ip 74.201.0.0/16
Require not ip 85.208.0.0/16
Require not ip 94.74.0.0/16
Require not ip 101.32.0.0/16
Require not ip 101.44.0.0/16
Require not ip 110.238.0.0/16
Require not ip 111.119.0.0/16
Require not ip 114.119.0.0/16
Require not ip 119.13.0.0/16
Require not ip 119.28.0.0/16
Require not ip 119.8.0.0/16
Require not ip 124.156.0.0/16
Require not ip 124.243.0.0/16
Require not ip 129.226.0.0/16
Require not ip 150.109.0.0/16
Require not ip 159.138.0.0/16
Require not ip 166.108.0.0/16
Require not ip 170.106.0.0/16
Require not ip 182.43.0.0/16
Require not ip 185.191.0.0/16
Require not ip 190.92.0.0/16
Require not ip 199.167.0.0/16
Require not ip 216.244.0.0/16
</RequireAll>
</Directory>
Re: Loss of service for several minutes
Posted: Sat Aug 31, 2024 3:40 am
by spot
The rogue bot stopped accessing the site at 8am today. Until then they'd continued trying to hammer the site from a thousand or so IP addresses, but all their requests were rejected. So was Alexa's bot which got caught up in the rejection list.
I've removed some of the constraints for today to see what happens.
I've left a couple in place though, requiring reverse proxy lookup success before responding. I'll take that off on Monday.
If anyone has found ForumGarden inaccessible this weekend, that will be the reason. PM me and I'll leave reverse proxy lookup permanently disabled. Otherwise if nobody speaks and the problem returns I may reinstate it.
Re: Loss of service for several minutes
Posted: Sun Mar 23, 2025 2:47 pm
by spot
Start: 2025-04-02 20:00 UTC
End: 2025-04-02 23:00 UTC
Hello,
During the above window, our Networking team will be making changes to the core networking infrastructure to improve performance and scalability in the LON1 region.
Expected impact:
During the maintenance window, users may experience delays or failures with event processing for a brief duration on Droplets and Droplet-based services, including Droplets, Managed Kubernetes, Load Balancers, Container Registry, and App Platform. We will endeavor to keep this to a minimum for the duration of the change.
If you have any questions related to this issue, please send us a ticket from your cloud support page.
https://cloudsupport.digitalocean.com/s/createticket
Thank you,
Team DigitalOcean
Re: Loss of service for several minutes
Posted: Sat Mar 29, 2025 12:35 pm
by spot
Hello,
We are reaching out again to inform you that the Network maintenance in LON1 region which was previously scheduled to be complete on 2025-04-02 20:00 UTC has been rescheduled to the following window:
Start: 2025-04-08 20:00 UTC
End: 2025-04-08 23:00 UTC
We apologize for any inconvenience this short notice causes and thank you for your understanding. You may find the initial maintenance notice along with a description of any expected impact related to this work included at the bottom of this message.
If you have questions or concerns about this maintenance, please reach out to us by opening up a ticket on your account.
Thank you,
Team DigitalOcean
Re: Loss of service for several minutes
Posted: Mon Apr 28, 2025 6:08 am
by spot
The site is visibly slow at the moment. Google is indexing the entire site by the look of it. Eventually I might intervene but it's potentially a benefit. Bear with, bear with...
Re: Loss of service for several minutes
Posted: Mon Apr 28, 2025 9:38 am
by spot
A helpful PM, thank you. That sod is using an Agent Name to which he he not entitled.
The last 24 hour graph is exactly as you'd expect. The 3am archive runs, the bad actor arrives, I take the site down for quarter of an hour's analysis, and it's straight back in when I restart.
I might leave it a bit and see if it sates itself, otherwise I'll put significant blocks in place.
- Screenshot_2025-04-28_17-32-03.png (86.42 KiB) Viewed 1783 times
Re: Loss of service for several minutes
Posted: Mon Apr 28, 2025 1:00 pm
by Bryn Mawr
And ignoring robots.txt which is bad form
Re: Loss of service for several minutes
Posted: Tue Apr 29, 2025 4:09 am
by spot
All out of China, like last time, these associated with Alibaba Cloud (Aliyun) rather than tencent. Blocked again now, the CPU is more modest, it may still feel less immediately responsive than it ought because the pummelling is still there. This evening I'll consider bricking up the port rather than a gatekeeper on the database.
Re: Loss of service for several minutes
Posted: Wed Apr 30, 2025 6:09 am
by spot
The backlog of vibrations settled overnight, response time is back to speedy. Do add observations to this thread if you find it sluggish again, it helps us focus on potential problems faster.
Speedy ought to be significantly less than a second.
Re: Loss of service for several minutes
Posted: Wed Apr 30, 2025 3:54 pm
by spot
The Chinabots came back 15 minutes ago. I've tightened the firewall and added rate limiting, and rebooted. It may again take a while to shake out the delay, I'll keep an eye on it.
Re: Loss of service for several minutes
Posted: Wed Apr 30, 2025 4:37 pm
by Bryn Mawr
Still up around 9 seconds as at 23:35 UTC
Re: Loss of service for several minutes
Posted: Wed Apr 30, 2025 4:38 pm
by spot
Bryn Mawr wrote: Wed Apr 30, 2025 4:37 pm
Still up around 9 seconds as at 23:35 UTC
It is, isn't it. And I'm exploring reasons. I'm moderately puzzled at the moment.
Re: Loss of service for several minutes
Posted: Wed Apr 30, 2025 5:06 pm
by spot
Sorted.
Re: Loss of service for several minutes
Posted: Sat May 03, 2025 6:51 am
by spot
I just crashed the site for ten minutes.
It just goes to show how much the world needs a Pope.
Back now, I'll leave it alone for a few hours.