Loss of service for several minutes

[spot]

Forumgarden has seen higher than usual web page access from illegitimate bots over the last 20 hours. For the time being I've put a stop to it. I can't tell whether it will develop into a DDoS attack - I've no reason to think we've done anything to warrant one. It had us repeatedly peaking at over 20MB/s disk accesses though, and touching 100% CPU, corresponding to consistently higher than 10 reads a second for hours on end. Insignificant figures for larger sites but not this one.

I'll sleep on it and consider how to deal with this more long-term. Geo-blocking, perhaps.

If anyone else noticed, or notices, do please add to the thread with a description.

[spot]

There we are, I got six hours sleep after sorting that - the graph indicates a successful bodyslam of a minor DoS in the tradition of that old-time operator Mick McManus.

Screenshot_2024-08-29_13-21-28.png (108.24 KiB) Viewed 131 times

What you're looking for as confirmation is /var/log/apache2/error.log showing authz_core:error announcements like (and note the attacks are continuing, this is now):

Code: Select all

[Thu Aug 29 12:25:40.063345 2024] [authz_core:error] [pid 19432:tid 19432] [client 43.133.38.100:59988] AH01630: client denied by server configuration: /var/www/xx/forums/viewforum.php, referer: https://google.com
[Thu Aug 29 12:25:40.215427 2024] [authz_core:error] [pid 19518:tid 19518] [client 43.133.43.227:49500] AH01630: client denied by server configuration: /var/www/xx/forums/viewforum.php, referer: https://google.com
[Thu Aug 29 12:25:40.328066 2024] [authz_core:error] [pid 19374:tid 19374] [client 43.153.5.20:60008] AH01630: client denied by server configuration: /var/www/xx/forums/viewforum.php, referer: https://google.com
[Thu Aug 29 12:25:40.404979 2024] [authz_core:error] [pid 19458:tid 19458] [client 43.159.146.48:34278] AH01630: client denied by server configuration: /var/www/xx/forums/viewforum.php

and the code to make it happen is in the virtual host directory configuration, such as:

Code: Select all

      <Directory /[directory masked for privacy reasons]>
            DirectoryIndex index.php
            AllowOverride All

# New syntax for Apache 2.4 and later
            <RequireAll>
                  Require all granted
                  Require not ip 43.0.0.0/8
                  Require not ip 49.51.0.0/16
                  Require not ip 66.249.64.0/16
                  Require not ip 85.208.96.0/16
                  Require not ip 94.74.0.0/16
                  Require not ip 101.32.0.0/16
                  Require not ip 101.44.0.0/16
                  Require not ip 110.238.0.0/16
                  Require not ip 111.119.0.0/16
                  Require not ip 114.119.0.0/16
                  Require not ip 119.8.0.0/16
                  Require not ip 119.13.0.0/16
                  Require not ip 124.156.0.0/16
                  Require not ip 124.243.0.0/16
                  Require not ip 129.226.0.0/16
                  Require not ip 150.109.0.0/16
                  Require not ip 154.54.249.0/16
                  Require not ip 159.138.0.0/16
                  Require not ip 166.108.0.0/16
                  Require not ip 170.106.0.0/16
                  Require not ip 185.191.171.0/16
                  Require not ip 190.92.0.0/16
                  Require not ip 199.167.138.0/16
                  Require not ip 216.244.66.0/16
                  Require not ip 217.113.194.0/16
            </RequireAll>
      </Directory>

It's my guess that the spiky nature as the attack builds is a timing effort to avoid fail2ban triggers for a frequency and volume trap, it allows the timers to expire before the next push instead of a potential 24 hour or one week or permanent ban.

And they all originate in:

ISP: Tencent Building, Kejizhongyi Avenue
Organization: Tencent Building, Kejizhongyi Avenue

and
ISP: Shenzhen Tencent Computer Systems Company Limited
Organization: 16 COLLYER QUAY # 18-29 INCOME AT RAFFLES (tencent.com)

in which case I may be slightly out of my league here.

Or it may be a rogue unwanted spider which ignores polite requests to go away.

[spot]

It's been back for 12 hours now and the fix appears to be holding:

Screenshot_2024-08-29_18-01-11.png (128.81 KiB) Viewed 117 times

[spot]

That's more like it:

Screenshot_2024-08-30_10-46-35.png (137.54 KiB) Viewed 103 times

For anyone watching who needs to fend off this particular bunch of antisocial data thieves, here's my blanket suppression of their current spider hosts. I'm aware my ranges are too broad but I was annoyed when I specified them.

Code: Select all

      <Directory /var/www/xx>
            DirectoryIndex index.php
            AllowOverride All

# New syntax for Apache 2.4 and later
            <RequireAll>
                  Require all granted

                  Require not ip 43.0.0.0/8

                  Require not ip 3.224.0.0/16
                  Require not ip 23.22.0.0/16
                  Require not ip 34.230.0.0/16
                  Require not ip 49.0.0.0/16
                  Require not ip 49.51.0.0/16
                  Require not ip 52.70.0.0/16
                  Require not ip 54.36.0.0/16
                  Require not ip 74.201.0.0/16
                  Require not ip 85.208.0.0/16
                  Require not ip 94.74.0.0/16
                  Require not ip 101.32.0.0/16
                  Require not ip 101.44.0.0/16
                  Require not ip 110.238.0.0/16
                  Require not ip 111.119.0.0/16
                  Require not ip 114.119.0.0/16
                  Require not ip 119.13.0.0/16
                  Require not ip 119.28.0.0/16
                  Require not ip 119.8.0.0/16
                  Require not ip 124.156.0.0/16
                  Require not ip 124.243.0.0/16
                  Require not ip 129.226.0.0/16
                  Require not ip 150.109.0.0/16
                  Require not ip 159.138.0.0/16
                  Require not ip 166.108.0.0/16
                  Require not ip 170.106.0.0/16
                  Require not ip 182.43.0.0/16
                  Require not ip 185.191.0.0/16
                  Require not ip 190.92.0.0/16
                  Require not ip 199.167.0.0/16
                  Require not ip 216.244.0.0/16

            </RequireAll>
      </Directory>
      

[spot]

The rogue bot stopped accessing the site at 8am today. Until then they'd continued trying to hammer the site from a thousand or so IP addresses, but all their requests were rejected. So was Alexa's bot which got caught up in the rejection list.

I've removed some of the constraints for today to see what happens.

I've left a couple in place though, requiring reverse proxy lookup success before responding. I'll take that off on Monday.

If anyone has found ForumGarden inaccessible this weekend, that will be the reason. PM me and I'll leave reverse proxy lookup permanently disabled. Otherwise if nobody speaks and the problem returns I may reinstate it.