WHAT ATTRACTED YOUR ATTENTION TODAY?

[Cathy]

Wendy, I will keep the FaceBook link in mind. Thanks.

[Stanley]

Peter, it's asking me for my password frequently as well. I don't think it's personal!

[PanBiker]

I have just managed to get this page up to reply after 15 minutes at least delay. This included 502 bad gateway and asking for my password.

In the few times I've been able to get in recently I've had to enter my password. Is this due to the main problem or have we set it in an attempt to make it harder for `bad actors'?

No changes to the password requirements. At the moment anything can happen I suppose a bit like "Stingray"

I have given the site 4 hours of my life today and I am at a point near to getting my ducks in a row. I will sprinkle the Firewall with some more IP ranges to block later. I'm going to have a break now and wash the car!

We seem to be in a trough at the moment.

[Gloria]

Ian,
It is very much appreciated all the effort you are putting in to sort this, I will start a round of applause for you

[PanBiker]

This is what we are up against. The graphs should be a relatively straight line somewhere up the scale when operating normally.

CPU and Ram.JPG

I have isolated and added to the firewall 16,903,955 IP addresses that are causing problems, that is roughly half of what is in the logs. All these belong to Amazon Web Servers that have been attacked by spammers who are using the Amazon addresses

[PanBiker]

Oh, didn't get round to washing the car.

[Big Kev]

Oh, didn't get round to washing the car.

It rained anyway

[Stanley]

It sounds like a massive task Ian. Thanks for the effort.

[Big Kev]

There is activity going on 'under the bonnet', Ian is researching another service that Ionos can provide to block the IP addresses. Hopefully he'll be able to provide an update on progress later on.

[PanBiker]

Kev is very good at ferreting the options out. Trying to find out what we can get for free as a start. There are subscription or paid options available also but no pricing yet. Spam blocking is probably very good business, probably replaced to some degree anti-virus threats. Apparently a Cloud Expert will be in contact soon.

[Big Kev]

I've upgraded the mySQL database that's 'behind' the site. The monitoring is currently showing a decrease in the CPU load average, not sure if we're out of the woods yet. I will continue to monitor.

[Stanley]

Only a very slight delay in connecting so far this morning. It is accompanied by the bouncing ball in the Oneguy tab marker.
Such a luxury to be able to function. We lose sight of the really important things of life and the last week has definitely reinforced how important Oneguy is to me!
Let me know if you need any subsidy for anti-spam activities.
Interesting comment there about the relative value of anti-spam compared to anti-virus....

[Big Kev]

I emailed abuse@amazonaws.com yesterday with a list of their server IP addresses that are spamming the site. I have been told they take these things seriously.

[PanBiker]

Site is operational this morning. I have checked the traces and the CPU is running at around 80% which is still too high and well above normal. We are no doubt still being hit with spam packets but I have given up trying to chase these. It's like flogging a dead horse, time consuming and very frustrating because as soon as you identify one lot that are doing damage they just steal another batch of different IP's to do the same. So you have to start again.

We have to move to an automated defence system which is what we are working on at the moment. Ball is in the IONOS court at the moment and I am waiting for feedback.

The option is not user configurable so we cant install it ourselves. Effectively it will be a separate database of known rogue IP addresses which is maintained by IONOS. The site traffic will be redirected, filtered, and packets compared with the database before being processed. Any packets from known bad IP's will be automatically blocked.

There is a free version and an enhanced version with more control, bells and whistles. I have asked for details of both. Free version will definitely be a benefit for the site and may well give us a bit of breathing space.

[Stanley]

"Site is operational this morning."
Depends what you mean by operational..... I'm still getting routine 502 errors and delays.
I've just been listening to Marta Lane Fox on 'The Week in Westminster' and I have a big problem with her and many others like her. They talk so fast that if you miss one word and the context goes it becomes gibberish. I had no problem with the other guests or the presenter so don't blame my hearing or the speakers!

[PanBiker]

By operational I meant it was working at the time I posted as it is now. If you look at the CPU trace I put up earlier you can see why the site is up and down. The graph is basically a square wave but the highs which flood the CPU are much longer duration than the lows which allows the site to function. As I said before, under normal usage and not under attack the CPU usually runs at 25% - 30%, anything above about 85% will reflect as the site slowing down when posting etc and eventually the 502 errors when it is fully overwhelmed.

To put into context what we are trying to deal with. The current 32 bit addressing used by the IPv4 network protocol creates 4,294,967,296 individual and unique IP addresses. Large IT corporations such as Microsoft, Amazon, Google etc own large numbers of IP addresses (hundreds of thousanad or even millions) from this pool which they either sell on or lease to smaller companies and users which is how they make money. It's a trickle down business with each owner making a few bob on each address. IONOS will have a pool which they vend out to their customers like us. We pay for use of the IP addresses that run the site.

The spammers relay on these large pools and steal huge chunks of them, not usually in continuous blocks and then use them effectively by allocating them to spam bots, thousands of them in fact. They set them off on their merry way with whatever payload or action they have been given. Once triggered the operation is fully automatic and only needs updating when the IP pool has been rumbled and added to thousand of firewalls and other anti spam defence systems. When this happens they simply refresh their pool with different stolen address. The other day I banned over 16 million addresses in our firewall that seem to belong to Amazon Web Servers but have been stolen by the spammers. DOS attacks (Denial of Service) are effectively the same as kidnapping as they can be used to hold target companies, institutions, businesses to ransom. Invariably the spammers are working from VPN's and using Proxy servers which effectively mask who and where they are.

The world has actually run out of 32 bit IP addresses and the entire system is now running on the legacy technology that supports it. Since it's inception all the security vulnerabilities have been found and are relied upon by the spammers.

To kill two birds with one stone a new protocol using 128bit encoding has been developed this is the next step in the evolution of the global networks. This is IPv6 and instead of the 4 x 8 bits used in 32bit addressing, IPv6 uses 6 x 128 bit addressing. If you multiply this out you arrive at 340, Trillion, Trillion, Trillion unique addresses. Each address in it's binary form will be 768 digits long! I'm glad I wont be around to have to deal with any problems with that network protocol.

[Big Kev]

I have managed to apply the firewall changes, the lists of AWS IP addresses that Ian added are now actively blocked. I have seen a big drop in CPU usage and will continue to monitor this afternoon for anything unusual.

A bit later, CPU is around 22% which is promising. I can see regular users logged into the site, good to see Mick Brett logged in as a lot of the blocked IPs are US based

CPU.JPG

The CPU trace for the last hour or so.

I'm still monitoring

[PanBiker]

It's continuing at 20 - 30% at the moment.

[Big Kev]

I'd best get back to my day job now

[Big Kev]

Last hour CPU

CPU2.JPG

Tomorrow morning will be the real test

[Gloria]

Excellent

[PostmanPete]

Thanks Ian and Kev for all the work you have done over the last few days. Hopefully you have got the site back to normal now and can get back to your normal routine It looks to be working fine at my end.

[Big Kev]

That's good to hear Pete, there was one hiccup when I accidentally blocked Wendy

All sorted now though. This did involve removing a couple of the newly applied firewall rules but, hopefully, as the range is UK based, it may not impact things.

CPU usage is still good and there's nowt weird in the logs.

[Wendyf]

Perhaps I was causing the trouble all along. I'll get my coat.....

[Big Kev]

Perhaps I was causing the trouble all along. I'll get my coat.....