Chinese Spiders

[russellw]

Good afternoon.

No, I'm not posting about some exotic 8-legged creature but about the web crawlers that have been hammering this site for the last 5 days or so.

These crawlers are ignoring the normal directives that robots are supposed to follow and thus we have several hundred of them crawling the site constantly.

The impact is that the server CPU's get maxxed out at 100% and overall performance slows to a (no pun intended) crawl.

They can normally be kept at bay by blocking them at the server level but for some reason this doesn't appear to be stopping this lot of Huawei crawlers.

There have been problems reported around the world from them so we aren't the only ones being impacted as Huawei try to get their search engine up to the standard of Google as part of their ongoing war with each other.

We are working with out ISP to find a solution.

[Citroënbender]

Thank you. I’d assumed the issue ran downstream and was just my problem.

[lumen8]

How about blocking large swathes of chinese IP's in cpanel, crude but could be a stopgap fix. I doubt youd exclude any real members with this approach.

I realise you may already have done this, thought id mention it.

[russellw]

Originally Posted by lumen8

How about blocking large swathes of chinese IP's in cpanel, crude but could be a stopgap fix. I doubt youd exclude any real members with this approach.

I realise you may already have done this, thought id mention it.

Already done that!

[aussiblue]

I was going to suggest "nuke em till they glow and then shoot 'em in the dark"


But they are the ones with all the nukes and they just want to be loved it seems https://www.bbc.com/news/world-asia-china-57327177

China's president has said he wants the country to "expand its circle of friends" by revamping its image.

Xi Jinping told senior Communist Party officials it was important to present an image of a "credible, loveable and respectable China", according to a report by state-run news agency Xinhua.

[aussiblue]

Any of the comments on this blog helpful https://www.johnlarge.co.uk/blocking...scrapers-bots/ ?

[russellw]

Originally Posted by aussiblue

Any of the comments on this blog helpful https://www.johnlarge.co.uk/blocking...scrapers-bots/ ?

I had read that but we aren't seeing quite the same thing. I've moved the directives into the base apache config file to see if that has any impact.

[russellw]

It 'might' be resolved but there are quite a few of the spiders still connected so we'll need to wait until their sessions expire before knowing for sure.

[Beastie]

Maybe closing the Covid Thread might have an impact...

[Bossxr8]

Have you tried aeroguard?

[lumen8]

Originally Posted by Beastie

Maybe closing the Covid Thread might have an impact...

Housing bubble thread too.


Maybe theyre here to try pick the bottom, get back into it and buy the dip.

[T3rminator]

Is it the sheer number of crawlers or are the crawlers doing weird stuff? Asked some of the folks at work, if the former, they were pointing as some form of DDOS protection....but I believe that is big $$$.

[Tickford.]

I think I've found what these "Chinese Spiders" are trying to attack...

https://fordforums.com.au/showthread.php?t=11484389

[wodahs]

your kidding right

we all know what thread their here (like every one else) for !!
https://fordforums.com.au/showthread.php?t=11483655

[Pis-ton broke]

there jealous because they don't have 351's and xa to xc coupes, oh and not forgetting xy gtho phase 3's.

[FoxtrotGolfXray 5.0]

I thought they were all just salivating over the delicious delicacies in this thread

[russellw]

Originally Posted by T3rminator

Is it the sheer number of crawlers or are the crawlers doing weird stuff? Asked some of the folks at work, if the former, they were pointing as some form of DDOS protection....but I believe that is big $$$.

It's mostly the number but also the speed they are crawling. They are doing what they normally do when they are here but the general rule is that the number of spiders is kept reasonably small so that the impact isn't great. For example, we generally have about 30 Google or MSN spiders at any one time but we had more like 150 of these Huawei ones and they were working 3x as fast as the Google or MSN ones.

The (useless) answer from our ISP was to install a Fortinet Firewall in front of our server for an extra US$50 / month but I've managed to block a number of them and I can still use the Linux iptables firewall if I need to.

To give you an idea of the impact (and why it is hurting us) here are some raw numbers....

This time of year our data usage averages 15 Gb / day but since May 28th the average has been 141 Gb / day.

Likewise, we would normally serve ~500k pages a day with 700k server hits but the averages since 28th May have been 3.3M pages and 3.5M hits.

It is improving gradually and there are currently no Huawei spiders active but there are some other Chinese ones like Baidu that I still need to knock on the head.

[mad2]

is that what happened earlier?

[russellw]

No - that was me being stupid and restarting mySQL without confirming that the 'forum' user had access to the database.

I'm going to have to take it down again to fix that problem as the temporary workaround in place isn't very good.

[FTE217]

Originally Posted by mad2

is that what happened earlier?

expect so for couldn't load page up some 10mins ago......edit, ah ok it was Russell stand corrected.

and people made noise about our stance re Huawei.
Glad the Poms went back on this as well, they quoted in the end it would run their IT period in time, more so CCP could just shut down/control whenever they wish. Baidu friggin their own google.
All power to you Russell.

[russellw]

Just as an FYI, I am going to shut down the server at 3:15 for the 5 minutes it will take to fix the database issue.

Apologies but it is necessary.

[russellw]

Took more like 12 minutes but it's resolved now.

[Tonz]

[MITCHAY]

Thanks Russ. I thought something was not quite right.

I'm not sure what model we use here but it could be quite a difference in cost with that sort of demand.

I know that Google collect all sort of **** here but as I'm given to understand there is a lot of censorship over there so not sure sure what they are trying to achieve except to compile a hit list

[T3rminator]

Have a look at these two references in case it happens again. The first link seems to recommend some plugins and tools which might help. The second link refers to Robot.txt, pretty standard, guessing you've already done it, relies on the crawlers to obey the rules, but obviously not mandatory.

https://codewithhugo.com/importance-crawler-bot-block/​

https://help.dreamhost.com/hc/en-us/...s-and-crawlers

Interesting, Huawei has been diversifying from their mobile and network equipment business, looks like they have now started their own search engine which would explain the crawlers.

[T3rminator]

Sorry, first link got screwed.

https://codewithhugo.com/importance-crawler-bot-block/

Here are some helpful tips to prevent malicious bots from attacking your website:

Adding an additional application plugin, such as the Wordfence security plugin for WordPress prevents such attacks.

It’s also advisable to set access rules to get rid of malicious requests. You can disallow a specific search engine by inputting the name of the search engine on the disallow user agent of your robots.txt file.

You can disallow other search engines from crawling your website except for Googlebot by only allowing Googlebot as your user agent in your robots.txt file.

Note: A robots.txt refers to a file that’s associated with your website. It’s used to ask web crawlers to crawl or not to crawl some parts or web pages of your website. In short, a robots.txt file specifies which of your web pages should be crawled by web crawlers or spiders.

[russellw]

Yes, we already had robots.txt in use but they ignore it so they have been blocked at the firewall level to keep them out. I've also set up a honey pot and fail2ban to keep hyperactive ones at bay in the future.