SharpCap Forums

Hi folks,

I noticed very slow performance on the forums today (2nd September), with pages sometimes failing to load. Along with that, I have noticed recently that forum attachments have very large download counts - an uploaded log file might have been downloaded 50 or 100 times within 24 hours, previously it might have been 1 or 2 times.

Working on the assumption that someone out there is busily trying to download forum attachments wherever they can find them, and that the excessive downloads are causing the performance issues, I have blocked 'Guest' users from downloading attachments. Guest users means people who do not have an account, or those who are not logged in. Logged in forum members will see no problems (all attachments available to download/view).

So far, this seems to have helped - within 30 minutes of making the change, the forums seem responsive again with no failed page loads. Unfortunately the block on attachments also blocks images in the gallery if they have been uploaded to the forums as an attachment (images on other web sites will still show).

I will be keeping the situation under observation and will see if I can fix the gallery image issue over the next few days.

cheers,

Robin

Hi Robin

It could be AI-bots crawling and downloading all kinds of stuff. You could use a robots.txt file in the root of the forum server which disallows those robots entrance?
Attached is an example of it which I use myself too.

Menno

Hi Menno,

yes, I might have to do something like that as I see today that there are still some remaining issues. I wonder if there is a way to default disallow and then allow the common search engines (Google,Bing, etc) specifically - that would avoid playing 'whack a mole' adding more items to the disallow list.

On the flip side if the AI search engines index the forums, maybe they can start answering some of the simple questions!

cheers,

Robin

Hi,

OK, further update - performance was still poor today with attachment downloads blocked to non members. I spent some time analysing the web server logs and found that 70% or so of the traffic was coming from a block of IP addresses registered to the Chinese Tech company 'Tencent'. I suspect that someone is using their public cloud compute services to scrape web content and whoever it is hasn't done a good job of preventing repeat visits or limiting hit rates.

I have now blocked a large chunk of IP addresses at the web server level (43.128.0.0 to 43.163.255.255) to try to block off this traffic. Right now the forums seem back to normal, but I will continue to monitor.

cheers,

Robin

That would make sense. Tencent is known to really hoard the net, especially for their AI development.
Also very aggressive with making it so that robots.txt is not honoured, so IP blocking is indeed the only way.
If needed, here some more IP-ranges from their cloud computing: https://www.ip2location.com/as133478

Menno