Forum needs to move to a new host

[Hated Moron]

The bbcbasic.co.uk website came very close to exceeding its bandwidth in March, the only reason it didn't do so was because I arranged an emergency 'overdraft' to carry it through to the end of the month. I can't risk that happening, because there are people who rely on the site to serve the in-browser edition of BBC BASIC (for example schools who use it for programs that pupils needs to be able to access).

The near-disaster happened because of some IP address (not recognised by nslookup) harvesting tens of thousands of pages, taking the total to more than 52 Gbytes for the month (the bandwidth limit is 50 Gbytes). The bulk of this download must have been Forum and/or Wiki pages because they are the only source of such a huge volume of data on the website.

Accordingly I want the forum to move to a new host. so that if another massive trawl is attempted it won't impact on the quota for bbcbasic.co.uk. I am happy to link to wherever it ends up going, so it can still be accessed at bbcbasic.co.uk/forum as now, but where downloads won't contribute to the bandwidth.

So can I urgently request the admin of this forum to initiate such a move as soon as possible. I assume the most practical way of moving the forum is to back it up and then restore it from the backup at the new host.

[Hated Moron]

On 08/04/2023 07:35, Sean Miller wrote (cross-posted from the DIscussion Group):

I use DigitalOcean for my hosting - for 25GB of storage and 1TB/month bandwidth I pay $6 (+VAT)/month.

That would presumably do for the forum? Might I suggest that whoever is actually the server admin considers DigitalOcean?

Thanks for the suggestion. The thorny issues of where the forum is moved to and how it will be paid for are ones I have deliberately not addressed! Let's wait and see what the admin has to say, there may already be somewhere to which he has access that could be used.

The current 50 Gbytes monthly bandwidth limit is the biggest non-reseller hosting packing my particular provider offers, but even if a larger limit was available putting the forum (low priority, high traffic) and the WASM server (high priority, low traffic) in the same hosting account probably isn't ever a good idea.

[Hated Moron]

On 08/04/2023 08:29, Robin Hodson wrote (cross-posted from the Discussion Group):

How about using bandwidth throttling, or at least limiting the amount of data one IP address can access per day? Setting it to eg 1Gb (or lower) per IP, wouldn't affect reasonable use.

Eg, https://www.simplified.guide/apache/limit-bandwidth

The second approach would probably require a flatfile database tracking usage per IP, so would be more complex. It might have to be implemented as a phpBB plugin, so it isn't removed by updates.

No idea, I don't know the first thing about it. All I can say is that my hosting provider, having been made aware of the recent crisis, didn't suggest anything of the sort. Maybe the facilities you mention aren't available in my particular hosting package - it is quite tied-down, I only gained access to a terminal after specifically requesting it!

[Hated Moron]

On 08/04/2023 01:25, Maksim AbuAjamieh wrote (cross-posted from the DIscussion Group):

You can detect and block suspicious traffic using your firewall, also using your web server configuration, is that not done already by your hosting provider?

As I said in a previous reply, I don't know the first thing about such matters, they are completely outside my experience. All i would say is that I've had more problems from blacklisted IPs being blocked, and unable to access the website, than from malicious access.

On a number of occasions I've been contacted by a legitimate potential user saying that they have been unable to access the site, and when I've asked my hosting provider it's been because their IP (quite possibly their ISP's IP) has been blacklisted, and I've had to request that the block be lifted.

nslookup might not always be useful in these scenarios, the person accessing your site might use VPN or tunnels or some remote bot to make identifying their real IP harder. And I hope they were just mining and not trying to take your host down.

I realise that not being identifiable by nslookup does not necessarily mean that they are suspicious. But there should be no legitimate reason to download the entire site in one go, even search-engine crawlers don't do that.

I suggest creating a public backup archive, a read only downloadable archive at some public location that could be updated monthly, and there you can archive all content you believe is valuable, so that others can mirror it and keep it alive even if your main site is down.

I can't say that I think any of the 'content' is particularly valuable, and in any case rtrussell.co.uk has always mirrored bbcbasic.co.uk as far as the main pages are concerned (not the Forum or Wiki, or course) hence providing a partial backup.

It's availability of the server for the in-browser (Web Assembly) edition of BBC BASIC for SDL 2.0 which is crucial. I can't back that up at another site, unless it used some kind of smart-redirection in the event of a failure (CloudFlare has been suggested but that's too expensive, and I don't know if it can serve the vital Cross-Origin-Opener-Policy, Cross-Origin-Embedder-Policy and Cross-Origin-Resource-Policy headers).

[DDRM]

Hi Richard et al,

Sorry for the delay - I've been away over Easter, and I couldn't face dealing with this on my phone....

I'm sorry the webcrawlers have been chewing up your data allowance again! My guess is they do it every few months to update their links. I can reactivate the need to log in, as we did last time this was a problem, if you think that's useful (and I can remember/find again how to do it...). The downside of course is that search engines will no longer be able to link to the site (or at least any pages within it).

One possibility that I learnt about recently on my course, is that you can have a "robots.txt" file on your site which well-behaved web crawlers will respect, which can limit what they can/do access. Of course, badly behaved robots may ignore it... I could look at doing something like that, but probably not until I've finished my finals (early June), unless I am smitten with the desire to have a go as part of my revision for that topic...

We have discussed multiple times your desire for me to move the forum elsewhere, and just to reiterate my position:

1) I am "admin" of this site purely in a caretaker role, to relieve you of the stress of day-to-say zapping of spam posts/accounts, and, rather imperfectly, to protect you from some of the criticism you perceive in some posts. I don't regard this as "my" forum or "my" responsibility in any way.

2) I lack the expertise, time, and frankly the desire to run a website.

3) This forum is already effectively mirrored in the groups.io site (I think all posts are cross-posted at present), you have your distillery site, and there is the unofficial shadow site that Kendall set up last time there was a major crisis. I can't see any point in having yet another unofficial site.

4) As we discovered some time ago, this BB software seems to allow us to make backups and restore them internally, but doesn't seem to provide any option to download/extract them so they could be used to set up an equivalent elsewhere.

Summary: I'm not proposing to move it anywhere, but I'm happy to help reduce your bandwidth load.

Best wishes,

D

[Hated Moron]

One possibility that I learnt about recently on my course, is that you can have a "robots.txt" file on your site which well-behaved web crawlers will respect

It wasn't a well-behaved web-crawler, it was an unidentifiable IP downloading multiple Gigabytes in one day, which no legitimate search engine would do.

I don't regard this as "my" forum or "my" responsibility in any way.

I'm afraid it's not relevant how you regard your role, you are currently the sole admin and as such the only person who can move the forum elsewhere. If you want to call for somebody else to support you in that rôle that is of course your prerogative.

As you know I gave up my admin and founder rôles a long time ago, the only 'administrative' control I have over the forum is that I can modify where bbcbasic.co.uk/forum goes to. Therefore I could effectively close down the forum, but I can't move it anywhere else.

2) I lack the expertise, time, and frankly the desire to run a website.

I'm not suggesting that you do any more than what you do now, indeed a relocated forum would not look any different at all, either to members or to you. Once moved, you won't know the difference!

you have your distillery site

Pardon? The Distillery forum is nothing to do with me, it was created by and is administered by Michael McConnell, and its main purpose is to support Matrix Brandy, (arguably) the main competitor to my BASICs.

I can't see any point in having yet another unofficial site.

Indeed not. I fear that you are completely misunderstanding what I am proposing, which is simply that this forum be moved to another host. That won't make it any less or more 'official', indeed it won't change in any way at all other than that its 'true' (non-redirected) URL will be different.

4) As we discovered some time ago, this BB software seems to allow us to make backups and restore them internally, but doesn't seem to provide any option to download/extract them so they could be used to set up an equivalent elsewhere.

As I'm sure you remember, I offered to set up an FTP account which would give you access to those backups. That offer remains open, but in the absence of that account you simply need to tell me the name of the backup file and I will download it for you.

Summary: I'm not proposing to move it anywhere, but I'm happy to help reduce your bandwidth load.

We've tried other approaches, they aren't sufficiently effective and foolproof. I now require that the forum is removed from being hosted at bbcbasic.co.uk so that no matter how much data is downloaded only the forum (and perhaps the Wiki if moved to the same place) would be affected.

To reiterate I'm not proposing that you do any management other than what you do now. It's a once-off process of moving the forum from A to B, thereafter nobody will know anything has changed.

Do you currently have access, perhaps via your work, to somewhere that it could be relocated to for little or no cost? If not, we will have to identify a suitable (and hopefully inexpensive) new host; I don't anticipate that funding it will be a problem because when similar issues have arisen before there have been offers of donations.

[DDRM]

Hi Richard,

No, I don't have anywhere it could go (remember, at the moment I am an undergraduate student, and even that only for another couple of months!).

If you have access to the backup files (I didn't realise that!), why not give it to Kendall, who already has a mirror site set up and functioning?

Anyone else fancy taking it on?

Best wishes,

D

[Hated Moron]

If you have access to the backup files (I didn't realise that!)

Maybe I've misunderstood something, but I thought we had established that the backup files were stored somewhere at bbcbasic.co.uk and therefore must be accessible to me - it's my domain! That's why I offered, quite some time ago, to set up an FTP account to allow you to access them.

If that's not right, where do you believe the backups are stored, and if neither you nor I can read them who can?!

why not give it to Kendall, who already has a mirror site set up and functioning?

Mirror site? It's news to me that Kendall (or anybody else) has set up their own phpBB forum. The only forum of his that I was aware of is at Freeforums.net. In any case you can't seriously expect that I would hand the forum over to somebody who I don't know and don't trust to run it responsibly.

I wasn't expecting this kind of pushback against what seemed to me a simple, quick and relatively painless way of solving the problem of bandwidth being exceeded by forum downloads. Once moved to a new host, nobody would have been aware of any change having occurred at all: not me, not you and not the members.

But clearly you're not prepared to contemplate that, and you seem to take the view that this forum isn't needed anyway. If that's also the view of the membership, I'll just shut it down and solve the problem that way.

[Hated Moron]

Anyone else fancy taking it on?

It seems not. Before I take the next, possibly irrevocable, step, I want to make absolutely sure that there is no possibility of a misunderstanding having arisen (something which my problems with cognition make more likely).

Can you confirm that you are not prepared to move the forum to a new host, despite that having no impact at all on the way the forum is accessed and no impact at all on your rôle as administrator? This would only involve taking a backup at the current host and restoring it at the new host.

[Hated Moron]

On 14/04/2023 10:34, Maksim AbuAjamieh wrote (cross-posted from the Discussion Group):

I am with keeping the forum, you mentioned that you are worried the most about the wasm server being inaccessible for those who rely on it (schools, etc) and that it shares the same host as the forum, if that was a decision caused by a cost issue, I am happy to host it on one of my servers

It's not at all a cost issue, I have already had an offer of free hosting.

Unfortunately moving the wasm server to a new host is problematic for several reasons. Firstly it has to be at the 'permanent' URL wasm.bbcbasic.co.uk because that's burnt into web pages all over the place; secondly it has to be secure (https://) because browsers won't execute multi-threaded Web Assembly code downloaded from an insecure site; and thirdly it must be able to serve the essential Cross-Origin-Opener-Policy, Cross-Origin-Embedder-Policy and Cross-Origin-Resource-Policy headers.

But perhaps the trickiest issue of all is that the transfer of the wasm server to a new host must be seamless, with little or no downtime. Maybe all these difficulties could be overcome, but it's much more straightforward to transfer the forum and wiki, which are non-critical and can go offline for a little while without it mattering.