Initial value of HIMEM

[Hated Moron]

In BBC BASIC for Windows version 6 onwards, all editions of BBC BASIC for SDL 2.0 and all the BBC BASIC Console Mode editions, HIMEM is initialised to 2 Mbytes above PAGE. It can typically be raised much higher (256 Mbytes or more) but that is the default.

Arguably this is anachronistic in 2024. With the memory available in all modern machines - even a Raspberry Pi has at least 1 Gbyte of RAM - requiring the programmer explicitly to raise HIMEM even to hold quite a modestly-sized array may be unreasonable.

I'm not in favour of increasing it to a huge degree because it only takes a simple coding error to result in a program which is accidentally recursive, or one which has a memory leak, and therefore gobbles up the entire heap and stack space between LOMEM and HIMEM.

Perhaps I can seek people's views of what might be a better choice for the initial value of HIMEM. If nobody expresses an opinion, I will leave it as it is now.

[JeremyNicoll]

I'm not sure that it's worth upping it. Whatever value you pick someone will think it should be higher.

Via groups.io "Andrew Cool" said:
> Can you interrogate the system to find out how RAM is on board, and set HIMEM to some percentage of that?

The thing is, it's not the amount of real RAM that matters; it's the amount of free virtual memory. That is, different OSes' page & swap space needs taken into account too - my ~8GB real RAM laptop has just over the same again in page space and when the machine is in heavy use nearly 16 GB is in use.

[Hated Moron]

On 06/02/2024 20:09, Andrew Cool wrote (cross-posted from the Discussion Group):

Can you interrogate the system to find out how RAM is on board, and set HIMEM to some percentage of that?

There's apparently a SDL_GetSystemRAM() function, although I've never tried to use it.

But, surely, it would be crazy to make the initial value of HIMEM variable, through being dependent on some machine-specific feature?

The consequence of that would be that the BASIC programmer could never be sure what HIMEM is, so he would always have to set it explicitly to whatever value his program needs!

The objective in, possibly, increasing the initial value of HIMEM is to reduce the need to set HIMEM explicitly, not to make that compulsory.

Or am I misunderstanding something?

[Flatlander]

Maybe just have the ‘Out Of Memory’(?) error message include something about raising HIMEM?

[Hated Moron]

I'm not sure that it's worth upping it. Whatever value you pick someone will think it should be higher.

You're right, of course, that someone will always think it should be higher, but that doesn't seem a good reason for not increasing it.

"Never improve anything because there will always be somebody who still isn't satisfied" seems an odd philosophy to live by!

Matrix Brandy BASIC initialises HIMEM - PAGE to 64 Mbytes so if we were to choose that value it would at least have the virtue of consistency.

[Hated Moron]

Maybe just have the ‘Out Of Memory’(?) error message include something about raising HIMEM?

Error messages are off-topic for this thread, but I would direct you to the existing documentation for the 'No room' error.

[DDRM]

My view would be that raising it substantially, but to a fixed value that is always likely to be available, would be sensible. Your suggestion of 64MB seems a reasonable one.

I don't think it would be good for BASIC to fail because the memory isn't available - I guess that might be more of a concern for the phone/tablet version? though even then typically have RAM in the multi-GB range these days....

About the only drawback I can think of is that programs written in the new version might fail on old machines? But there are probably lots of other compatibility issues that mean you aren't going to be back-porting a BB4W program to a BBC-B....

Best wishes,

D

[Hated Moron]

\

Your suggestion of 64MB seems a reasonable one.

It's not "my suggestion", I mentioned that value only because it's what Matrix Brandy defaults to. It seems on the high side to me, increasing it from 2 Mbytes to 64 Mbytes does feel rather a 'step too far'. Perhaps 16 Mbytes or 32 Mbytes might be more appropriate, but I really don't know - hence asking here!

I don't think it would be good for BASIC to fail because the memory isn't available - I guess that might be more of a concern for the phone/tablet version?

I agree that we ought to choose a value that 'cannot' fail, but what value would that be? Currently the in-browser edition is the one with the least heap memory available (16 Mbytes), because it is running in a sandboxed environment.

I'm not at all sure how to achieve a consensus.

[Hated Moron]

On 07/02/2024 23:05, J.G.Harston wrote (cross-posted from the Discussion Group):

Having read the other posts, I think I'd go with leaving the behaviour
as it is. There are calls to ask for more memory than allocated, so if
a program specifically needs more memory, rather than modifying its
behavoir to fits its environment, then a programmer can choose to write
their program to do that with, eg HIMEM=LOMEM+4096*1024.

I can't help thinking that some confusion has arisen. My question is about the initial value of HIMEM not about the allocation of (virtual) memory! I've checked back through the thread and I'm pretty sure I've never mentioned memory allocation anywhere.

For the record I do not propose making any changes at all to the allocation of memory (in 32-bit editions anyway), I expect that to remain exactly the same as it is now: 512 Mbytes in BBC BASIC for Windows and 256 Mbytes in 'BBC BASIC for SDL 2.0' (16 Mbytes in the browser edition).

The initial value of HIMEM determines how much memory (heap + stack) is available to a BASIC program by default, and it determines how much real (physical) memory could be used up if a program got stuck in a recursive loop, say. But it doesn't have any effect on the amount of memory allocated.

A statement such as HIMEM = PAGE + 64000000 does not allocate any more virtual memory, it simply moves HIMEM up within the block of memory that has already been allocated. You can easily check that in Windows, for example, by looking in Task Manager.

(BTW, is there a way to release memory back to the system, for a program
to say something like "I only need 32K of workspace, HIMEM=LOMEM+32768,
give everything above that back to the system.

No (other than by OS-dependent SYS calls), but if the memory is no longer being used the OS Memory Manager should eventually notice that and offload it to the swap file, so it won't be occupying any physical RAM anyway.

[Hated Moron]

On 08/02/2024 11:23, J.G.Harston wrote (cross-posted from the Discussion Group):

Ah, that is me misinterpreting the manual entry for HIMEM then.

"By default, HIMEM is set at slightly less than 2 Mbytes above LOMEM. If this is insufficient for your program and data, you can raise HIMEM (subject to there being sufficient memory in your computer)."

I read that as being: when Basic starts, it allocates itself 2M(ish) of memory,
and sets HIMEM to the top of that 2M of memory (and PAGE to the bottom).

The documentation tries to avoid the complication of virtual memory and describes things in terms that somebody more used to programming a BBC Micro would understand!

In all modern Operating Systems, when you allocate memory no physical memory is reserved for you, just virtual memory. In effect what is allocated is an address range, that is the OS guarantees that all memory addresses in that range now 'belong' to your program. The actual assignment of real, physical, RAM only happens 'behind the scenes' when you later write to an address in that allocated range.

So when BBC BASIC for SDL 2.0 starts it requests the allocation of 256 Mbytes of contiguous address space. Although no physical memory is actually set aside for it, that allocation can still fail because of 'fragmentation'.

If the memory address space is already fragmented (typically because of the loading of DLLs etc., especially with ASLR enabled which is the default) and there is no contiguous region big enough, the allocation will fail.

That's precisely why it requests 256 Mbytes rather than, say, 512 Mbytes or 1 Gbyte. Requesting a larger address space won't make any difference to the amount of RAM actually used but it will greatly increase the chance that the allocation fails due to fragmentation.

I was reading the initial question as whether the initial default should
be changed. To which I answered: I don't see a need to change it, if a
programmer decides that their program needs more, there is an established
method of doing so.

What you seem to be saying is that there is no pressing reason to change it (which I agree with) but I cannot see anything in what you or anybody else has argued that provides a good reason not to change it.

Surely we can all agree that increasing the initial value of HIMEM would be beneficial because it would reduce the number of occasions on which explicitly increasing HIMEM in your program is necessary (and not every BBC BASIC programmer even knows that they can do that).

So to argue that it shouldn't be changed requires a compensatory disadvantage to balance that benefit. What is that disadvantage? The only one I can think of is that if the program goes mad and gobbles up all the memory from PAGE to HIMEM, more physical RAM will be assigned.

But even then the OS will eventually realise that the memory is no longer being used and will offload it to the swapfile (pagefile).