Per-function RLE return stacks

Notation: a { b c }
calling a
entering a
calling b
calling c
leaving a

Example: a { b { c1{d{e}} c2{d{e@}} } c { d { c { d { c { f i{a} i{a@} b{c1} } f } f } f } f } c {df} }

Stack at first @: (e_locals)(d)(d_locals)(c2)(c2_locals)(b)(b2_locals)(a)(a_locals)

Stack at second @: top(a_locals)(i)(i_locals)(c)(c_locals)(d)(d_locals)(c)(c_locals)(d)(d_locals)(c)(c_locals)(a)(a_locals)

Problems: Stack grows even when local variables take up 0 bytes because return addresses take up space too.

Proposed solution: One returnstack with RLE for each function.

At first @:
b: (a)x1
c2: (b)x1
d: (c2)x1
e: (d)x1

At second @:
a: (i)x1
c: (d)x2 (a)x1
d: (c)x2
i: (c)x1

Result: Tailcall that's not at the tail.

Variation: The per-function stack can be fixed-sized. In that case old entries that fall off the per-function stack go into the main-stack.

Discuss.

Locals and return addresses together considered harmful.

Local linked stack:
{a's locals}{b's locals}{c's locals}
Return stack:
{a's return address}{b's return address}{c's return address}
The contiguous parts of the local area grow upwards instead of downwards so a buffer overflow will only affect the function's own locals before it runs into unmapped memory and causes an exception. You can use a canary value to detect buffer overflows without destroying useful data. The local stack can be a linked list so "stack overflow" only happens when the heap cannot allocate any more memory. Arguments are a sub-block of the locals. The return stack does not contain pointers into any part of itself so it can be relocated anywhere in memory or split into multiple linked blocks. Only call, return, unwind (pop n entries and return), and tail call work on the return stack. This is how it works on z/Architecture, except registers are also stored on the return stack and the stack is in a separate protected address space that can only be modified by linkage instructions with a stack pointer that can only be accessed by those instructions or privileged code.