[no subject]

> I've tried to work as space efficient as possible
> (e.g. doing matrix operations in place),

part. If you're computing your matrix under GP, then the result is not put
on the stack but on the heap, and doesn't count towards the stack limit.

So I'm assuming you have a clean 150MB stack when invoking matsolve.

Second, setting DEBUGMEM to 2 (\gm 2 under GP), you get debugging messages
each time 'random garbage collecting' occurs, as well as a rough indicator on
the progress of your computation. Here, it would produce something like

(17:09) gp > 1/a
  ***   collecting garbage in gauss. i=1.
  ***   collecting garbage in gauss. i=2.
  ***   collecting garbage in gauss. i=3.

Then you can see exactly when the overflow occurs (probably right at the
start in your case).

You might want to check that ALL your matrix entries are indeed floating
point reals (multiply by 1. in doubt). In that case, at default precision,
the value of sizebyte(m x n matrix) should be roughly equal to 24 mn [4*5 (28
digits = 5 words) for each entry + 4 * mn to hold them]