Re: GMP kernel and stack abuse

[Ilya Zakharevich <ilya@Math.Berkeley.EDU>]

On Tue, Apr 08, 2003 at 11:21:45AM +0200, Bill Allombert wrote:
> > > Does not it require recompiling libgmp ?
> > 
> > So do it.  ;-)
> 
> ... then I am afraid we cannot really avoid pointing at the FAQ. Most
> people just use whatever libgmp is provided by their software
> distribution/sysadmin and the default is to use the real alloca.

Hmm; do sysadmins provide GMP?  I would think that one usually does it
himself; same as with PARI.  Given that gmp-build of PARI is not a
default, I see no big problem with one recompiling GMP as well.

Anyway, merging the fix into the GMP CVS will quickly (in the software
pipeline timeframe ;-() fix this too.  Just check

   char fence;

   if (gmp_stack_bottom && &fence - *gmp_stack_bottom < limit)
	malloca();
   else
	alloca();

By default gmp_stack_bottom is NULL, so alloca() is used.  PARI would
set gmp_stack_bottom, and gmp will be safe.

> > > The stack overflow here is of the order of tens of megabytes.
> > 
> > So it is not hard to detect.  ;-)
> 
> PARI is able to catch small stack overflow and output a deep recursion
> error. Is it really possible to catch large stack overflow?

What I think PARI does is just checking the stack position within
stack_start/stack_end before a possible stack-eater operation.
Similar to what I did above.

I think you are thinking about guard-page approach, which IIUC PARI is
not using.

Yours,
Ilya