The mem_limit() function is suspicious

Submitted by Jiri Dluhos

Assigned to Stuart Anderson @anderson

Link to original bug (#6964)

Description

The mem_limit() function, in libXtTest/avs_func.c, is very hard to comprehend, and I suspect it may be incorrect on Linux because malloc() is not guaranteed to return NULL when there is not enough memory.

The code looks like this:

size_t mem_limit() { char *ptr; size_t hbit,i,j;

j = 0;
for (hbit=(((size_t)-1)>>1)+1;hbit;hbit=(ptr)?(i>>1):(i>>2)){
	for(i=1;
		(ptr=malloc(i|j)) && (free(ptr),1) && (i<hbit);
	i<<=1);
		j |= (ptr) ? i : i >> 1;
}
return(j);

}