GAP does not contain a multivariate GCD algorithm. The following operations are used internally to try to keep the denominators as small as possible
RationalFunctionByExtRepWithCancellation( rfam, num, den ) F
constructs a rational function as RationalFunctionByExtRep does but
tries to cancel out common factors of numerator and denominator, calling
TryGcdCancelExtRepPolynomials.
TryGcdCancelExtRepPolynomials( fam, a, b ) F
Let f and g be two polynomials given by the ext reps a and b. This function tries to cancel common factors between a and b and returns a list [ac,bc] of cancelled numerator and denominator ext rep. As there is no proper multivariate GCD cancellation is not guaranteed to be optimal.
HeuristicCancelPolynomials( fam, ext1, ext2 ) O
is called by TryGcdCancelExtRepPol to perform the actual work. It will
return either fail or a new list [num,den] of cancelled numerator
and denominator. The cancellation performed is not necessarily optimal.
GAP 4 manual