UCI-GUIs should use the parameter window to set hash size or turn on logging.
Winboard GUIs have to use the commandline-options.

Available commandline options:
   -hash <size in MB> Sets size of transposition table (default: 9MB).
                      Size will be rounded to next possible table size:
                        1 MB
                        2 MB
                        4 MB
                        9 MB
                       18 MB
                       36 MB
                       72 MB
                      144 MB
                      288 MB
                      576 MB

   -xboard            Use Xboard/Winboard to communicate with GUI
                      else use UCI to communicate with GUI.
	 -egtb <path>       Set path to endgame tablebases (separate multiple paths by ; )
   -log               Creates a log file in current directory.



If you are low on memory keep in mind that hagrid loads its opening book to memory, so the big book will use several MBs.