Use GPUs

May I know MXNET by default will store all variable in memory instead of GPU memory even I have install CUDA version MXnet, unless I specifically assign ctx=gpu(0)?
Is there any way to make it by default everything store in GPU memory, just like TensorFlow default behaviour?