Use <c-o> to go to previous spot in the 'jumplist' and <c-i> to go forward.