Hmm, that's a tricky one, I've never bought a PhD grad gift before. My sense ses tell me you can go one of 2 routes. You could try to find him something serious that will celebrate his entry into the professional world (for instance laywers traditionally recieve a very nice breifcase for a graduation gift), or you could go for something fun. The professional gift might be a little tough to figure out for a newly minted mathematician since our tools are primarily our brains. For fun stuff, you could try looking at www.thinkgeek.com.

What field of pure mathematics is he studying? What is he planning on doing after graduation? One thing you and others may want to consider is having a whole bunch of people chip in to send him on a much deserved vacation trip.