This Neo-Nazi likes me? What do I do?

Just tell him that you aren't ready for a serious commitment, that you have a lot of stuff on your plate, that you're really busy. Tell him that the timing just isn't good. And that you wish it were otherwise, but it's not.