Pick up an nintendo DSi (The i is important because they are the ones with an sd slot) or an PSP and mod them, they can play a lot of what came before other than their native games and build quality will be much higher.
PSP can play PS1 games "natively" as well if you convert them.
Also DSi games are awesome and you need two screens to play them right.
There is no shortage of second hand portable consoles while a Pi is pretty hard to get.
I would say nintendo (new) 3ds and vita but those get a bit too expensive for just "I would like to try that" .