Pick up an nintendo DSi (The i is important because they are the ones with an sd slot) or an PSP and mod them, they can play a lot of what came before other than their native games and build quality will be much higher.
PSP can play PS1 games "natively" as well if you convert them.
Also DSi games are awesome and you need two screens to play them right.
There is no shortage of second hand portable consoles while a Pi is pretty hard to get.
I would say nintendo (new) 3ds and vita but those get a bit too expensive for just "I would like to try that" .
Yes, I heard DSi is pretty good actually. But are they still selling the original DSi ? Hard to find a NEW one in stock, except the XL variant.
I'm actually planning to get the SWITCH console pretty soon, just to play The Legend of Zelda: Tears of the Kingdom. Although, there are Emulators by which we can also play it on the PC, but I want to experience it on Switch. Of course, Switch is on different level though, as compared to the older portable con soles.
My most favorite game was the good old DONKEY KONG on the "Game and Watch series'.