It's probably just because the scaler in your TV is really bad. The PS4 may render games at their native 1080p or 900p resolution internally, but dynamically scales it to your TV's native resolution. And the scaler in the PS4 is pretty good.
The scaler in the RX580 is also really good. You use that by setting the output resolution at 1366 x 768 and it will resample any other resolution videos, etc to that and look fine. While it's overkill for gaming at 768p, you can either crank up the antialiasing to not waste its power, or render the game at 1080p and downsample it to 768p using
Virtual Super Resolution (AMD's equivalent to DSR)