A little hard to say. The Vega 8 is hampered by the fact that it has to access DDR4 system RAM vs the GDDR5 on the HD5770.
Looking at the legacy section of the
GPU hierarchy chart, the HD 5770 is about on the level of the HD 7750(GDDR5 version)/R7 250E(GDDR5 version)/R7 250 in terms of performance. This is one tier under the GT 1030.
Then, going back up to the main, non-legacy section of the hierarchy chart, the Vega 8 is at 6.1% vs the GT 1030's 7.2%. Relative to each other, the Vega 8 performs at about 84% of the capability of the GT 1030 (or, inversely, the GT 1030 performs about 18% faster than Vega 8).
Now, these are approximations... and, unfortunately, I can't really guess what the percentage difference is between two adjacent tier levels are on the legacy part of the hierarchy chart.
If I were to guess . . about the same? In some places the 5770 might have a slight edge, and in others, the Vega 8 will have an edge. The 5770 is, after all, about 4 generations earlier than the underlying Vega architecture.
Even if they perform equally, with the 5770, you've got two issues:
AMD does offer drivers for download, their old Catalyst drivers, and (only as Beta) the newer Crimson drivers (which, for current products, has now evolved to the Adrenaline drivers). The drivers date back to 2015 for Catalyst, and 2016 for the Crimson beta. To wit, the older architecture isn't really supported anymore.
It couldn't hurt to try, just out of curiosity. I know the curiosity would definitely get me to try it. But I would guess that the performance difference, even if the 5770 had some advantage, might not be enough to justify the power draw and that the drivers are outdated.
I know this really was a kind of "eh, maybe the one, maybe the other" answer, but I think their performance would be pretty close to each other. MAYBE the 5770 has an advantage.