In terms of real-world usage scenarios, the performance would generally be closer than that, likely only around 15% or so behind the RX 580 8GB in most existing games running at 1080p. The GPU hierarchy chart combines results for 1080p, 1440p and 4K, and the higher-resolution results undoubtedly cripple performance a lot worse on a 3GB card. Even the RX 580 doesn't really have the processing power to properly handle 1440p and 4K in most modern titles, so including those resolutions in the comparison is not really representative of the typical performance difference one would encounter. The 1060 3GB was more of an RX 570 4GB competitor though, having 10% of it's cores disabled from the 6GB version, in addition to less VRAM.
Getting a 1050 Ti in place of an RX 580 4GB would be a lot worse though. Even for 1080p gaming, a 1050 Ti performs around 40% below an RX 580 in most games. The two cards are in entirely different performance tiers. The RX 580 4GB tends to perform rather similar to the 8GB version in most titles.
It is kind of amusing that anyone would think AMD would have a recall and trade Nvidia cards for AMD ones though.