Mining is only part of it. HBM is expensive and at release Vega 64 cost about the same as a 1080 Ti while only performing like a 1080. So Vega started with a disadvantage. AMD had a hard time with the architecture. It uses way more power than current Nvida cards and doesn't clock nearly as high. Vega 56 was competitive though. Vega 56 made Nvidia release the 1070 Ti.
I think at release 8GB HBM stacks cost over $150 each and that was from the factory. Vega 64 uses 2 of them. Then AMD had problems because they used 2 fabrication facilities to make the GPUs before sending them to board partners ( Asus, MSI, Gigabyte etc. ) to finish. Those different fabs didn't make identical GPUs. The HBM stacks were just slightly different heights making it almost impossible for the partners to make a single cooler that fit all the cards. Screw ups like that also affect price.