When you buy a passively cooled card it's because low noise is important to you. This comes with a trade off, and that is the card will get hotter under load.
So if you buy a card for gaming and you choose a passively cooled card, you're doing it wrong. If you buy a passively cooled card and then attach fans to it, you're doing it wrong.
You can do whatever you like, it's just that components are designed for certain purposes and to work in certain ways. When you buy a component meant for one kind of use and try to force it into another kind of use that's just the hard way of doing things. That's when you can expect inferior results. It's best to identify your needs, then find the component that best fits those needs.