It really does depend.
Here are a few common factors at play:
-Some companies refurbish their returned products better than others.
-Some problems that are 'fixed' are worse than others. If you have a repaired CPU for example that would be a much more significant repair than replacing a USB port.
-Some refurbished products have reduced warranties (not all, but some). This can be a deterrent for many people.
I have personally owned plenty of refurbished products in my time and not have issues with them. That said, I have been very careful about what companies I buy refurbished products from.