It's definitely possible to build a computer without a case, but you'd have to be extra careful about where you put everything, and you'd need to build some kind of frame to secure the GPU and any other expansion cards or they'll work themselves out of the slots.
What is your case? It sounds like your case doesn't support the form factor of that motherboard.