I'm wanting to build my first hackintosh. I'm relatively new to this so excuse me if I ask a dumb question, but will a quad-core Skylake processor work on a H170 motherboard? What about a H110?
Not in a small case with good airflow (there aren't that many); unfortunately you haven't provided any info about the case you plan on buying. You also need a good cooler; which one will you use that's good enough and fits a very small case? There are thousands of development hours in a Mac Mini; Apple can design a small and excellent case and cooling solution.
With decent cooling and airflow it shouldn't. A mac mini appears to be more like an intel nuc and the only one's I see available are using lower powered model i5's (1.4ghz to 2.8ghz dual core i5's). I guess the question comes down to how small of a case?