Depends on the media server software, for example, Plex does not use any GPU excelerated rendering and require pure CPU horse power.
My Plex server is a Phenom II X4 840 and is happy to transcode 6 HD1080 streams, if direct streaming not an option.
If you something like Kodi and use the server as a NAS, then the host will do any rendering and server just needs to handle the file thoughput, a dual core will do that.
One note, 16GB way overkill, my server runs with 8GB, just becasue I had it lying around !
If anybody is interested ..
Server 2012 R2 x64
Gigabyte GA-870A-UD3 R3
AMD Phenom II X4 X840
4x2GB DDR3-13333
AMD HD5700
64GB OCZ O/S
6 x 4TB WD Purple Raid-5 Storage
2 x 1TB Samsung 1TB Raid-0 Downloads, Temp, Swapfile etc etc
2 x Intel Pro 1Gbps NICS balanced to AC router (soon to be replaced by proxy server and switch)
25/250Mbps cable broadband