The advantages of Windows Home Server is that it is a little easier to setup for a Windows only environment. Downsides is it costs money. I've never really used it so I don't know of any other benefits.
*nix based systems are obviously free, and might be a bit harder to setup. However, they run better on older hardware and can easily be setup for a mixed OS environment.
I setup my file/backup server with Ubuntu Server 10.10 (upgraded to 11.04 just recently). My decision to use Ubuntu was based on the fact that I have limited linux experience and wanted something easy, with a large user base to ask questions. I also figured that I could use my server for other purposes. I used Webmin as my interface and while convenient it doesn't make configuring it that much easier. I've also heard of ebox which is supposed to simplify things, but doesn't give you as detailed control. Since I used a dedicated boot drive (8GB CF to IDE) I can easily swap out the OS to anything that supports ext4 and try those out. If I don't like it, then I just re-image my boot drive and everything is working again.
There are other NAS specific OSs to choose from too like OpenFiler and FreeNAS. OpenFiler is linux based and comes with everything you need for a NAS built in. It comes with a web interface for easy setup. Since it supports the ext4 filesystem, if you setup a Ubuntu based server with ext4 data drives you could switch to OpenFiler without having to do something with your data. web interface.
FreeNAS is BSD based and supports the ZFS file system. ZFS allows software raid and dynamically adding disks to the pool. It is a convenient file system for a NAS. It has a web interface built in, and comes with everything you need for a NAS. It is small so it'll fit on whatever you decide to use as a boot drive.
As for which to use, I'd just pick a *nix based system and try it out since there is nothing to lose except some time, but hopefully you'll learn something as you go. One big suggestion I have is that you should image your boot drive as soon as the OS is installed, since you'll probably screw up the settings beyond your own repair. Then you can just re-image the drive and everything is installed again and ready to tweak. I learned the hard way when I had to wait for Ubuntu to install a second time