The problem with a lot of these browsers is that many of them try to reinvent the user interface paradigm to varying degrees. I realize visual appeal is a subjective matter, but a consistent look/feel is supposed to be a core philosophy of modern graphical operating systems; so if you're going to create software that looks outlandish or overly "pretty" in some way, the look should at least be user-defeatable such that the application can be set to use OS-standard layouts and GUI elements.
With an open mind, I downloaded Chrome a while back, but was immediately horrified by how bizarre and non-standard it looks. That's great if you like it, but I couldn't find any configuration option to make it look/feel like a standard Windows app. Instant fail, in my book, no matter how fast it runs.
But even Microsoft is going somewhat against, er, Microsoft's standards with the layout of IE 7/8. I much preferred the look of IE6, and that was the sole reason I held out for ages before "upgrading" to 8; but no matter how you look at it, sandwiching navigation buttons and the URL box between the window title bar and the menu bar is non-standard in Windows, and just plain weird. And worse yet? You can't change it (as far as I can tell). And why can't the tab bar be configured to disappear when you only have one tab open, like in Firefox? Why waste the screen real estate?
Ironically, Safari has a more Windows-ish layout than IE, but the actual design of its GUI elements is unmistakably Apple. No thanks. If I want an Apple look, I'll get a Mac.
I haven't used Opera so I can't comment, but among the others, Firefox is the *only* browser that still looks like a Windows web browser is "supposed" to look. But for me at least, it feels slower. Personally, then, I'm left torn between IE and Firefox, and I can't find true love for either.
Chrome may be fast, but its non-user-defeatable, non-standard look is a GUI no-no.
Another thought: the way IE stores bookmarks as discrete files/folders makes maintenance a lot easier, since you don't have to use IE's proprietary bookmark manager. Also, you can create a lovely taskbar toolbar to quickly access any Favorites folder. I don't believe this is possible with these other browsers (sure, they will launch in whatever your default browser is, but it's still IE's bookmarks). Personally, I'd like to see all browsers move towards a standard way of using a shared bookmarks repository, since this would make life a lot easier for people who use multiple browsers and would like their bookmarks uniform from one to the next; but at the very least, using standard filesystem files/folders (like IE does) would be a huge enhancement.
Some of the factors I've brought up are subjective, but some are about clear usability violations, and none of these browsers are totally innocent. But Firefox seems to come out on top of IE, Chrome, and Safari in this very important, but often overlooked, category.