Help Us With the Web Browser Grand Prix Scoring System

[freggo]

Compliance with HTML and CSS specifications is the most important thing. Speed is secondary.

When I work on websites my customers expect the actual site layout to match the layout given by the designer. The problem is that still every browser seems to have a different interpretation of font sizes, markings and other relatively basic stuff. It just drives me up the wall wasting so much time on fine tuning things that should fall into place according to the W3C specs.

Example... try using a simple JPG or PNG as a table background with or without CSS. Getting it to work on every browser takes some major bending backwards instead of a simple command. These things need to get cleaned up; not adding fancy 3D tabs or other browser cosmetics.

 

[PreferLinux]

[citation][nom]afmenez[/nom]Let's say browser C wins a benchmark with a score of 1000, browser F achieves 940, browser O gets 890 and browser S gets 820.I believe that an absolute scoring system (like 5 for browser C, 4 for browser F, 3 for O and 2 for S) is unfair, specially when you have categories with similar scores, like this example. Browser C seems much better than browser S, with more than twice the score, but it is not that much faster. This may end up favoring a browser that is marginally better than the others on some categories, but much worse on the others (like a browser that is a little faster with JavaScript, but way worse on memory use).It would be more fair to attribute 10 to the best score, and proportional grades for the other browsers. Then the score would more fair to a browser that is does good (but not necessarily best) on every category.[/citation]
Yes, I think that is a very good idea. Although it might be better to give a base of more like 100.

 

[Trewyy]

[citation][nom]adamovera[/nom]trewyy: I put startup time and memory efficiency in level 3 for a few reasons: 1) Most people only start their browser once per boot. 2) There is no evidence that memory efficiency correlates to better performance. 3) Both are measurements that can apply to ANY locally-installed application, and not just Web browsers.I have to disagree on HTML5 simply because, how can it be more essential than Flash when soooo many more sites have flash content than HTML5 code? Ditto for JS, and I just don't get the DOM downgrade. And while the average user might not consciously care about standards conformance, they will when they can't properly load a page.[/citation]

I was reconsidering where I placed these two, and I would agree now that they aren't maybe as important as I said previously. Certainly for memory efficiency I can see why it would now fall under non-essential, but that is a tight squeeze, and I think when you give out points, browsers that fail memory efficiency should suffer when it comes to distributing points.

In terms of start-up time, I would put this under important. I would no longer say it is essential, but I think start-up time is something that most people would like to reduced. As an example, this was so key in my decision a few years ago to drop Firefox in place for Chrome. Firefox had horrendous start up times previously, and Chrome was much faster then (still is, just not significantly). Firefox has definitely made up ground since then, which is why I have gone back to Firefox. This is a good reason as to why it is important - it can cause a person to leave a browser when they become frustrated.

As to your point about HTML5 and Flash, I would say that I never visit sites with Flash, and I have a long list of sites i visit regularly. I'm not quite sure what sites you are visiting (most sites might have their ads in flash, but that isn't really necessary when it comes down to user experience). The reason most websites lack HTML5 is because it is a technology/code that all browsers fail to conform completely to it. This is why it is essential that browsers complete all the code necessary, so that websites can actually start developing HTML5 (cause and effect here).

Also your comment about standards conformance - HTML5 is a standard, and a standard that will be prevalent in the years to come. This is why it is more important the most other standards.

Robert

 

[phate]

I just want to add my fervent concurrence that Page reliability and Standards Conformance be essential.

I don't really understand people like this

The main user doesn't care about things like Standard Conformance and DOM.

Really? Everything is really secondary in my opinion. It doesn't matter how fast a browser is, how much memory it uses or how many features it has, if it doesn't display and work correctly.

Obligatory car-analogy:
A/C is nice and all, but if the car doesn't drive it's useless.

 

[Guest]

I think the basic approach is fine. Details of which category gets which weighting will depend on the individual; no one arrangement will suit everybody. Just go with something fairly logical and let the reader make adjustements as they see fit.

I've always been surprised that privacy isn't a consideration. I'd like to see tests that reveral how often each browser phones home or contacts other sites that the user didn't explicitly request. Perhaps a surrogate measure would be how many bytes are transferred for a given request (fewer is obviously better). This could be done in the browser's as-installed state (which is what most users would experience) and again in a customised state (for power users).

 

[Guest]

admovera:

>> "I also see your point on page load times, but once you average the page load times of multiple sites, you're measuring differences in full seconds. True you only navigate to one page at a time, but think about a search session: Google a topic, click a link, not relevant, click back, choose another link... opening several pages in quick succession is relatively common."

I doubt you'll get more than a couple of seconds difference over the accumulated time of such an operation. An operation I remind you may take a several minutes of your time. And this is exactly the problem; this false perception that seconds count on event-driven usage like that of a web browser. How relevant is exactly a 5 second gain (and I'm stretching it here) on an accumulated operation of web searching that takes you around 3 minutes to execute? You see the problem?

For once I'd like someone to not fall prey to the False Doctrine of Speed. There are good reasons for speed to be accounted for. And there are bad reasons. Once we reached a certain milestone in browsers rendering time, speed has become a non-issue. Someone needs at some point to acknowledge this and properly categorize speed in modern browser usage as a being a secondary concern only. Will you be that guy?

>> "Java is essential to you?"

Indeed. Many scientific and specialized intranet and internet websites make extensive use of Java for modeling or graphical simulations, for instance. And you always have Runescape. But I see your point. Does a limited adoption mean one should consider it essential? My argument is that on what comes to web technologies, yes. It's irrelevant how much it is used. What should be relevant is whether a web browser will stop you dead from accessing information on the web, or on the other hand will indeed behave as a true window to internet content. For this reason, the choice to separate web technologies like you are doing is something I find incorrect. I'd rather use an inclusive term like "Web Technologies" and place it on the Essential category. And then discriminate each technology and whether the browser supports it or not.

 

[frostmachine]

14 categories: Normalized score against the winner. Reflects relatives differences between browsers. You'll get a percentage with the winner = 100% and the others anything in between.

4 Brackets: Average score of categories within bracket. Gives a even weight to all categories involved.

Final score: Sum of average score multiply by relative weights of each bracket.

Weights each bracket: 40% - Essential, 30% - Important, 20% - Unessential and 10% - Nonimportant.

Essential should at least be twice of Unessential. Important somewhere in between and Nonimportant dead last.

Final score should be a nice percentage. Easy to understand and hopefully reflects the differences between browsers. Someone needs to try it out on past GPs to confirm though.

 

[alidan]

here, i will make a list of what should be top and bottom and what i dont know what it does or shows.

Essential:
Flash - almsot all wrb pages have it, and many webpages can be unusable if flash is poorly rendered (its why i use chrome and firefox, chrome handles flash great, firefox uses less memory)
JavaScript - we are at a point where java is fast enough for most people, many scripts we use to customize out browsers are javascript if i am correct, so it is important.
Memory Efficiency - this is a deciding factor in what browser i use as main, and as media. firefox can take a heavy load and come out with far less memory usage than chrome.
Silverlight - netflix is still a big part of what people use the internet for, even though it lost subscribers... for stupid reasons.
Important:
Page Load Reliability - its important, but not so much, i have a 50 mbit line, if i need to reloading a webpage takes no effort and is painless, but is still an important feature
Nonessential:
Java - it may just be me, but i have to go out of my way to find anything java on the internet, java does not equal javascript
HTML5 Hardware Acceleration - when html5 really takes off, this will be important, but right now, i have to go out of my way to find any of it
HTML5 - very little really uses it at this point
Unimportant:
Page Load Time - whats the difference between 1ms and 1 second? when we are talking about page loads, and the likely ness that you are going to stay on that page for a minute or so, that time to load the page today is not important.
WebGL - like html5 but even less used
Startup Time - i keep browers open 24/7, people who cant leave a browser open all day long (in my case 78 days, with restarting them only to apply updates) than there computers are under powered. 4gb of ram is all you need to keep one open all day, so start up shouldn't be weighed at all
i dont know enough to comment: CSS, DOM, Standards Conformance

this is the list as i see it, css and dom, i have a light understanding of, but not enough to comment, if i have to put it another way, i have all of the browsers but safari on my computer, i notice very little difference in the look of web pages, performance i notice, but look all the same, so any minor difference in css or dom are pretty much unimportant. seeing as even if they arent fully supported or conformed to them, people who make webpages test it on multipul browsers and make sure it works, so end result they dont matter mush, but i dont know where to rank them. and standard conformance... i dont know what that means just general... or what? again i just dont know where to rank that one.

 

[dotaloc]

*impressed by the reasonable comments, in general*

 

[jimmysmitty]

here, i will make a list of what should be top and bottom and what i dont know what it does or shows.

Essential:
Flash - almsot all wrb pages have it, and many webpages can be unusable if flash is poorly rendered (its why i use chrome and firefox, chrome handles flash great, firefox uses less memory)
JavaScript - we are at a point where java is fast enough for most people, many scripts we use to customize out browsers are javascript if i am correct, so it is important.
Memory Efficiency - this is a deciding factor in what browser i use as main, and as media. firefox can take a heavy load and come out with far less memory usage than chrome.
Silverlight - netflix is still a big part of what people use the internet for, even though it lost subscribers... for stupid reasons.
Important:
Page Load Reliability - its important, but not so much, i have a 50 mbit line, if i need to reloading a webpage takes no effort and is painless, but is still an important feature
Nonessential:
Java - it may just be me, but i have to go out of my way to find anything java on the internet, java does not equal javascript
HTML5 Hardware Acceleration - when html5 really takes off, this will be important, but right now, i have to go out of my way to find any of it
HTML5 - very little really uses it at this point
Unimportant:
Page Load Time - whats the difference between 1ms and 1 second? when we are talking about page loads, and the likely ness that you are going to stay on that page for a minute or so, that time to load the page today is not important.
WebGL - like html5 but even less used
Startup Time - i keep browers open 24/7, people who cant leave a browser open all day long (in my case 78 days, with restarting them only to apply updates) than there computers are under powered. 4gb of ram is all you need to keep one open all day, so start up shouldn't be weighed at all
i dont know enough to comment: CSS, DOM, Standards Conformance

this is the list as i see it, css and dom, i have a light understanding of, but not enough to comment, if i have to put it another way, i have all of the browsers but safari on my computer, i notice very little difference in the look of web pages, performance i notice, but look all the same, so any minor difference in css or dom are pretty much unimportant. seeing as even if they arent fully supported or conformed to them, people who make webpages test it on multipul browsers and make sure it works, so end result they dont matter mush, but i dont know where to rank them. and standard conformance... i dont know what that means just general... or what? again i just dont know where to rank that one.

Not everyone has a newer/modern PC that can support or has 4GB of RAM. A large population still uses XP and Pentium 4s with 1GB of RAM.

Either way, I don't think anything is more important than security overall and then stability along with compatibility. The rest is either just flashy or pointless (such as web page load times which are now almost all under 1 second per browser).

A more secure browser is important. And I mean in identifying known bad sites or files, not one that cannot be hacked as thats impossible since it will be hacked either way such as by either a OS flaw or a application flaw. Nothing is unhackable.

Stability is also a major factor. Luckily most browsers are very stable these days, normally issues are caused by bad website coding.

But compatibility is a hard one. Until there is a unified standard, it will be hard to say a browser is always compatible. For a while they were designed mainly for older IE compatibility. IE9 is more like FF and others but Chrome is WebKit based so there will be a speration of standards. Myabe HTML5 will change that, maybe not. We shall see.

Still all the competition is nice. Not sure I am a fan of the monthly FF releases. Not sure that makes a better browser instead of releasing a fully stable version tested over a year like IE.

Still, to better browsers.

 

[Trewyy]

[citation][nom]alidan[/nom]here, i will make a list of what should be top and bottom and what i dont know what it does or shows. Essential:Flash - almsot all wrb pages have it, and many webpages can be unusable if flash is poorly rendered (its why i use chrome and firefox, chrome handles flash great, firefox uses less memory) JavaScript - we are at a point where java is fast enough for most people, many scripts we use to customize out browsers are javascript if i am correct, so it is important. Memory Efficiency - this is a deciding factor in what browser i use as main, and as media. firefox can take a heavy load and come out with far less memory usage than chrome. Silverlight - netflix is still a big part of what people use the internet for, even though it lost subscribers... for stupid reasons. Importantage Load Reliability - its important, but not so much, i have a 50 mbit line, if i need to reloading a webpage takes no effort and is painless, but is still an important featureNonessential:Java - it may just be me, but i have to go out of my way to find anything java on the internet, java does not equal javascriptHTML5 Hardware Acceleration - when html5 really takes off, this will be important, but right now, i have to go out of my way to find any of itHTML5 - very little really uses it at this pointUnimportantage Load Time - whats the difference between 1ms and 1 second? when we are talking about page loads, and the likely ness that you are going to stay on that page for a minute or so, that time to load the page today is not important. WebGL - like html5 but even less usedStartup Time - i keep browers open 24/7, people who cant leave a browser open all day long (in my case 78 days, with restarting them only to apply updates) than there computers are under powered. 4gb of ram is all you need to keep one open all day, so start up shouldn't be weighed at alli dont know enough to comment: CSS, DOM, Standards Conformancethis is the list as i see it, css and dom, i have a light understanding of, but not enough to comment, if i have to put it another way, i have all of the browsers but safari on my computer, i notice very little difference in the look of web pages, performance i notice, but look all the same, so any minor difference in css or dom are pretty much unimportant. seeing as even if they arent fully supported or conformed to them, people who make webpages test it on multipul browsers and make sure it works, so end result they dont matter mush, but i dont know where to rank them. and standard conformance... i dont know what that means just general... or what? again i just dont know where to rank that one.[/citation]

Flash is actually hardly ever used. I read online that 24.6% of websites have some kind of flash - and by my own guess almost 90% of the websites w/ flash only use them for advertisements (right click one next time you see it, it will say Flash Player). So to say almost all websites use flash is probably the farthest off thing i've heard. (btw, flash usage by webpages is dropping fast, REALLY fast)

Silverlight is used by 0.3% of web sites. I see this as almost not important at all. I can see why netflix might be important to you - except the issue is netflix will be dead in 5 years because they have the worst management in the modern world. If netflix was at all cutting edge they would have done 100 things different - including switching to a client that people actually have (silverlight is hardly on anybodys computer relative to Flash), and this includes HTML5.

I will agree with you on Java. It is the least used "major" web technology. Only 0.2% of websites have it, and yet it the greatest cause of viruses (Oracle fails at making safe software).

As for saying HTML5 is not important, this is probably crazier than anything i've heard before. HTML5 is the future of the web - and it is already here. Any modern page uses HTML5 code. It allows for faster browsing (less code to read) and less hardware intensive animations - among other things.

FYI: - WebGL is used to speed up everything you do on the web (pretty much). It is not code like HTML.
- CSS is used to make things look super pretty without much code or any kind of software (like flash)
- DOM is used to make the structure of the web page (how the boxes look).

Robert

 

[alidan]

[citation][nom]jimmysmitty[/nom]Not everyone has a newer/modern PC that can support or has 4GB of RAM. A large population still uses XP and Pentium 4s with 1GB of RAM.Either way, I don't think anything is more important than security overall and then stability along with compatibility. The rest is either just flashy or pointless (such as web page load times which are now almost all under 1 second per browser).A more secure browser is important. And I mean in identifying known bad sites or files, not one that cannot be hacked as thats impossible since it will be hacked either way such as by either a OS flaw or a application flaw. Nothing is unhackable.Stability is also a major factor. Luckily most browsers are very stable these days, normally issues are caused by bad website coding.But compatibility is a hard one. Until there is a unified standard, it will be hard to say a browser is always compatible. For a while they were designed mainly for older IE compatibility. IE9 is more like FF and others but Chrome is WebKit based so there will be a speration of standards. Myabe HTML5 will change that, maybe not. We shall see.Still all the competition is nice. Not sure I am a fan of the monthly FF releases. Not sure that makes a better browser instead of releasing a fully stable version tested over a year like IE.Still, to better browsers.[/citation]
even back before i had more than 1gb of ram, i ran firefox 24/7, my assumption is that anyone with less uses the browser in ways their computer can handle.

stability is a big thing that i would love to see done... but how do you test that? and most of the time its going to be a flash fail first, followed by a javascript fail that might take the rest of the browser with it.

i think that if a ff update would kill extension compatibility, than a version upgrade is necessary, and yes, even if you force extension compatibility, some extensions will not function in newer builds.

[citation][nom]trewyy[/nom]Flash is actually hardly ever used. I read online that 24.6% of websites have some kind of flash - and by my own guess almost 90% of the websites w/ flash only use them for advertisements (right click one next time you see it, it will say Flash Player). So to say almost all websites use flash is probably the farthest off thing i've heard. (btw, flash usage by webpages is dropping fast, REALLY fast)Silverlight is used by 0.3% of web sites. I see this as almost not important at all. I can see why netflix might be important to you - except the issue is netflix will be dead in 5 years because they have the worst management in the modern world. If netflix was at all cutting edge they would have done 100 things different - including switching to a client that people actually have (silverlight is hardly on anybodys computer relative to Flash), and this includes HTML5.I will agree with you on Java. It is the least used "major" web technology. Only 0.2% of websites have it, and yet it the greatest cause of viruses (Oracle fails at making safe software).As for saying HTML5 is not important, this is probably crazier than anything i've heard before. HTML5 is the future of the web - and it is already here. Any modern page uses HTML5 code. It allows for faster browsing (less code to read) and less hardware intensive animations - among other things.FYI: - WebGL is used to speed up everything you do on the web (pretty much). It is not code like HTML. - CSS is used to make things look super pretty without much code or any kind of software (like flash) - DOM is used to make the structure of the web page (how the boxes look).Robert[/citation]

24% of webpages... you do realize how big a number that really is... now lets also go a step further and exclude unpopular websites with less than 10k total hits a year, ill assume that makes flash an even bigger percent. but lets take every website out of the question but one... youtube. that still uses flash, most video places use flash, and thats what gets consumed the most is flash video... so flash plays a LARGE portion of importance, from places that use it for ads (hogs resources, and can crash browser) to full movies played through flash (or very long videos, some up to 2 hours)

silverlight, i would personally use it over flash, because in my experience it plays a hell of allot nicer than flash, and netflix uses it because of obscurity, as in i can rip damn near any video i want from a flash website that i want, and html5 if memory serves isn't much harder, but i have yet to see a silverlight ripper, i googled it, and got some results, but nothing i want to put on my computer.

show me a few big websites that use html5, im not saying they dont exist, but i want some examples. and not things in beta, or side projects, websites that fully swiched to html5, i just wikied it, because as i remember its still not finished, and it appears to still not be finished. i also took note of this

"led by search engines and social networks" 34 top 100 sites use it, so, exclude social networks, and search engines from what you can site as an example, as most of the world uses google, and social networks are a disease as far as im concerned.

 

[alidan]

what i mean with html5 not being important is that its not right now, its not a set standard, and is still changing, till its a set in stone feature its not a "oh my god cant use this browser any more" type of feature...

if i saw chrome at 100% and fire fox at 50% id still use fire fox before chrome because there are more important features

 

[tlmck]

Need to add Waterfox to the list.

 

[virtualban]

Can I haz UI of Opera and rendering power of Chrome?

Chrome is too simplified for my tastes, so it never becomes my main browser.
Opera has bugs in the way it opens pages and executes the code in them after the full load. Though I like it how Opera does not consume too much Cpu on pages, and flash gets full priority once clicked instead of as soon as it loads.
Currently I am staying with a non-up-to-date version of Firefox because of the UI and a couple of addons. When needing to venture out of safe realms, Chrome -incognito is the tool.

 

[AndrewJacksonZA]

Thanks for taking the time to gather our input, we appreciate it.

[citation][nom]annymmo[/nom]The most important things of course if it does what it should do in the first placeage reliability and correctness.Speed always comes after that.[/citation]+1 It's the browser's MOST IMPORTANT job to render the pages correctly.

I have noticed a disturbing trend with browsers once memory usage goes above +-800MB/1024MB: they seem to die. To try and reproduce what I'm talking about, load up 20-40 tabs that have big pages in them (e.g. lots of images, adverts and interactive content like js/flash) and then browse around- how well does the browser perform? Is it still responsive? Does it crash? Do graphical artefacts appear?
I disagree on the topics assigned to the brackets. My breakdown:

Essential (in descending order)
Page Load Reliability
CSS
Standards Conformance
HTML5
DOM
JavaScript
Ability for the browser to still work once memory usage goes above +-800MB/1024MB - how well does the browser perform? Is it still responsive? Does it crash? Do graphical artefacts appear?

Important
Page Load Time
Flash
Memory Efficiency for one tab that is a blank page and then 10, 20 & 40 tabs that contain pages.

Nonessential
Startup Time - How many times do you start up a browser a day? 10 times? 10 x 20sec = 2min 20sec.
Java
Silverlight

Unimportant
Hardware Acceleration - What, besides games, use this?
WebGL

Thanks

 

[randyzie]

Accuracy, reliability, customizable, speed, memory usage.

 

[mayankleoboy1]

i suggest some subjective tests:

1. load 10 tabs together and then see how smoothly can you switch between those tabs while they are loading

2. while tabs are loading, how smooth is the UI of the browser(like menus, animations)
3. performance when memory usage is ~1GB. some browsers get slow when they use that much memory. i dont care if it uses more memory. but it shouldnt get slow when using that much memory.
4. the top 40 sites you mentioned in the previous GP, open all of them in all the browsers, and then scan for any obvious artefacts in the page rendering. i suspect Opera is going to get creamed in this test.
5. any memory leakage? by that i mean that suppose the task manager says your browser is using 200MB of RAM. but when you close the browser, you free 250MB RAM. so 50MB is the leakage.
happens all the time on my office IE7. Task manager says it is using 30MB. but when i close it, 300MB ram is freed!

category order: perfectly fine. except maybe the standard conormance should be in Important category. And why do you even test silverlight?

 

[srap]

[citation][nom]mayankleoboy1[/nom]5. any memory leakage? by that i mean that suppose the task manager says your browser is using 200MB of RAM. but when you close the browser, you free 250MB RAM. so 50MB is the leakage.happens all the time on my office IE7. Task manager says it is using 30MB. but when i close it, 300MB ram is freed!category order: perfectly fine. [/citation]
Freeing up 250 MB when it namely uses 200 MB is either the difference between Explicit and Resident memory /explanatory link/, or the fragmentation of memory (see the former link at the RSS question).
Also, IE is an exceptional monste- browser, it is really hard to get correct results due to some reasons.

 

[srap]

[citation][nom]mayankleoboy1[/nom]...[/citation]
Also, that is not memory leak. A memory leak is when it uses far more memory than it should, and this can be seen in about:memory, or the amount of used memory steadily increases even if the browser is idle.
And a tip: never use task managers for memory measurements, about:memory gives more correct numbers.

 

[spotify95]

I would personally suggest that the winner, strong, acceptable and weak values do not have set point scores. I would do it so that the winner gets (say) 5 points, and the others get a score relative to how they performed in comparison to the winner - for example, if Browser A scores 50 and browser B scores 48, the scores that are given to both browsers are quite close - say 5 points and 4.5 points? Then if Browser C only scored 5, then perhaps a very low, or even negative score should be given to those scores that are extremely weak and way off the scores of all other browsers?
Also, in terms of the essential/important etc, I believe that a multiplier should be used for the score, based on how important the test is? That way a true comparison can be shown, in the important things, about how the browsers perform.

 

[computeruser777]

You should include Maxthon browser. It is the top scoring browser at html5test.com. It also claims to be fastest. i know you are trying to test the most popular, but I think a browser that has been shown to compete at a high level should be tested against the popular ones.