Help Us With the Web Browser Grand Prix Scoring System

[adamovera]

Requests for a points-based scoring system have appeared in the comments section of the Web Browser Grand Prix articles for some time now. Today we're ready to implement one, but we need help. And who better to ask than you, the Tom's Hardware readers.

Help Us With the Web Browser Grand Prix Scoring System : Read more

 

[Trewyy]

First of all, I think your importance system is terribly flawed and wrong. Here is the revised system I recommend:

Essential: Page Load Time, CSS, Page Load Reliability, HTML5, Startup Time, Memory Efficiency
Important: Flash, JavaScript, DOM, Standards Conformance
Nonessential: Java, Silverlight
Unimportant: HTML5 Hardware Acceleration, WebGL

The importance system you had was based on what developers and power users are interested in. What is in important is how the main user experiences the browser. The main user doesn't care about things like Standard Conformance and DOM. They want their webpage to load beautifully and fast.

I will not post a points recommendation until you make your importance scheme actually make sense. Your old scheme is far better than this new one you propose.

Robert

 

[annymmo]

The most important things of course if it does what it should do in the first place:
page reliability and correctness.
Speed always comes after that.

Features and WIP are something that is about correctness.
Maybe give a feature categorie divided in essential and experimental???

 

[MooseMuffin]

[citation][nom]trewyy[/nom]First of all, I think your importance system is terribly flawed and wrong. Here is the revised system I recommend:Essential: Page Load Time, CSS, Page Load Reliability, HTML5, Startup Time, Memory EfficiencyImportant: Flash, JavaScript, DOM, Standards ConformanceNonessential: Java, SilverlightUnimportant: HTML5 Hardware Acceleration, WebGLThe importance system you had was based on what developers and power users are interested in. What is in important is how the main user experiences the browser. The main user doesn't care about things like Standard Conformance and DOM. They want their webpage to load beautifully and fast.I will not post a points recommendation until you make your importance scheme actually make sense. Your old scheme is far better than this new one you propose.Robert[/citation]

I thought this was a website for power users. I start my browser like once per reboot and leave it open forever. I don't care how long it takes to start up. My system has plenty of memory, so I'm not terribly concerned about efficiency either. What I care about is if using browser x will result in the page displaying quickly and correctly, which is why load time/standards conformance/dom/js belong at the top.

 

[JamesSneed]

Exactly my thoughts as well ^ annymmo I would like to see a huge amount of weighting on reliability/correctness of page loads. IE9 had issues with this at first and drove me crazy. I live in the real world so test the top 100 most visited web sites and tell me did it load the page properly and how long did it take. You know for testing graphics cards prowess in games we don't test how fast a graphics card can render a polygon, no we test what the end user sees(Image Quality) and feels(FPS), the same abstraction from the tech should be done here. Thanks.

 

[Trewyy]

[citation][nom]MooseMuffin[/nom]I thought this was a website for power users. I start my browser like once per reboot and leave it open forever. I don't care how long it takes to start up. My system has plenty of memory, so I'm not terribly concerned about efficiency either. What I care about is if using browser x will result in the page displaying quickly and correctly, which is why load time/standards conformance/dom/js belong at the top.[/citation]

Don't get me wrong, this website is definitely for power users. But you do not represent power users and developers. Unfortunately we all have different features that we feel are more important, but these tests weren't created to show what will be the best for developers - if so, Internet Explorer never would have the Grand Prix.

I just think that the way Adam Overa (author of the Grand Prix) is taking these tests is wrong way. We shouldn't be doing these tests for us - power users already know whats best for us. These tests are for the public who may not be aware of speed improvements.

Robert

 

[Guest]

You'll get so many different opinions on this, I'm not sure how we can help really. And popularity of an opinion won't necessarily mean you'll get the right choices.

Anyways, I don't agree with the importance categories on just a few details:

Page Load Times is non-essential on modern web browsers which already have so very little differences between them. Some millisecond or even 1 second difference of page load times between two browsers is absolutely something not important. Web browsers are used in an event-like basis, not as a web spider collecting thousands of websites in a serial fashion. I need to hit a button, a link, or a toolbar item to navigate to a page. Often the time I take to move my mouse there is longer than whatever benefit I may gain from a faster web browser. I never consider page load times.

Startup Time is however essential. Not everyone likes, or can afford, to keep a browser opened all the time. It's nice to have a web browser that becomes itself available quickly. That said, the only reason I consider this Essential is because of the categories you established. Saying it is Nonessential is a bit to much since it does have some weight on my decisions.

Flash, Java and HTML5 are essential. It's irrelevant whether one likes or not flash. It's still an important part of the modern web and it will remain so for many years, unfortunately. As for HTML5, it is an emerging technology and not having it on a browser means not being able to properly use HTML5 web pages. Now, not being able to use ANY web page on the web is to me absolutely unacceptable. The same goes for Java. One cannot just choose to categorize web technologies as essential or non essential when looking at them from the POV of a web browser. Any web browser that doesn't support a web technology that has expression on the web is a crippled browser and one that bars me from web content. AOL again.

 

[Estix]

What it sounds like to me is that, rather than a single tell-all score, there should be multiple scoring systems; that is, a score for people who care about speed, a score for people who care about standards compliance, a score for what will run best on modest or old hardware, etc...

 

[pharoahhalfdead]

I would like to see a how secure each browser is with and without addons. We all know FF is notorious for having lots of addons that make us feel safe, but do they work as advertised? Security could be tested with something along the lines of how easy it is to click jack, or automatically install malware once a site is visited, etc.

I would also like to know how truthful the browsers are in private browsing, or "do not track." I know my writing skills need a lot of improvement but hopefully you understand what I am trying to say.

Thanx

 

[adamovera]

trewyy: I put startup time and memory efficiency in level 3 for a few reasons: 1) Most people only start their browser once per boot. 2) There is no evidence that memory efficiency correlates to better performance. 3) Both are measurements that can apply to ANY locally-installed application, and not just Web browsers.
I have to disagree on HTML5 simply because, how can it be more essential than Flash when soooo many more sites have flash content than HTML5 code? Ditto for JS, and I just don't get the DOM downgrade. And while the average user might not consciously care about standards conformance, they will when they can't properly load a page.

 

[of the way]

There are two issues that I would love to see addressed (sorry for not answering the actual question). With the test system used, you push as much as you can on the browser so that we see what the browsers are truly capable of. That's good. But how different would certain things (mainly speed) be on less powerful systems? Some browsers might be affected more than others, and that seems important to me. The second issue regards memory efficiency. Do we want low memory usage? Yes. Do we want our memory back after we close tabs? Yes. But not necessarily immediately. If I have memory to spare, and I reopen a recently closed tab, I wouldn't mind if the browser had held onto that memory. When there isn't memory to spare then it is an issue, and so once again, I feel a less powerful test system could be beneficial. The 'memory given back after X minutes' is nice, but I don't feel the prix ever explains memory efficiency well enough.

Maybe I should actually give the feedback Tom's is looking for....I think a weak should give 0 points for any category, but I would forgive a particularly horrendous showing getting negative points. Average 1 point for unimportant and nonessential, 2 points for important and essential. Strong should give 2 points for unimportant and nonessential, 3 points for important, and 4 for essential. Winner should be 3 for unimportant and nonessential, 4 for important, and 5 for essential. Did I treat unimportant and nonessential the same? Yes. Are the points distributions fairly arbitrary? So what if they are?

 

[adamovera]

Marfig: I see your point regarding Flash, but the reason I did not make it a level 1 category is that 1) it's a plug-in, not a Web standard, and 2) It's importance is eroding - iOS users don't seem to care, iPads/Pods/Phones are still selling like hotcakes.
I also see your point on page load times, but once you average the page load times of multiple sites, you're measuring differences in full seconds. True you only navigate to one page at a time, but think about a search session: Google a topic, click a link, not relevant, click back, choose another link... opening several pages in quick succession is relatively common.
Startup times... I guess we'd have to test on much older hardware to see how much of an impact that can have, but I think that's a demographic issue, and from my experience both power users and granny open the browser about once per boot, the former because they can spare the system resources to leave it open always, and the latter because that's the only app ever used.
Java is essential to you?

 

[Guest]

Teach me how to improperly load a page on IE9.

Anyone?

 

[adamovera]

[citation][nom]lolburghurs[/nom]Teach me how to improperly load a page on IE9.Anyone?[/citation]
Open 40 tabs with different live Web sites in each, all at once. Unless you have a seriously high-end rig, it will give you a ton of borked pages, or crash.

 

[glenricky]

[citation][nom]adamovera[/nom]trewyy: I put startup time and memory efficiency in level 3 for a few reasons: 1) Most people only start their browser once per boot. 2) There is no evidence that memory efficiency correlates to better performance. 3) Both are measurements that can apply to ANY locally-installed application, and not just Web browsers.I have to disagree on HTML5 simply because, how can it be more essential than Flash when soooo many more sites have flash content than HTML5 code? Ditto for JS, and I just don't get the DOM downgrade. And while the average user might not consciously care about standards conformance, they will when they can't properly load a page.[/citation]

I would like to give comments on your number 2. When I was using my 1GB Ram laptop, the memory efficiency is very important for me. I can't use chrome with that laptop because the memory just run out with just 5-8 tabs, and that's why I use Aurora as my browser in that laptop. And even my new 3GB Ram laptop still can run out of memory if I use IE10 with 10 tabs, why I use IE10? Because it's the fastest browser I think but it lacks of feature and compability

 

[akamrcrack]

[citation][nom]adamovera[/nom]Open 40 tabs with different live Web sites in each, all at once. Unless you have a seriously high-end rig, it will give you a ton of borked pages, or crash.[/citation]

Something I do everyday.

Rhymes with Barn.

 

[Guest]

Let's say browser C wins a benchmark with a score of 1000, browser F achieves 940, browser O gets 890 and browser S gets 820.
I believe that an absolute scoring system (like 5 for browser C, 4 for browser F, 3 for O and 2 for S) is unfair, specially when you have categories with similar scores, like this example. Browser C seems much better than browser S, with more than twice the score, but it is not that much faster. This may end up favoring a browser that is marginally better than the others on some categories, but much worse on the others (like a browser that is a little faster with JavaScript, but way worse on memory use).
It would be more fair to attribute 10 to the best score, and proportional grades for the other browsers. Then the score would more fair to a browser that is does good (but not necessarily best) on every category.

 

[gm0n3y]

Unlike most of the commenters here, I think that your brackets make sense. I would personally move memory efficiency into tier 1 or at least tier 2. It's not a big deal until you have a browser that decides that it wants over 1GB of RAM when you only have one tab open doing a google search. Other than that though, I agree with your choices.

I also agree with afmenez that there should be a score out of say 1000 instead of just rankings.

 

[mightymaxio]

Opera for the WIN since you can use voice commands XD

 

[Shin-san]

I want stability. How long before the browser freezes. I'm getting this more and more on Firefox

 

[Guest]

I search for stability, speed and correct rendering.
A missing point in the tests is the add ons (presence of popup blockers, ads remover,...) and easy and efficient download feature ( suspend, resume, list of downloads,...)

 

[weatherdude]

The brackets look fine to me. Of course everyone has their own preference but this seems to be the most pragmatic design. The point system could be based on having the lowest performer being the baseline and giving points from there. Ehh... probably too tricky to work out though.

 

[eddieroolz]

In following the outline, I will rank the features in terms of importance to me as an individual. These rankings are for suggestion only and by no means do I claim to represent any demographic.

Essential: Page Load Time, CSS, Javascript, Page Load Reliability, Memory Efficiency, Standards Conformance
Important: Flash, DOM, Startup Time, HTML5 Hardware Acceleration, HTML5
Nonessential: Java
Unimportant: WebGL, Silverlight

Now, the reasoning.

I usually launch a browser once and keep it open until next reboot. However, there are times when I must close and re-launch the browser. This happens because of memory inefficiency and so this explains why I place Memory Efficiency in the essential category. In this respect, Firefox and its derivatives are notorious offenders.

Page Load Time is there because some browsers like IE9 takes slightly longer than others to load certain sites, especially if the site is very complex and rich in content.

Page Load Reliability is there because some browsers such as IE9 and Safari fails to load a page and instead spends an eternity just spinning in circles.

Standards Conformance is there because we are moving towards converging on one web standard that works for everybody. We don't want a repeat of the IE/Netscape/Firefox/whatever era.

I've truncated some of the explanations as I don't feel warranted to explain everything, but this should give the editors a snapshot of where this particular enthusiast and power user places his attention.

 

[ik242]

my priority (in order of importance) is stability, security, correct rendering, speed, memory use.
what ticks me off is that sites check browser version to discriminate against this or that.
for example website only lets you through if you use IE or FireFox - plus backdoor: you can use any other browser if you click on 'login' link N times. Supposedly that makes it more secure...

 

[iamtheking123]

Cold start time IMO is the most important aspect. On my non-ssd laptop FF can often times thrash for 30 seconds before loading just because I have a lot of addons and plugins (all of which I use).

Absolute memory usage is unimportant. With everyone nowadays running >= 4gb of ram (even in laptops), whether a browser uses 400 mb or 20 mb doesn't matter anymore. Test memory reclamation....record starting memory, open 40 tabs, close 40 tabs. Repeat 20 times. Whichever browser is closest to the starting memory wins.

Page load time is average importance. Again, who cares about synthetic js benchmarks or opening one tab at a time. Open 40 tabs and then see how long it takes for a "Reload All Tabs" operation to complete. Also examine what happens to CPU and memory usage during this period. I know FF really chokes when I try to reload 40 tabs at the same time (mouse freezes up).

I'd like to see some attention paid to addon selection as well. Opera may be the fastest browser in the world but who cares if there's no customizations available?