On Lawn Mark, and accusations of cheating

There was a recent article on Softonic claiming “Microsoft’s browser benchmark cheats, gives IE 11 speed crown”. I attempted to comment on the article, but it seems that they are being eaten by the site’s comment system. Instead, I’m posting it here, as I think there are some important inaccuracies that need highlighting. After all, “cheating” is a serious allegation.

The evidence doesn’t seem to back up the claims of cheating that are levelled in this article.

The accusation in the article states:

What they found was that Lawn Mark 2013 was improperly setting the smallest timeout value (setTimeout) improperly for the competition. While IE11 was allowed a “0” value for setTimeout, the competition was set to the standard 4 milliseconds.

If we check the code, instead we’ll find the following:

window.requestFertilization = (
    function ()
    {
        return window.setImmediate ||
                window.msSetImmediate ||
                window.mozSetImmediate ||
                window.oSetImmediate ||
                window.webkitSetImmediate ||
                function (bagOfFertilization) { window.setTimeout(bagOfFertilization, 0); };
    }
)();

We will see that

• IE uses window.setImmediate not setTimeout(0)
• All vendor prefixes are used for setImmediate, so any browser that implements this API will get the same code as IE10+
• If browsers do not support setImmediate (currently, it is only supported in IE, and Node.js), it is given setTimeout with a value of 0, not 4 as stated above. i.e., the test gives Chrome the code that this article is claiming is given to IE11.

Now, it is true that Chrome uses a value of 4ms for setTimeout, as that is what the HTML5 spec says the minimum value setTimeout should be clamped to:

If the currently running task is a task that was created by the setTimeout() method, and timeout is less than 4, then increase timeout to 4. – HTML5 Spec

It is also worth pointing out that setImmediate used by IE (and prefixed for other browsers if they support it, as mentioned above) is a proposed W3C standard. Node.js also implements this spec, and there has been positive noises about it from respected people in the JavaScript community, such as Nicholas Zakas, and Domenic Denicola. Even Douglas Crockford uses it in one of his Monads & Gonads presentation.

So the crux of the matter is that IE supports setImmediate while other browsers do not, and thus the other browsers use setTimeout as a polyfill. Is this polyfill valid and honest? Chrome engineers seem to think not, but lets look at that.

It is certainly not completely unreasonable to use setTimeout at polyfill. For example, when debating if setImmediate should be added to WebKit/Blink, a Chromium engineer was skeptical suggested:

Shim it as “setTimeout(…, 0)”. That will be the same effect as setImmediate in most places (we only clamp when timers are nested several layers deep). – jamesr@chromium.org

This shows that:

• It likely isn’t a crazy fallback to use, as Google themselves are suggesting it.
• There probably is a use case for implementing setImmediate after all.

It is claimed, by Tony Gentilcore from Google, that the test runs intentionally slow in non-IE browsers because of an incorrectly used polyfill, and that the MDN documentation for setImmediate suggests a better polyfill. This makes it sound like this better polyfill (using postMessage instead of setTimeout is something that is clearly obvious and easy to find. It sounds like IE is ignoring this better suggestion.

Even ignoring that Google had in the past suggested the same fallback as IE used in the benchmark, if you check the history of said MDN page, you’ll see that the suggested polyfill was only added, by Paul Irish of Google, on the 27th of June of this year. The IE11 preview came out one day earlier on the 26th. Unless Microsoft have invented a time machine, it is implausible that they were ignoring this advice. It now doesn’t seem as obvious as it once did that IE is cheating, and purposely ignoring a better polyfill.

One can also debate if using postMessage actually is a better polyfill than setTimeout. For one, postMessage is a very different API than setImmediate. setTimeout is a much more obvious drop in, with a similar API signature, and something that will be more familiar to developers; they’re both timer based APIs.

It also seems like a hack to me to use postMessage in such a way, avoiding the normal security checks used by that API. The polyfill linked in the MDN article even mentions this:

It’s quite the abuse, using a cross-document messaging protocol within the same document simply to get access to the event loop task queue, but until there are native implementations, this is the best option.

While using postMessage will speed up Chrome, it has negative implications for UI responsiveness. IE also becomes less responsive when using postMessage over setImmediate.

In summary, I don’t think there is any evidence to back up accusations that the IE team was cheating when creating the Lawn Mark benchmark, and it isn’t even cut and dry whether using postMessage is the most appropriate fallback. Indeed, the clearest take away from this seems to be that there is a valid use case for setImmediate after all.