sabato 1 dicembre 2012

Saltarelle vs JSIL -- Raytracer Demo

I've compared the performance of Saltarelle-compiler vs. another C# to Javascript compiler: JSIL by Kevin Gadd.

Differently from Saltarelle, JSIL operates at intermediate language (IL) level. After the C# program is compiled by Visual Studio, JSIL examines the MSIL instruction codes contained in the dll files and translates them into javascript.

This allows a greater compatibility and portability of .NET applications, but probably introduces an unnecessary layer of abstraction between the original C# code and the javascript output but the additional code needed to simulate all the .NET library has a negative impact on the execution of the compiled code.

This is shown by the following test.

I've converted the Raytracer Demo featured on the JSIL website to the Saltarelle compiler, with the aim to compare the execution speed of the two versions.

Adapting the code to Saltarelle was easy but not straightforward. Saltarelle was missing some needed System classes that I had to re-implement, namely BitmapColorStopWatch, Console and Random. I tried to keep them as similar as possible to the JSIL version so to not affect the result of the test. I also had to write the import code for mersenne.js, a small library the demo uses for generating random numbers (probably because javascript's Math.random() lacks the random seed feature).

Result of the test

I ran the test on a four core Athlon AMD processor with Windows 7 and Chrome browser. The result is summarized in the table:

Script code size (.js)108 KB15 MB
Average time/pixel0.006 ms0.093 ms

So, on average, Saltarelle is 15 times faster than JSIL. Impressive!

The output script code is also very small-sized with no loading time, while the JSIL version has to download 15 MB of stuff before it can start.

Check by yourself

Run the above test directly in your browser:
You can also download the sources hosted on GitHub .

3 commenti:

  1. Cool to see someone do this comparison - I didn't realize Saltarelle was able to compile unmodified application code in this fashion (or did you have to modify the actual code? It looks equivalent to me).

    A couple complaints though;

    first of all while your measurements for execution time seem more or less right, your conclusions about performance are totally wrong. I don't think you spent any time looking in the profiler or even reading the output.

    There's no 'unnecessary layer of abstraction' anywhere in the JSIL output that results in any performance degradation; you can easily confirm this by looking at the generated JS from Saltarelle and JSIL for the functions that show up as hotspots in the profiles, they are equivalent. For example:

    // Saltarelle
    $simpleray_$RayTracer.$checkIntersection = function(ray) {
    for (var $t1 = 0; $t1 < $simpleray_$RayTracer.$objects.length; $t1++) {
    var obj = $simpleray_$RayTracer.$objects[$t1];
    // loop through objects, test for intersection
    var hitDistance = obj.intersect(ray.$);
    // check for intersection with this object and find distance
    if (hitDistance < ray.$.closestHitDistance && hitDistance > 0) {
    ray.$.closestHitObject = obj;
    // object hit and closest yet found - store it
    ray.$.closestHitDistance = hitDistance;
    ray.$.hitPoint = $simpleray_Vector3f.op_Addition(ray.$.origin, $simpleray_Vector3f.op_Multiply(ray.$.direction, ray.$.closestHitDistance));
    // also store the point of intersection

    // JSIL
    $.Method({Static:true , Public:false}, "CheckIntersection",
    $sig.make(0x253A, null, [$jsilcore.TypeRef("JSIL.Reference", [$asm01.TypeRef("simpleray.Ray")])], []),
    function RayTracer_CheckIntersection (/* ref */ ray) {

    for (var a$0 = $thisType.objects._items, i$0 = 0, l$0 = $thisType.objects._size; i$0 < l$0; i$0++) {
    var obj = a$0[i$0];
    var hitDistance = obj.Intersect(ray.value);
    if (!((hitDistance >= ray.value.closestHitDistance) || (hitDistance <= 0))) {
    ray.value.closestHitObject = obj;
    ray.value.closestHitDistance = hitDistance;
    ray.value.hitPoint = $T01().op_Addition(ray.value.origin, $T01().op_Multiply(ray.value.direction, ray.value.closestHitDistance));

    Obviously there are differences in variable names, Saltarelle includes comments, and a couple minor semantic differences (JSIL produces slightly more opaque, but faster code for the foreach construct) but there is no unnecessary abstraction in either function. Both compilers are preserving meaning and not adding any unnecessary overhead to this operation.

    The reason the JSIL version is slower is simply due to modern JS runtimes still being unable to effectively inline simple functions and generate efficient in-memory data structures in a variety of cases. Saltarelle's output is much easier to inline since it does not actually implement the full semantics of .NET. The *actual code* generated by both compilers to do things like loops and computations is effectively equivalent, and you are measuring the performance hit(s) caused by JavaScript runtimes' failure to inline it in all cases.

    It's definitely true that the JSIL output is pretty big, but more or less for the same reason: You compiled a real .NET application, so you got all the junk that comes along with it: .NET types, etc. The fact that you had to implement Console and Random yourself kind of goes to illustrate that this is probably the wrong way to compare these two tools: Saltarelle is perfect if your goal is to build something with a C#-like language, but the wrong choice if you want to port an existing application. Likewise, I would not recommend JSIL to someone who's aiming to build an app from scratch for the web - the burden of real .NET would just get in your way.

    1. Hi Kevin, that you for your comment.

      Yes the code is almost unmodified, apart from missing types I had to change the pixel writer routine to make it work with Canvas directly.

      I agree, now that I reread it, the "unnecessary layer of abstraction" is misleading, it leads to think that there is a sort of IL emulator which is not the case.

      I started to think of JS runtime issue too, recently there was a new Saltarelle release which changed how .NET methods are simulated and that lead to a significant drop in performance in the Raytracer, not completely justified by the change introduced. (For curiosity, it's because .NET methods are no longer in the prototype, eg mystring.Length() is now compiled as something like ss.StringLength(mystr);)

      BTW, your work with JSIL is amazing, when I have to impress people with C# running in the browser I usually show them "Escape Goat"!

    2. It's interesting to hear that such a simple change on Saltarelle's end caused a performance hit; I'd encourage you (if not Saltarelle's developer) to file bugs with the developers of V8 and SpiderMonkey in scenarios where it seems like a change shouldn't cause a performance hit. At the very least, sometimes they can provide guidance on how to coax their runtimes into doing the right thing; in other cases it can result in fixes.

      Thanks again for taking the time to do this test and write it up. I appreciate it :)