Repository

Translating a Javascript Minifier from Perl to Javascript by James Heywood

So I've set myself another challenge, this time the goal is to gain a bit more experience and understanding of Perl syntax, so what better way to do that than to try and translate some Perl into a language I already know something about, that way I have to research Perl syntax and find its counterparts in a language I already understand.

As Perl is a dynamic interpreted language it seems only fair to give myself a fighting chance and use something similar, thinking about dynamic interpreted languages that I have had exposure to, of those the one I'm most familiar with is good old JavaScript, so let's use that.

Next we need something to translate, let's not go overboard here with complexity, although at the same time I need something a little more challenging than hello world, and if it was something of real utility it would help to make the results something useful, after a bit of digging around I found this Perl module; https://github.com/zoffixznet/JavaScript-Minifier a JavaScript minifier written in Perl. This kind of completes a nice circle in that if I'm successful I'll have a JavaScript minifier of JavaScript, which is nice!

Taking the goal of this exercise a little further, it would be nice to use some of the new shiny ES6 features of JavaScript too, to push this on beyond simple vanilla JS, so we'll have a go at that too.

The results of this challenge can be seen in my repository here; https://github.com/jdheywood/node-js-minifier what follows is a description of the process I went through to get this up and running, including decisions I made and my approach to the solution along with some thoughts on the outcome and where to go next, I hope you enjoy my little ramble!

Approach

So how best to approach the problem, after a brief review of the Perl module I gave myself a little vertigo, perhaps this was too much of an ask, but that's just the fear talking, and as Paul Muad'Dib once said, fear is the mind killer, (yes I recently re-read Dune, still great 10+ years after my first trip to Arrakis) so let's park that, take a step back and break the problem down into smaller more manageable pieces, as I see it there are six main steps to cracking this;

  1. Identify and set up some means of running my JavaScript minifier
  2. Translate the Perl to vanilla JavaScript 
  3. Test the translation during development 
  4. Pick a reasonable piece of JavaScript to test with
  5. Compare results with those of the Perl modules
  6. Once happy start re-writing with ES6 features

Running my JavaScript

So a few steps to take care of there, starting at the beginning how do I run my as yet non-existent JavaScript? There are a few ways to do this, I could use an on-line tool like http://jsfiddle.net or one of the many similar tools a quick Google throws up like http://jsbin.com or http://codepen.io/ but they don't really offer the kind of full hands on experience I'm after.

On reflection I decided to write a little node application that I could submit a JavaScript file to, that way I could write my minifier within node and have a framework in place to manually and possibly automatically test the use of my minifier against actual JavaScript files.

Luckily I already had a basic node application that allows you to upload and display image files, that'll do nicely for my purposes. I wrote this ages ago following a basic node tutorial (if you've searched for a node tutorial on-line you have possibly stumbled over this), you can find it here; http://www.nodebeginner.org/ only the first part is on-line, I bought the book a while ago to complete the tutorial, its a great introduction to node so well worth it in my humble opinion.

So adapting my basic node application to upload a JavaScript file and minify this then save the minified file and respond with the minified script in the body seems like a nice fit for purpose. Also it gives me some time playing around with node which is enjoyable.

A brief overview of the node application is probably a good idea then, so this is a couple of features more than the most basic node server you can write, MVP and all that, it consists of the following files; 

  • index.js - the entry point of the application, executed to start the app via $ node index.js
  • router.js - responsible for routing requests to handlers
  • server.js - the server that accepts requests and passes control to the router
  • requestHandlers.js - the handlers themselves that do the work and return responses

Translating Perl to Vanilla JS

That's the first of my steps taken care of then (with a bit of code development), how do I start translating Perl a language I don't know to JavaScript? Well it turns out I do know some Perl, or at least I know enough of similar dynamically typed and interpreted languages, such as JavaScript and a little Python and Ruby. Also most high level languages share some commonality, especially with the great granddaddy C so once you know how to code you stand a good chance of learning a new programming language.

Armed with a little knowledge then let's have a read through of the Perl module, using the (very) basic pseudo code below we can describe the flow of the Perl minify module

  • Accept the input (file or string, in my case this will be a file uploaded and converted to a string
  • Buffer the first four characters of the JavaScript in four variables, a, b, c and d
  • Initialise some other useful variables for last char processed, last whitespace found and conditional comment flag
  • Start the main loop, 
    • Check character a 
      • If forward slash; check b and/or c to determine how to process comment blocks 
      • Else if string literal; write out the literal and preserve necessary white space
      • Else if + or - character; process carefully so as to not break logic of JavaScript
      • Else if alphanumeric; write out and remove any unnecessary white space
      • Else if a closing square bracket/parentheses/brace; write and preserve endspace
      • Else if stripDebug flag set and debug comment found ';;;'; remove debug comments
      • Else; print character and skip any whitespace
      • Whatever branch is taken the buffer variables for a, b, c and d are moved on so that we progress through the input JavaScript string
      • Wash, rinse, repeat
  • End of loop when variable a is empty
  • Return output

So now we have a basic understanding of the flow how do we start translating this code? I decided to start with the various helper functions that the Perl module contains, the main routine uses several helper functions to help either identify the expression that the current part of the input represents or to help chop up the input and remove stuff we don't need like comments and whitespace etc.

So let's add a new file for my minifier then, we'll call this minifier.js to keep things nice and simple, and starting with the supporting Perl functions we'll add JavaScript functions for the following; 

  • readFile
  • isAlphanumeric
  • isEndspace
  • isWhitespace
  • isInfix
  • isPrefix
  • isPostfix
  • defined

During the development I'll also need to test functions for getChar and putChar, and finally I'll need to run the actual minify function and check it's output, but we can probably get away with using the node app itself to test these functions by trying to minify an uploaded file.

So starting at the top and working my way down gave me exposure to Perl syntax, armed with my trusty Google-Fu skills I started piecing together these helper functions, but once I understand what these should do how can I ensure what I am writing is fit for purpose? 

Test the Translation During Development

We need some tests, given that I have enough new technology on my hands at the moment I decided to shy away from implementing a JavaScript testing framework such as Jasmine, instead as I have a nice little server I decided to add a new request handler for /test and use this to test these helper functions by setting up test cases, calling the helper functions and writing out the results to the response body, try it out by calling the /test route and you'll see how far I got with these tests.

Given a bit more time I would like to implement Jasmine BDD testing, it looks like it has a nice node integration and seems fairly straight forward to set up.

So by reading through Perl syntax on-line, and writing the helper functions in JavaScript I gained an understanding of how the minifier works, once the helper functions and a means for me to execute the minifier (via my node app) were in place the next step was to write the main function that actually does the minification.

This part took the longest, especially as it is based on a while loop. There were a few head scratching moments when I entered an infinite loop, but I got there in the end. 

Pick a Reasonable Piece of JavaScript to Test With

As I was writing the main minifier function I realised I needed something realistic to test this on, looking through some of my previous projects the first thing that jumped out at me was jQuery, why not, its a well known and proven piece of code, let's have a crack at that then.

I downloaded the uncompressed development version of 1.11.3 from here; http://code.jquery.com/jquery-1.11.3.js 

Compare Results With Those of the Perl Module

There's one easy way to check the veracity of the output from my translated minifier, which is obvious really, run the same JavaScript through the Perl minifier and compare the two outputs, so let's do that then!

To do that we'll need to set up a Perl envirnoment, luckily I have one at hand as described by this post. So after finding the package on CPAN I was able to install this on my Perl VM and run my jQuery JavaScript file through it so I can compare it to my JavaScript minified version.

Initially there was a significant difference, which after a little debugging was due to an error I had made in transposing the boolean logic around /* comments. 

In Perl there is a nice bit of loop syntax, the do {...} until (condition) loop. There is no direct equivalent in JavaScript, but we can replicate this with a while (condition) {...} loop.

The error I made was in my transposition of the boolean logic, changing 

until (!defined($s->{b}) || ($s->{a} eq '*' && $s->{b} eq '/'))

to 

while (defined(myCharB) && !(myCharA == '*' && myCharB == '/'))

Initially I had the following; 

while (defined(myCharB) && (myCharA != '*' && myCharB != '/'))

Which is not the same logic, and meant I had loads of left over comment content and end comment markers that had not been removed as they should be, so my minified JavaScript file was in fact broken.

Once I resolved this I got pretty close to the Perl minified version, however I still have a few tabs or white spaces left over which the Perl version has removed. I decided to park the debugging of this as the overall difference between the Perl and my JavaScript minified scripts is only 2kb. 

You can see the comparative output of my version and the Perl module in the reference folder of my repository, the files jquery-1.11.3-min-js.js and jquery-1.11.3-min-perl.js show the two outputs.

Given that I want to look at using some of the new ES6 JavaScript syntax I figured my version is close enough for now for me to move on, and given enough time later I will redouble my efforts to get this last whitespace removed from my minified version.

Once happy start re-writing with ES6 features

So what features does ES6 provide that we might want to use, a quick bit of Google-Fu provides us a few likely candidates so I decided to pick three and implement these, below are my thoughts on these features;


Modules

There are a ton of tutorials on the net covering modules so I won't waste space discussing these here, I'll quickly mention two things here though about these. 

Firstly its great that there is now a more modular way to write and manage JavaScript, this has been long overdue in my opinion and solves a lot of horrible namespacing and scoping issues we've had to deal with in the past, for example when writing your own namespacing off the window or global scope. 

Secondly when requiring files in node (which is what your import and export module syntax is actually transpiled down to when you get this working) I found a gotcha, if you write;

import { minify } from "minifier";

Once transpiled you are asking node to require the 'minifier' module installed via npm

What you need to do to require custom modules (that you've written and have locally in your application code) is the following;

import { minify } from "./minifier";

That ./ indicates the current path and therefore node knows to look there and not try and resolve via npm installed packages, obvious I know but it stumped me for a while!


Iterators

The iterator syntax of ES6 is much cleaner than vanilla JavaScript, one example of this is the 'for of' loop construct, taking one of my test cases as an example we can see that in good old JavaScript we have the following; 

for (var i = 0; i < whitespace_cases.length; i++) {
whitespace_results.push(isWhitespace(whitespace_cases[i]));
};

This Loops over an array of test cases and pushes each result to the results array, pretty simple. The for loop is classic JavaScript syntax, with the need to reference the iterator value to pull out the entry at that position. Using the 'for of' loop construct we get the following;

for (let x of whitespace_cases) {
whitespace_results.push(isWhitespace(x));
};

As you can see this is much cleaner, just let me loop over each entry in my array, I don't care about its position in this scenario I just want to do something to each variable and crack on. I really like this, as with modules its about time we had something like this. This is also just the surface of iterators in ES6, there's much more to learn about these which can only benefit our code in future.


Template Strings

If we look at the putLiteral helper function in our minifier.js we can see that we do some string concatenation to throw an error if we find an unterminated literal, the block of code dealing with throwing the exception in my original implementation of this method looked something like this (copied directly from the way it is done in the Perl module); 

if (myLast !== delimiter) {
throw 'unterminated ' + (delimiter == '\'' ? 'single quoted string' : delimiter == '"' ? 'double quoted string' : 'regular expression') + ' literal, stopped';
}

If we choose to rewrite this using the shiny new template string syntax we get something like the following;

if (myLast !== delimiter) {

var start = 'unterminated '
var descriptive = '';
switch(delimiter) {
case '\'':
descriptive = 'single quoted string';
break;
case '"':
descriptive = 'double quoted string';
break;
default:
descriptive = 'regular expression';
}
var end = 'literal, stopped processing';

throw `${start} ${descriptive} ${end}`;
}

As you can see the actual throw statement is much cleaner, mainly as we've moved the logic out to a switch, rather than using an immediate if, but the use of back ticks and ${var} placeholders is quite nice.

When transpiled to ES5 this throw ends up as the classic; 

throw start + ' ' + descriptive + ' ' + end;

In this trivial case I'm not sure it's worth the extra lines of code, but it demonstrates string templating, I can see this helping to clean up lots of messaging code, dealing with + ' ' + to build up a string is a bit of a pain and its all too easy to introduce bugs due to missing operators, so this templating should save us debugging time if nothing else!

Running our ES6 Javascript

So how do we run our nice ES6 JavaScript? Given that support is not yet universal if we want to run it through node we'll need to transpile it down to ES5 syntax for it to execute.

This sounds like a job for some automation, let's introduce a task runner. The one I've used in the past is grunt, surely there's a grunt tool for transpiling ES6, let's have a look, yep of course there is, I'm rolling with babel you can see the package.json and Gruntfile.js I set up to run the compilation/transpilation in the route of my repository.

So the development process now is;

  • write my ES6 flavoured JS
  • run grunt
  • execute the compiled/index.js with node
  • enjoy!

This seems a little odd, write code in one format but execute in another, but its no different to using coffeescript or typescript, in fact its similar to writing code in a compiled language like C#, although admittedly the output of our compilation here is still code that needs interpreting rather than a binary executable or package.

So anyway once you have transpiled navigate to the compiled version and fire it up as before, all being well it should behave itself and work as it did before, et voila we have a JavaScipt minifier written using some ES6 javascript, very nice indeed.

Summary

So to round off this very long winded post, here are my final thoughts;

What I Like

  • The fact that my JS minifier actually minifies javascript, using javascript!
  • The use of node as a development framework
  • That I've learnt a bit about perl, setting up a VM, installing modules, syntax, the $_ default variable (or topic)
  • Grunt, babel and ES6

What I Don't like

  • That it's not quite an exact match due to some small bug I've yet to track down
  • That online JS lint tools are super strict and a bit of a pain when just trying to validate your javascript
  • That use strict kills global variables, although I shouldn't have been using them anyway

What I'd do if I had more time

  • Implement proper testing using Jasmine
  • Debug the Perl to see the control flow rather than just picturing this in my head
  • Add some defensive coding, catch thrown exceptions for example
  • Add some basic styling to the node app pages
  • Introduce more ES6 features, specifically refactor the minifier as a class

 

I've learnt quite a few things during this exercise, and it's been a fun little challenge which I've mostly achieved, as with all first passes there is plenty of room for improvement but that just gives me a reason to revisit this project and learn some more.


Ok that's all for now folks, until next time, stay classy

MongoDb C# Driver 2 - A Nice(ish) Abstraction by James Heywood

I've been doing a bit of work in the evenings of late to put something together that better demonstrates some of my back end development as I realised there was no example data access code in my bitbucket repos. 

So I've resurrected part of an old project, the project was an on-line car owners portal for Porsche, in fact some previous posts on this blog relate to this, I had to remove the name Porsche from those articles (and screen grabs) under threat from a previous employer, but I feel enough time has passed now that it's ok to mention their name, if not well we'll see what happens!

You can see my efforts in the following repository; https://github.com/jdheywood/4um - a rough estimate of the time I've spent on this is somewhere between 10 and 15 hours over the course of a week.

Overview

Anyway the part I have chosen to rewrite to show what I can do is a mini forum, where users can ask questions and other users can reply. There is a search function and the ability to favourite questions and answers so users can revisit these as and when needed. 

When we built this for the client it was a single .NET MVC application using forms authentication, knockout.js to enhance the user interface and MongoDb for the data storage. I've decided to rewrite this as an API and an SPA so that I can better separate the data from the interface, my plan is to recreate the knockout interface once the API is ready and then after that have a go at writing interfaces using a variety of JS frameworks to contrast and compare these approaches.

The user interface (when ready) will consist of two forms, one for users and another for admins (or maybe one form that adapts based on the user's role?), with a JS framework using ajax to call the API for data operations. A stretch goal would be to add OAuth support to ease registration and access, but that's way down the line.

 

Progress

For now I've been concentrating on getting the back end up and running, I thought this would be relatively quick to do but Mongo have rewritten their C# driver completely since I last looked at it so I've ended up re-writing this part completely. Also when I built this a couple of years ago we had very little test coverage so I've tried to address this shortcoming.

So what I have so far is a nice(ish) abstraction over the Mongo C# drivers (if I do say so myself!) and the beginnings of an API that uses the repositories I've created to access the MongoDb collections.

Yes I know repositories are so 2000s and in the 20 teens it's all about CQRS, but hey I can always refactor these at a later date, I wanted to concentrate on getting the mongo abstraction in place first before I changed too much.

I have written unit tests for the various factories that abstract the MongoDb drivers and integration tests to demonstrate and prove the repositories functionality. 

I've also created a simple console application to load sample data to mongo, as the integration tests set up and tear down the data they use when they finish you have no data, hence the need for loading sample data to demonstrate the API.

If you have access to a Windows machine and Visual Studio 2013 you can pull down the repo, build the solution and see this running. You'll need to install MongoDb first though, instructions/links can be found in the readme of my github repo.

If you run the integration tests you can see the repositories in action, similarly if you run the Forum.Tools.Samples console application then fire up the API and hit one of the endpoints; /api/questions, /api/answers, /api/searchterms you can see the API returning data from MongoDb.

 


Good, Bad, Better

Some things I like about this
- The abstraction over MongoDb via MongoContext, MongoClientFactory, MongoDatabaseFactory and Generic MongoCollectionFactory
- The use of SimpleInjector for IOC/Dependency Injection, especially the ability to verify the container
- The use of MongoContext in the API Startup as a single entry point to set up the data access
- The integration tests covering the repositories

Some things I would change given time;
- Refactor the repositories into Commands and Queries to allow easier testing
- Introduction of API data transfer objects and mapping between domain objects and these DTOs
- Swap out the use of an installed MongoDb for an embedded or hosted version
- Use AutoFixture customisations to build test and sample data instead of semi-hand rolling these objects

The next things for me to do to get on with this project;
- Define the API routes and build out the API layer
- Change the API to return JSON instead of XML
- Unit and Integration tests of the API
- Install swagger to self-document the API; https://github.com/swagger-api/swagger-ui 
- Set up a simple web application
- Markup and styling for the web app
- Introduction of knockout to provide UI functionality

So quite a few things to do before this is a functioning application, however the data access layer is well defined, covered by tests and helps to demonstrate the kinds of applications I've built, I hope you like it.

 

Summary

One final point to note is that I had trouble unit testing my repositories due to issues substituting the MongoCollection objects that the repository methods use, I have used integration test to add test coverage because of this issue. Even though we need integration tests anyway to actually test data storage and access we should be able to write unit tests for our repositories which run quickly and are less expensive in terms of maintenance, deployment and execution, so how do we do this?

The issue stems from the collection objects and the fluent API that the MongoDb C# driver provides, when trying to set the .Returns(object) for the chained fluent API methods off a collection NSubstitute throws errors as it doesn't know which method you are trying to set the .Returns() on.

After some googling on this and how best to resolve what I've determined is that this is a problem with my code and abstraction, rather than NSubstitute (way to state the bleeding obvious eh?). The way to resolve this is to introduce another layer to the abstraction, something along the lines of an IForumMongoCollection, and have this implement methods of IMongoCollection, this way my MongoCollectionFactory can return my ForumMongoCollections and I can mock/substitute these in my unit tests.

So a simple idea when you step back and think it through, it alluded me at the time I was writing this code though, but that's what you get for coding late into the night. So that's now top of my to do list, once that's in place I can start building out the application properly.

Anyway enough blogging for now folks, hope you like this article and the repository, as ever throw comments and questions at me below if you are so inclined.

Cheers