Node

Creativity & Productivity Coaching Bot by James Heywood

Evening all, just a quick post to point you at my latest project page.

I've just finished documenting a side project I undertook last year, building an online coaching 'bot' called Pep for a client of mine.

Hello from Pep!

My involvement with the project started when I was asked whether I could build a working application based on an initial prototype in InVision which the client presented to me, from here I was involved in market research, planning of the project, scoping of the product, requirement gathering, development, deployment and user testing.

The project page is a full write up of the project, as such, it turned into a bit of an epic but hopefully, it will be of interest to someone, and if nothing else it helps to clarify my thoughts and learning from the project, as well as add to my online portfolio of work.

If you have 10 minutes spare and are interested in the process of bringing a prototype product to market (or at least my experience of attempting this) then head on over and give it a read.

As ever comments are welcome here, hope you enjoy the journey.

Cheers,

James

Translating a Javascript Minifier from Perl to Javascript by James Heywood

So I've set myself another challenge, this time the goal is to gain a bit more experience and understanding of Perl syntax, so what better way to do that than to try and translate some Perl into a language I already know something about, that way I have to research Perl syntax and find its counterparts in a language I already understand.

As Perl is a dynamic interpreted language it seems only fair to give myself a fighting chance and use something similar, thinking about dynamic interpreted languages that I have had exposure to, of those the one I'm most familiar with is good old JavaScript, so let's use that.

Next we need something to translate, let's not go overboard here with complexity, although at the same time I need something a little more challenging than hello world, and if it was something of real utility it would help to make the results something useful, after a bit of digging around I found this Perl module; https://github.com/zoffixznet/JavaScript-Minifier a JavaScript minifier written in Perl. This kind of completes a nice circle in that if I'm successful I'll have a JavaScript minifier of JavaScript, which is nice!

Taking the goal of this exercise a little further, it would be nice to use some of the new shiny ES6 features of JavaScript too, to push this on beyond simple vanilla JS, so we'll have a go at that too.

The results of this challenge can be seen in my repository here; https://github.com/jdheywood/node-js-minifier what follows is a description of the process I went through to get this up and running, including decisions I made and my approach to the solution along with some thoughts on the outcome and where to go next, I hope you enjoy my little ramble!

Approach

So how best to approach the problem, after a brief review of the Perl module I gave myself a little vertigo, perhaps this was too much of an ask, but that's just the fear talking, and as Paul Muad'Dib once said, fear is the mind killer, (yes I recently re-read Dune, still great 10+ years after my first trip to Arrakis) so let's park that, take a step back and break the problem down into smaller more manageable pieces, as I see it there are six main steps to cracking this;

  1. Identify and set up some means of running my JavaScript minifier
  2. Translate the Perl to vanilla JavaScript 
  3. Test the translation during development 
  4. Pick a reasonable piece of JavaScript to test with
  5. Compare results with those of the Perl modules
  6. Once happy start re-writing with ES6 features

Running my JavaScript

So a few steps to take care of there, starting at the beginning how do I run my as yet non-existent JavaScript? There are a few ways to do this, I could use an on-line tool like http://jsfiddle.net or one of the many similar tools a quick Google throws up like http://jsbin.com or http://codepen.io/ but they don't really offer the kind of full hands on experience I'm after.

On reflection I decided to write a little node application that I could submit a JavaScript file to, that way I could write my minifier within node and have a framework in place to manually and possibly automatically test the use of my minifier against actual JavaScript files.

Luckily I already had a basic node application that allows you to upload and display image files, that'll do nicely for my purposes. I wrote this ages ago following a basic node tutorial (if you've searched for a node tutorial on-line you have possibly stumbled over this), you can find it here; http://www.nodebeginner.org/ only the first part is on-line, I bought the book a while ago to complete the tutorial, its a great introduction to node so well worth it in my humble opinion.

So adapting my basic node application to upload a JavaScript file and minify this then save the minified file and respond with the minified script in the body seems like a nice fit for purpose. Also it gives me some time playing around with node which is enjoyable.

A brief overview of the node application is probably a good idea then, so this is a couple of features more than the most basic node server you can write, MVP and all that, it consists of the following files; 

  • index.js - the entry point of the application, executed to start the app via $ node index.js
  • router.js - responsible for routing requests to handlers
  • server.js - the server that accepts requests and passes control to the router
  • requestHandlers.js - the handlers themselves that do the work and return responses

Translating Perl to Vanilla JS

That's the first of my steps taken care of then (with a bit of code development), how do I start translating Perl a language I don't know to JavaScript? Well it turns out I do know some Perl, or at least I know enough of similar dynamically typed and interpreted languages, such as JavaScript and a little Python and Ruby. Also most high level languages share some commonality, especially with the great granddaddy C so once you know how to code you stand a good chance of learning a new programming language.

Armed with a little knowledge then let's have a read through of the Perl module, using the (very) basic pseudo code below we can describe the flow of the Perl minify module

  • Accept the input (file or string, in my case this will be a file uploaded and converted to a string
  • Buffer the first four characters of the JavaScript in four variables, a, b, c and d
  • Initialise some other useful variables for last char processed, last whitespace found and conditional comment flag
  • Start the main loop, 
    • Check character a 
      • If forward slash; check b and/or c to determine how to process comment blocks 
      • Else if string literal; write out the literal and preserve necessary white space
      • Else if + or - character; process carefully so as to not break logic of JavaScript
      • Else if alphanumeric; write out and remove any unnecessary white space
      • Else if a closing square bracket/parentheses/brace; write and preserve endspace
      • Else if stripDebug flag set and debug comment found ';;;'; remove debug comments
      • Else; print character and skip any whitespace
      • Whatever branch is taken the buffer variables for a, b, c and d are moved on so that we progress through the input JavaScript string
      • Wash, rinse, repeat
  • End of loop when variable a is empty
  • Return output

So now we have a basic understanding of the flow how do we start translating this code? I decided to start with the various helper functions that the Perl module contains, the main routine uses several helper functions to help either identify the expression that the current part of the input represents or to help chop up the input and remove stuff we don't need like comments and whitespace etc.

So let's add a new file for my minifier then, we'll call this minifier.js to keep things nice and simple, and starting with the supporting Perl functions we'll add JavaScript functions for the following; 

  • readFile
  • isAlphanumeric
  • isEndspace
  • isWhitespace
  • isInfix
  • isPrefix
  • isPostfix
  • defined

During the development I'll also need to test functions for getChar and putChar, and finally I'll need to run the actual minify function and check it's output, but we can probably get away with using the node app itself to test these functions by trying to minify an uploaded file.

So starting at the top and working my way down gave me exposure to Perl syntax, armed with my trusty Google-Fu skills I started piecing together these helper functions, but once I understand what these should do how can I ensure what I am writing is fit for purpose? 

Test the Translation During Development

We need some tests, given that I have enough new technology on my hands at the moment I decided to shy away from implementing a JavaScript testing framework such as Jasmine, instead as I have a nice little server I decided to add a new request handler for /test and use this to test these helper functions by setting up test cases, calling the helper functions and writing out the results to the response body, try it out by calling the /test route and you'll see how far I got with these tests.

Given a bit more time I would like to implement Jasmine BDD testing, it looks like it has a nice node integration and seems fairly straight forward to set up.

So by reading through Perl syntax on-line, and writing the helper functions in JavaScript I gained an understanding of how the minifier works, once the helper functions and a means for me to execute the minifier (via my node app) were in place the next step was to write the main function that actually does the minification.

This part took the longest, especially as it is based on a while loop. There were a few head scratching moments when I entered an infinite loop, but I got there in the end. 

Pick a Reasonable Piece of JavaScript to Test With

As I was writing the main minifier function I realised I needed something realistic to test this on, looking through some of my previous projects the first thing that jumped out at me was jQuery, why not, its a well known and proven piece of code, let's have a crack at that then.

I downloaded the uncompressed development version of 1.11.3 from here; http://code.jquery.com/jquery-1.11.3.js 

Compare Results With Those of the Perl Module

There's one easy way to check the veracity of the output from my translated minifier, which is obvious really, run the same JavaScript through the Perl minifier and compare the two outputs, so let's do that then!

To do that we'll need to set up a Perl envirnoment, luckily I have one at hand as described by this post. So after finding the package on CPAN I was able to install this on my Perl VM and run my jQuery JavaScript file through it so I can compare it to my JavaScript minified version.

Initially there was a significant difference, which after a little debugging was due to an error I had made in transposing the boolean logic around /* comments. 

In Perl there is a nice bit of loop syntax, the do {...} until (condition) loop. There is no direct equivalent in JavaScript, but we can replicate this with a while (condition) {...} loop.

The error I made was in my transposition of the boolean logic, changing 

until (!defined($s->{b}) || ($s->{a} eq '*' && $s->{b} eq '/'))

to 

while (defined(myCharB) && !(myCharA == '*' && myCharB == '/'))

Initially I had the following; 

while (defined(myCharB) && (myCharA != '*' && myCharB != '/'))

Which is not the same logic, and meant I had loads of left over comment content and end comment markers that had not been removed as they should be, so my minified JavaScript file was in fact broken.

Once I resolved this I got pretty close to the Perl minified version, however I still have a few tabs or white spaces left over which the Perl version has removed. I decided to park the debugging of this as the overall difference between the Perl and my JavaScript minified scripts is only 2kb. 

You can see the comparative output of my version and the Perl module in the reference folder of my repository, the files jquery-1.11.3-min-js.js and jquery-1.11.3-min-perl.js show the two outputs.

Given that I want to look at using some of the new ES6 JavaScript syntax I figured my version is close enough for now for me to move on, and given enough time later I will redouble my efforts to get this last whitespace removed from my minified version.

Once happy start re-writing with ES6 features

So what features does ES6 provide that we might want to use, a quick bit of Google-Fu provides us a few likely candidates so I decided to pick three and implement these, below are my thoughts on these features;


Modules

There are a ton of tutorials on the net covering modules so I won't waste space discussing these here, I'll quickly mention two things here though about these. 

Firstly its great that there is now a more modular way to write and manage JavaScript, this has been long overdue in my opinion and solves a lot of horrible namespacing and scoping issues we've had to deal with in the past, for example when writing your own namespacing off the window or global scope. 

Secondly when requiring files in node (which is what your import and export module syntax is actually transpiled down to when you get this working) I found a gotcha, if you write;

import { minify } from "minifier";

Once transpiled you are asking node to require the 'minifier' module installed via npm

What you need to do to require custom modules (that you've written and have locally in your application code) is the following;

import { minify } from "./minifier";

That ./ indicates the current path and therefore node knows to look there and not try and resolve via npm installed packages, obvious I know but it stumped me for a while!


Iterators

The iterator syntax of ES6 is much cleaner than vanilla JavaScript, one example of this is the 'for of' loop construct, taking one of my test cases as an example we can see that in good old JavaScript we have the following; 

for (var i = 0; i < whitespace_cases.length; i++) {
whitespace_results.push(isWhitespace(whitespace_cases[i]));
};

This Loops over an array of test cases and pushes each result to the results array, pretty simple. The for loop is classic JavaScript syntax, with the need to reference the iterator value to pull out the entry at that position. Using the 'for of' loop construct we get the following;

for (let x of whitespace_cases) {
whitespace_results.push(isWhitespace(x));
};

As you can see this is much cleaner, just let me loop over each entry in my array, I don't care about its position in this scenario I just want to do something to each variable and crack on. I really like this, as with modules its about time we had something like this. This is also just the surface of iterators in ES6, there's much more to learn about these which can only benefit our code in future.


Template Strings

If we look at the putLiteral helper function in our minifier.js we can see that we do some string concatenation to throw an error if we find an unterminated literal, the block of code dealing with throwing the exception in my original implementation of this method looked something like this (copied directly from the way it is done in the Perl module); 

if (myLast !== delimiter) {
throw 'unterminated ' + (delimiter == '\'' ? 'single quoted string' : delimiter == '"' ? 'double quoted string' : 'regular expression') + ' literal, stopped';
}

If we choose to rewrite this using the shiny new template string syntax we get something like the following;

if (myLast !== delimiter) {

var start = 'unterminated '
var descriptive = '';
switch(delimiter) {
case '\'':
descriptive = 'single quoted string';
break;
case '"':
descriptive = 'double quoted string';
break;
default:
descriptive = 'regular expression';
}
var end = 'literal, stopped processing';

throw `${start} ${descriptive} ${end}`;
}

As you can see the actual throw statement is much cleaner, mainly as we've moved the logic out to a switch, rather than using an immediate if, but the use of back ticks and ${var} placeholders is quite nice.

When transpiled to ES5 this throw ends up as the classic; 

throw start + ' ' + descriptive + ' ' + end;

In this trivial case I'm not sure it's worth the extra lines of code, but it demonstrates string templating, I can see this helping to clean up lots of messaging code, dealing with + ' ' + to build up a string is a bit of a pain and its all too easy to introduce bugs due to missing operators, so this templating should save us debugging time if nothing else!

Running our ES6 Javascript

So how do we run our nice ES6 JavaScript? Given that support is not yet universal if we want to run it through node we'll need to transpile it down to ES5 syntax for it to execute.

This sounds like a job for some automation, let's introduce a task runner. The one I've used in the past is grunt, surely there's a grunt tool for transpiling ES6, let's have a look, yep of course there is, I'm rolling with babel you can see the package.json and Gruntfile.js I set up to run the compilation/transpilation in the route of my repository.

So the development process now is;

  • write my ES6 flavoured JS
  • run grunt
  • execute the compiled/index.js with node
  • enjoy!

This seems a little odd, write code in one format but execute in another, but its no different to using coffeescript or typescript, in fact its similar to writing code in a compiled language like C#, although admittedly the output of our compilation here is still code that needs interpreting rather than a binary executable or package.

So anyway once you have transpiled navigate to the compiled version and fire it up as before, all being well it should behave itself and work as it did before, et voila we have a JavaScipt minifier written using some ES6 javascript, very nice indeed.

Summary

So to round off this very long winded post, here are my final thoughts;

What I Like

  • The fact that my JS minifier actually minifies javascript, using javascript!
  • The use of node as a development framework
  • That I've learnt a bit about perl, setting up a VM, installing modules, syntax, the $_ default variable (or topic)
  • Grunt, babel and ES6

What I Don't like

  • That it's not quite an exact match due to some small bug I've yet to track down
  • That online JS lint tools are super strict and a bit of a pain when just trying to validate your javascript
  • That use strict kills global variables, although I shouldn't have been using them anyway

What I'd do if I had more time

  • Implement proper testing using Jasmine
  • Debug the Perl to see the control flow rather than just picturing this in my head
  • Add some defensive coding, catch thrown exceptions for example
  • Add some basic styling to the node app pages
  • Introduce more ES6 features, specifically refactor the minifier as a class

 

I've learnt quite a few things during this exercise, and it's been a fun little challenge which I've mostly achieved, as with all first passes there is plenty of room for improvement but that just gives me a reason to revisit this project and learn some more.


Ok that's all for now folks, until next time, stay classy