Menu Search
Jump to the content X X
Smashing Conf Barcelona

You know, we use ad-blockers as well. We gotta keep those servers running though. Did you know that we publish useful books and run friendly conferences — crafted for pros like yourself? E.g. our upcoming SmashingConf Barcelona, dedicated to smart front-end techniques and design patterns.

How To Harness The Machines: Being Productive With Task Runners

Task runners are the heroes (or villains, depending on your point of view) that quietly toil behind most web and mobile applications. Task runners provide value through the automation of numerous development tasks such as concatenating files, spinning up development servers and compiling code. In this article, we’ll cover Grunt, Gulp, Webpack and npm scripts. We’ll also provide some examples of each one to get you started. Near the end, I’ll throw out some easy wins and tips for integrating ideas from this post into your application.

There is a sentiment that task runners, and JavaScript advances in general, are over-complicating the front-end landscape. I agree that spending the entire day tweaking build scripts isn’t always the best use of your time, but task runners have some benefits when used properly and in moderation. That’s our goal in this article, to quickly cover the basics of the most popular task runners and to provide solid examples to kickstart your imagination regarding how these tools can fit in your workflow.

Further Reading on SmashingMag: Link

A Note on the Command Line Link

Task runners and build tools are primarily command-line tools. Throughout this article, I’ll assume a decent level of experience and competence in working with the command line. If you understand how to use common commands like cd, ls, cp and mv, then you should be all right as we go through the various examples. If you don’t feel comfortable using these commands, a great introductory post is available5 on Smashing Magazine. Let’s kick things off with the granddaddy of them all: Grunt.

Grunt Link

Grunt456 was the first popular JavaScript-based task runner. I’ve been using Grunt in some form since 2012. The basic idea behind Grunt is that you use a special JavaScript file, Gruntfile.js, to configure various plugins to accomplish tasks. It has a vast ecosystem of plugins and is a very mature and stable tool. Grunt has a fantastic web directory7 that indexes the majority of plugins (about 5,500 currently). The simple genius of Grunt is its combination of JavaScript and the idea of a common configuration file (like a makefile), which has allowed many more developers to contribute to and use Grunt in their projects. It also means that Grunt can be placed under the same version control system as the rest of the project.

Grunt is battle-tested and stable. Around the time of writing, version 1.0.0 was released8, which is a huge accomplishment for the Grunt team. Because Grunt largely configures various plugins to work together, it can get tangled (i.e. messy and confusing to modify) pretty quickly. However, with a little care and organization (breaking tasks into logical files!), you can get it to do wonders for any project.

In the rare case that a plugin isn’t available to accomplish the task you need, Grunt provides documentation on how to write your own plugin9. All you need to know to create your own plugin is JavaScript and the Grunt API10. You’ll almost never have to create your own plugin, so let’s look at how to use Grunt with a pretty popular and useful plugin!

An Example Link

Screenshot of the grunt-example directory11
What our Grunt directory looks like (View large version12)

Let’s look at how Grunt actually works. Running grunt in the command line will trigger the Grunt command-line program that looks for Gruntfile.js in the root of the directory. The Gruntfile.js contains the configuration that controls what Grunt will do. In this sense, Gruntfile.js can be seen as a kind of cookbook that the cook (i.e. Grunt, the program) follows; and, like any good cookbook, Gruntfile.js will contain many recipes (i.e. tasks).

We’re going to put Grunt through the paces by using the Grunticon plugin13 to generate icons for a hypothetical web app. Grunticon takes in a directory of SVGs and spits out several assets:

  • a CSS file with the SVGs base-64-encoded as background images;
  • a CSS file with PNG versions of the SVGs base-64-encoded as background images;
  • a CSS file that references an individual PNG file for each icon.

The three different files represent the various capabilities of browsers and mobile devices. Modern devices will receive the high-resolution SVGs as a single request (i.e. a single CSS file). Browsers that don’t handle SVGs but handle base-64-encoded assets will get the base-64 PNG style sheet. Finally, any browsers that can’t handle those two scenarios will get the “traditional” style sheet that references PNGs. All this from a single directory of SVGs!

The configuration of this task looks like this:

module.exports = function(grunt) {

  grunt.config("grunticon", {
    icons: {
      files: [
        {
          expand: true,
          cwd: 'grunticon/source',
          src: ["*.svg", ".png"],
          dest: 'dist/grunticon'
        }
      ],
      options: [
        {
          colors: {
            "blue": "blue"
          }
        }
      ]
    }
  });

  grunt.loadNpmTasks('grunt-grunticon');
};

Let’s walk through the various steps here:

  1. You must have Grunt installed globally14.
  2. Create the Gruntfile.js file in the root of the project. It’s best to also install Grunt as an npm dependency in your package.json file along with Grunticon via npm i grunt grunt-grunticon --save-dev.
  3. Create a directory of SVGs and a destination directory (where the built assets will go).
  4. Place a small script in the head15 of your HTML, which will determine what icons to load.

Here is what your directory should look like before you run the Grunticon task:

|-- Gruntfile.js
|-- grunticon
|   `-- source
|       `-- logo.svg
`-- package.json

Once those things are installed and created, you can copy the code snippet above into Gruntfile.js. You should then be able to run grunt grunticon from the command line and see your task execute.

The snippet above does a few things:

  • adds a new config object to Grunt on line 32 named grunticon;
  • fills out the various options and parameters for Grunticon in the icons object;
  • finally, pulls in the Grunticon plugin via loadNPMTasks.

Here is what your directory should look like post-Grunticon:

|-- Gruntfile.js
|-- dist
|   `-- grunticon
|       |-- grunticon.loader.js
|       |-- icons.data.png.css
|       |-- icons.data.svg.css
|       |-- icons.fallback.css
|       |-- png
|       |   `-- logo.png
|       `-- preview.html
|-- grunticon
|   `-- source
|       `-- logo.svg
`-- package.json

There you go — finished! In a few lines of configuration and a couple of package installations, we’ve automated the generation of our icon assets! Hopefully, this begins to illustrate the power of task runners: reliability, efficiency and portability.

Gulp: LEGO Blocks For Your Build System Link

Gulp emerged sometime after Grunt and aspired to be a build tool that wasn’t all configuration but actual code. The idea behind code over configuration is that code is much more expressive and flexible than the modification of endless config files. The hurdle with Gulp is that it requires more technical knowledge than Grunt. You will need to be familiar with the Node.js streaming API16 and be comfortable writing basic JavaScript.

Gulp’s use of Node.js streams is the main reason it’s faster than Grunt. Using streams means that, instead of using the file system as the “database” for file transformations, Gulp uses in-memory transformations. For more information on streams, check out the Node.js streams API documentation17, along with the stream handbook18.

An Example

Screenshot of the Gulp example directory19
What our Gulp directory looks like (View large version20)

As in the Grunt section, we’re going to put Gulp through the paces with a straightforward example: concatenating our JavaScript modules into a single app file.

Running Gulp is the same as running Grunt. The gulp command-line program will look for the cookbook of recipes (i.e. Gulpfile.js) in the directory in which it’s run.

Limiting the number of requests each page makes is considered a web performance best practice (especially on mobile). Yet collaborating with other developers is much easier if functionality is split into multiple files. Enter task runners. We can use Gulp to combine the multiple files of JavaScript for our application so that mobile clients have to load a single file, instead of many.

Gulp has the same massive ecosystem of plugins as Grunt. So, to make this task easy, we’re going to lean on the gulp-concat plugin21. Let’s say our project’s structure looks like this:

|-- dist
|   `-- app.js
|-- gulpfile.js
|-- package.json
`-- src
    |-- bar.js
    `-- foo.js

Two JavaScript files are in our src directory, and we want to combine them into one file, app.js, in our dist/ directory. We can use the following Gulp task to accomplish this.

var gulp = require('gulp');
var concat = require('gulp-concat');

gulp.task('default', function() {
  return gulp.src('./src/*.js')
    .pipe(concat('app.js'))
    .pipe(gulp.dest('./dist/'));
});

The important bits are in the gulp.task callback. There, we use the gulp.src API22 to get all of the files that end with .js in our src directory. The gulp.src API returns a stream of those files, which we can then pass (via the pipe API23) to the gulp-concat plugin. The plugin then concatenates all of the files in the stream and passes it on to the gulp.dest24 function. The gulp-dest function simply writes the input it receives to disk.

You can see how Gulp uses streams to give us “building blocks” or “chains” for our tasks. A typical Gulp workflow looks like this:

  1. Get all files of a certain type.
  2. Pass those files to a plugin (concat!), or do some transformation.
  3. Pass those transformed files to another block (in our case, the dest block, which ends our chain).

As in the Grunt example, simply running gulp from the root of our project directory will trigger the default task defined in the Gulpfile.js file. This task concatenates our files and let’s us get on with developing our app or website.

Webpack Link

The newest addition to the JavaScript task runner club is Webpack4725. Webpack bills itself as a “module bundler,” which means it can dynamically build a bundle of JavaScript code from multiple separate files using module patterns such as the CommonJS26 pattern. Webpack also has plugins, which it calls loaders27.

Webpack is still fairly young and has rather dense and confusing documentation. Therefore, I’d recommend Pete Hunt’s Webpack repository28 as a great starting point before diving into the official documentation. I also wouldn’t recommend Webpack if you are new to task runners or don’t feel proficient in JavaScript. Those issues aside, it’s still a more specific tool than the general broadness of Grunt and Gulp. Many people use Webpack alongside Grunt or Gulp for this very reason, letting Webpack excel at bundling modules and letting Grunt or Gulp handle more generic tasks.

Webpack ultimately lets us write Node.js-style code for the browser, a great win for productivity and making a clean separation of concerns in our code via modules. Let’s use Webpack to achieve the same result as we did with the Gulp example, combining multiple JavaScript files into one app file.

An Example Link

Screenshot of the webpack-example directory29
What our Webpack directory looks like (View large version30)

Webpack is often used with Babel31 to transpile ES6 code to ES5. Transpiling code from ES6 to ES5 lets developers use the emerging ES6 standard while serving up ES5 to browsers or environments that don’t fully support ES6 yet. However, in this example, we’ll focus on building a simple bundle of our two files from the Gulp example. To begin, we need to install Webpack and create a config file, webpack.config.js. Here’s what our file looks like:

module.exports = {
    entry: "./src/foo.js",
    output: {
        filename: "app.js",
        path: "./dist"
    }
};

In this example, we’re pointing Webpack to our src/foo.js file to begin its work of walking our dependency graph. We’ve also updated our foo.js file to look like this:

//foo.js
var bar = require("./bar");

var foo = function() {
  console.log('foo');
  bar();
};

module.exports = foo;

And we’ve updated our bar.js file to look like this:

//bar.js
var bar = function() {
  console.log('bar');
};

module.exports = bar;

This is a very basic CommonJS example. You’ll notice that these files now “export” a function. Essentially, CommonJS and Webpack allow us to begin organizing our code into self-contained modules that can be imported and exported throughout our application. Webpack is smart enough to follow the import and export keywords and to bundle everything into one file, dist/app.js. We no longer need to maintain a concatenation task, and we simply need to adhere to a structure for our code instead. Much better!

Extending Link

Webpack is akin to Gulp in that “It’s just JavaScript.” It can be extended to do other task runner tasks via its loader system. For instance, you can use css-loader32 and sass-loader33 to compile Sass into CSS and even to use the Sass in your JavaScript by overloading the require CommonJS pattern34! However, I typically advocate for using Webpack solely to build JavaScript modules and for using another more general-purpose approach for task running (for example, Webpack and npm scripts or Webpack and Gulp to handle everything else).

npm Scripts Link

npm scripts35 are the latest hipster craze, and for good reason. As we’ve seen with all of these tools, the number of dependencies they might introduce to a project could eventually spin out of control. The first post I saw advocating for npm scripts as the entry point for a build process was by James Halliday36. His post37 perfectly sums up the ignored power of npm scripts (emphasis mine):

There are some fancy tools for doing build automation on JavaScript projects that I’ve never felt the appeal of because the lesser-known npm run command has been perfectly adequate for everything I’ve needed to do while maintaining a very tiny configuration footprint.

Did you catch that last bit at the end? The primary appeal of npm scripts is that they have a “very tiny configuration footprint.” This is one of the main reasons why npm scripts have started to catch on (almost four years later, sadly). With Grunt, Gulp and even Webpack, one eventually begins to drown in plugins that wrap binaries and double the number of dependencies in a project.

Keith Cirkel has the go-to tutorial38 on using npm to replace Grunt or Gulp. He provides the blueprint for how to fully leverage the power of npm scripts, and he’s introduced an essential plugin, Parallel Shell39 (and a host of others40 just like it).

An Example Link

In our section about Grunt, we took the popular module Grunticon and created SVG icons (with PNG fallbacks) in a Grunt task. This used to be the one pain point with npm scripts for me. For a while, I would keep Grunt installed for projects just to use Grunticon. I would literally “shell out” to Grunt in my npm task to achieve task-runner inception (or, as we started calling it at work, a build-tool turducken). Thankfully, The Filament Group41, the fantastic group behind Grunticon, released a standalone (i.e. Grunt-free) version of their tool, Grunticon-Lib42. So, let’s use it to create some icons with npm scripts!

This example is a little more advanced than a typical npm script task. A typical npm script task is a call to a command-line tool, with the appropriate flags or config file. Here’s a more typical task that compiles our Sass to CSS:

"sass": "node-sass src/scss/ -o dist/css",

See how it’s just one line with various options? No task file needed, no build tool to spin up — just npm run sass from the command line, and you’re Sass is now CSS. One really nice feature of npm scripts is how you can chain script tasks together. For instance, say we want to run some task before our Sass task runs. We would create a new script entry like this:

"presass": "echo 'before sass',

That’s right: npm understands the pre- prefix. It also understands the post- prefix. Any script entry with the same name as another script entry with a pre- or post- prefix will run before or after that entry.

Converting our icons will require an actual Node.js file. It’s not too serious, though. Just create a tasks directory, and create a new file named grunticon.js or icons.js or whatever makes sense to those working on the project. Once the file is created, we can write some JavaScript to fire off our Grunticon process.

Note: All of these examples use ES6, so we’re going to use babel-node to run our task. You can easily use ES5 and Node.js, if that’s more comfortable.

import icons from "grunticon-lib";
import globby from "globby";

let files = globby.sync('src/icons/*');
let options = {
  colors: {
    "blue": "blue"
  }
};

let icon = new icons(files, 'dist/icons', options);

icon.process();

Let’s get into the code and figure out what’s going on.

  1. We import (i.e. require) two libraries, grunticon-lib and globby. Globby43 is one of my favorite tools, and it makes working with files and globs so easy. Globby enhances Node.js Glob44 (select all JavaScript files via ./*.js) with Promise support. In this case, we’re using it to get all files in the src/icons directory.
  2. Once we do that, we set a few options in an options object and then call Grunticon-Lib with three arguments:
    • the icon files,
    • the destination,
    • the options.

    The library takes over and chews away on those icons and eventually creates the SVGs and PNG versions in the directory we want.

  3. We’re almost done. Remember that this is in a separate file, and we need to add a “hook” to call this file from our npm script, like this: "icons": "babel-node tasks/icons.js".
  4. Now we can run npm run icons, and our icons will get created every time.

npm scripts offer a similar level of power and flexibility as other task runners, without the plugin debt.

Breakdown Of Task Runners Covered Here Link

Tool Pros Cons
Grunt456 No real programming knowledge needed The most verbose of the task runners covered here
Gulp46 Configure tasks with actual JavaScript and streams Requires knowledge of JavaScript
Adds code to a project (potentially more bugs)
Webpack4725 Best in class at module-bundling More difficult for more generic tasks (for example, Sass to CSS)
npm scripts Direct interaction with command-line tools. Some tasks aren’t possible without a task runner.

Some Easy Wins Link

All of these examples and task runners might seem overwhelming, so let’s break it down. First, I hope you don’t take away from this article that whatever task runner or build system you are currently using needs to be instantly replaced with one mentioned here. Replacing important systems like this shouldn’t be done without much consideration. Here’s my advice for upgrading an existing system: Do it incrementally.

Wrapper Scripts! Link

One incremental approach is to look at writing a few “wrapper” npm scripts around your existing task runners to provide a common vocabulary for build steps that is outside of the actual task runner used. A wrapper script could be as simple as this:

{
  "scripts": {
    "start": "gulp"
  }
}

Many projects utilize the start and test npm script blocks to help new developers get acclimatized quickly. A wrapper script does introduce another layer of abstraction to your task runner build chain, yet I think it’s worth being able to standardize around the npm primitives (e.g. test). The npm commands have better longevitiy than an individual tool.

Sprinkle in a Little Webpack Link

If you or your team are feeling the pain of maintaining a brittle “bundle order” for your JavaScript, or you’re looking to upgrade to ES6, consider this an opportunity to introduce Webpack to your existing task-running system. Webpack is great in that you can use as much or as little of it as you want and yet still derive value from it. Start just by having it bundle your application code, and then add babel-loader to the mix. Webpack has such a depth of features that it’ll be able to accommodate just about any additions or new features for quite some time.

Easily Use PostCSS With npm Scripts Link

PostCSS48 is a great collection of plugins that transform and enhance CSS once it’s written and preprocessed. In other words, it’s a post-processor. It’s easy enough to leverage PostCSS using npm scripts. Say we have a Sass script like in our previous example:

"sass": "node-sass src/scss/ -o dist/css",

We can use npm script’s lifecycle keywords to add a script to run automatically after the Sass task:

"postsass": "postcss --use autoprefixer -c postcss.config.json dist/css/*.css -d dist/css",

This script will run every time the Sass script is run. The postcss-cli package is great, because you can specify configuration49 in a separate file. Notice that in this example, we add another script entry to accomplish a new task; this is a common pattern when using npm scripts. You can create a workflow that accomplishes all of the various tasks your app needs.

Conclusion Link

Task runners can solve real problems. I’ve used task runners to compile different builds of a JavaScript application, depending on whether the target was production or local development. I’ve also used task runners to compile Handlebars50 templates, to deploy a website to production and to automatically add vendor prefixes that are missed in my Sass. These are not trivial tasks, but once they are wrapped up in a task runner, they became effortless.

Task runners are constantly evolving and changing. I’ve tried to cover the most used ones in the current zeitgeist. However, there are others that I haven’t even mentioned, such as Broccoli51, Brunch52 and Harp53. Remember that these are just tools: Use them only if they solve a particular problem, not because everyone else is using them. Happy task running!

(da, ml, al, il)

Footnotes Link

  1. 1 https://www.smashingmagazine.com/2015/07/become-command-line-power-user-oh-my-zsh-z/
  2. 2 https://www.smashingmagazine.com/2015/12/introduction-to-postcss/
  3. 3 https://www.smashingmagazine.com/2013/10/get-up-running-grunt/
  4. 4 https://www.smashingmagazine.com/2014/06/building-with-gulp/
  5. 5 https://www.smashingmagazine.com/2012/01/introduction-to-linux-commands/
  6. 6 http://gruntjs.com
  7. 7 http://gruntjs.com/plugins
  8. 8 http://gruntjs.com/blog/2016-02-11-grunt-1.0.0-rc1-released
  9. 9 http://gruntjs.com/creating-plugins
  10. 10 http://gruntjs.com/api/grunt
  11. 11 https://www.smashingmagazine.com/wp-content/uploads/2016/05/01-grunt-example-opt.png
  12. 12 https://www.smashingmagazine.com/wp-content/uploads/2016/05/01-grunt-example-opt.png
  13. 13 https://github.com/filamentgroup/grunticon
  14. 14 http://gruntjs.com/getting-started
  15. 15 https://github.com/filamentgroup/grunticon/blob/master/site/src/js/head.js
  16. 16 https://nodejs.org/api/stream.html
  17. 17 https://nodejs.org/api/stream.html
  18. 18 https://github.com/substack/stream-handbook
  19. 19 https://www.smashingmagazine.com/wp-content/uploads/2016/05/02-gulp-example-opt.png
  20. 20 https://www.smashingmagazine.com/wp-content/uploads/2016/05/02-gulp-example-opt.png
  21. 21 https://github.com/contra/gulp-concat
  22. 22 https://github.com/gulpjs/gulp/blob/master/docs/API.md#gulpsrcglobs-options
  23. 23 https://nodejs.org/api/stream.html#stream_readable_pipe_destination_options
  24. 24 https://github.com/gulpjs/gulp/blob/master/docs/API.md#gulpdestpath-options
  25. 25 https://webpack.github.io
  26. 26 https://webpack.github.io/docs/commonjs.html
  27. 27 http://webpack.github.io/docs/using-loaders.html
  28. 28 https://github.com/petehunt/webpack-howto
  29. 29 https://www.smashingmagazine.com/wp-content/uploads/2016/05/03-webpack-example-opt.png
  30. 30 https://www.smashingmagazine.com/wp-content/uploads/2016/05/03-webpack-example-opt.png
  31. 31 https://babeljs.io
  32. 32 https://github.com/webpack/css-loader
  33. 33 https://github.com/jtangelder/sass-loader
  34. 34 https://webpack.github.io/docs/stylesheets.html#using-css
  35. 35 https://docs.npmjs.com/misc/scripts
  36. 36 https://github.com/substack
  37. 37 http://substack.net/task_automation_with_npm_run
  38. 38 http://blog.keithcirkel.co.uk/how-to-use-npm-as-a-build-tool/
  39. 39 https://github.com/keithamus/parallelshell
  40. 40 https://github.com/mysticatea/npm-run-all/issues/10
  41. 41 https://www.filamentgroup.com
  42. 42 https://github.com/filamentgroup/grunticon-lib
  43. 43 https://github.com/sindresorhus/globby
  44. 44 https://github.com/isaacs/node-glob
  45. 45 http://gruntjs.com
  46. 46 http://gulpjs.com
  47. 47 https://webpack.github.io
  48. 48 https://github.com/code42day/postcss-cli
  49. 49 https://github.com/code42day/postcss-cli#--config-c
  50. 50 http://handlebarsjs.com
  51. 51 https://github.com/broccolijs/broccoli
  52. 52 http://brunch.io
  53. 53 http://harpjs.com

↑ Back to top Tweet itShare on Facebook

Adam Simpson is a frontend developer working for Sparkbox in lovely Dayton, OH. He digs React.js, ancient text-editors, cookies, and his wife Christi (not necessarily in that order).

  1. 1

    Tyler Paulson

    June 22, 2016 2:43 pm

    Adam, thanks for writing this! As a freelancer, I feel like over the last six months I have worked on projects other developers have built that have used all four of these. This overview really helped me understand the differences between them, what they are most suited for, and why their configuration files look they way they do.

    1
  2. 2

    hm .. so, what about regular unix task runners that have done their job properly for ages?

    Writing stuff in JS “just because we can” is not really my favorite. And having to install it all, possibly bogging down my IDE, is even worse. NTARS and system stability over all.

    So, are we gonna see a quick roundup on eg. at, cron and anacron as a future article / part of the series? ;)

    cu, w0lf.

    -2
  3. 6

    Benny Bottema

    June 22, 2016 8:13 pm

    I feel you left out the greatest advantage of WebPack over the other tools: convention over configuration.

    With webpack I reduced pages of grunt / gulp scripts to a minimum of code, and everything is standardized. I don’t need to do any folder pinpointing at all anymore. I thought it was a breath of fresh air!

    0
  4. 7

    Right now webpack has the greatest momentum, gulp is a useful glue/fallback and grunt is pretty much legacy.

    8
    • 8

      A legacy which I think will continue to live for long cause of the community around it.

      0
  5. 9
  6. 10

    Charl Kruger

    June 28, 2016 8:07 am

    I would include the importance of –save-dev, you won’t believe how many times I have worked with a repo where guys install their plugins locally and forget to add them to the package.json.

    I know you, “assume a decent level of experience” though you have gone quite in-depth and think it can’t hurt. Also when viewing a plugin, the instructions for installation don’t always include the save dev.

    0
  7. 11

    Jack McNicol

    July 1, 2016 2:48 am

    Good stuff @Adam – we use pretty much everything there – but one take away for me was the auto `pre-` `post-` npm scripts.

    0

↑ Back to top