I’ve written hundreds of Bash scripts over my career, but I still suck at Bash. I have to look up the syntax for simple logical structures every single time. If I want to do anything fancy with curl or sed, I have to go and look up man pages too. I spend hours brute forcing every possible combination of single and double quotes and escaping and double-escaping every character in my regular expressions until I get something that looks like abstract ASCII art, all while trying to remember the difference between grep and perl regular expressions.
Then, one day, I looked at the last six letters of the language that I’ve been using every day for the last decade and had a lightpalm facebulb moment. It turns out you can use JavaScript… for scripting!
In this tutorial I’m going to give you my tips for building a script or command line tool using Node.js and npm. Specifically we’ll be covering:
- packaging new shell commands with npm
- parsing command line options
- reading text and passwords from stdin
- dodging callbacks with ES6 generators
- error output and codes
- coloring terminal output
- rendering an ASCII progress bar
I’m a big fan of worked examples, so to illustrate these concepts we’ll be creating a new shell command named snippet that creates a Bitbucket Snippet from a file on our local disk.
Here’s what we’re aiming for:
Packaging shell commands
npm is not only for managing dependencies for your apps and web pages. You can also use it for packaging and distributing new shell commands.
The first step is to create a new npm project with npm init:
$ npm init
name: bitbucket-snippet
version: 0.0.1
description: A command-line tool for creating Bitbucket snippets.
entry point: index.js
license: Apache-2.0
This generates a new package.json file for our project. Then we’ll need to create a JS file that will contain our script. Let’s follow Node.js conventions and call it index.js.
+ #!/usr/bin/env node
+ console.log('Hello, world!');
Note that we have to add a shebang to tell our shell how to invoke this script.
Next we need to add a bin section to the top level of our package.json. The property key (snippet, in our case) will become the command that the user invokes in their shell. The property value is the path to the script relative to the package.json.
...
"author": "Tim Pettersen",
"license": "Apache-2.0",
+ "bin": {
+ "snippet": "./index.js"
+ }
}
Now we have a working shell command! Let’s install it and test it out.
$ npm install -g
$ snippet
Hello, world!
Neat! npm install -g actually links the script to a location on our path, so we can use it like any other shell command.
$ which snippet
/usr/local/bin/snippet
$ readlink /usr/local/bin/snippet
../lib/node_modules/bitbucket-snippet/index.js
During development it’s convenient to make the symlink on our path point to the index.js we’re actually working on, using npm link.
$ npm link
/usr/local/bin/snippet -> /usr/local/lib/node_modules/bitbucket-snippet/index.js
/usr/local/lib/node_modules/bitbucket-snippet -> /Users/kannonboy/src/bitbucket-snippet
When we’re ready, we can publish our script to the public npm registry with npm publish. Then anyone in the world will be able to install it on their own machine by running:
$ npm install -g bitbucket-snippet
But let’s get our script working first!
Parsing command line options
Our script is going to need a few pieces of input from the user: their Bitbucket username, their password, and the file to upload as a snippet. The typical pattern for scripts is to pass these values in as arguments to the command.
You can get the raw arguments passed to your node script with process.argv, but there are several npm packages that provide nice abstractions for parsing arguments and options for you. My favorite is commander, inspired by the Ruby gem of the same name.
A simple:
$ npm install --save commander
Will add the latest version to our package.json. We can then define our options in a simple declarative fashion:
#!/usr/bin/env node
- console.log('Hello, world!');
+ var program = require('commander');
+
+ program
+ .arguments('<file>')
+ .option('-u, --username <username>', 'The user to authenticate as')
+ .option('-p, --password <password>', 'The user's password')
+ .action(function(file) {
+ console.log('user: %s pass: %s file: %s',
+ program.username, program.password, file);
+ })
+ .parse(process.argv);
This is pretty easy to read. Actually, that’s an understatement. It’s a work of art compared to what handling switches looks like in Bash. At least, the kind of Bash that I write.
Let’s give it a quick test.
$ snippet -u kannonboy -p correcthorsebatterystaple my_awesome_file
user: kannonboy pass: correcthorsebatterystaple file: my_awesome_file
Sweet! commander also generates some simple help output for us, based on the configuration we provided above.
$ snippet --help
Usage: snippet [options] <file>
Options:
-h, --help output usage information
-u, --username <username> The user to authenticate as
-p, --password <password> The user's password
So we have our args. However, getting a user to input their password in clear text as an option is a little clunky. Let’s fix that.
Prompting for user input
Another common way for scripts to retrieve content from the user is to read it from standard input. This is available as process.stdin, but again, there are several npm packages that provide a nice API for us to use. Most of these are based on callbacks or promises, but we’re going to use co-prompt (built on co) so we can take advantage of the magic ES6 yield keyword. This lets us write async code without callbacks that looks and feels more… scripty.
$ npm install --save co co-prompt
In order to use yield in conjunction with co-prompt we need to wrap our code in some co magic:
+ var co = require('co');
+ var prompt = require('co-prompt');
var program = require('commander');
...
.option('-u, --username <username>', 'The user to authenticate as')
.option('-p, --password <password>', 'The user's password')
.action(function(file) {
+ co(function *() {
+ var username = yield prompt('username: ');
+ var password = yield prompt.password('password: ');
console.log('user: %s pass: %s file: %s',
- program.username, program.password, file);
+ username, password, file);
+ });
})
...
Now a quick test.
$ snippet my_awesome_file
username: kannonboy
password: *************************
user: kannonboy pass: correcthorsebatterystaple file: my_awesome_file
Great! The only trick is that yield was introduced in ES6, so this will only work out of the box if the user is running node 4.0.0+. But we can make it backwards compatible all the way back to 0.11.2 by adding the --harmony flag to our shebang.
- #!/usr/bin/env node
+ #!/usr/bin/env node --harmony
var co = require('co');
var prompt = require('co-prompt');
...
POSTing the snippet
Bitbucket has a pretty sweet API for working with snippets. For this example I’m going to focus on posting a single file, but we could post whole directories, change access settings, add comments etc. if we wanted. My favorite node HTTP client is superagent, so let’s add it to the project.
$ npm install --save superagent
Now lets use the data we’re collecting from the user to POST the file to the server. One of the advantages of superagent is that it has a really nice API for working with file attachments.
+ var request = require('superagent');
var co = require('co');
var prompt = require('co-prompt');
...
.action(function(file) {
co(function *() {
var username = yield prompt('username: ');
var password = yield prompt.password('password: ');
- console.log('user: %s pass: %s file: %s',
- file, username, password);
+ request
+ .post('https://api.bitbucket.org/2.0/snippets/')
+ .auth(username, password)
+ .attach('file', file)
+ .set('Accept', 'application/json')
+ .end(function (err, res) {
+ var link = res.body.links.html.href;
+ console.log('Snippet created: %s', link);
+ });
});
});
...
Let’s try it out.
$ snippet my_awesome_file
username: kannonboy
password: *************************
Snippet created: https://bitbucket.org/snippets/kannonboy/yq7r8
Our snippet was POSTed! o/
Handling error cases
So we’re handling the happy case OK, but what if the upload fails or the user enters the wrong credentials? The UNIX-y way to handle it would be to write an message to standard error and exit with a non-zero code, so let’s do that.
...
request
.post('https://api.bitbucket.org/2.0/snippets/')
.auth(username, password)
.attach('file', filename, file)
.set('Accept', 'application/json')
.end(function (err, res) {
+ if (!err && res.ok) {
var link = res.body.links.html.href;
console.log('Snippet created: %s', link);
+ process.exit(0);
+ }
+
+ var errorMessage;
+ if (res && res.status === 401) {
+ errorMessage = "Authentication failed! Bad username/password?";
+ } else if (err) {
+ errorMessage = err;
+ } else {
+ errorMessage = res.text;
+ }
+ console.error(errorMessage);
+ process.exit(1);
});
That should do it.
Coloring terminal output
If your users are using a decent shell, there are also some packages you can use to colorize your terminal output. I like chalk as it has a clean, chainable API and automatically detects whether the user’s shell supports colors. This is handy if you want to share your script with Windows users.
$ npm install --save chalk
The output of chalk commands can be coerced to the colored and styled string and easily concatenated with regular strings.
+ var chalk = require('chalk');
var request = require('superagent');
var co = require('co');
...
.set('Accept', 'application/json')
.end(function (err, res) {
if (!err && res.ok) {
var link = res.body.links.html.href;
- console.log('Snippet created: %s', link);
+ console.log(chalk.bold.cyan('Snippet created: ') + link);
process.exit(0);
}
var errorMessage;
if (res && res.status === 401) {
errorMessage = "Authentication failed! Bad username/password?";
} else if (err) {
errorMessage = err;
} else {
errorMessage = res.text;
}
- console.error(errorMessage);
+ console.error(chalk.red(errorMessage));
process.exit(1);
});
Let’s give it a whirl (screenshot this time, so you can see the fabulous colors).
So we have a pretty neat little utility for creating text snippets now. But what about images, PDFs and other large binary files?
Adding a progress bar
The snippets API actually supports files of any type (up to 10mb), but large file sizes or slow internet connections are going to result in the command appearing to hang while the file is uploaded. The command line solution to this is a stylish ASCII loading bar.
progress is currently the most popular npm package for rendering progress bars.
$ npm install --save progress
The progress API is simple and pretty flexible, the only problem is that the node version of superagent doesn’t have an event that we can subscribe to to track how our upload is going.
We can work around this by creating a readable stream for our file attachment and adding a listener that is triggered as data is streamed from disk to the request. Then we can initialize the progress bar with the total size of the file, and increment its progress whenever our listener is triggered.
+ var fs = require('fs');
+ var ProgressBar = require('progress');
var chalk = require('chalk');
var request = require('superagent');
...
var username = yield prompt('username: ');
var password = yield prompt.password('password: ');
+ var fileSize = fs.statSync(file).size;
+ var fileStream = fs.createReadStream(file);
+ var barOpts = {
+ width: 20,
+ total: fileSize,
+ clear: true
+ };
+ var bar = new ProgressBar(' uploading [:bar] :percent :etas', barOpts);
+
+ fileStream.on('data', function (chunk) {
+ bar.tick(chunk.length);
+ });
request
.post('https://api.bitbucket.org/2.0/snippets/')
.auth(username, password)
- .attach('file', file)
+ .attach('file', fileStream)
.set('Accept', 'application/json')
...
Here it is in action with a ~6mb file on a fast internet connection.
Great! Users now have something to look at whilst their upload completes.
Summary
We’ve barely scraped the surface of what’s possible with command line tooling in node. As per Atwood’s Law, there are npm packages for elegantly handling standard input, managing parallel tasks, watching files, globbing, compressing, ssh, git, and almost everything else you did with Bash. Plus, there are nice APIs for forking child processes if you need to fall back on another shell script or command that you can’t find a decent JavaScript implementation of.
The source code for the example we built above is liberally licensed and available on Bitbucket and, of course, published to npm. I’ve also implemented a couple of features that aren’t shown here, like OAuth, so you don’t have to keep typing in your username and password. You can start using it yourself with a simple:
$ npm install -g bitbucket-snippet
$ snippet --help
If you found this useful, found a bug or have any other cool Node.js scripting tips, drop me a line on Twitter (I’m @kannonboy).
You might also enjoy our ebook, “Hello World! A new grad’s guide to coding as a team” – a collection of essays designed to help new programmers succeed in a team setting. Grab it for yourself, your team, or the new computer science graduate in your life. Even seasoned coders might learn a thing or two.