Deployment with Scarlett ( Hubot + HipChat + Nodejs + CoffeScript + Phing )

Clase Premier / Scarlett Johansson by César Moreno on Behance

Clase Premier / Scarlett Johansson by César Moreno on Behance

Here at Behance, we deploy a lot of code, very VERY frequently. We are constantly adding new features to our applications, hotfixing bugs, and changing things to give everyone a better user experience.

The Problem:

Currently, we have 3 operations engineers who have the credentials to build our applications to our pre-production and production environments ( Myself, Ko Uchiyama, and Chris Henry ). If none of us are available for whatever reason, changes don’t get pushed. At the same time, if we ARE available, on a really bad deployment day we can receive anywhere from 12-30 requests to push changes which can REALLY interrupt our work flow or weekend ( Especially mine since I’ve become the “main build engineer guy” lately). How can we make this workflow better?

The Solution:

Build a robot to do it for you! Duh.

Introducing our newest team member:

Scarlett J ( Named after Scarlett Johansson )

So what is Scarlett?

Scarlett is a bot ( built from Github’s open source project Hubot ) that sits in our HipChat rooms and does things for us! Her basic functionality was pretty awesome from the get go ( See the source here: https://github.com/github/hubot ), but we needed to extend her functionality a bit to allow her to actually build our applications. How do we do that? First let’s talk about all of the technologies that need to play nicely with each other in order for this to happen.

The Tech Behind Scarlett ( Hubot ):

Hipchat — Group Chat and IM built for teams. We use this to communicate with each other throughout the day. We’re huge fans of HipChat especially because it is available on a large number of platforms including mobile phones ( iPhone, Android ), and we believe communication is key. HipChat also has a really awesome API which became critical in creating Scarlett.

Nodejs — Hubot is essentially a nodejs server that can run either on Heroku or your own personal web server. We decided to spin up an Ubuntu 12.04 LTS cloudserver for Scarlett to live on.

CoffeeScript — A programming language that compiles into JavaScript ( inspired by Ruby, Python and Haskell ). All Hubot tasks / functionality is added via CoffeeScript.

Phing — Based on Apache ant, Phing is a PHP project build system. We use it in each of our build scripts to deploy our applications. It’s very flexible and easy to maintain ( You simply need to change an XML build file for each build script ). We have a phing script for our image service, Behance Network, ProSite, Outfitter, billing service, Action Method, search service, blog, 99U, and DLA. …Yeah, we have a lot of products to maintain!

Redis — Redis is an open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets. Redis allows Hubot to maintain persistence ( if needed ). It has an in-memory key-value store called robot.brain that can be used to store and retrieve data in a Redis database.

Puppet — One of my favorite tools. Puppet is an open source configuration management tool written in Ruby. In this particular instance, we can use puppet to push the correct / up-to-date scarlett scripts to all servers that need them. ( This will be explained more later. )

Hipchat-cli — Created by Garret Heaton. Command line bash script used to perform HipChat API calls – specifically, sending messages to a given HipChat.

Phing-Hipchat — Created by Rob Crowe. A Phing task for sending messages to a HipChat room. We use this to send status messages back to a given HipChat room to let the dev team know the status of the build.

Hippy — Created by Rob Crowe. Hippy is a simple PHP client for sending messages to a HipChat room. It is designed for incidental notifications from an application. This is a dependency for Phing-HipChat to work correctly.

HOW:

1. Find a server for Hubot to run on.

When testing Hubot out originally, I created an account on Heroku and deployed Scarlett to that. After I got her basic functionality working, I moved her over to one of our own servers ( Rackspace cloud server Ubuntu 12.04 LTS ). This move was essential because we needed this machine to serve as a jump box which would allow us to do remote commands on serval servers within our network ( Build / staging servers ). Simple guide on how to deploy Hubot on Unix ( https://github.com/github/hubot/wiki/Deploying-Hubot-onto-UNIX )

Hubut @ Github Chat Room

Hubut @ Github Chat Room

 

2. Created a LINUX user to run build script / deploy commands.

Next I created a Linux user that is capable of only one task, using an ssh private key to connect to one of our given staging/Pre-Production servers and execute our build wrapper script. This script contains logic to figure out which app you want to build out and run the correct commands to deploy new code to the correct servers. It also has logic in there to determine whether a user is authorized to run the command and a “devops only” mode to prevent developers from building during a maintenance window.

3. Created a build.coffee script file to listen for commands and execute them.

Following that, I created a Coffeescript file that parses responses from hipchat sent to Scarlett ( via Regex ), and translates them into the correct command line options to pass into the wrapper script. If the command is unrecognized, Scarlett throws an error message directly back at the user who tried to execute the command and logs it. She’s very nice with her responses though ! Example:

Scarlett Response to @bossjones

Scarlett Response to @bossjones

 

Other Fun Things Scarlett Can Do

 

1. Integration with Behance API

One of the beautiful things about Scarlett ( Hubot ) is that it can integrate with virtually any API that you have access to. The obvious first choice? The Behance API! I created a feature called bequery, which allows you to look up specific information about a given user / project / comment on the Behance. A lot of times we receive errors in our Distributed Logging Application ( DLA! ) that sends us a user ID and the exception that was thrown … in the past, for us to figure out which user was affected and how it changed their user experience, we had to go onto one of the DB slaves and query the user by their user ID — not very efficient, and not everyone on the team has access to those servers or even knows how to do that. Now, all we need to do is simply say to Scarlett:

@scarlett bequery user 557703

and…….

Scarlett finds user @ aintaer

Scarlett finds user @ aintaer

Simple tools like this will allow our community team to be more proactive and efficient in dealing with our users!

The Coffeescript code for that is here ( Don’t rip me apart, I’m not a CoffeeScript expert! But that’s the beauty of it: Anyone can do it! ):

# Description:
# Query the network for information on users / things
#
# Commands:
# bequery user [name|uid]
#
# URLS:
# /hubot/bequery
#
# Notes:
# Hit behance api

util = require 'util'
url = require 'url'

failureCodes =
  '403': 'Forbidden'
  '404': 'Not Found'
  '429': 'Too Many Requests'
  '500': 'Internal Server Error'
  '504': 'Service Unavailable'

module.exports = (robot) ->
  robot.respond /(bequery user) (.*)/i, (msg) ->
    endpoint = url.format
      protocol: 'http'
      host: 'www.behance.net'
      pathname: util.format 'v2/users/%s', msg.match[2]

    msg
      .http(endpoint)
      .query
        api_key: process.env.BE_API_KEY
      .get() (err, res, body) ->
        return msg.send failureCodes[res.statusCode] if failureCodes[res.statusCode]
        try
          results = JSON.parse body
          user = results.user
          msg.send util.format "%s - %s - %s - %s - %s - %s", user.id, user.first_name, user.last_name, user.username, user.display_name, user.url
          msg.send util.format "Profile Picture: %s", user.images[115]

Scarlett in the future

1. Remind people to check/confirm their hotfixes in HipChat ( store a time stamp, the name of the developer who requested the build, git commit hash, and number of reminds [ using Redis ] )

2. Devops status. Any time one of the 3 devops engineers are not available, return a message to the developer requesting a production build and remind them that the devops engineer is not available.

3. Fun feature: Create memes of your coworkers using http://version1.api.memegenerator.net/ to convey how you feel at the moment.

Point is, the sky is the limit with Scarlett ( Hubot )!

Do you use Hubot? If so, how? Do you have any ideas/suggests? Let us know in the comments below!

Till next time ! Happy deploying!

– Malcolm, Devops Engineer

Find me on Twitter: @bossjones 

My Behance Profile !