Here at Behance, we deploy a lot of code, very VERY frequently. We are constantly adding new features to our applications, hotfixing bugs, and changing things to give everyone a better user experience.
Currently, we have 3 operations engineers who have the credentials to build our applications to our pre-production and production environments ( Myself, Ko Uchiyama, and Chris Henry ). If none of us are available for whatever reason, changes don’t get pushed. At the same time, if we ARE available, on a really bad deployment day we can receive anywhere from 12-30 requests to push changes which can REALLY interrupt our work flow or weekend ( Especially mine since I’ve become the “main build engineer guy” lately). How can we make this workflow better?
Build a robot to do it for you! Duh.
Introducing our newest team member:
Scarlett J ( Named after Scarlett Johansson )
So what is Scarlett?
Scarlett is a bot ( built from Github’s open source project Hubot ) that sits in our HipChat rooms and does things for us! Her basic functionality was pretty awesome from the get go ( See the source here: https://github.com/github/hubot ), but we needed to extend her functionality a bit to allow her to actually build our applications. How do we do that? First let’s talk about all of the technologies that need to play nicely with each other in order for this to happen.
The Tech Behind Scarlett ( Hubot ):
Hipchat — Group Chat and IM built for teams. We use this to communicate with each other throughout the day. We’re huge fans of HipChat especially because it is available on a large number of platforms including mobile phones ( iPhone, Android ), and we believe communication is key. HipChat also has a really awesome API which became critical in creating Scarlett.
Phing — Based on Apache ant, Phing is a PHP project build system. We use it in each of our build scripts to deploy our applications. It’s very flexible and easy to maintain ( You simply need to change an XML build file for each build script ). We have a phing script for our image service, Behance Network, ProSite, Outfitter, billing service, Action Method, search service, blog, 99U, and DLA. …Yeah, we have a lot of products to maintain!
Redis — Redis is an open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets. Redis allows Hubot to maintain persistence ( if needed ). It has an in-memory key-value store called robot.brain that can be used to store and retrieve data in a Redis database.
Puppet — One of my favorite tools. Puppet is an open source configuration management tool written in Ruby. In this particular instance, we can use puppet to push the correct / up-to-date scarlett scripts to all servers that need them. ( This will be explained more later. )
Phing-Hipchat — Created by Rob Crowe. A Phing task for sending messages to a HipChat room. We use this to send status messages back to a given HipChat room to let the dev team know the status of the build.
Hippy — Created by Rob Crowe. Hippy is a simple PHP client for sending messages to a HipChat room. It is designed for incidental notifications from an application. This is a dependency for Phing-HipChat to work correctly.
1. Find a server for Hubot to run on.
When testing Hubot out originally, I created an account on Heroku and deployed Scarlett to that. After I got her basic functionality working, I moved her over to one of our own servers ( Rackspace cloud server Ubuntu 12.04 LTS ). This move was essential because we needed this machine to serve as a jump box which would allow us to do remote commands on serval servers within our network ( Build / staging servers ). Simple guide on how to deploy Hubot on Unix ( https://github.com/github/hubot/wiki/Deploying-Hubot-onto-UNIX )
2. Created a LINUX user to run build script / deploy commands.
Next I created a Linux user that is capable of only one task, using an ssh private key to connect to one of our given staging/Pre-Production servers and execute our build wrapper script. This script contains logic to figure out which app you want to build out and run the correct commands to deploy new code to the correct servers. It also has logic in there to determine whether a user is authorized to run the command and a “devops only” mode to prevent developers from building during a maintenance window.
3. Created a build.coffee script file to listen for commands and execute them.
Following that, I created a Coffeescript file that parses responses from hipchat sent to Scarlett ( via Regex ), and translates them into the correct command line options to pass into the wrapper script. If the command is unrecognized, Scarlett throws an error message directly back at the user who tried to execute the command and logs it. She’s very nice with her responses though ! Example:
Other Fun Things Scarlett Can Do
1. Integration with Behance API
One of the beautiful things about Scarlett ( Hubot ) is that it can integrate with virtually any API that you have access to. The obvious first choice? The Behance API! I created a feature called bequery, which allows you to look up specific information about a given user / project / comment on the Behance. A lot of times we receive errors in our Distributed Logging Application ( DLA! ) that sends us a user ID and the exception that was thrown … in the past, for us to figure out which user was affected and how it changed their user experience, we had to go onto one of the DB slaves and query the user by their user ID — not very efficient, and not everyone on the team has access to those servers or even knows how to do that. Now, all we need to do is simply say to Scarlett:
@scarlett bequery user 557703
Simple tools like this will allow our community team to be more proactive and efficient in dealing with our users!
The Coffeescript code for that is here ( Don’t rip me apart, I’m not a CoffeeScript expert! But that’s the beauty of it: Anyone can do it! ):
# Description: # Query the network for information on users / things # # Commands: # bequery user [name|uid] # # URLS: # /hubot/bequery # # Notes: # Hit behance api util = require 'util' url = require 'url' failureCodes = '403': 'Forbidden' '404': 'Not Found' '429': 'Too Many Requests' '500': 'Internal Server Error' '504': 'Service Unavailable' module.exports = (robot) -> robot.respond /(bequery user) (.*)/i, (msg) -> endpoint = url.format protocol: 'http' host: 'www.behance.net' pathname: util.format 'v2/users/%s', msg.match msg .http(endpoint) .query api_key: process.env.BE_API_KEY .get() (err, res, body) -> return msg.send failureCodes[res.statusCode] if failureCodes[res.statusCode] try results = JSON.parse body user = results.user msg.send util.format "%s - %s - %s - %s - %s - %s", user.id, user.first_name, user.last_name, user.username, user.display_name, user.url msg.send util.format "Profile Picture: %s", user.images
Scarlett in the future
1. Remind people to check/confirm their hotfixes in HipChat ( store a time stamp, the name of the developer who requested the build, git commit hash, and number of reminds [ using Redis ] )
2. Devops status. Any time one of the 3 devops engineers are not available, return a message to the developer requesting a production build and remind them that the devops engineer is not available.
3. Fun feature: Create memes of your coworkers using http://version1.api.memegenerator.net/ to convey how you feel at the moment.
Point is, the sky is the limit with Scarlett ( Hubot )!
Do you use Hubot? If so, how? Do you have any ideas/suggests? Let us know in the comments below!
Till next time ! Happy deploying!
– Malcolm, Devops Engineer
Find me on Twitter: @bossjones
My Behance Profile !