Preparing for the future: Part II

By December 10, 2013clypd Blog, Product

In a previous post, Jason introduced some of the infrastructure that we are building out now, in preparation for scaling in the future.

We decided early in the life of the company to drive all of our DevOps from chef. System configuration and deployment should be a software development activity: under source control, repeatable, sane.

The other thing that we wanted to be able to do was to stand up a server farm with one click of a mouse. This would allow anyone to browse on over to the web app (called toolclyp), select what type of farm they wanted, say a sandbox production farm for testing, click, voila! Ooo. Wouldn’t it be really cool if toolclyp also seeded the databases with a (scrubbed) backup of last night’s production database? Ooo. Ooo. How about if it also created the Jenkins jobs, all preconfigured, to point to the apps in Github? Also cool if you could point it at a branch in Github so that when you push a change, it automatically deploys the updated app?

Yup. We did it. It’s awesome.

We have different “farm templates” that we use, depending on what we are trying to do. A farm template is a collection of nodes (VMs), configured for various OS packages and applications. One farm template that we use, for example, has a PostgreSQL cluster (2 nodes), an app1 cluster (2 nodes), an app2 cluster (2 nodes) and a log aggregator cluster (2 nodes, using logstash). All of the applications on these nodes are configured automatically to point to the right stuff: Database server, log aggregator, etc. The load balancer is also set up for you, as if by magic.

It gets better.

The other thing that toolclyp sets up for you when you create a farm is all of the DNS stuff. New DNS CNAME records are created for all of the nodes and all of the services running on those nodes. If you want to find a service in a farm, you just reference service.farm.domain.com. You don’t have to try to remember which node service X runs on in farm template A versus farm template B.

If you are testing Database migrations, you can also use toolclyp to seed the database in your sandbox farm with the click of a mouse. Very cool.

Because of all of this, we don’t need to wear the cowboy hat NEARLY as much.

What cowboy hat are you talking about?!?

Right. OK. So we have a few props that we use around here in clypd-land. We have a superman cape, a monkey and a cowboy hat. The cape is for those that do superhuman feats. The build monkey is the monkey on your back if you break the build or tests. The cowboy hat is worn by those who shell into the production system to fix something that they broke. Yee Haw! Thankfully, with controlled devops, deployment process, chef and toolclyp, we rarely have to do that anymore.
OK. Rarely. Here I am wearing the hat after breaking production. Sumit is helping me figure out what I broke. :)

jeffCowboy
This is Richard wearing the cape. He tends to wear it a lot.

richardCape
And, our mascot. She generally wanders around the office causing trouble.

monkey
It was a lot of work to build out our devops infrastructure, but it was well worth it. You might want to check out Scalr, IronFan and RightScale for some ideas on how you bootstrap your own devops world.

When not sporting the cowboy hat, Jeff leads the Engineering team at clypd

Leave a Reply