Counting bytes and measuring latencies

Evolving enterprise infrastructure using Chef

People consider chef as a configuration management tool. You specify certain state using the infrastructure dsl that chef provides. Apply them , and get reproducible infrastructure. I have found chef more than that.

  1. Chef is a modular infrastructure management toolkit. There are several way to look at chef: At its core chef is an api layer over day to day system administration works. All other chef components leverage this api.
  2. chef-client and chef-solo are two binaries that execute chef scripts in client/server   or server less chef setups.
  3. knife is a command line interface to chef-managed infrastructure.

All these things together gives you traditional configuration management style workflows, a chef managed infrastructure. 

Now, there are ample one off tasks that happens only once in a server's life time. Incident management, audits are other example. There are configurations related to third party hosted solutions.  Enterprise's own service deployments.  Most of these workflows dont model well in conventional configuration management style infrastructure management.

For such workflows chef can be easily extended. Chef provides multiple avenues to extend its capabilities:

  •  Chef internally distributed as a set of different libraries.   
    1. at its core chef has Mixlib::ShellOut , which is a separate rubygem. This is the component that provides portable command execution. Since 90% of the time configuration management systems dispatch command , this is the component that can be called as heart of chef. If you have lots of shell-script based tools you can easily port them with mixlib-shellout. I have rarely seen bash scripts that checks the return value of every command it invoke. Its more difficult to maintain long shell scripts. With mixlin-shellout its much easy to express safe guard measures.
    2. chef uses mixlib-json to convert json data. This is also a separate gem, and you can independently use it. You might be wondering why this is important. JSON is the principle marshalling solution in chef and most other modern system. mixlib-json  facilitates the inclusion of a json conversion library from the available ones. You can use mixlib-json independently to grab raw data and serialize them and feed into other programs in a relatively portable way.
    3. mixlib-cli and mixlib-config . Mixlib::CLI lets you easily parse command line arguments. While mixlib-config lets you read,access override configuration values. Again both of them are distributed as separate gems. You can used these two and build your custom command line tool chains rapidly.
    4. knife-plugins. Knife can be extended using plugins. Its a more formal way of extending knife via sub-commands. The best part of chef is its pure ruby everywhere, so you can pretty much use any rubygem (fog,aws-sdk,openvz) to have any cloud based infrastructure automation tool also. There are ample cases where your configurations are not applied against a server (linux or windows etc). They can be hosted solutions. Like IAM entries inside aws, or ultradns based resource records, pagerduty escalation policies, saleforce settings campfire notifications etc. knife-plugin ecosystem along with the other libraries provide an elegant way to develop such tooling.

Instead of using directly an organization can start automating smaller chunks of work and then refactor them into portable recipe so that a conventions configuration management style infrastrcuture is possible. Chef's philosophy emphasizes on resource as the primary building block and idempotency is a desired criteria. It is desired because there is an implicit assumption of repetetive invocation of chef runs within a system. Once you have get your infrastructure to this level, there are lot magic that can be leveraged from chef , also lot other tool that work on top of chef can be adopted.



  • Chef is also available as a single monolithic operating system package. Its called omnibus installers. They are present for most of the popular os. For instance the linux installer bundles everything on top of glibc. So you have an isolated,battle tested working chef binary that you can introduce to your system without inflicting major drifts.

This is crucial, as its lets you minimize iterative migrate manual or semi-automated infrastructure under full configuration management. Or if not traditional configuration management, still reproducible and automatic infrastructure.  Start with only omnibus installer and then iteratively automate the most repetitive and  time consuming tasks. Whatever it is.

  • Once you have adopted chef significantly you'll realize the volume of chef-scripts is large. This is by product of treating  large infrastructure as code. Code needs maintenance. And the maintenance can be easy if appropriate software development practices are applied. Having tests, keeping your chef scipts under a CI adds tremendous value. But the entire paradigm depends upon how easy to write tests against chef script. Last one year has seen tremendous efforts on this aspect. Now we have decently effective unit testing , functional testing and integration testing frameworks (chefspec,cucumber-chef,minitest-handler are examples respectively). Chef allows you to treat infrastructure as any other functional domain , like mobile or banking or health care. Any developers can work with domain experts to develop neat chef scripts (an operations guy is an example of domain expert).

 Together these things makes it much easier for an enterprise to iteratively improve and evolve its infrastructure to accommodate newer innovations(like cloud adoption, automatic failovers, disposable servers) with greater agility using chef....