Over the past few years i have got oppertunities to learn, adopt and extend different configuration management systems. I have used it for a wide variety of use cases. From conventional enterprise wide change management to recent days enabling continuous delivery. Each of these use cases points to certain strengths and pitfalls with individual configuration management systems. But now as i start thinking about the arean as a whole, these systems are evolving, in their design, in their capacity to scale, the kind of uses cases they solve, the kind of end users who will be adopting them.
But aside from these obvious and somewhat predictable evolution theres a more interesting development happening on how you model system, how you percieve configuration, and whether you correlate the state of an infrastructure with its configuration.
Most of the established configuration management systems (like chef , puppet, cfengine3) takes a system determines it current state, compiles a list of configuration that needs to be applied , which generates a desired state, and then does the bare minimal work to bring the current state to desired state. And if it succeeds in doing that, the system is declared as converged. Though there are certain assumptions for the convergence to be idealistic (like individual resources needs to be idempotent) , these are pretty straight forward to understand and to use.
The varieties exist within these systems are on implementation level. Puppet is fairly declarative (unless you want to extend the RAL) , hence can be easily adopted by the ops folks (assuming the ops folks code less than the devs), while chef is pretty flexible uses vanialla ruby as dsl, hence pretty popular in the dev community. lcfg uses xml, hence compaaratively easy if you want the configurations to be generated by another automated system. Palllet uses clojure which runs on jvm, so if you are a j2ee shop and fascinated with FP, pallet might be a good choice. There are other things like how easily you can scale them or how easily they mingle with other infrastructure frameworks etc..but i think there are ample resources available on those topics.
But, there are other systems like babushka which is much lighter , in way they dont assume you have the whole config that needs to be applied in the same place. So, individual system can have their own dependency modeled in babusha (deps as they call it), and each of them can apply it on the system they are running on. So, the configs are not really in a central place , neither do they applied at once, but they are applied on demand.
Also, on the same line, its a common practice to run the the configuration management system periodically or as a service, to ensure the system is in desired state., also for auditing against possible changes. But i have seen certain scenarios where this is a overkill, and running them only when required is more than enough.
It is definitely possible to run puppet or chef like systems in the later mode also. But generally they are not designed to be used like that.
One common theme thats emerging is the importance of testing. Automated, multi level (integration testing, unit testing ) testing and testing frameworks are of immense value. But, theres a no easy way to do it , though virtualization, and few other tools made it easy, its still not a satisfactory state, where you can at least address your problem, and then improvise on the solution. We're even struggling to solve the happy path testing.