About

Collins exists to drive infrastructure automation.

Someone recently asked me to describe collins in a sentence. At Tumblr, it's the infrastructure source of truth and knowledge. Everything about Tumblr production environments is stored and encoded in Collins, and that data is used to drive all of our automation. Sometimes people refer to systems like this as a CMDB, or Configuration Management DataBase.

Collins started as a system to manage all of the physical servers, switches, racks, etc in Tumblr production environments. As we started to inventory hardware, IP addresses, software, and so on, we found the API and data gave us an excellent way to drive automation processes. Today Collins can do push button cluster (HBase, Hadoop, web, etc) deployment, drive configuration generation when hardware cluster topologies change, drive infrastructure updates when software configuration changes, and help manage software deploys.

Because of the loosely coupled design of Collins, consistently applied conventions are a system requirement. This document serves as a guide to those conventions as well as the basic core concepts of the collins system. If you're just interested in the basic howto or screenshots, click here.

Features

Collins consists of the core application, a set of client libraries, the collins-shell, and an automation toolchain. Nothing but collins core is required.

• Documented, consistent REST API
• Client API libraries in , Python, and Go.
• A collins command line shell for scripting and automation
• A callback system for hooking into asset lifecycle events
• Multiple authentication backends (LDAP, files, etc)
• Per-feature and per-endpoint configurable permissions
• Flexible Web and API based provisioning (PXE, SSH/grub/kickstart, API (e.g. AWS))
• Power management support
• IP Address allocation and management support
• Support for running collins in distributed mode across multiple datacenters
• Flexible data-model supporting multi-dimensional keys/values per asset
• Simple key/value queries as well as a more advanced query language
• A script engine for modifying collins behavior
• Internationalized language support

Design

Collins provides an easy to use API on top of a simple data model. You can extend collins through callbacks, plugins, and light-weight collins scripts.

Collins is extremely dumb. It knows about assets, their meta-data and asset logs. You can think of collins as a key/value store where each asset has its own set of key/value pairs. There are no relationships between assets other than the ones you, through convention, derive. The API makes it trivial to create and manage the tags (meta-data, key/value pairs) associated with an asset, and to query based on those tags.

Collins is intentionally dumb. It worries about basic authentication, clean API interactions, and data persistence. If you start thinking, "Hey, I should build X into Collins", you probably shouldn't. Collins supports both a plugins architecture (for things that actually in some way change the behavior of collins) as well as a very usable API (including clients in Python, Ruby and Bash). Nearly everything you might want to do can be accomplished via the API and anything that can't is doable as a plugin.

Motivation

By infrastructure engineers, for infrastructure engineers.

We looked at a dozen or so open source and commercial products before coding began. Having an API and flexible workflow were requirements. The intake support for our specific hardware intake process also needed to be accounted for. We also needed something that would support not just virtualized systems, but was more tailored towards physical hardware. We found nothing that met the requirements we had. Things we looked at included:

• openstack
• Ubuntu Orchestra
• dcTrack
• IPplan
• OpenNetAdmin
• openIPAM
• Spoiled

Some of these are strictly for IPAM, some handled parts of what collins does. None of these were good fits for the initial problems we were trying to solve.

Collins can sit on top of existing asset management systems (we have done this at Tumblr with both SoftLayer and Amazon) or be the infrastructure source of truth for your entire environment. I personally found openstack to be the closest fit for what we were looking for but the number of changes required to make it work for Tumblr were substantial.

Quickstart

Up and running in 5 minutes.

Software Requirements

Collins has one software requirement - Java 1.7 (with the unlimited strength JCE). Collins is developed using Java 1.7 and 1.8, and we currently release builds in Java 1.7 bytecode.
If you would like your data to persist between restarts, you should also have MySQL 5.5 (5.1 may work but is untested).

Docker

The simplest way to get started playing with Collins is to use the Docker image. Make sure you have a functioning Docker install. Docker repository

1. Start the container: docker run -p 9000:9000 tumblr/collins
2. Hit http://localhost:9000/ and log in with the username blake and password admin:first.
3. If desired, add some assets with ./scripts/populate.sh.

Use our build

1. Download the latest collins zip from here
2. Unpack the zip file
3. run scripts/setup and follow the instructions

Note that the above method will only work with MySQL. If you don't have MySQL available you can follow the instructions below which will use an in memory database.

Get your hands dirty and build it yourself

1. Download the Play Framework activator
2. Unpack the zip into $HOME/src/activator-1.3.6-minimal
3. cd ~/src/ & & git clone git@github.com:tumblr/collins.git
4. In ~/src/collins type ~/src/activator-1.3.6-minimal/activator and hit enter (this will take a minute the first time)
5. Once you are in the console type compile and hit enter, you should see no errors. We're just testing your environment, you normally won't do this.
6. Type run and then visit http://localhost:9000 in your browser. To run with a different configuration file (the default is conf/application.conf), instead of typing run type run -Dconfig.file=/path/to/alt.conf. Login with username blake and password admin:first.

Support Scripts

There is a supplied population script at scripts/populate.sh which will create 10 assets in your local collins instance for you to play around with.
There is a supplied package script at scripts/package.sh which will produce a fat zip with all the requirements and dependencies. This is what we deploy to production at Tumblr.
There is a supplied init script at scripts/collins.sh. This is a useful starting point for running the app and is what we use to run the app in production at Tumblr. This script is also bundled by the scripts/package.sh script. Note that the supplied init script requires daemon.

Authentication

By default the authentication is provided via the MockAuthenticationProvider (set in conf/application.conf as authentication.type=default). Usernames/passwords for this provider are below.

You can change the auth provider by setting authentication.type to file or ldap in conf/application.conf. If you set authentication.type to file, the default file used is conf/users.conf (which can be changed by setting authentication.file.userfile=/path/to/users.conf). The usernames/passwords/groups in the default users.conf file are below.

The script used to generate the file based passwords is available at scripts/gen_passwords.sh

There is more complete authentication configuration documentation available here

Contributing to Collins

If you want to help develop collins, you'll need to follow the instructions in "Software Setup" above. Then, to use your favorite IDE, follow the instructions on how to set it up using play. Setting up your preferred IDE with Play Framework

Docker

Vendorizing the Docker image

If you want to use the Docker build as the base for your production instance, you ought to vendorize the image with your own configurations. A good list of things you should override is listed below, along with a sample Dockerfile to produce your build. The Tumblr provided build uses the H2 in-memory database, development credentials, and lots of sample features turned on and configured. You will certainly want to tune the settings in the build (see https://github.com/tumblr/collins/tree/master/conf/docker for all the included configs).

• /opt/collins/conf/production.conf
• /opt/collins/conf/users.conf
• /opt/collins/conf/profiles.yaml
• /opt/collins/conf/database.conf
• /opt/collins/conf/permissions.yaml
• /opt/collins/conf/authentication.conf

Here is an example Dockerfile that will vendorize the Tumblr build with your own configs that live in ./conf/vendor/

FROM tumblr/collins
  ADD ./conf/vendor/production.conf /opt/collins/conf/production.conf
  ADD ./conf/vendor/database.conf /opt/collins/conf/database.conf
  ADD ./conf/vendor/validations.conf /opt/collins/conf/validations.conf
  ADD ./conf/vendor/authentication.conf /opt/collins/conf/authentication.conf
  ADD ./conf/vendor/users.conf /opt/collins/conf/users.conf
  ADD ./conf/vendor/profiles.yaml /opt/collins/conf/profiles.yaml
  ADD ./conf/vendor/permissions.yaml /opt/collins/conf/permissions.yaml

Then you can docker build -t omnicorpllc/collins:123 . and docker run -p 9000:9000 omnicorpllc/collins:123

Links

Quick links to useful resources

• Submit issues
• Find help
• Hang out at #collins on freenode
• Code
• Downloads
• Docs
• Docker Repository

Thanks

A number of people were responsible for Collins growing into the useful tool that it has.

• Dan Simon for being the only other collins contributor for months
• Steve Salevan for jumping in at the last minute to help get collins into shape to open-source
• Joshua Hoffman for working with me on the initial specs
• Dallas Marlow for figuring out all the ways to use the API that I didn't anticipate
• Brad McDuffie for providing feedback on collins-shell (and making it usable)
• Everyone else who helped with testing or provided feedback