Collins exists to drive infrastructure automation.
Someone recently asked me to describe collins in a sentence. At Tumblr, it's the infrastructure
source of truth and knowledge. Everything about Tumblr production environments is stored and
encoded in Collins, and that data is used to drive all of our automation. Sometimes people refer
to systems like this as a CMDB, or Configuration Management DataBase.
Collins started as a system to manage all of the physical servers, switches,
racks, etc in Tumblr production environments. As we started to
inventory hardware, IP addresses, software, and so on, we found the API and
data gave us an excellent way to drive automation processes. Today Collins can
do push button cluster (HBase, Hadoop, web, etc) deployment, drive configuration
generation when hardware cluster topologies change, drive infrastructure updates
when software configuration changes, and help manage software deploys.
Because of the loosely coupled design of Collins, consistently applied
conventions are a system requirement. This document serves as a guide
to those conventions as well as the basic core concepts of the collins system.
If you're just interested in the basic howto or screenshots, click
Collins consists of the core application, a set of client libraries, the collins-shell,
and an automation toolchain. Nothing but collins core is required.
- Documented, consistent REST API
- Client API libraries in , Python, and Go.
- A collins command line shell for scripting and automation
- A callback system for hooking into asset lifecycle events
- Multiple authentication backends (LDAP, files, etc)
- Per-feature and per-endpoint configurable permissions
- Flexible Web and API based provisioning (PXE, SSH/grub/kickstart, API (e.g. AWS))
- Power management support
- IP Address allocation and management support
- Support for running collins in distributed mode across multiple datacenters
- Flexible data-model supporting multi-dimensional keys/values per asset
- Simple key/value queries as well as a more advanced query language
- A script engine for modifying collins behavior
- Internationalized language support
Collins provides an easy to use API on top of a simple data model. You can extend
collins through callbacks, plugins, and light-weight collins scripts.
Collins is extremely dumb. It knows about assets, their
meta-data and asset logs. You can think of collins as a key/value
store where each asset has its own set of key/value pairs. There are
no relationships between assets other than the ones you, through
convention, derive. The API makes it trivial to create and manage
the tags (meta-data, key/value pairs) associated with an asset, and
to query based on those tags.
Collins is intentionally dumb. It worries about basic
authentication, clean API interactions, and data persistence. If you
start thinking, "Hey, I should build X into Collins", you probably
shouldn't. Collins supports both a plugins architecture (for things
that actually in some way change the behavior of collins) as well as
a very usable API (including clients in Python, Ruby and Bash).
Nearly everything you might want to do can be accomplished via the
API and anything that can't is doable as a plugin.
By infrastructure engineers, for infrastructure engineers.
We looked at a dozen or so open source and commercial products before coding began.
Having an API and flexible workflow were requirements. The intake support for our
specific hardware intake process also needed to be accounted for. We also needed
something that would support not just virtualized systems, but was more tailored
towards physical hardware. We found nothing that met the requirements we had.
Things we looked at included:
Some of these are strictly for IPAM, some handled parts of what collins does. None of these were good fits for the
initial problems we were trying to solve.
Collins can sit on top of existing asset management systems (we have done this at Tumblr with both SoftLayer and Amazon)
or be the infrastructure source of truth for your entire environment. I personally found openstack to be the closest fit for
what we were looking for but the number of changes required to make it work for Tumblr were substantial.
Up and running in 5 minutes.
Collins has one software requirement - Java 1.7 (with the unlimited strength JCE).
Collins is developed using Java 1.7 and 1.8, and we currently release builds in Java 1.7 bytecode.
If you would like your data to persist between restarts, you should also have MySQL 5.5 (5.1 may work but is untested).
The simplest way to get started playing with Collins is to use the Docker image. Make sure you have a functioning Docker install. Docker repository
- Start the container:
docker run -p 9000:9000 tumblr/collins
- Hit http://localhost:9000/ and log in with the username
blake and password
- If desired, add some assets with
Use our build
- Download the latest collins zip from here
- Unpack the zip file
scripts/setup and follow the instructions
Note that the above method will only work with MySQL. If you don't have MySQL available you can follow the instructions
below which will use an in memory database.
Get your hands dirty and build it yourself
- Download the Play Framework activator
- Unpack the zip into
cd ~/src/ & & git clone email@example.com:tumblr/collins.git
~/src/activator-1.3.6-minimal/activator and hit enter (this will take a minute the first time)
- Once you are in the console type
compile and hit enter, you should see no errors. We're just testing your environment, you normally won't do this.
run and then visit
http://localhost:9000 in your browser. To run with a different configuration file (the default is
conf/application.conf), instead of typing run type
run -Dconfig.file=/path/to/alt.conf. Login with username
blake and password
There is a supplied population script at
scripts/populate.sh which will create 10 assets in your local collins instance for you to play around with.
There is a supplied package script at
scripts/package.sh which will produce a fat zip with all the requirements and dependencies. This is what we deploy to production at Tumblr.
There is a supplied init script at
scripts/collins.sh. This is a useful starting point for running the app and is what we use to run the app in production at Tumblr. This script is also bundled by the
scripts/package.sh script. Note that the supplied init script requires daemon.
By default the authentication is provided via the
authentication.type=default). Usernames/passwords for this provider are below.
You can change the auth provider by setting
conf/application.conf. If you set
file, the default file used
conf/users.conf (which can be changed by setting
The usernames/passwords/groups in the default
users.conf file are below.
The script used to generate the file based passwords is available at
There is more complete authentication configuration documentation available here
Contributing to Collins
If you want to help develop collins, you'll need to follow the instructions in "Software Setup" above. Then, to use your favorite IDE, follow the instructions on how to set it up using play.
Setting up your preferred IDE with Play Framework
Vendorizing the Docker image
If you want to use the Docker build as the base for your production instance, you ought to vendorize the image with your own configurations. A good list of things you should override is listed below, along with a sample Dockerfile to produce your build. The Tumblr provided build uses the H2 in-memory database, development credentials, and lots of sample features turned on and configured. You will certainly want to tune the settings in the build (see https://github.com/tumblr/collins/tree/master/conf/docker for all the included configs).
Here is an example Dockerfile that will vendorize the Tumblr build with your own configs that live in ./conf/vendor/
ADD ./conf/vendor/production.conf /opt/collins/conf/production.conf
ADD ./conf/vendor/database.conf /opt/collins/conf/database.conf
ADD ./conf/vendor/validations.conf /opt/collins/conf/validations.conf
ADD ./conf/vendor/authentication.conf /opt/collins/conf/authentication.conf
ADD ./conf/vendor/users.conf /opt/collins/conf/users.conf
ADD ./conf/vendor/profiles.yaml /opt/collins/conf/profiles.yaml
ADD ./conf/vendor/permissions.yaml /opt/collins/conf/permissions.yaml
Then you can
docker build -t omnicorpllc/collins:123 . and
docker run -p 9000:9000 omnicorpllc/collins:123
Quick links to useful resources
A number of people were responsible for Collins growing into the useful tool that it has.
- Dan Simon for being the only other collins contributor for months
- Steve Salevan for jumping in at the last minute to help get collins into shape to open-source
- Joshua Hoffman for working with me on the initial specs
- Dallas Marlow for figuring out all the ways to use the API that I didn't anticipate
- Brad McDuffie for providing feedback on collins-shell (and making it usable)
- Everyone else who helped with testing or provided feedback