There is no such thing as a "best" CI Tool. There is no one size fits all. There are several other options out there and they all fit very well to various problems. I recently had to evaluate which CI Tool fits out needs so I'll give you a few suggestions on how to approach that topic.
As my answer got a little bit longer than i expected, here is the sum up:
- Learn how to use CI in your business. You can't choose the right tool if you don't know what your specific needs are.
- Figure out all important stake holders. This includes different roles as developers, architects and managers as well as people who work on different topics/projects.
- Evaluate the needs of all stake holders. Try to find common ground among all requirements and try to figure out what they need, which is not necessarily what they say.
- Get a list of all available CI Tools.
- Eliminate all tools that don't fulfill the exclusion criteria. This might be self-hosting, a price tag or similar concerns.
- There should not be that much alternatives left. Evaluate which of them fit your business the best. This might be the right place to get a trial and try out the candidates on real world problems. Build a sample workflow that you can realize with the tools. Get the stakeholders involved again, as they might know their requirements much more precise when they can use a demo. Be careful to differentiate real criticism from the fear of new tools.
- Choose the tool that is right for your specific domain and define your workflow adequately. Write it down, really, document the whole process.
- Write short tutorials for all your stakeholders on how to use the new system, how to adapt from the old system, how the terms and actions translate from the old one to the new. Be sure that all stakeholders know about the tutorials and how to access them. After all, your CI tool isn't worth much if you don't have a good, documented workflow and your whole team reasonably following the workflow. Having a tool that can monitor how well they adopt the workflow (use the given templates and similar) really helps here.
And here is the long answer:
First of all, there is no way around meticulously analyzing your specific needs.
If you haven't used a CI tool yet, it might be a good idea to try a tool that is very easy to setup and get into. Bamboo and Travis CI are relatively easy to start with. Bamboo is free for Open Source Projects and you can get a Trial version for $10 I guess. Travis CI can currently be used with GitHub only. Jenkins is a good option too, probably the most versatile out there. It is free, really easy to set up, not so difficult to use and you can do nearly everything with it using the tremendous amount of plugins available. Either way, the question should not just be about using the best CI Tool, but to start doing CI. You have to define the processes, detect the obstacles along the way, find a way to deal with them. Until you truly understand what CI is about in your particular business, this is what you have to focus on. If you got that far, here is what I'd do next.
If you have used CI before and feel like you know how CI should work for your business, you have to analyze what needs you have. This is, when the biggest downsides of Jenkins CI might kick in. Theoretically you could do nearly everything with Jenkins Community-Based Plugins. But the core functionality of Jenkins is much smaller than most of it's competitors. That means, that you have to work out how to utilize the different plugins on your own. It is very likely that you'll end up using a lot of plugins for things they were never designed for. Just because they can do the trick. But then the plugins change or your requirements do and the trick doesn't work anymore, so you have to come up with a completely new way. The plugins also clutter the already semi-neat interface of Jenkins. We ended up with Job configuration pages that were around 3-4 DIN A4 pages long, when most jobs only differentiated in about 2-3 lines, that were somewhere hidden on page 3. We spent more time on evaluating new plugins, fixing problems with the old ones and on teaching new employees how to do even the simplest things with our setup. After a good amount of evaluation work it looks like TeamCity fits our needs very nicely. It offers a nearly as good integration with other Atlassian Tools like Bamboo does, but you can do a huge amount of admin and Dev-PL side configurations, templates, parameterization and workflow definition that most users won't have to work into. This leads to extremely low duplication of definitions, parameters and scripts. But you have to design the whole design process and workflow and you'll need to put a good amount of time into learning about all the features it has to offers and tailor them to your needs. It's main advantages over Jenkins CI are a much cleaner interface and lot's of things integrated as first class concepts instead of additional, community-based barely reviewed plugins.
On the other hand, it's possible that there is this one feature absolutely crucial to your business which a Jenkins plugins offers but TeamCity CI (or every other CI server) doesn't and it's plugins doesn't. That's why you just have to evaluate your needs, and - more important - the needs of all stakeholders that are going to use your CI System. That may include the following:
- Low Level Developer, who don't have that deeper understanding of the higher workflow and process but none the less should know when and why their changes break the build and maybe should even be able to alter the build very easy without breaking the well defined workflow and build architecture.
- Higher Level Developers who might want to optimize their builds, get deeper insight into build problems and the given workflow and might be responsible for the initial build config setup of new projects as well as maintaining the old configs. Be aware that even if it's less likely that they break the build config itself, they still might break the defined workflow just because they think their idea is better instead of reworking the workflow definition itself.
- Architects who are responsible for defining the workflow definition. On the one hand they might love it to have powerful tools at their hands for defining all around your automated builds on a higher level of abstraction so that every real build just inherits their templates and parameters and uses their pre-defined build-steps. On the other hand they might not want to put in days or weeks of work defining such a complicated setup at all. They'd also need to understand the complex features of the CI tool well enough to use them accordingly.
- Project managers might want to access the tool to check build status and history, get statistical build or code quality data and even access the produced artifacts or rebuild artifacts of given commits/branches/tags. Based on their technical understanding and the given CI Tool this might be an easy task for them or not.
- Operators will have very dissimilar effort for setup and maintenance of different CI Tools. There are also a lot of security concerns about self-hosting vs. cloud services and more.
- Even if you got all stake holders from the different levels, don't forget to get information from stakeholders of all affected departments/projects/products as they might have vastly different requirements for the CI Tool.
As always, get the right tool for the right job.