Svbtle

Menu

November 3, 2014

Walter, a simple build pipeline tool

Today I announce Walter, a simple build pipeline tool with a collaborator @takahi_i.

Beriefly, Walter is a tiny tool for executing build pipelines (such as automated build workflows) in Continuous Integration / Delivery(CI / CD).

 Background of development

 Build pipeline settings heavily depends on Jenkins GUI

When managing a CI environment with Jenkins, we commonly use some sort of build pipeline plugins (ex. buildpipeline-plugin) to define deployment flows and execute the tasks in the flow one by one.

Although these Jenkins plugins are useful to manage and control CI, we still have some problems in the deployment of our services.

• None of us is able to run build pipeline without our Jenkins server
• No one is able to check if the pipeline works until starting the CI jobs in Jenkins server
• We feel the difficulties to migrate the build pipeline settings to other CI environments

Although we have built our system that maintains the servers settings with configuration files of Chef or Ansible in the Git repositories, only the deployment pipeline settings are still in Jenkins server… Modern CI services, [travis-ci][travis], wercker provide the solution to this problem. Unfortunately, we can’t execute these services locally.

To manage the deployment settings with the local repositories, we have developed a tiny deployment workflow engine in Golang.

 About Walter

The followings are the benefits of adopting Walter.

• Store build pipeline as a settings file: Walter makes the deployment easy, more portable in managing CI settings.
• Execute build pipeline locally: We can run CI workflows without a CI server.
• Simple to use: Does anyone like to understand horribly complex GUIs or DSLs? With Walter, we just need shell and yaml.

We can adopt Walter without the special training. The pipeline configuration is yaml formatted (It is similar to the configuration file of TravisCI). We just add multiple stages containing shell commands in the pipeline setting.

 Use case

In the settings yaml for Walter, we describe the workflow control and tasks in pipeline: section.
The definition of each task is written in stage: section.

stage: section has the properties below:

• stage_name: Name of stage.
• stage_type: Type of stage. command is applied by default when stage_type is not specified.
• command: Description of the system command to execute.
• run_after: Definition of the stages that you want to execute after finished executing parent stage. run_after has more than one stage sections. the stages in run_after section is executed in parallel.

The following example shows a sample setting of Walter pipeline:

# Hipchat notification settings
messenger: 
   type: hipchat
   room_id: ROOM_ID
   token: TOKEN
   from: USER_NAME

# Workflow configuration 
pipeline:
    # step 1
    - stage_name: Notify the beggining of process pipeline
      command: echo "build pipeline is started"
      message: true
      run_after:
          # step 2
          -  stage_name: Provisioning server with Ansible.
             command: Ansible-playbook -i hosts site.yml
          -  stage_name: Test application with rspec 
             command: bundle exec rake test
    # step 3
    - stage_name: Deploy application with Capistrano
      command: bundle exec cap production deploy
    # step 4
    - stage_name: Test server settings by serverspec
      command: bundle exec rake spec
    # step 5
    - stage_name: Notify the finish of process pipeline
      message: true
      command: echo "build pipeline is finished"

Walter executes each stage described at the top level of pipiline sequencially.
However there is one exception run_after. Only stages in run_after section are executed in parallel.

The follwoing diagram shows how the some stages in the pipeline execute in palallel.

We also provide one relatively large build pipeline example for Spring demo application deployment.

You can download and run the Walter to understand the usage. Please check it out!

 Try Walter

We provide Walter compiled binaries at github-releases. You can try Walter quickly.

As a demonstration, we show how to execute Walter in Mac.

1. Get the binary for Mac.

   wget https://github.com/Walter-cd/Walter/releases/download/v0.1.0/walter_darwin_amd64.zip
   unzip walter_darwin_amd64.zip

2. Get the sample of build pipeline settings

   cd walter_darwin_amd64
   wget https://gist.githubusercontent.com/ainoya/78971d4b1e2797a59af6/raw/992467e22597e7a6de5834e6a21bf6c624286970/pipeline_sample.yml

3. Run Walter. you can specify the build pipeline settings file by -c option. (by default, use ./pipeline.yml)

   ./walter -c pipeine_sample.yml

When you run Walter and execute the build pipelines and find the bugs, please submit to GitHub issues :)

 Microservices with Walter

If a system architecture consists of multiple services / servers, it has often deploy dependencies each other.

In such cases, Walter helps us to deploy the services with multiple components. We can deploy the services following deploy dependencies automatically with Walter as the below diagram.

Actually, Ansible already has wait_for of parallel execution option for clustering multiple services. However Ansible playbooks become huge and intractable in our existing services. Therefore we think it is a good idea to split huge playbooks with Walter to make build pipelines more maintainable.

 Future plans

Though The Code quality of Walter is still prototype level, we’re going to improve the quality periodically. Especially we will enhance error handring and retrying feature.

Furthermore, we will provides below features:

• Communicate data between jobs: We will try message-passing feature between jobs with goroutine
• More simple definition of settings: Settings by YAML is still complex; we improve the the syntax to make it easy.
• Ready to ChatOps: We will enhance notification to various sevices (such as slack) and more details of notification settings.
• Integration with Jenkins: Feedback mechanism to tell the progress of executing Walter build pipeline to Jenkins (or other CI service) is provided.

We hope you to try Walter, and contribute the development of Walter. Let’s improve build pipeline system with us!

Thanks.