Our Guide to CI Processing for Automated Versioning

/ January 6, 2022

In our first article we provided an outline of continuous integration/continuous delivery (CI/CD) and we’ll now go into greater detail about how you can build CI/CD processes for customers that use microservice architecture. Based on this tutorial you will be able to achieve the same flow at your company. This approach was designed for the Kaa IoT Platform and supports multiple releases.

Versioning

Versioning is one of the most important parts of the CI/CD process. Usually, the versioning process is manual which represents a bottleneck that can prevent the release process occurring quickly. This can occur because there’s no possibility to build a continuous testing process. Instead, we automate the versioning process for each component and only release required manual intervention.

You need versions of components for mark build artifacts and versioning for the platform to provide an aggregate of all platform component versions. For platform components, we increment the version for any changes which go to the main branch, and each change of service increments the version of the platform.

For storing versions we use git tags, because all popular git servers provide a convenient UI and API to work with.

Versioning format

We suggest you use Semantic Versioning 2. This format is supported by most popular packages and can be used as git tags. Jar, helm, docker image, semver 2 can be used everywhere.

We will use format only, meaning the version number you can change based on your company workflow. For example in the Kaa Iot platform company we release our platform each sprint, so we use the following process:

MAJOR = version of the application,
MINOR = sprint number,
PATCH = commit number in sprint.

As a next step, we need to have some mechanism for aggregate versions of all services and set the version for this aggregation as the platform version.

Version agregation (Repository of versions)

The simplest way to track and store changes to versions of services is to have a separate git repository with a file where you record service versions. You can use the following structure:

                    version.yaml: |
                    ${git-repo-path}/${git-repo-name}: $version
                    ${git-repo-path}/${git-repo-name}: $version
                    ...

Where:
git-repo-path - The path of the repository in the git repo if such a feature is supported (gitlab/github)
git-repo-name - The name of the git repository

Here’s a real KaaIot platform example:

                    version.yaml: |
                    core:
                    client:
                        python-simulator: 0.3.0
                    dev-tools:
                        keycloak-configurator: 0.3.3
                        test-framework: 0.2.8
                    docs:
                        kaa-docs: 1.4.21
                    service:
                        adx: 0.0.17
                        asf: 0.0.20
                        bcx: 0.2.12
                        branding-service: 0.1.1
                        ccm: 0.2.15
                        ccm-cm-compatible: 0.0.6
                        cex: 1.4.35
                        cm: 1.4.34
                        cmx: 1.4.16
                        dcx: 1.4.18
                        ecr: 1.4.20
                        epl: 1.4.24
                        epmx: 1.4.14
                        epr: 1.4.71
                        epts: 1.4.12
                        iam: 0.0.144
                        kdca: 1.4.14
                        kpc: 1.4.24
                        otao: 1.4.22
                        ote: 0.2.7
                        rci: 0.0.0-AGILE-3284
                        rsx: 0.1.4
                        tekton: 1.3.24
                        tenant-manager: 0.2.43
                        tsa: 0.0.16
                        tsx: 1.4.4
                        wd: 1.4.223
                    devops:
                    automated-follow-up: 0.0.9
                    charts:
                        aws-autoscaler: 9.9.2
                        cert-manager: 1.3.1
                        certmanager-issuer: 0.2.2
                        elastic-exporter: 4.4.0
                        external-dns: 3.4.6
                        filebeat: 3.1.0
                        global-chart: 0.1.35
                        kafka: 0.21.2
                        keycloak: 0.2.18
                        kube-prometheus-stack: 15.4.6
                        logstash: 7.10.2
                        minio: 0.1.1
                        nats: 4.5.5
                        nginx-ingress: 3.31.0
                        opendistro: 0.2.8
                        prometheus-operator: 9.2.1
                        prometheus-pushgateway: 1.10.1
                        vault: 0.1.1
                    cloud-portal: 0.1.6
                    gitlab-ci-templates: 0.0.6
                    kaa-installer: 1.4.9
                    pbm: 0.1.13
                    ...

For the next step, we need to define git flow and branch naming conventions.

Git flow

We usually use rebase flow because this approach allows you to utilize a cleaner git history, which is very useful when multiple team members work on the same service. You can read which approach is more suitable for your company here.

Name of the Branch

To support multiple releases of software you should have a platform as a service selling model and a mechanism to deliver bug fixes to your old release. We decided to use a branch for each release.

master branch like a stable upstream branch.
rel_${release-number} like a stable release branch.
${issue-number}-${feature-name} like the unstable feature branch where the issue number of the ticket ID tracking system and the short feature-name are present.

Merge Request rules

There are various rules based on your project requirements but our basic rule set includes:

Branch must match pattern: ${issue-number}-${feature-name} as this is required to create integration with a ticket tracking system
Each feature should be developed in one branch feature-$jira-ticket-name and can be merged to master only by merge request, and this must be one commit.
Each merge request must have at least one approval from members on the approvers list.
Each merge request should be built without errors
Each merge request should ensure quality check tests are passed (including code quality).

We also need to have protected branches. We protect:

master
rel_*

Also, the master should be the latest stable branch, and rel_* should be released.

Tags also should be protected because they are managed by CI:

*.*.*

After we apply all previously described concepts we will reach the following schema:

CI flow for versioning

When the developer starts working on a feature they should create a branch with the name ${issue-number}-${feature-name}.

Afterwards, they should create a merge request for the master branch. After this feature is automatically merged into master, a git tag $(version of the application).$(sprint number).$(commit number in sprint) will be created.

Once the sprint ends, we create a release from the latest version repo tag, and rel_$(version of the application).$(sprint number).$(commit number in sprint) - branch will be created in the version repository. Each service should be recorded in the version repo.

For bug fixing, apply the same process, and the target branch which the developer should select should be rel_$(version of the application).$(sprint number).$(commit number in sprint). Such commits will be automatically cherry-picked to version rel_$(version of application).$(sprint number + 1).$(commit number in sprint) and merge requests will be created. If the cherry-pick fails, the committee will get an email notification and will be able to remove conflicts, which will then trigger a new commit.

For simple implementation on gitlab ci we prepared a versioning container which you can use in your project:

Link to repo: https://gitlab.kaaiot.net/devops/versioning-manager

Or use your docker image: https://dev-hub.kaaiot.net/devops/versioning-manager/versioning-manager (expose before publishing)

To integrate this container you can use these gitlab.ci templates:

                    For the build version:
                    get_build_version:
                    image: https://dev-hub.kaaiot.net/devops/versioning-manager/versioning-manager
                    script:
                        - export BUILD_VERSION=$(main versioning --token ${CI_JOB_TOKEN} generate-commit-tag ${CI_COMMIT_SHA} ${CI_PROJECT_ID} ${CI_BUILD_REF_NAME})
                        - echo build version is [$BUILD_VERSION]
                        - "[ -z $BUILD_VERSION ] && exit 1"
                        - echo $BUILD_VERSION > current_version
                    artifacts:
                        paths:
                        - current_version

This step will generate a version to build and record to current_version file

After you read this file you can mark all of build artifacts with this version

                    Create version tag in git server:
                    create_version_tag:
                    image: https://dev-hub.kaaiot.net/devops/versioning-manager/versioning-manager
                    only:
                        - master
                        - /^rel_.*$/  
                    script:
                        - main versioning --token ${CI_JOB_TOKEN} create-commit-tag --project-id ${CI_PROJECT_ID} --branch-name ${CI_BUILD_REF_NAME} --commit-sha ${CI_COMMIT_SHA}

And record it to version repository:

                    update_version_in_version_repo:
                    except:
                        - tags
                    only:
                        - master
                        - /^rel_.*$/
                    image: https://dev-hub.kaaiot.net/devops/versioning-manager/versioning-manager
                    script:
                        - export VERSION=$(cat current_version)
                        - main versioning --token ${CI_JOB_TOKEN} update-version-in-version-repo --version-repo-project-id ${CI_VERSION_REPO_ID} --project-id ${CI_PROJECT_ID} --project-version ${VERSION} --branch-name ${CI_BUILD_REF_NAME}

This guide will have prepared you to build CI/CD processes for microservice architecture, but be sure to check out our other article on the subject. If you do become stuck, or need some assistance, you can also send us an email and one of our team members will get back to you. We’re passionate about providing help to developers, so let us know how we can help you.

Read more about DevOps and combining CI/CD with multi-repository as an effective way to improve your company’s performance.