Your First Workflow in Kraken CI

Kraken CI is a new Continuous Integration tool. It is a modern, open-source, on-premise CI/CD system that is highly scalable and focused on testing. It is licensed under Apache 2.0 license. Its source code is available on Kraken CI GitHub page.

This tutorial is the second installment of the series of articles about Kraken CI. Part 1, Kraken CI, New Kid on the CI block, presented the installation of Kraken. This time we will cover how to prepare a workflow for a simple Python project. The workflow will:

1. fetch sources from GitHub,
2. run tests using Pytest,
3. build a wheel package,
4. upload built package to Kraken's artifacts storage.

Here we are using fresh Kraken installation using Docker Compose with built-in Kraken Agent running in Docker container. Installation procedure details can be found in part 1 or in the installation manual from https://kraken.ci project web page.

Project Preparation

Details about a new project setup in Kraken CI can be found in the previous article, part 1 as well.

Here, in short, let's do:

1. Create a project with a name: Demo
2. Create a branch with a name: main
3. Use a pre-created stage and rename it to: Hello World

Stage Hello World

The stage should look as follows:

And the original workflow schema code looks as follows:

def stage(ctx):
    return {
        "parent": "root",
        "triggers": {
            "parent": True,
        },
        "parameters": [],
        "configs": [],
        "jobs": [{
            "name": "hello",
            "steps": [{
                "tool": "shell",
                "cmd": "echo 'hello world'"
            }],
            "environments": [{
                "system": "any",
                "agents_group": "all",
                "config": "default"
            }]
        }]
    }

Now let's try if it works. Click Run CI Flow button.

Git Repo Checkout Step

Ok, let's start extending our schema to make it more real life. First, let's add a step for checking out a Git repository:

"steps": [{
    "tool": "git",
    "checkout": "https://github.com/Kraken-CI/sample-project-python.git"
}, {
    "tool": "shell",
    "cmd": "ls -al"
}],

Now run new CI flow. The output should look like this:

The logs show that the repository has been cloned and ls -al shows sample-project-python folder.

Run Tests Step

Now let's replace the step with listing the current directory with something more interesting, like running tests. Kraken provides several built-in tools for running tests. In the case of our Python sample project, we will use pytest built-in tool.

"steps": [{
    "tool": "git",
    "checkout": "https://github.com/Kraken-CI/sample-project-python.git"
 }, {
    "tool": "pytest",
    "cwd": "sample-project-python",
    "params": "-vv",
    "pythonpath": "src"
 }],

Change the steps in schema editor, save it and run new CI flow. And the job output should be:

Building Step

So if tests succeeded we could build a package of our Python project. Add step for building a wheel package using project's setup.py script:

"steps": [{
    "tool": "git",
    "checkout": "https://github.com/Kraken-CI/sample-project-python.git"
 }, {
    "tool": "pytest",
    "cwd": "sample-project-python",
    "params": "-vv",
    "pythonpath": "src"
 }, {
    "tool": "shell",
    "cwd": "sample-project-python",
    "cmd": "python3 setup.py sdist bdist_wheel"
 }],

Now run CI flow again. The output should look as follows:

It seems that there is missing a package that provides wheel support for Python. Let's install it.

"steps": [{
    "tool": "git",
    "checkout": "https://github.com/Kraken-CI/sample-project-python.git"
 }, {
    "tool": "pytest",
    "cwd": "sample-project-python",
    "params": "-vv",
    "pythonpath": "src"
 }, {
    "tool": "shell",
    "cmd": "sudo apt update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends python3-wheel python3-setuptools"
 }, {
    "tool": "shell",
    "cwd": "sample-project-python",
    "cmd": "python3 setup.py sdist bdist_wheel"
 }],

New output with building:

Storing Artifacts Step

The results of building the sample project are two packages:

• dist/sampleproject-2.0.0-py3-none-any.whl
• dist/sampleproject-2.0.0.tar.gz

We could store it Kraken storage and make it available for user for downloading. For that purpose there is another built-in tool: artifacts.

"steps": [{
    "tool": "git",
    "checkout": "https://github.com/Kraken-CI/sample-project-python.git"
}, {
    "tool": "pytest",
    "cwd": "sample-project-python",
    "params": "-vv",
    "pythonpath": "src"
 }, {
    "tool": "shell",
    "cmd": "sudo apt update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y - no-install-recommends python3-wheel python3-setuptools"
 }, {
    "tool": "shell",
    "cwd": "sample-project-python",
    "cmd": "python3 setup.py sdist bdist_wheel"
 }, {
    "tool": "artifacts",
    "action": "upload",
    "cwd": "sample-project-python/dist",
    "source": [
        "sampleproject-2.0.0-py3-none-any.whl",
        "sampleproject-2.0.0.tar.gz"
    ],
    "public": True
 }],

And the job output:

Here the log shows that artifacts have been stored. Now, as they were marked as public in artifacts step, they are available in the UI:

This is it. In this installment we learn how to clone Git repository, how to run tests using Pytest, how to install dependencies and how to build artifacts and store them in Kraken's storage. More about Kraken CI you can find on https://kraken.ci.

Comments

[jacksonhead]

It is intereting to see whenever a new ci platform comes out it has a dsl for describing the pipeline. Using a complete script language (like scripted pipelines in Jenkins) can be a security risk, I understand that but nieche requirements are bound to come in. Wouldn't it worth to reserach the possibility to use an interpreter instead of a dsl and make it secure?

[Michal Nowikowski]

Here in Kraken CI, Python is used to define pipeline. Long term plan is to have Starlark which is a safe subset of Python and it is used in e.g. Bazel, Buck or Caddy.

[jacksonhead]

Oh sorry i have misinterpret the pipeline descriptor. In case i want to extend the current logic with something of my own i can explicitly do that in the 'stage' python function. But where does that function go? Where is the python file containing the function?

[Michal Nowikowski]

You can put the Python code directly in the UI where a stage is defined or store the Python code in a file and put it to Git repo and indicate this in the UI (check the first picture in the article, Stage Hello World section, there is a 'Schema from Repository' tab that allows this).

[Alexey Melezhik]

Hi. Nice posts on Kraken, thanks. However what is a pros over already existing (billions) ci/cd solutions ?

[Michal Nowikowski]

Most of them are offered as a service. Kraken CI is an on-premise solution. Beside of that it is focused on testing. When you have thousands of test cases and there is never 100% pass ratio then it is hard to spot regressions or unstable tests (eg. Jenkins does not have good support for such things). All of that and more is present in Kraken CI out of the box. There is much more, hard to list all. Please, check Kraken CI web page and its docs or contact me directly ([e-mail]) or chat here :)

[Alexey Melezhik]

Thanks for reply. Thousands of test cases? It should be very specific development imho. And even in this case I am not sure what certain problem you are trying to solve here? I guess most of test frameworks provides a user with some sort of reports? Or I miss the point here ...

[Michal Nowikowski]

Medium and big projects can have 10k, 100k or even more test cases. But even for 1k if there is hard to achieve 100% then it is hard to track changes of results over time. E.g. can you say which tests are the least stable in the last month so that you could fix or disable them? Or could you say which tests started failing during the last week? Or if you have some performance tests, lots of them, then which of them degraded? I think that it is very hard to get answers for such questions in Jenkins. I was writing some scripts to do such analysis and generate some reports using Jenkins API but this is a dead end. In the past I wrote similar testing system to Kraken but it was proprietary and it was used only internally. Kraken is an open source version of the concepts that emerged in the original system.

[Alexey Melezhik]

Ok. I get you, thanks for clarification. You might be then interested in tool, I've created - Tomty and see how it deals with tests subgroups - dev.to/melezhik/write-devopsish-te... , see tags section.