Kargo - promote your application changes in a controlled (GitOps) way!

Sep 27, 2024

Docs

Github

CRDs (neat page to view CRDs in GitHub projects!)

Kargo?

Yes, Kargo.

Kargo and Argo, two pals in the cloud,
Promoting those images, they're doing us proud.
Kargo says, "Hey, I'm promoting a new build!"
Argo replies, "I’ve got the deploy, chill!"

What is that?

Simply told, Kargo will bump up your image tag reference in the git whenever new version is released.

Not simply told, Kargo is a release management tool with environment as a pipeline delivery solution (few hard to understand phrases and voule'a!)

Source: https://akuity.io/blog/why-continuous-promotion/

Components (Custom Resources)

You can read about all of them in official documentation, however, I will try to make a short TLDR:

Project - a Kubernetes namespace to group CR (Custom Resources) of all other components (IMPORTANT - when submitting Project CR, Kargo will create a namespace for us with the same name as the project name, namespace must not exist before project creation)
Stage - a step in a delivery pipeline (similar to the environment, like "staging" and "production") that groups resources (container images, helm charts)
Freight - an artifact (container image, helm chart) version (i.e.: "1.0.0", "1.0.1")
Warehouse - a source of Freights (like container registry or helm repository)
Promotion - version upgrade of resources for a stage

Requirements?

As for now, official documentation states that the following projects must be installed first to make it work:

cert-manager
ArgoCD
Argo Rollouts

However, it is possible to install it without those projects. Kargo maintainers stated that their goal is to separate Kargo from those projects - this will allow to have one Kargo instance deployed, regardless of the number of K8s clusters and ArgoCD instances.

So one Kargo instance rules all environments and promotions.

One instance to rule them all, one instance to find them, one instance to bring them all, and in the darkness... oh, wait, too much Lord of The Rings.

Neat features

Promotion mechanisms

Git manifest update
- can be any valid YAML - Kubernetes Pod Manifest, Helm chart values, etc.
- can write directly to the main branch or create a pull request
ArgoCD manifest update
- will update the ArgoCD manifest directly in the cluster (can speed up things a bit or... if you can use that if you don't like GitOps)

Promotion policies

This can be used to automatically upgrade staging environments with the newest image/chart/values but wait for manual approval before going into production.

apiVersion: kargo.akuity.io/v1alpha1
kind: Project
metadata:
  name: kargo-demo
spec:
  promotionPolicies:
  - stage: staging
    autoPromotionEnabled: true
  - stage: production
    autoPromotionEnabled: false

CLI

Kargo has a CLI that can connect to an API and allows to:

create/list/delete custom resources
promote changes
approve promotions
and more:

❯ kargo             
Usage:
  kargo [flags]
  kargo [command]

Available Commands:
  apply       Apply a resource from a file or from stdin
  approve     Manually approve a piece of freight for promotion to a stage
  completion  Generate the autocompletion script for the specified shell
  config      Manage Kargo CLI configuration
  create      Create a resource from a file or from stdin
  dashboard   Open the Kargo Dashboard in your default browser
  delete      Delete resources by file and names
  get         Display one or many resources
  grant       Grant a role to a user or grant permissions to a role
  help        Help about any command
  login       Log in to a Kargo API server
  logout      Log out of the Kargo API server
  promote     Promote a piece of freight
  refresh     Refresh a stage or warehouse
  revoke      Revoke a role from a user or revoke permissions from a role
  server      Start a local Kargo API server
  update      Update a resource
  verify      Verify a stage
  version     Show the client and server version information

Flags:
  -h, --help   help for kargo

Use "kargo [command] --help" for more information about a command.

UI (Dashboard)

Despite all of the resources are stored in Kubernetes API part and can be managed by GitOps, kubectl , Kargo CLI there is also an UI that allows that as well. It is useful for:

visualizing a project architecture (how stages, warehouses, freights relate to each other)
grant access to approve a promotion

Demo

💡

Source code for this demo is available in the repository and you can try it yourself!

Let's start with initial state:

Kubernetes cluster up and running
ArgoCD running

Kargo running

Now, let's define Kargo resources.

Project

apiVersion: kargo.akuity.io/v1alpha1
kind: Project
metadata:
  name: echo-server
spec:
  promotionPolicies:
  - stage: development
    autoPromotionEnabled: true

Project will group all application delivery stages between many environments
for development environment (in this case environment == stage) we want to have automatic promotions (update automatically whenever new version is detected)

Warehouse

apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
  name: echo-server-image
  namespace: echo-server
spec:
  subscriptions:
  - image:
      repoURL: ealen/echo-server
      # semverConstraint: ^0.6.0
      discoveryLimit: 5

Warehouse defines where we look for new release of artifacts (I use artifacts word here as Kargo can observe images, helm charts and hopefully more in the future
we can limit amount of discoverable artifacts using semverConstraint's or discoveryLimit
for all discoverd artifacts, a Freight object will be created

Stage

Development

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: development
  namespace: echo-server
spec:
  requestedFreight:
  - origin:
      kind: Warehouse
      name: echo-server-image
    sources:
      direct: true
  promotionMechanisms:
    gitRepoUpdates:
    - repoURL: https://github.com/krzwiatrzyk/blog.git
      writeBranch: master
      helm:
        images:
        - valuesFilePath: snippets/02-kargo/app/development.yaml
          key: image.tag
          value: Tag
          image: ealen/echo-server

Stage defines source of data (new release) and target to update (git repository)
TL;DR: update image.tag YAML field in snippets/02-kargo/app/development.yaml file in https://github.com/krzwiatrzyk/blog.git in git repository in master branch using data from echo-server-image Warehouse

Staging

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: staging
  namespace: echo-server
spec:
  requestedFreight:
  - origin:
      kind: Warehouse
      name: echo-server-image
    sources:
      stages:
      - development
  promotionMechanisms:
    gitRepoUpdates:
    - repoURL: https://github.com/krzwiatrzyk/blog.git
      writeBranch: master
      helm:
        images:
        - valuesFilePath: snippets/02-kargo/app/staging.yaml
          key: image.tag
          value: Tag
          image: ealen/echo-server

almost the same as in development phase but
- use staging.yaml file instaed of development.yaml
- use development stage as source of image version

Production

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: production
  namespace: echo-server
spec:
  requestedFreight:
  - origin:
      kind: Warehouse
      name: echo-server-image
    sources:
      stages:
      - staging
  promotionMechanisms:
    gitRepoUpdates:
    - repoURL: https://github.com/krzwiatrzyk/blog.git
      writeBranch: master
      helm:
        images:
        - valuesFilePath: snippets/02-kargo/app/production.yaml
          key: image.tag
          value: Tag
          image: ealen/echo-server

Same as staging

Result

After applying those manifests, we can see following project in Kargo UI:

However... there are no freights! (They will be shown shortly, we must wait a little, Kargo checks registered warehouses every few minutes).

There is a new release (Freight) visible, version 0.9.2. , codename mouthy-hedgehog ready to be deployed. There is a small warning icon on development stage, let's check that.

I haven't created a secret for Kargo to update my resources in git! Kargo also mentions that in Events panel

💡

This event is marked as "Normal", I guess "Warning" or "Error" would be better here to indicate that something is not right.

The secret for Kargo should look like this:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: repo
  namespace: echo-server
  labels:
    kargo.akuity.io/cred-type: git
stringData:
  repoURL: https://github.com/krzwiatrzyk/blog.git
  username: krzwiatrzyk
  password: ${GITHUB_PAT}

Label kargo.akuity.io/cred-type: git is required for Kargo to discover that secret.

A short kubectl apply and secret is ready.

Now we can see discovered Freights properly:

and automatic promotion kicked in:

and that creates a commit similar to that one:

Now, let's promote staging by clicking this button:

And after confirming:

For production, we will use kargo cli for promotion.

First, let's see stages:

❯ kargo get stages -p echo-server
NAME          SHARD   CURRENT FREIGHT                            HEALTH   PHASE           AGE
development           b13f48c114ad8819e992495b8fbfff83a9663f61            Steady          24h
production            0/1 Fulfilled                                       NotApplicable   24h
staging               b13f48c114ad8819e992495b8fbfff83a9663f61            Steady          24h

and freights:

❯ kargo get freight -p echo-server
NAME                                       ALIAS          ORIGIN                        AGE
b13f48c114ad8819e992495b8fbfff83a9663f61   lucky-jaguar   Warehouse/echo-server-image   27m

And now promote:

❯ kargo promote --project=echo-server --freight-alias=lucky-jaguar --stage=production 
promotion.kargo.akuity.io/production.01j8t8rkkzymzncvf7qes5pjj7.b13f48c promotion created

❯ kargo get stages -p echo-server                                                         
NAME          SHARD   CURRENT FREIGHT                            HEALTH   PHASE    AGE
development           b13f48c114ad8819e992495b8fbfff83a9663f61            Steady   25h
production            b13f48c114ad8819e992495b8fbfff83a9663f61            Steady   25h
staging               b13f48c114ad8819e992495b8fbfff83a9663f61            Steady   25h

❯ kargo get promotions -p echo-server
NAME                                             SHARD   STAGE         FREIGHT                                    PHASE       AGE
staging.01j8t8bk3jypbfgceesp7tgnyy.b13f48c               staging       b13f48c114ad8819e992495b8fbfff83a9663f61   Succeeded   10m
production.01j8t8xmf66bvw0528dq97ge9g.b13f48c            production    b13f48c114ad8819e992495b8fbfff83a9663f61   Succeeded   30s
production.01j8t8rkkzymzncvf7qes5pjj7.b13f48c            development    b13f48c114ad8819e992495b8fbfff83a9663f61   Succeeded   32m

And now when new freight is added, we can see which stage uses what:

ℹ️

I have assembled freight manually, adding a previous version. Auto promotions will kick-in in that scenario as well.

A soft red color indicates that development stage was on version 0.9.2 and was "updgraded" to 0.9.1 (bold red color).

Materials

Talk with Let's Go DevOps

Krzysztof Wiatrzyk

Big love for Kubernetes and the entire Cloud Native Computing Foundation. DevOps, biker, hiker, dog lover, guitar player, and lazy gamer.