CI and Autostaging - Things and Stuff

In my last post about leveraging GitLab CI and Autopkg to build packages I mentioned that all packages falling out of the CI pipeline are first put into the testing catalog. I have used munki-staging.py and a Trello board to shift packages around from the testing environment into production in a mostly manual process. Since then I have been thinking about how to integrate the staging into that CI process.

It took me a bit but in the end I decided to use GitLab issues to handle the staging part as following:

Whenever a new package version is merged into the master branch of the munki repository in GitLab, a new issue is created with a distinct label and due date.
Periodically, some script runs that promotes packages in overdue issues.

Note

In the following article I will be using the word repository a lot. Actually we will be dealing with multiple repositories as following:

Autopkg repository - Contains recipe overrides and build scripts, runs the package building CI pipeline
Munki repository - The actual munki repository, runs CI Pipelines that merge new package branches into master and CI to deploy master branch on distribution infrastructure
Build repository - The checked out munki repository on the package building Mac Mini, built packages land here first and are pushed onto the GitLab server

So while reading, make sure to set it into the right context.

Issue creation

Let’s revisit what happens after a new package is built:

graph TD; A[Feature branch with new ispackage pushed onto munki repository] -->|CI pipeline triggered| B[lint pkginfo] B --> C[test installation] C --> D[Create MR + trigger merge on pipeline complete]

When the merge request is created, we can be sure that the new package will actually be merged into the master branch of the munki repository so at that point we can safely create the issue through a little python script.

create_staging_issue.py

This is the script that creates the issue in GitLab:

$ cat create_staging_issue.py
#!/usr/bin/python

import gitlab
import os
import sys
import datetime

token = sys.argv[1]

gl = gitlab.Gitlab('https://gitlab', private_token=token)

project_id = os.environ.get('CI_PROJECT_ID')
source_br = os.environ.get('CI_COMMIT_REF_NAME')

autostage_due_date = (datetime.datetime.now() + datetime.timedelta(days=3)).strftime('%Y-%m-%d')

project = gl.projects.get(project_id)

issue = project.issues.create({'title': source_br,
                               'description': 'autostage for production',
                               'due_date': autostage_due_date,
                               'labels': ['autostage']})

Script needs a GitLab API token as parameter

The script creates a new issue with the following characteristics:

Issue title is <PackageName>-<PackageVersion>
Due date three days in the future
Label autostage

Later on, we will be using the title to find the pkginfo file. The due date is used to determine when to promote the package into production.

The script in its current form is a very simple approach. After a three days testing window, the packages are promoted into production. Future ideas include:

Different staging windows through custom attributes in the pkginfo file
Manual staging through different labels
More staging tiers such as dev -> testing -> production
Setting of a force install date based on labels

.gitlab-ci in the munki repository

The python script is added to the deploy step in .gitlab-ci:

merge:
  stage: deploy
  tags:
    - osx
  script:
    - /usr/bin/python build/create_staging_issue.py $PRIVATE_TOKEN
    - /usr/bin/python build/create_and_merge.py $PRIVATE_TOKEN
  except:
    - master

Promotion into production

At this point we have created issues for our newly built packages. As the CI pipeline in the autopkg repository is running on a nightly schedule, we are going to add the actual promotion step there.

autostage_packages.py

This python script is added to the autopkg repository:

#!/usr/bin/python

import os
import sys
import datetime
import subprocess
import FoundationPlist
import gitlab

token = sys.argv[1]

gl = gitlab.Gitlab('https://gitlab', private_token=token)

project_id = os.environ.get('MUNKI_PROJECT_ID')

build_repo_dir = os.environ.get('BUILD_REPO')

GIT = '/usr/bin/git'

autostage_due_date = datetime.date.today() + datetime.timedelta(days=3)

project = gl.projects.get(project_id)

issues = project.issues.list(state='opened')

class Error(Exception):
    """generic error"""

class GitError(Error):
    """git error"""

class FileError(Error):
    """file error"""

class BranchError(Error):
    """Branch-related exceptions."""

def run_cmd(cmd):
    proc = subprocess.Popen(
        cmd,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE
    )
    (out, err) = proc.communicate()
    results_dict = {
        'stdout': out,
        'stderr': err,
        'status': proc.returncode,
        'success': proc.returncode == 0
    }
    return results_dict

def git_run(arglist):
    gitcmd = [GIT]
    for arg in arglist:
        gitcmd.append(str(arg))
    old_pwd = os.getcwd()
    os.chdir(build_repo_dir)
    results = run_cmd(gitcmd)
    os.chdir(old_pwd)
    if not results['success']:
        raise GitError("Git error: %s" % results['stderr'])
    return results['stdout']

def change_feature_branch(branch, new=False):
    gitcmd = ['checkout']
    if new:
        gitcmd.append('-b')
    gitcmd.append(branch)
    try:
        git_run(gitcmd)
    except GitError as e:
        raise BranchError("Couldn't switch to '%s': %s" % (branch, e))

def create_commit(branchname, message):
    change_feature_branch(branchname)
    print "Adding items..."
    gitaddcmd = ['add']
    gitaddcmd.append(build_repo_dir)
    git_run(gitaddcmd)
    print "Creating commit..."
    gitcommitcmd = ['commit', '-m']
    gitcommitcmd.append(message)
    #git_output = git_run(gitcommitcmd)
    git_run(gitcommitcmd)

def push_branch(branchname):
    change_feature_branch(branchname)

    gitcmd = ['push', '--set-upstream', 'origin', branchname]
    try:
        git_run(gitcmd)
        print "Pushed %s to origin" % branchname
    except GitError as e:
        raise BranchError("Couldn't push %s to origin: %s" % (branchname, e))

def pull_branch(branchname):
    change_feature_branch(branchname)

    gitcmd = ['pull', 'origin', branchname]
    try:
        git_run(gitcmd)
        print "Pushed %s to origin" % branchname
    except GitError as e:
        raise BranchError("Couldn't pull %s from origin: %s" % (branchname, e))

def getPlist(pkgname, repo_dir):
    pkgsinfo_dir = repo_dir + '/pkgsinfo'
    plist_filename = pkgname + '.plist'
    for root, dirs, names in os.walk(pkgsinfo_dir):
        if plist_filename in names:
            return os.path.join(root, plist_filename)
    raise FileError("Couldn't find plist for: %s" % plist_filename)

for issue in issues:
    if datetime.datetime.now() >= datetime.datetime.strptime(issue.due_date, "%Y-%m-%d"):
        pull_branch('master')
        plist_file = getPlist(issue.title, build_repo_dir)
        plist = FoundationPlist.readPlist(plist_file)
        plist['catalogs'].append('production')
        FoundationPlist.writePlist(plist, plist_file)
        issue.state_event = 'close'
        issue.save()
        commit_msg = "Autostaged " + issue.title + " into production"
        create_commit('master', commit_msg)

        print commit_msg
    else:
        print "Not staging %s until %s" % (issue.title, issue.due_date)

push_branch('master')

Note

This script is run during the autopkg GitLab CI pipeline and operates on the build repository on the package building box. Two variables need to be set in GitLab:

MUNKI_PROJECT_ID - Set to the GitLab Project ID of the munki repo
BUILD_REPO - Set to the path of the build repository on the package building box

Also, a GitLab API token is needed as a parameter.

Due to all the git handling code (mostly taken from Facebook’s autopkg tools script, thank you) this script is a bit lengthy. The main stuff happens in the for loop at the end of the script:

Check if an issue’s due date is reached.
Add production to the catalogs in the package’s pkginfo plist.
Close the issue.
Create the commit and push it onto GitLab (munki repo).

As we are pushing the changes onto the master branch in GitLab the following CI Pipeline is triggered:

graph TD; A[master branch changed] -->|CI pipeline triggered| B[lint pkginfo] B --> C[make catalogs] C --> D[deploy to distribution infrastructure]

CI configuration in the autopkg repository

The following is added to .gitlab-ci in the autopkg repository:

autostage:
  stage: deploy
  tags:
    - osx
  script:
    - (if [[ -z "$RECIPE_NAME" ]]; then build/autostage_packages.py $PRIVATE_TOKEN; fi)
  only:
    - triggers
    - schedules

This makes sure autostaging is run whenever a package build is scheduled.

Conclusion

Wrapping it all up we have automated the whole package building process. Packages get automatically built every night, after three days they are automatically staged into the production catalog.

The whole workflow for building a single package now looks like this:

graph TD; X((Schedule time reached)) -->|CI pipeline in autopkg repo triggered| A[Create branch PackageName in build repo] A --> B[run autopkg] B -->|new version built| C[Rename branch to PackageName-Version] C --> D[Commit changes] D --> E[push branch onto GitLab] E -->|CI Pipeline triggered in munki repo| F[lint pkginfo] F --> G[test installation] G --> H[Create merge request and set to merge on pipeline completion] H --> I[create autostaging issue] I --> J[Branch is merged onto master] J -->|CI pipeline triggered on master| K[lint pkginfo] K --> L[make catalogs] L --> M[deploy to distribution]

The next post might be about cleaning old packages from the munki repository.