Caching CI stuff from Gitlab with MinIO

sysadmin

Some notes on how I've gotten a cache to work with Gitlab Runner.

First, install MinIO somewhere. Not going to repeat what's already on the internet a bunch. Personally, I installed it on my NAS, a Synology, using the SynoCommunity package.

One thing to watch out for, the character set for the keys are somewhat limited. I ended up sticking to alphanumerics and making the values very long.

MinIO client

In order to administrate a MinIO server, you use it's API. The built-in web browsing interface doesn't have an admin panel.

Arch Linux packages the client (community/minio-client), so that is a single command away: pacman -S minio-client. To avoid conflicting with Midnight Commander, per the upstream recommendation, the binary is called mcli and not mc. I'll use that name as well throughout.

Now that it's installed, you must create an alias:

mcli alias set syno http://10.0.0.2:9000 ROOT_ACCESS_KEY ROOT_SECRET_KEY

Preparing MinIO

After that's done, in a fresh MinIO installation you will have only 1 user, the root account. I didn't want to give gitlab-runner that much power, so I created a second account for it:

mcli admin user add syno/ gitlab-runner-1234 SECRET_KEY

I'm keeping the access key somewhat descriptive so that it's obvious what it's purpose is when I come back to this server.

That user doesn't have permission to do anything, so let's make it a bucket for them:

mcli mb syno/gitlab-runner

Then create an security/access policy file that lists the permissions we want to grant. The list of supported actions can be found in minio's source code. Unfortunately it's not better documented elsewhere.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::gitlab-runner/*", "arn:aws:s3:::gitlab-runner"
      ]
    }
  ]
}

And attach it to the server:

mcli admin policy add syno gitlab-runner-perms gitlab_runner_policy.json

If you need to make changes to the policy, rerunning the same command is the same as an update.

Now, glue the permissions to the user:

mcli admin policy set syno gitlab-runner-perms user=gitlab-runner-1234

Configuring gitlab-runner

Cool, now we have a user that can do some stuff in MinIO, time to teach gitlab-runner about that. All of this can be done in the configuration file. Or as environment variables, but I'm using files in my deployment.

  [runners.cache]
  Type = "s3"
  Path = "cache"
  Shared = true
    [runners.cache.s3]
    ServerAddress = "10.0.0.2:9000"
    AccessKey = "gitlab-runner-1234"
    SecretKey = "SUPER_SECRET_KEY"
    BucketName = "gitlab-runner"
    Insecure = true # since minio is over HTTP

Future work

The gitlab runner server is on the same LAN as my NAS, so it's not the worst that it's over HTTP, but I do want to improve that. I've been working on setting up an internal CA for HTTPS deployments, so hopefully I can flip that Insecure to false!

All of these details are done using CLI tools, which can be scripted, but they aren't super idempotent as they stand. Thankfully, there are projects that can help with that already, like this terraform provider for minio.

Thanks

This was put together from reading documentation, but also from these lovely blog posts:

|