Setting up my own GitLab CI/CD runner

Oh boy, this one's been a long time coming. I was first shown the idea of CI/CD back in 2018, and it took me until fairly recently to actually set up my own runner. So if you've been putting this off, you're in good company. CI/CD stands for continuous integration and continuous deployment, which is a fancy way of saying this: every time I push my code up, a process kicks in on its own and gets the site live for me. No logging into the server, no typing the same handful of commands in the same order every single time. I set it up once, and after that something else does the boring bit. I'd watched agencies do this for years. Commit your changes, wander off, and a few minutes later the site's updated (depending on how big it is). It looked like magic. Turns out it's fiddly rather than clever, I just hadn't seen the setup behind it. I have now, and I can walk you through it. A quick word on my setup: I self-host GitLab on a private server, and I use GitLab's own CI/CD rather than bolting on something like Jenkins or paying for a service like DeployHQ. Why muddy the waters? It's already there, so I'd rather learn the tool I've got. ## Where to begin? First thing, make sure CI/CD is actually switched on for your project. It's usually on by default, but I'd turned it off site-wide at one point, because the "hey, you've got this thing and it doesn't work, do something about it" warning on every project was doing my head in. GitLab lets you enable or disable it per project in the settings, which suits me because I keep non-DevOps bits in there too. Once it's on, you'll get a CI/CD menu down the side. GitLab is pretty forthcoming with its prompts, so it more or less tells you what it wants next. What it wants is a pipeline. ## What's a pipeline, then? A pipeline is just a file that lives inside your repo. Inside it is a list of instructions for the runner to carry out once certain conditions are met, say, when a particular branch gets updated. You can set different instructions for different branches (your main branch, a staging branch, a developer's branch, whatever), and the commands can be anything from SSH commands to running scripts to pulling in stored variables. I'll be honest, I've only ever needed the simple stuff. All my runner does is notice its branch has changed and copy the files up to wherever they need to go. There's a bit more to it in practice (backups, file permissions, that sort of thing), but the whole point is to take the job off my hands. ## What's this "runner" I keep mentioning? Right, yes. A runner is a service that acts as a user on your behalf. Think of it as a robot butler (hey, "Jenkins") that sits about quietly, keeps an eye on your repos, and the moment something changes it works through your list of instructions and gets on with it. Why pay someone to sit and wait when you can build a machine to do it? It's recommended you run it on a separate machine, since it doesn't need much grunt and you don't want it hogging the box you actually work on. But you can run it on the same server as the repo, or even on your own computer, as long as you install the runner and turn the service on. I followed GitLab's own install instructions and they worked fine, though I had a rough time getting it going on Windows and PowerShell. In the end I used Ubuntu (I'm well versed in Linux now... I think I actually prefer it). ## Here's a bit of free advice: sort your SSH keys first This is the bit that had me stuck for ages, so I'm putting it before the config rather than after. For the runner to reach your remote server, it needs the right access, and that means SSH keys. SSH is just the secure way one machine talks to another and runs commands on it. I didn't think properly about the keys, and I lost a good while to it, everything else was right and it still wouldn't connect. Store your keys as variables in GitLab rather than hardcoding them anywhere, and make sure the remote's host key is in the runner's known_hosts file. Get that squared away and the rest goes far more smoothly. ## The YAML file (the list of instructions) This is the pipeline I mentioned. The way I built mine was to write down what I'd normally type by hand when deploying, then translate it into the format GitLab expects. It's not quite a straight copy-paste, there are rules to follow, but that's the gist. There are loads of tutorials out there. In the example below I'm storing my keys as GitLab variables, and I used ssh-keyscan to grab the host keys (handy if you haven't got them saved somewhere already). # The stages we've got (think of these as the tasks, in order) stages: - deploy # The job itself (you can have more than one) deploy_staging: stage: deploy # The actual list of instructions script: # Load the SSH key so the runner can reach the remote - eval $(ssh-agent -s) - echo "$SSH_REPO_KEY" | tr -d '\r' | ssh-add - > /dev/null - mkdir -p ~/.ssh && touch ~/.ssh/known_hosts - echo "$SSH_KNOWN_STAGING" >> ~/.ssh/known_hosts - chmod 644 ~/.ssh/known_hosts # The bit that actually deploys, run whatever you'd normally type by hand - ssh -p[port] [user]@[ip] "[the command you'd normally run manually]" # Only run this job on the branch you want only: - [branch] And that's about the size of it. You can pile on far more if you want, more stages, more jobs, running migrations, clearing caches, firing off notifications, all sorts. My advice is keep it simple, at least to start. The runner and the remote both use real resources while a job runs, and I've watched a machine run itself out of RAM trying to do too much at once. Get the basic thing working (connect, copy, done), and only add to it once that's solid. ## Is it worth it? For me, yes, though I'll not pretend it was a smooth ride. The payoff is that I mostly stop thinking about deployments. I push the code and wander off, same as those agencies I used to watch, except now I know what's going on behind it. If you're juggling a few projects at once, that saved faff adds up quickly. If you've got a cleaner way of doing any of this, I'd genuinely like to hear it. I'm always happy to be shown a better approach.