Secrets and the Problem with Build Servers
I am not totally sure I understand this, so let’s see if I do when I write it down.
I decided that gitlab needs to stop accessing hosts using keys from the gitlab runner, because this means:
- Everything on gitlab runner can access every VM
- If the gitlab runner moves, nothing can access any VM
So 1 is too open, and 2 just doesn’t work. While I could configure a gitlab runner per environment, it also strikes me that the secrets are in the ansible build directory and shouldn’t be. We need a better approach.
Gitlab will have the public key in armoured format (i.e. in 7 bit ascii) in a file variable (i.e. it writes the contents pasted into the box into a file, and sets a variable to the location of that file. It will have a blank password, as adding a password seems to add complication for no benefit.
The secrets will be moved to the inventory repository. Programs will be given access to
the secrets, so they can be used to build VMs. Also added to these secrets will be the password (Access token) to any git repositories it needs, and the root password of any VMs it needs to build.
Hopefully we can set up a way for the same process to be run from gitlab runner, or an engineers PC, so long as they have access to the secrets (via regpg) and the git repositories (via e.g. host keys).
I discovered that you can tell gpg to use another keyring. This will be very useful. So I do this by:
Now the key is created, let’s export it:
This creates runner.asc, which is the armoured public key file, and runnersec.asc which is the secret key file. This latter key is pasted into the gitlab variable, and labelled as a file variable.
To give the key access to the secrets, in another window (Or having unset GNUPGHOME, so I can access my key that can give the other key access) I enrol an admin:
So now I should be able to decrypt a key from the gitlab runner job. Let’s test:
gpg-preload.asc contains the value
True as per the
Setting up GitLab
This is using the key I created for GitLab runner because the
variable is set. I can do this for a GitLab CI job. As I say, the contents
runnersec.asc needs to be pasted into a GitLab variable. In the project, navigate to
Settings -> CI/CD -> Variables. Expand the variables section, and type in a name
for the variable in the Key field. Paste the private key into the value, and click save.
Assembling the Environment with a script
Now when the CI is run, a file is created with these contents. They key can’t
be used as it is, we need to import it into a keyring. We discovered above
that it is possible to create a special
GPGHOME for this job. So we create a script
that looks like this:
As I did above manually, I tested using the
gpg-preload.asc file that
sets up. I proved the gpg decrypt works! I can detect if we are running from
GitLab runner by checking if my file variable exists. If not, I assume the
user has access to the secrets through their own GPG key.
Getting the Other Repositories
As I mentioned, we also need to access other repositories, the one containing the playbooks and the one with the code to deploy.
If we are running in GitLab runner, the job can access other repositories using
CI_JOB_TOKEN. A user running the job from their PC would use the host keys
they have set up. We need to detect whether we are
running as a user or in gitlab runner. I use the following code to do this
So if the build is being run from gitlab runner,
it is set to:
email@example.com: Hopefully have host keys set up. The
repository name can then just be appended to this URI.
The reason for this was to ensure we don’t have to log in using host keys because any process running on a gitlab runner would be able to access that host. So let’s address that now.
I decided that if Ansible connected with a username and password rather than with host keys, we would be able to control which hosts each CI job connects to. So we need to configure two Ansible variables. This can be done in the inventory.
But of course the password should be encrypted. Create a file with the encrypted password:
This is on my desktop, so I don’t mind that the password is briefly in the process
listing, or in memory, or in the clear on the screen. I do like my passwords
not to end up in my bash history, so that is what the
set +o history does.
Using bash without history is surprisingly annoying, so I switch it back on again.
The password doesn’t contain a carriage return, so
echo -n ensures it doesn’t
write a newline at the end of the string.
Now the password is encrypted, so we can change the inventory:
Beautiful. The windows username and password is different, so I have a similar setting in the windows group which defines that, and it overrides this one nicely.
The testing so far shows the approach is valid. No doubt there is some work to get everything working, but hopefully that won’t take too long!