Cloud Jenkins Single Master Controller CI/CD Upgrade

[Gone]

Hello all,

 

This doesn't cleanly fit in any sub-forum we have here, it is possibly a split between this forum and development. Since I am dealing with cloud infrastructure and CI/CD, I'll post this here.

 

I have a non-AWS cloud provider that is substantially cheaper where I run single Jenkins Master Controller on a very cheap instance, within a Docker container. This controller monitors several Github repos for changes and automagically builds and redeploys parts of my personal infrastructure and apps (I am working on a product I'd like to sell so I can work for myself one day).

 

One of my problems with the cloud provider I use is a lack of AWS-like managed services, which while they would make life easy, would probably very easily bypass my current living expenses.

 

I am trying to brainstorm ideas for redeploying this single master controller if the GIt repo that hosts its configuration changes. Here are some things I though of, but I would like to hear from this community for different perspectives.

1. I deploy an extra instance on a cheap cloud instance to use something such as Terraform to mange my whole infrastructure.
2. I have a secondary Jenkins "master" controller that is occasionally spun up to facility self-configuration changes, which is responsible for terminating the old Jenkins controller.
3. I create a "function" specifically for the task of replacing the lone Jenkins controller (this is good because I have a storage volume that can only be attached to one instance at a time. I would like this volume that contains my Jenkins data to persists and be the sole volume). This is more difficult because I have to find a way to pass secrets that Jenkins would normally hold for the rest of my infrastructure, but it is not something that can't be done by deploying a secrets manager such as HashiCorp Vault.
4. I create a redeploy script hat if the container goes down for any reason, it pulls the updated Git repo with credentials that are already available to the host, then runs a redeploy script that spins up a new instance, waits for my cloud-init scripts to run, then detaches the volume that holds persistent Jenkins data and attaches it to the new instance, where Jenkins can then start up. The instance would then terminate itself after verifying Jenkins is up and ready.

Does anybody have any recommendations that are budget friendly? At work, I have the luxury of funding which cost becomes a secondary concern to a point and we have a whole team dedicated to managing config changes and software upgrades, but at home I don't.  Lots of my team recommended using K8s, but I'd much rather not deal with that if possible.