How to upgrade builders

This article explains how to upgrade generate the Copr builder images (so called “golden images”) and how to migrate the Fedora Copr to them. Namely for

This HOWTO page is useful for upgrading images to a newer Fedora release or simply updating the packages contained within the builder images to the latest versions. This process of image “refreshing” significantly improves subsequent VM startup times and resolves bugs. Because when a builder machine starts from a relatively up-to-date image, only a minimal set of tasks is required at startup.

Keep amending this page if you find something not matching reality or expectations.

Requirements

Building an image locally

This is useful for development or debugging but the produced image will not be used production.

$ git clone https://github.com/fedora-copr/copr-image-builder.git
$ cd copr-image-builder
$ IMAGE_TYPE=qcow2 BUILD_OCI=true ./copr-build-image-bootc.sh

Run virt-manager and boot the image.

Build OCI images in Konflux

We have set up a GitOps process, so you may simply commit to the image building repo to trigger a new multi-arch OCI image build in Konflux.

Alternatively, go to this Konflux component, click on the “Actions” tab, and then select “Start a new build.” This will build OCI images for all architectures. Wait until it is finished, and then continue and build bootable images.

Locate the built image ID in Konflux UI

Check the last “push” Pipelinerun, and go to the “Details” tab. You should find the IMAGE_URL field that you want to copy-paste. If the pipeline failed but at least the build-image-index Task succeeded, go to the “Details” tab on that Task and copy the IMAGE_URL from there. Define a variable like this:

$ ssh root@copr-be-dev.aws.fedoraproject.org
# su - resalloc
IMAGE_URL=quay.io/redhat-user-workloads/fedora-copr-tenant/copr-image-builder:348c42d4e6249e3f5fef4534e77cf17d61d32fd9

Prepare the images on staging backend machine

This step allocates arch-specific VMs, creates qcow2/raw/ami images on them, and downloads them back to a directory structured as /var/lib/copr/public_html/images/2025-07-28/:

$ test -d copr-image-builder || git clone https://github.com/fedora-copr/copr-image-builder
$ cd copr-image-builder && git pull
$ make clean
$ make IMAGE=$IMAGE_URL<from the previous step>
... takes ~1h ...

Upload the images

Now it’s time to upload the built images to appropriate places:

STAMP=$(date -I) copr-upload-builder-images /var/lib/copr/public_html/images/2025-07-25

Check the stdout/stderr of the upload script, and recall image IDs.

Testing

If the images for all supported architectures are updated (according to previous sections), the staging copr instance is basically ready for testing. Update the Ansible git repo for all the changes in playbooks above, and also update the copr_builder_images option in inventory/group_vars/copr_dev_aws so it points to correct image names.

Increment the copr_builder_fedora_version number.

Once the changes are pushed upstream, you should re-provision the backend configuration from batcave:

$ ssh batcave01.iad2.fedoraproject.org
$ sudo rbac-playbook \
    -l copr-be-dev.aws.fedoraproject.org groups/copr-backend.yml \
    -t provision_config

You might well want to stop here for now, and try to test for a week or so that the devel instance behaves sanely. If not, consider running Running Sanity tests against local dev instance (or at least try to build several packages there).

You can try to kill all the old currently unused builders, and check the spawner log what is happening:

[copr@copr-be-dev ~][STG]$ resalloc-maint resource-delete --unused

Production

There is a substantially less work for production instance. You just need to equivalently update the production configuration file ./inventory/group_vars/copr_aws, so the copr_builder_images config points to the same image names as development instance does. And re-run playbook from batcave:

$ sudo rbac-playbook \
    -l copr-be.aws.fedoraproject.org groups/copr-backend.yml \
    -t provision_config

Optionally, when you need to propagate the new images quickly, you can terminate the old but currently unused builders by:

$ su - resalloc
$ resalloc-maint resource-delete --unused

Fixing just one architecture/cloud

Sometimes you want to fix just one architecture or cloud, e.g., aarch64 in EC2. Please never regenerate the OCI image in such a situation; otherwise, the freshly created images would diverge too much from the old. Conversely, if you really need to regenerate the OCI base image, please regenerate all the cloud images. But back to our example, this is how to do it:

make clean
IMAGE=$IMAGE_URL make download.aarch64
STAMP=$(date -I) ARCHES=aarch64 TARGETS=ec2 copr-upload-builder-images /var/lib/copr/public_html/images/2025-10-02/

Don’t worry the command says “make download”, it rebuilds the image first, then downloads it.

Cleanup

When everything is up and running the new version, do not forget to delete the old AMIs and associated snapshots from AWS.

Run make clean in the copr-image-builder directory.

Remove the new subdirectory in /var/lib/copr/public_html/images/ (>= 40GB).