How to release Copr
Go through this page well before you will do the release. Maybe you will want to do some steps in different order, and in any case, it’s good to know what’s ahead.
Keep amending this page if you find something not matching reality or expectations.
Copr projects used during the release
- @copr/copr-dev
Automatic test builds from every push to
main. The CI workflow (.github/workflows/copr-builds.yml) rebuilds only the packages that changed in each push (test builds, not tagged releases). This is the repo you upgrade dev machines from before tagging.- @copr/copr
Tagged release builds. After you run
tito tagand push, you manually trigger builds here. These are the packages that go into Koji / infra repos / production.
Warning
Version skew in @copr/copr-dev: Because CI only rebuilds changed
packages, the repo can accumulate builds from different commits. For
example, python-copr-common might be from commit A while
copr-backend (which depends on it) is still from an older commit B.
Before relying on @copr/copr-dev for pre-release testing, always run
build_aux/rebuild-copr-stack to rebuild all packages in correct
dependency order from the current main HEAD.
Pre-release
The goal is to do as much work pre-release as possible while focusing only on important things and not creating a work overload with tasks, that can be done post-release.
The key principle: test before tagging. If you tag first and tests fail,
the tags are already pushed and you need fix-up tags on top. Instead, deploy
test builds from @copr/copr-dev to dev machines, run sanity tests, and only
create release tags once everything passes.
Rebuild @copr/copr-dev from current main
Make sure you are on the main branch and that it is up-to-date with
the upstream repository:
git checkout main
git pull --rebase origin main
Note
Commands in this document assume origin points to fedora-copr/copr.
If your origin is a personal fork, substitute your upstream remote name
(commonly upstream):
git remote -v # check which remote points to fedora-copr/copr
Ensure all PRs intended for this release are merged. Then rebuild the entire
stack into @copr/copr-dev in proper dependency order:
./build_aux/rebuild-copr-stack
This ensures all packages in the repo are built from the same main HEAD
with no version skew. Wait for all builds to finish.
Verify that all builds at
https://copr.fedorainfracloud.org/coprs/g/copr/copr-dev/builds/ show
“succeeded” status. If any build fails, investigate the build log, fix the
issue on main, push the changes, and re-run ./build_aux/rebuild-copr-stack.
Upgrade -dev machines
Run on batcave01.rdu3.fedoraproject.org (if you do not have account there ask Mirek or somebody from fedora-infra):
sudo rbac-playbook -l copr-be-dev.aws.fedoraproject.org groups/copr-backend.yml
sudo rbac-playbook -l copr-keygen-dev.aws.fedoraproject.org groups/copr-keygen.yml
sudo rbac-playbook -l copr-fe-dev.aws.fedoraproject.org groups/copr-frontend.yml
sudo rbac-playbook -l copr-dist-git-dev.aws.fedoraproject.org groups/copr-dist-git.yml
Note
If there is a new version of copr-rpmbuild, follow the Terminate resalloc resources instructions.
Make sure expected versions of Copr packages are installed on the dev instances:
./releng/run-on-all-infra --devel 'rpm -qa | grep copr'
If versions do not match, see the safe install tip below and fix before proceeding.
Tip
Rather than trusting a blanket dnf update, verify that the installed NVRs
match what you expect. You can use releng/install-copr-packages to install
packages from a Copr directory, identified by the upstream commit whose short
hash appears in the RPM Release field:
# On the target machine:
./releng/install-copr-packages @copr/copr-dev <commit-sha> copr-frontend copr-backend
Test on dev machines
Run the sanity tests from a Podman container (alternatively this can also be run from Beaker directly).
Call for QA
Move MODIFIED+ bugzillas to ON_QA.
Ask people to test, verify bugs, and generally help with QA. They will ignore it but you will feel good about giving them a chance.
Tag untagged packages that have changes in them
Prerequisite: All sanity tests on dev machines pass (previous step).
Run:
tito report --untagged-commits
and walk the directories of packages listed, and tag them. During development, we sometimes put a .PATCH number into our packages versions. See what packages has such version:
find . -maxdepth 2 -name "*.spec" -exec grep -H "Version:" {} + | grep -E '[0-9]+\.[0-9]+\.[0-9]+'
If a package has .PATCH number, manually increment its
version to the subsequent Major.Minor version with:
tito tag --use-version X.Y
For others, new version can be bumped automatically:
tito tag
Make sure that the %changelog is nice and meaningful, i.e. remove the
frontend:, rpmbuild:, etc. prefixes and filter-out entries which are not
interesting for the package end-users (git-log != %changelog). Later on, if
properly polished, the %changelogs’ contents may be used for filling the Bodhi
update text.
Push all new tags at once to the upstream repository:
git push --follow-tags origin
Build packages
Build all the updated packages into @copr/copr copr project:
copr build-package @copr/copr --nowait --name python-copr
copr build-package @copr/copr --nowait --name copr-frontend
...
Verify: All builds in @copr/copr succeed at
https://copr.fedorainfracloud.org/coprs/g/copr/copr/builds/.
Build packages for production
Make sure that .tito/releasers.conf has up to date list of branches.
Make sure you are co-maintainer of those packages in Fedora:
copr-backend
copr-cli
copr-dist-git
copr-frontend
copr-keygen
copr-messaging
copr-mocks
copr-rpmbuild
copr-selinux
python-copr
python-copr-common
For each package do:
cd <package subdir>
# run this for python-copr and copr-cli
tito release fedora-git-clients
# run this for python-copr-common and copr-messaging packages
tito release fedora-git-common
# run this for the copr-rpmbuild package
tito release fedora-git-rpmbuild
# run this for other (server) packages (copr-frontend, copr-backend, ...)
tito release fedora-git
Note
Koji doesn’t automatically put successfully built packages into the buildroot
for the following builds and therefore you can easily encounter failures of
copr-cli or copr server pacakges because of a missing dependency to
python3-copr or python3-copr-common that you have just built in Koji. To
fix this, you need to create a
Bodhi override for those dependencies
for example you can use this snippet:
sudo dnf install python3-fedora-distro-aliases
cd your/checkout/directory/copr-cli
for i in $(/usr/bin/resolve-fedora-aliases fedora-branched epel-all -o branch); do
git checkout $i
git pull
fedpkg override create --duration 1 --notes "Copr Release"
done
It takes up to 30 minutes to for the override to be available in the buildroot:
koji wait-repo f34-build --build=python-copr-common-0.13-1.fc34
for ver in 38 39 40 41; do
koji wait-repo f$ver-build --build=python-copr-common-0.22-1.fc$ver || echo ERROR $ver
done
Successfully waited 0:00 for python-copr-common-0.22-1.fc38 to appear in the f38-build repo
...
for ver in 8 9; do
koji wait-repo epel$ver-build --build=python-copr-common-0.22-1.el$ver || echo ERROR $ver
done
Successfully waited 0:00 for python-copr-common-0.22-1.el8 to appear in the epel8-build repo
...
Warning
Tito doesn’t work properly with more than one source, and when releasing
backend, it removes test-data-copr-backend-2.tar.gz from the DistGit
sources file. Until it gets resolved,
fix this way.
Prepare release notes
Go over bugs, which were resolved. Write some nice announce. It is useful to prepare the release notes beforehand because developers usualy don’t remember what they worked on and therefore don’t know what to test once production instances are upgraded. Sharing the prepared notes with team members before doing the actuall release is appreciated.
See previous release notes and try to format them in the same way. Then create a pull request with this release notes against Copr git repository.
Schedule and announce the outage
See a specific document Fedora Copr outage announcements, namely the “planned” outage state.
Release window
If all the pre-release preparations were done meticulously and everything was tested properly, the release window shouldn’t take more than ten minutes. That is, if nothing goes terribly sideways…
Let users know
See Fedora Copr outage announcements again, ad “ongoning” issue.
Post-release
At this moment, every Copr service should be up and running.
Generate documentation
The Copr project documentation
is generated automatically for every commit in the main branch.
Generate package specific documentation by going to:
And hitting “Build” button for each of those projects.
If schema was modified you should generate new Schema documentation.
Announce the end of the release
See a specific document Fedora Copr outage announcements, the “resolved” section.
Release packages to PyPI
Make sure you have ~/.pypirc correctly set up and run:
dnf install twine
python3 setup.py sdist
twine upload dist/<NAME-VERSION>.tar.gz
If you cannot run that, tell somebody with access to run that (msuchy, praiskup, jkadlcik).
This needs to be run for copr-common, python, copr-cli and copr-messaging.
Submit Bodhi updates
Create updates in Bodhi for every package built in Koji.
It is useful to do updates in batches, e.g. to group several packages into one
update. You can do this by fedpkg update, with the following template:
[ copr-backend-1.127-1.fc31, copr-frontend-1.154-1.fc31]
type=enhancement
notes=copr-frontend
- change 1 in frontend
- change 2 in frontend
copr-backend
- change 1 in backend
- change 2 in backend
It is often good idea to put new (filtered) %changelogs entries there.
Final steps
Check if the MODIFIED bugs (that are not ON_QA) are fixed in released Copr or not, move them ON_QA.
Change status of all ON_DEV, ON_QA, VERIFIED, and RELEASE_PENDING bugs to CLOSED/CURRENTRELEASE with comment like ‘New Copr has been released.’
Fix this document to make it easy for the release nanny of the next release to use it.