--- title: "How to version bump and branch" author: "Bioconductor Maintainer" date: "`r format(Sys.time(), '%B %d, %Y')`" vignette: > %\VignetteEngine{knitr::rmarkdown} %\VignetteIndexEntry{Version bumping and branching} %\VignetteEncoding{UTF-8} output: BiocStyle::html_document: number_sections: yes toc: true --- # Version-bump-and-branch-creation-HOWTO ## A. Introduction On the day prior to the release, we will need to apply and push the following changes to all the Bioconductor packages that go in the new release, **in this order**: - **First version bump**: bump x.y.z version to even y **in the `devel` branch** - **Branch creation**: create the new release branch - **Second version bump**: bump x.y.z version to odd y **in the `devel` branch** For example, for the BioC 3.19 release, we will need to do this for all the packages listed in the `software.txt`, `data-annotation.txt`, `data-experiment.txt`, `workflows.txt`, and `books.txt` files of the `RELEASE_3_19` branch of the `manifest` repo. However, note that for the packages listed in the `data-annotation.txt`, we won't perform any version bump i.e. we will only create the new release branch. Important exceptions: - The BiocVersion package (software package): This package will only need the new branch and a simple y -> y + 1 version bump. - The data-annotation packages: For these packages we will create the new branch but we won't bump versions. We'll use Python script `bump_version_and_create_branch.py` to apply and push these changes. This will need to be done on the day prior to the release, and BEFORE THE BioC 3.19 BUILDS START FOR SOFTARE, DATA-ANNOTATION, DATA-EXPERIMENT, WORKFLOW, AND BOOK PACKAGES. Look for the prerun jobs in the crontab for the `biocbuild` user on the central BioC 3.19 builder to get the times the software and data-experiment builds get kicked off. Make sure to check the crontab again a couple of days before the release as we sometimes make small adjustments to the crontabs on the build machines. Also be sure to translate to your local time if you are not on the East Coast. ## B. Preliminary steps These steps should be performed typically a couple of days before the steps in sections **C.** and **D.**. * Update this document to reflect the BioC version to be released i.e. replace all occurrences of `3.19` and `RELEASE_3_19` with appropriate version. This will avoid potentially disastrous mistakes when copying/pasting/executing commands from this document. * Choose a Linux machine with enough disk space to clone all the software, data-experiment, workflow, and book packages (as of October 2021, the total size of all the package clones is about 122G). The machine needs to have the `git` client and Python. The procedure described here doesn't require `sudo` privileges. Make sure to pick up a machine that has fast and reliable internet access. The Linux build machines are a good choice. If you want to use one of them, use your personal account or the `biocpush` account. Do NOT use the `biocbuild` account to not interfere with the builds. Using a Mac server might work but was not tested. * Make sure to use the `-A` flag to enable forwarding of the authentication agent connection when you `ssh` to the machine e.g.: ```sh ssh -A hpages@malbec1.bioconductor.org ``` * Clone (or update) the `BBS` git repo: ```sh # clone git clone https://github.com/Bioconductor/BBS # update cd ~/BBS git pull ``` * Create the `git.bioconductor.org` folder: ```sh mkdir git.bioconductor.org ``` * Populate `git.bioconductor.org` with git clones of the `manifest` repo and all the package repos (software, data-annotation, data-experiment, workflows, and books). This takes more than 3h so is worth doing in advance e.g. a couple of days before the release. It will save time when performing the steps described in sections **C.**, **D.**, and **E.** below on the day prior to the release. ```sh export BBS_HOME="$HOME/BBS" # clone `manifest` repo $BBS_HOME/utils/update_bioc_git_repos.py manifest RELEASE_3_19 # clone software package repos (takes approx. 2.5h) time $BBS_HOME/utils/update_bioc_git_repos.py software devel RELEASE_3_19 # clone data-annotation package repos (takes a couple of minutes) time $BBS_HOME/utils/update_bioc_git_repos.py data-annotation devel RELEASE_3_19 # clone data-experiment package repos (takes approx. 2.5h) time $BBS_HOME/utils/update_bioc_git_repos.py data-experiment devel RELEASE_3_19 # clone workflow package repos (takes approx. 9 min) time $BBS_HOME/utils/update_bioc_git_repos.py workflows devel RELEASE_3_19 # clone book repos (takes < 1 min) time $BBS_HOME/utils/update_bioc_git_repos.py books devel RELEASE_3_19 ``` * Make sure you can push changes to the BioC git server (at git.bioconductor.org): ```sh # check config file cat ~/.gitconfig # make any necessary adjustment with git config --global user.email "you@example.com" git config --global user.name "Your Name" # also make sure to set push.default to matching git config --global push.default matching # check config file again cat ~/.gitconfig # try to push cd ~/git.bioconductor.org/software/affy git push # should display 'Everything up-to-date' ``` * Clone (or update) the `bioc_git_transition` git repo: ```sh # clone git clone https://github.com/Bioconductor/bioc_git_transition # update cd ~/bioc_git_transition git pull ``` * Find packages with duplicate commits Historically the GIT-SVN mirror was the primary source of duplicate commits. Since the transition to git, we should not be seeing many new duplicates. Additionally, we've implemented a gitolite hook to prevent any new duplicate commits from getting through. For these reasons, this check for duplicates is becoming obsolete and can probably be removed at the next release in Spring 2019. ```sh # Local copy of bioc_git_transition export BIOC_GIT_TRANSITION="$HOME/bioc_git_transition" # software packages export WORKING_DIR="$HOME/git.bioconductor.org/software" export MANIFEST_FILE="$HOME/git.bioconductor.org/manifest/software.txt" cd $WORKING_DIR pkgs_in_manifest=`grep 'Package: ' $MANIFEST_FILE | sed 's/Package: //g'` # Check last 30 commits in each package (like the duplicate commits hook # does): for pkg in $pkgs_in_manifest; do echo "" echo ">>> check $pkg package for duplicate commits" python3 $BIOC_GIT_TRANSITION/misc/detect_duplicate_commits.py $pkg 30 done > duplicatecommits.out 2>&1 ``` * Anyone involved in the "bump and branch" procedure should temporarily be added to the `admin` group in `gitolite-admin/conf/gitolite.conf`. Membership in this group enables the creation of a new branch and _pushing_ to any branch regardless of inactivated (commented out) lines in `gitolite-admin/conf/packages.conf`. ## C. Version bumps and branch creation for software packages Perform these steps on the day prior to the release. They must be completed _before_ the software builds get kicked off (see **A. Introduction**). The full procedure should take about 3 hours. Make sure to reserve enough time. ### C1. Comment out all builds in crontab on central BioC 3.19 builder **IMPORTANT: No 3.19 build should start until:** - the bump and branch process is complete, - **and** `BBS_BIOC_GIT_BRANCH` is switched from `devel` to `RELEASE_3_19` on the central BioC 3.19 builder (this switch is documented in section F3 below). Maybe no build is scheduled to start during the next 3 hours, in which case everything is fine. Otherwise, comment out all the builds that are at the risk of starting before step F3 below is completed. Commenting out the `prerun` and `run` entries for each build should be enough. ### C2. Ask people to stop committing/pushing changes to the BioC git server Announce or ask a core team member to announce on the bioc-devel mailing list that people must stop committing/pushing changes to the BioC git server (git.bioconductor.org) for the next 3 hours. ### C3. Modify packages.conf to block all commits The `RELEASE_3_18` lines in `gitolite-admin/conf/packages.conf` were commented out when the release builds were frozen. At this point, only the `devel` lines are still active. Deactivate all push access by commenting out the `devel` lines in `gitolite-admin/conf/packages.conf`. NOTE: Do not change the branch from RELEASE_3_18 to RELEASE_3_19, it is not a good solution. Maintainers now will be able to push their own RELEASE_3_19 branch before we are able to create it at release time. This issue reflects the issue . Using vim, it is possible with a one liner: ```vi :g/RW devel/s/^/# ``` Or Lori created `sed` commands see `scripts/sedCommandsForVariousTasks.txt` Once `packages.conf` is updated, push to `gitolite-admin` on the git server to make the changes effective. ### C4. Disable hooks Log on to git.bioconductor.org as the `git` user. - Comment out the hook lines in `packages.conf`. - Remove the `pre-receive.h00-pre-receive-hook-software` file from the `hook/` directory in each package, e.g., `/home/git/repositories/packages/.git/hooks` ```sh rm -rf ~/repositories/packages/*.git/hooks/pre-receive.h00-pre-receive-hook-software ``` Or (trying as of release 3.19) try using hooks exclusion file. See scripts/sedCommandsForVariousTasks.txt. This changes is made in the hook_maintainer repo. ### C5. Log on the machine where you've performed the preliminary steps Make sure to use the `-A` flag to enable forwarding of the authentication agent connection e.g.: ```sh ssh -A hpages@malbec1.bioconductor.org ``` See **B. Preliminary steps** above for the details. ### C6. Checkout/update the `RELEASE_3_19` branch of the `manifest` repo ```sh cd ~/git.bioconductor.org/manifest git pull --all git checkout RELEASE_3_19 git branch git status ``` ### C7. Set the `BBS_HOME`, `WORKING_DIR` and `MANIFEST_FILE` environment variables - `BBS_HOME`: ```sh export BBS_HOME="$HOME/BBS" ``` IMPORTANT NOTE: The settings for `WORKING_DIR` and `MANIFEST_FILE` are specific to the type of packages. The following settings are for the _software_ packages. Make sure to adapt for data-experiment packages, workflows, and books. - `WORKING_DIR`: Point `WORKING_DIR` to the folder containing the software packages: ```sh export WORKING_DIR="$HOME/git.bioconductor.org/software" ``` All the remaining steps in section **C.** must be performed from _within_ this folder: ```sh cd $WORKING_DIR ``` - `MANIFEST_FILE`: Point `MANIFEST_FILE` to the manifest file for software packages. This must be the file from the `RELEASE_3_19` branch of the `manifest` repo: ```sh export MANIFEST_FILE="$HOME/git.bioconductor.org/manifest/software.txt" ``` ### C8. Run `bump_version_and_create_branch.py` We're going to run the script on all the software packages. It will take care of applying and pushing the changes described in **A. Introduction**. ```sh cd $WORKING_DIR pkgs_in_manifest=`grep 'Package: ' $MANIFEST_FILE | sed 's/Package: //g'` $BBS_HOME/utils/bump_version_and_create_branch.py --push RELEASE_3_19 $pkgs_in_manifest >bump_version_and_create_branch.log 2>&1 & ``` Alternatively, the `bump_version_and_create_branch.py` run above can be replaced with a 2-pass run: ```sh # First pass (apply all the changes but do NOT push them): $BBS_HOME/utils/bump_version_and_create_branch.py RELEASE_3_19 $pkgs_in_manifest >bump_version_and_create_branch.log1 2>&1 & # Second pass (push all the changes): $BBS_HOME/utils/bump_version_and_create_branch.py --push RELEASE_3_19 $pkgs_in_manifest >bump_version_and_create_branch.log2 2>&1 & ``` The 2-pass run can be useful if one wants to inspect the changes before pushing them. Notes: * You can follow progress with `tail -f bump_version_and_create_branch.log`. * The BiocVersion package will automatically receive special treatment. * In the 2-pass run, the second pass checks the packages and applies the changes only if needed (i.e. if a package does not already have the `RELEASE_3_19` branch) before pushing the changes. * If for some reason the `bump_version_and_create_branch.py` script stops prematurly, it can be safely re-run with the same arguments. This is because the script only does things to packages that need treatment. Packages that were already successfully treated will no longer be touched. * A typical error is `Error: duplicate commits` (happened for affyPLM and Rdisop first time I tested this). Report these errors to `gitolite` experts Lori or Jen. Once the problem is fixed, re-run the script. ### C9. Check `bump_version_and_create_branch.log` After completion of the `bump_version_and_create_branch.py` run, open `bump_version_and_create_branch.log` and make sure everything looks ok. ## D. Version bumps and branch creation for data-experiment packages, workflows, and books ### D1. Data-experiment packages Repeat steps C6 to C9 above **but for C7 define the environment variables as follows**: ```sh export WORKING_DIR="$HOME/git.bioconductor.org/data-experiment" export MANIFEST_FILE="$HOME/git.bioconductor.org/manifest/data-experiment.txt" ``` ### D2. Workflows Repeat steps C6 to C9 above **but for C7 define the environment variables as follows**: ```sh export WORKING_DIR="$HOME/git.bioconductor.org/workflows" export MANIFEST_FILE="$HOME/git.bioconductor.org/manifest/workflows.txt" ``` ### D3. Books Repeat steps C6 to C9 above **but for C7 define the environment variables as follows**: ```sh export WORKING_DIR="$HOME/git.bioconductor.org/books" export MANIFEST_FILE="$HOME/git.bioconductor.org/manifest/books.txt" ``` ## E. Branch creation for data-annotation packages Note that for data-annotation packages we **create the new branch WITHOUT BUMPING VERSIONS**. Repeat steps C6 to C9 above **but for C7 define the environment variables as follows**: ```sh export WORKING_DIR="$HOME/git.bioconductor.org/data-annotation" export MANIFEST_FILE="$HOME/git.bioconductor.org/manifest/data-annotation.txt" ``` Then for C8, **make sure to call the `bump_version_and_create_branch.py` script with the `--no-bump` option**: ```sh cd $WORKING_DIR pkgs_in_manifest=`grep 'Package: ' $MANIFEST_FILE | sed 's/Package: //g'` $BBS_HOME/utils/bump_version_and_create_branch.py --no-bump --push RELEASE_3_19 $pkgs_in_manifest >bump_version_and_create_branch.log 2>&1 & ``` ## F. Finishing up ### F1. Enable push access to new `RELEASE_3_19` branch This is done by editing the `conf/packages.conf` file in the `gitolite-admin` repo (`git clone git@git.bioconductor.org:gitolite-admin`). - If not done already, replace all instances of `RELEASE_3_18` with `RELEASE_3_19`. - Uncomment all `RELEASE_3_19` and `devel` lines. See scripts/sedCommandsForVariousTasks.txt for help on re-enabling access - If using hooks exclusion file for allowing version bump. Restore original hooks file. - Run `gitolite setup` from /home/git/repositories to re-enable the hooks. - Test that a non-super user can push access is enabled on the dummy package BiocGenerics_test. Check: ```sh git push git checkout RELEASE_3_19 git pull ``` ### F2. Tell people that committing/pushing to the BioC git server can resume Announce or ask a core team member to announce on the bioc-devel mailing list that committing/pushing changes to the BioC git server (git.bioconductor.org) can resume. ### F3. Switch `BBS_BIOC_GIT_BRANCH` from `devel` to `RELEASE_3_19` on central BioC 3.19 builder DON'T FORGET THIS STEP! Its purpose is to make the BioC 3.19 builds grab the `RELEASE_3_19` branch of all packages instead of their `devel` branch. Log on the central BioC 3.19 builder as `biocbuild` and replace ```sh export BBS_BIOC_GIT_BRANCH="devel" ``` with ```sh export BBS_BIOC_GIT_BRANCH="RELEASE_3_19" ``` in `~/BBS/3.19/config.sh` Also replace ```sh set BBS_BIOC_GIT_BRANCH=devel ``` with ```sh set BBS_BIOC_GIT_BRANCH=RELEASE_3_19 ``` in `~/BBS/3.19/config.bat` Then remove the `manifest` and `MEAT0` folders from `~/bbs-3.19-bioc/`, `~/bbs-3.19-data-annotation/`, `~/bbs-3.19-data-experiment/`, `~/bbs-3.19-workflows/`, and `~/bbs-3.19-books/`. They'll get automatically re-created and re-populated when the builds start. Once you're done, DON'T FORGET TO UNCOMMENT ALL THE BUILDS THAT WERE TEMPORARILY COMMENTED OUT IN THE CRONTAB (see section C1 above). ### F4. Sync all core Bioconductor packages hosted on GitHub The Bioconductor organization on GitHub hosts repositories that may also be packages in Bioconductor. To identify and update these packages with the latest version bump from a Bioconductor release, use the `ReleaseLaunch` R package at . First, create a fine-grained Personal Access Token (PAT) under User \> Settings \> Developer Settings \> PATs \> Fine-grained tokens. When generating a new token, select `Bioconductor` (or organization) as the resource owner, then 'All repositories' and under 'Repository permissions' select 'Contents' \> 'Read and Write'. If you are also modifying default branches, select 'Read and Write' under 'Administration'. It is recommended to use `gitcreds::gitcreds_set()` to store the fine-grained PAT. **Note:** Depending on the organization's settings, R / Bioconductor packages hosted on GitHub may also be updated with a classic PAT. These commands only work if the user has admin access to the organization's GitHub account. To update all packages, run the following command: ```r add_gh_release_branches(release = "RELEASE_3_19", org = "Bioconductor") ``` Be sure to edit the `release` argument in the function. To update an individual package, run the following function: ```r add_gh_release_branch( package_name = "updateObject", release = "RELEASE_3_19", gh_branch = "devel" ) ``` Note that the `gh_branch` corresponds to the default branch on GitHub. On some repositories, the `gh_branch` can be `devel`. Packages will be cloned into the current working directory.