How to make new xPack QEMU RISC-V releases

This page is intended for those who maintain the xPack QEMU RISC-V project and documents how to make new releases.

Prerequisites

The build scripts run on GNU/Linux and macOS. The Windows binaries are compiled on x64 GNU/Linux, using mingw-w64.

For details on installing the prerequisites, please read the XBB prerequisites page.

Get project sources

The project is hosted on GitHub:

https://github.com/xpack-dev-tools/qemu-riscv-xpack.git

To clone the stable branch (xpack), run the following commands in a terminal (on Windows use the Git Bash console):

rm -rf ~/Work/xpack-dev-tools/qemu-riscv-xpack.git && \
git clone https://github.com/xpack-dev-tools/qemu-riscv-xpack.git \
  ~/Work/xpack-dev-tools/qemu-riscv-xpack.git

For development purposes, clone the xpack-development branch:

rm -rf ~/Work/xpack-dev-tools/qemu-riscv-xpack.git && \
mkdir -p ~/Work/xpack-dev-tools && \
git clone \
  --branch xpack-development \
  https://github.com/xpack-dev-tools/qemu-riscv-xpack.git \
  ~/Work/xpack-dev-tools/qemu-riscv-xpack.git

Or, if the repo was already cloned:

git -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git pull

Get the writable helper sources

The project has a dependency to a common helper, that is normally installed as a read-only dependency; for development purposes, to be able to make changes to the scripts located inside the helper, clone the xpack-development branch and link it to the user global xPacks store:

rm -rf ~/Work/xpack-dev-tools/xbb-helper-xpack.git && \
mkdir -p ~/Work/xpack-dev-tools && \
git clone \
  --branch xpack-development \
  https://github.com/xpack-dev-tools/xbb-helper-xpack.git \
  ~/Work/xpack-dev-tools/xbb-helper-xpack.git && \
xpm link -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git

Or, if the repo was already cloned:

git -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git pull
xpm link -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git

Other repositories in use are:

https://github.com/xpack-dev-tools/qemu - to create the small macOS 10.13 patch

Release schedule

This distribution generally follows the official QEMU releases, but retains only about one release per year.

Prepare the new releases

Before starting the build, perform some checks and tweaks.

Download the build scripts

The build scripts are available in the build-assets/scripts folder of the xpack-dev-tools/qemu-riscv-xpack Git repo.

To download them on a new machine, clone the xpack-development branch, as seen above.

Check Git

In the xpack-dev-tools/qemu-riscv-xpack Git repo:

switch to the xpack-development branch
pull new changes
if necessary, merge the website branch

caution

This is really important, otherwise the next steps will build again the previous release!

The xpack branch should be unchanged since the previous release and will be updated when the new release is out.

Update helper & other dependencies

Check the latest versions at https://github.com/xpack-dev-tools/ and update the dependencies in build-assets/package.json.

Increase the version and update it in the top `package.json`

determine the version (like 8.2.6-1)
update the version in top package.json
use the new version, suffixed by .pre, like 8.2.6-1.1.pre.

Update the `customFields` in `build-assets/package.json` (if necessary)

If the project uses customFields in build-assets/package.json, update them.

Start the local web server

Execute the npm script clean then start in the website sub-project, or run the following in a terminal:

(cd ~/Work/xpack-dev-tools/xbb-helper-xpack.git/website; npm run clean; npm run start)

Navigate to the Maintainer Info page, the Update the version specific code section.

Increase the version and update VERSION

Determine the version (like 8.2.6-1) and update the build-assets/scripts/VERSION file; the format is 8.2.6-1. The fourth number is the xPack release number of this version. A fifth number will be added to the version in `package.json` when the package is published to the `npm` server.

Check the latest upstream release

The latest releases must have been announced in the QEMU web, and reflected in the Git tags.

Compare the latest upstream release with the latest xPack release. If necessary, update the triplet, otherwise increase the fourth number, as below.

Update the `customFields` (if necessary)

If the project uses customFields in build-assets/package.json

update the release specific properties (if any)
restart the local web server

Update the version specific code

open the build-assets/scripts/versioning.sh file
add a new if with the new version before the existing code

Fix possible open issues

Check GitHub issues and pull requests:

https://github.com/xpack-dev-tools/qemu-riscv-xpack/issues/

and fix them; assign them to a milestone (like 8.2.6-1).

Update `CHANGELOG.md`

open the CHANGELOG.md file
check if all previous fixed issues are in
check the latest commits xpm run git-log; if necessary, copy/paste lines, group by dates and edit them using the below regular expressions
add a new entry like * v8.2.6-1 prepared
commit with a message like prepare v8.2.6-1

To turn the dates into headings, change from:

([0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]) [*]

to:

## $1

*

To remove the rest of the dates, change from:

^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] [*]

to:

Push the changes to the `xpack-development` branch

Be sure that the latest commit that updates VERSION is pushed to the xpack-development branch!

caution

This is really important, otherwise the next steps will build again the previous release!

Run the development builds

The builds currently run on 5 dedicated machines (x64 GNU/Linux, x64 macOS, arm64 macOS, arm64 GNU/Linux, arm GNU/Linux).

Before the real build, run test/development builds on all platforms.

Visual Studio Code extension

All actions are defined as xPack actions and can be conveniently triggered via the VS Code graphical interface, using the xPack C/C++ Managed Build Tools extension.

Patches

The build uses two patches, one done with sed directly in the source tree, to add xPack to the greeting message, and one generated with git diff, to add support for macOS 10.13 (located in the helper patches folder).

Restart the docker daemons

If, for any reasons, the docker builds need to be canceled, it is very likely that some processes will continue to run in the background.

To completely terminate them, and ensure that docker runs from a clean slate, restart the docker daemons on all GNU/Linux machines:

sudo systemctl restart docker

Check if both projects are pushed

Check if both the current project and the helper are on the xpack-development branch and the latest commits are pushed to GitHub.

Build the x64 macOS binaries

For x64 macOS, first run the build on the development machine (wksi, a recent macOS):

export XBB_ENVIRONMENT_SKIP_CHECKS="y"

rm -f ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets/package-lock.json
git -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git pull

xpm run install -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

git -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git pull
xpm link -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git

xpm run link-deps -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

xpm run deep-clean --config darwin-x64  -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm install --config darwin-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run build-development --config darwin-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

For a debug build:

xpm run build-development-debug --config darwin-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

The build takes about 33 minutes to complete.

When functional, push the xpack-development branch to GitHub.

Run the native build on the production machine (xbbmi, an older macOS); start a VS Code remote session, or connect with a terminal:

caffeinate ssh xbbmi

To check the available space:

xpm run check-space -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

To free the space used by all previous builds of all xpack-dev-tools packages:

xpm run clear-all-project-builds -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

To update the build scripts and build the development binaries:

rm -f ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets/package-lock.json && \
git -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
git -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git pull && \
xpm link -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git && \
xpm run link-deps -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
\
xpm run deep-clean --config darwin-x64  -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm install --config darwin-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run build-development --config darwin-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

About 30 minutes later, the output of the build script is a compressed archive and its SHA signature, created in the buils-assets/build/darwin-x64/deploy folder:

xpack-qemu-riscv-8.2.6-1-darwin-x64.tar.gz
xpack-qemu-riscv-8.2.6-1-darwin-x64.tar.gz.sha

To update the build scripts and build the release binaries:

rm -f ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/package-lock.json && \
git -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
\
xpm run deep-clean --config darwin-x64  -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm install --config darwin-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run build --config darwin-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

Build the arm64 macOS binaries

Run the native build on the production machine (xbbma, an older macOS); start a VS Code remote session, or connect with a terminal:

caffeinate ssh xbbma

To check the available space:

xpm run check-space -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

To free the space used by all previous builds of all xpack-dev-tools packages:

xpm run clear-all-project-builds -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

To update the build scripts and build the development binaries:

rm -f ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets/package-lock.json && \
git -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
git -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git pull && \
xpm link -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git && \
xpm run link-deps -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
\
xpm run deep-clean --config darwin-arm64  -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm install --config darwin-arm64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run build-development --config darwin-arm64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

About 19 minutes later, the output of the build script is a compressed archive and its SHA signature, created in the buils-assets/build/darwin-arm64/deploy folder:

xpack-qemu-riscv-8.2.6-1-darwin-arm64.tar.gz
xpack-qemu-riscv-8.2.6-1-darwin-arm64.tar.gz.sha

To update the build scripts and build the release binaries:

rm -f ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/package-lock.json && \
git -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
\
xpm run deep-clean --config darwin-arm64  -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm install --config darwin-arm64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run build --config darwin-arm64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

Build the x64 GNU/Linux binaries

Run the docker build on the production machine (xbbli); start a VS Code remote session, or connect with a terminal:

caffeinate ssh xbbli

To check the available space:

xpm run check-space -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

To free the space used by all previous builds of all xpack-dev-tools packages:

xpm run clear-all-project-builds -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

To update the build scripts and build the development binaries:

rm -f ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets/package-lock.json && \
git -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
git -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git pull && \
xpm link -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git && \
xpm run link-deps -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
\
xpm run deep-clean --config linux-x64  -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-prepare --config linux-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-link-deps --config linux-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-build-development --config linux-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

About 17 minutes later, the output of the build script is a compressed archive and its SHA signature, created in the buils-assets/build/linux-x64/deploy folder:

xpack-qemu-riscv-8.2.6-1-linux-x64.tar.gz
xpack-qemu-riscv-8.2.6-1-linux-x64.tar.gz.sha

To update the build scripts and build the release binaries:

rm -f ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/package-lock.json && \
git -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
\
xpm run deep-clean --config linux-x64  -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-prepare --config linux-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-build-development --config linux-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

Build the x64 Windows binaries

Run the docker build on the production machine (xbbli); start a VS Code remote session, or connect with a terminal:

caffeinate ssh xbbli

To check the available space:

xpm run check-space -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

To free the space used by all previous builds of all xpack-dev-tools packages:

xpm run clear-all-project-builds -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

To update the build scripts and build the development binaries:

rm -f ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets/package-lock.json && \
git -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
git -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git pull && \
xpm link -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git && \
xpm run link-deps -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
\
xpm run deep-clean --config win32-x64  -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-prepare --config win32-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-link-deps --config win32-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-build-development --config win32-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

About 10 minutes later, the output of the build script is a compressed archive and its SHA signature, created in the buils-assets/build/win32-x64/deploy folder:

xpack-qemu-riscv-8.2.6-1-win32-x64.zip
xpack-qemu-riscv-8.2.6-1-win32-x64.zip.sha

To update the build scripts and build the release binaries:

rm -f ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/package-lock.json && \
git -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
\
xpm run deep-clean --config win32-x64  -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-prepare --config win32-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-build-development --config win32-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

Build the arm64 GNU/Linux binaries

Run the docker build on the production machine (xbbla); start a VS Code remote session, or connect with a terminal:

caffeinate ssh xbbla

To check the available space:

xpm run check-space -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

To free the space used by all previous builds of all xpack-dev-tools packages:

xpm run clear-all-project-builds -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

To update the build scripts and build the development binaries:

rm -f ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets/package-lock.json && \
git -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
git -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git pull && \
xpm link -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git && \
xpm run link-deps -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
\
xpm run deep-clean --config linux-arm64  -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-prepare --config linux-arm64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-link-deps --config linux-arm64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-build-development --config linux-arm64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

About 92 minutes (1h32m) later, or 29 minutes on ampere, the output of the build script is a compressed archive and its SHA signature, created in the buils-assets/build/linux-arm64/deploy folder:

xpack-qemu-riscv-8.2.6-1-linux-arm64.tar.gz
xpack-qemu-riscv-8.2.6-1-linux-arm64.tar.gz.sha

To update the build scripts and build the release binaries:

rm -f ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/package-lock.json && \
git -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
\
xpm run deep-clean --config linux-arm64  -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-prepare --config linux-arm64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-build-development --config linux-arm64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

Build the arm GNU/Linux (32-bit) binaries

Run the docker build on the production machine (xbbla32); start a VS Code remote session, or connect with a terminal:

caffeinate ssh xbbla32

To check the available space:

xpm run check-space -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

To free the space used by all previous builds of all xpack-dev-tools packages:

xpm run clear-all-project-builds -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

To update the build scripts and build the development binaries:

rm -f ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets/package-lock.json && \
git -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
git -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git pull && \
xpm link -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git && \
xpm run link-deps -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
\
xpm run deep-clean --config linux-arm  -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-prepare --config linux-arm -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-link-deps --config linux-arm -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-build-development --config linux-arm -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

About 84 minutes (1h24m) later, the output of the build script is a compressed archive and its SHA signature, created in the buils-assets/build/linux-arm/deploy folder:

xpack-qemu-riscv-8.2.6-1-linux-arm.tar.gz
xpack-qemu-riscv-8.2.6-1-linux-arm.tar.gz.sha

To update the build scripts and build the release binaries:

rm -f ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/package-lock.json && \
git -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
\
xpm run deep-clean --config linux-arm  -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-prepare --config linux-arm -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets && \
xpm run docker-build-development --config linux-arm -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

Update the trees in the Install Guide page

Copy the trees displayed at the end of the test builds and paste to Folder hierarchy section in the web file:

website/docs/install/_miscellaneous.mdx

If present, replace the actual version (like 8.2.6-1) with the following expression:

${customField('xpackVersion')}

tip

macOS does not have a tree command; either compile it from sources, or install it via HomeBrew, and make a link to /usr/local/bin.

Update the list of links in top package.json

Copy/paste the full list of links displayed at the end of the build, in sequence, for each platform (Windows, macOS, GNU/Linux), and check the differences compared to the repository.

Commit if necessary.

Update the durations of the development builds

In website/docs/maintainer/_development-durations.mdx, update the durations of the builds.

Commit if necessary.

How to build a debug version

In some cases it is necessary to run a debug session with the binaries.

For these cases, the build script accepts the --debug options.

There are also xPack actions that use this option (build-development-debug and docker-build-development-debug).

Files cache

The XBB build scripts use a local cache such that files are downloaded only during the first run, later runs being able to use the cached files.

However, occasionally some servers may not be available, and the builds may fail.

The workaround is to manually download the files from alternate locations (like https://github.com/xpack-dev-tools/files-cache/tree/master/libs), place them in the XBB cache (Work/cache) and restart the build.

Run the production builds

The automation is provided by GitHub Actions and three self-hosted runners.

Start the self-hosted runners

on the development machine (wksi) open ssh sessions to the build machines (xbbli, xbbmi, xbbma, xbbla, xbbla32):

caffeinate ssh xbbli
caffeinate ssh xbbmi
caffeinate ssh xbbma
caffeinate ssh xbbla
caffeinate ssh xbbla32

Create a screen session, to provide a persistent standard output for the runners:

screen -S ga

To quit it, use # Ctrl-a Ctrl-d.

For xbbli & xbbla, which have more memory, start two runners:

~/actions-runners/xpack-dev-tools/1/run.sh &
~/actions-runners/xpack-dev-tools/2/run.sh &

On all other machines start a single runner:

~/actions-runners/xpack-dev-tools/run.sh &

To check the status of the runners locally:

ps -a | grep 'Runner.Listener'

To kill them:

killall Runner.Listener

Check the status of the runners

The status of all self-hosted runners is available at the GitHub Runners page.

Publish the helper

Publish a new release of the helper on npmjs.

Update the dependency to the new helper

In build-assets/package.json, update the reference to "@xpack-dev-tools/xbb-helper": (possibly in all xPack projects).

Commit the changes to `build-assets/package.json`

Use the message build-assets/package.json: bump deps.

Check for disk space

Check if the build machines have enough free space and eventually do some cleanups:

xpm run check-space -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

This is equivalent to running a df command.

macOS
GNU/Linux

df -gH /

df -BG -H /

To free the space used by all previous builds of all xpack-dev-tools packages:

xpm run clear-all-project-builds -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

This is equivalent to running a wide rm to remove the build folders:

rm -rf ~/Work/xpack-dev-tools/*/build-assets/build

Generate the GitHub workflows

Run the generate-workflows xPack action to update the GitHub workflow files.

xpm run generate-workflows -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

Commit and push

commit with the message Re-generate workflows
push the xpack-development branch to GitHub

Manually trigger the build GitHub Actions

To trigger the GitHub Actions builds, use the xPack actions:

trigger-workflow-build-darwin-x64
trigger-workflow-build-darwin-arm64
trigger-workflow-build-linux-x64
trigger-workflow-build-win32-x64
trigger-workflow-build-linux-arm64
trigger-workflow-build-linux-arm

xpm run trigger-workflow-build-darwin-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets
xpm run trigger-workflow-build-darwin-arm64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets
xpm run trigger-workflow-build-linux-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets
xpm run trigger-workflow-build-win32-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets
xpm run trigger-workflow-build-linux-arm64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets
xpm run trigger-workflow-build-linux-arm -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

The scripts behind these actions require the GITHUB_API_DISPATCH_TOKEN variable to be present in the environment, and the organization PUBLISH_TOKEN to be visible in the Settings → Action → Secrets page.

These commands use the xpack-development branch of this repo.

Durations & resulting binaries

The builds take about 2 hours to complete:

darwin-x64: 29m (nuc)
darwin-arm64: 18m (m1)
linux-x64: 21m (amd)
win32-x64: 13m (amd)
linux-arm64: 1h25m (rpi4)
linux-arm: 1h18m (rpi4)

The workflow results and logs are available from the GitHub Actions page.

Update the values in website/docs/maintainer/_github-actions-durations.mdx with those shown by GitHub Actions.

The resulting binaries are available for testing from pre-releases/test.

Run the tests

Automated tests

The automation is provided by GitHub Actions.

To trigger the GitHub Actions tests, run the xPack actions:

trigger-workflow-test-prime
trigger-workflow-test-docker-linux-x64
trigger-workflow-test-docker-linux-arm

xpm run trigger-workflow-test-prime -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets
xpm run trigger-workflow-test-docker-linux-x64 -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets
xpm run trigger-workflow-test-docker-linux-arm -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

The scripts behind these accesible require the GITHUB_API_DISPATCH_TOKEN variable to be present in the environment.

These actions use the xpack-development branch of this repo and the pre-releases/test binaries.

The tests results are available from the GitHub Actions page.

After running the prime tests, download the test results and compare with previous runs.

More macOS tests

Since GitHub Actions provides a limited range of versions for the macOS runner, the multi-version macOS tests run on Travis.

To trigger the Travis test, run the trigger-travis-macos xPack action:

xpm run trigger-travis-macos -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

This script requires the TRAVIS_COM_TOKEN variable to be present in the environment.

The test results are available from Travis CI.

Manual tests

To download the pre-released archive for the specific platform and run the tests, use:

git -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git pull
xpm run install -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets
xpm run test-pre-release -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

For even more tests, on each platform, download the archive from pre-releases/test and check the binaries.

On macOS, remove the com.apple.quarantine flag:

xattr -cr ${HOME}/Downloads/xpack-*

Publish the release

Create a new GitHub pre-release

in CHANGELOG.md, add the release date and a message like * v8.2.6-1 released
commit with CHANGELOG update
check and possibly update the build-assets/templates/body-github-release-liquid.mdx
push the xpack-development branch to GitHub
run the trigger-workflow-publish-release xPack action:

xpm run trigger-workflow-publish-release -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

The workflow result and logs are available from the GitHub Actions page.

The result is a draft pre-release tagged like v8.2.6-1 (mind the dash in the middle!) and named like xPack QEMU RISC-V v8.2.6-1 (mind the dash), with all binaries attached.

edit the draft and be sure it is attached to the xpack-development branch (important!)
perform the final edits (maintenance vs. new release) and check if everything is fine
keep the pre-release button enabled
do not enable Discussions yet
click the Publish release button

Note: at this moment the system should send a notification to all clients watching this project.

Prepare a new blog post

check and possibly update the website/blog/_templates/blog-post-release-part-[12]-liquid.md
run the generate-website-blog-post xPack action; this will add a file in the website/blog folder:

xpm run generate-website-blog-post -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

edit the front matter properties
select the correct summary
commit with blog post release 8.2.6-1 published

Update the web install page

check and possibly update the output of the --version runs in
- website/docs/install/_automatic-install-quick-test.mdx
- website/docs/install/_manual-install-quick-test.mdx
commit changes

Check the list of links in top package.json

open the package.json file
check if the links in the bin property cover the actual binaries
if necessary, also check on Windows

Update the package.json list of binaries

select the xpack-development branch
be sure that the binaries were attached to the pre-release
run the update-package-binaries xPack action:

xpm run update-package-binaries -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

open the top package.json file
check the baseUrl: it should match the file URLs (including the tag/version); no terminating / is required
from the release, check the SHA & file names
compare the SHA sums with those shown by cat *.sha
check the executable names
commit all changes, use a message like package.json: update urls for 8.2.6-1.1 release

Publish on the npmjs.com server

select the xpack-development branch
check the latest commits xpm run git-log
update CHANGELOG.md, add a line like * v8.2.6-1 published on npmjs.com
commit with a message like CHANGELOG: publish npm v8.2.6-1.1
npm pack and check the content of the archive, which should list only the package.json, the README.md, LICENSE and CHANGELOG.md; possibly adjust .npmignore
configure the version; the first 4 numbers are the same as the GitHub release; the fifth number is the npm specific version:

npm version 8.2.6-1.1

the commits and the tag should have been pushed by the postversion script; if not, push them with git push origin --tags to GitHub
publish and add the test tag:

npm publish --tag test

when publishing for the first time, use:

npm publish --access public

After a few moments the version will be visible at npmjs.com.

Test if the binaries can be installed with xpm

Run the trigger-workflow-test-xpm xPack action:

xpm run trigger-workflow-test-xpm -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

This will install the package via xpm install on all supported platforms.

The tests results are available from the GitHub Actions page.

Tag the npm package as `latest`

When the release is considered stable, promote it as latest:

npm dist-tag ls @xpack-dev-tools/qemu-riscv
npm dist-tag add @xpack-dev-tools/qemu-riscv@8.2.6-1.1 latest
npm dist-tag ls @xpack-dev-tools/qemu-riscv

In case the release proves problematic and needs to be unpublished:

npm unpublish @xpack-dev-tools/qemu-riscv@8.2.6-1.1

Build the static website locally

Validate the web site content in a local build via the npm build script:

npm --prefix website run build

Publish the website

merge xpack-development into website
push the website branch to GitHub

At this moment an automated GitHub Action will generate and publish the website.

the new release blog post is in https://xpack-dev-tools.github.io/qemu-riscv-xpack/blog/
remember the post URL, since it must be used to update the release page and the X/Twitter announcement

The website branch may be subsequently updated, as long as the version in package.json is not changed.

Update the `xpack` branch

merge xpack-development into xpack
push the xpack branch to GitHub

note

Avoid further updates to the xpack branch until the next release.

Create the final GitHub release

go to the GitHub Releases page
check the download counter, it should match the number of tests
add a link to the Web blog [Continue reading »](); use an same blog URL
remove the tests only notice
disable the pre-release button
click the Update Release button

Check SourceForge mirror

https://sourceforge.net/projects/qemu-riscv-xpack/files/

Cleanups

Remove the pre-release binaries

go to https://github.com/xpack-dev-tools/pre-releases/releases/tag/test/
remove the test binaries

Clean the work area

Run the trigger-workflow-deep-clean xPack action

xpm run trigger-workflow-deep-clean -C ~/Work/xpack-dev-tools/qemu-riscv-xpack.git/build-assets

This will remove the build folders on all self-hosted runners.

The results are available from the GitHub Actions page.

in a separate browser windows, open X/Twitter
using the @xpack_project account
paste the release name like xPack QEMU RISC-V v8.2.6-1 released
paste the link to the blog post release
click the Tweet button

Analytics

GitHub xpack-dev-tools/qemu-riscv-xpack repo
- latest xPack release
- all xPack releases
- individual file counters (grouped by release)
npmjs.com @xpack-dev-tools/qemu-riscv xPack
- latest release, per month
- all releases

Credit to Shields IO for the badges and to Somsubhra/github-release-stats for the individual file counters.

Prerequisites​

Get project sources​

Get the writable helper sources​

Release schedule​

Prepare the new releases​

Download the build scripts​

Check Git​

Update helper & other dependencies​

Increase the version and update it in the top package.json​

Update the customFields in build-assets/package.json (if necessary)​

Start the local web server​

Increase the version and update VERSION​

Check the latest upstream release​

Update the customFields (if necessary)​

Update the version specific code​

Fix possible open issues​

Update CHANGELOG.md​

Push the changes to the xpack-development branch​

Run the development builds​

Visual Studio Code extension​

Patches​

Restart the docker daemons​

Check if both projects are pushed​

Build the x64 macOS binaries​

Build the arm64 macOS binaries​

Build the x64 GNU/Linux binaries​

Build the x64 Windows binaries​

Build the arm64 GNU/Linux binaries​

Build the arm GNU/Linux (32-bit) binaries​

Update the trees in the Install Guide page​

Update the list of links in top package.json​

Update the durations of the development builds​

How to build a debug version​

Files cache​

Run the production builds​

Start the self-hosted runners​

Check the status of the runners​

Publish the helper​

Update the dependency to the new helper​

Commit the changes to build-assets/package.json​

Check for disk space​

Generate the GitHub workflows​

Commit and push​

Manually trigger the build GitHub Actions​

Durations & resulting binaries​

Run the tests​

Automated tests​

More macOS tests​

Manual tests​

Publish the release​

Create a new GitHub pre-release​

Prepare a new blog post​

Update the web install page​

Check the list of links in top package.json​

Update the package.json list of binaries​

Publish on the npmjs.com server​

Test if the binaries can be installed with xpm​

Tag the npm package as latest​

Build the static website locally​

Publish the website​

Update the xpack branch​

Create the final GitHub release​

Check SourceForge mirror​

Cleanups​

Remove the pre-release binaries​

Clean the work area​

Share on Twitter​

Analytics​

Prerequisites

Get project sources

Get the writable helper sources

Release schedule

Prepare the new releases

Download the build scripts

Check Git

Update helper & other dependencies

Increase the version and update it in the top `package.json`

Update the `customFields` in `build-assets/package.json` (if necessary)

Start the local web server

Increase the version and update VERSION

Check the latest upstream release

Update the `customFields` (if necessary)

Update the version specific code

Fix possible open issues

Update `CHANGELOG.md`

Push the changes to the `xpack-development` branch

Run the development builds

Visual Studio Code extension

Patches

Restart the docker daemons

Check if both projects are pushed

Build the x64 macOS binaries

Build the arm64 macOS binaries

Build the x64 GNU/Linux binaries

Build the x64 Windows binaries

Build the arm64 GNU/Linux binaries

Build the arm GNU/Linux (32-bit) binaries

Update the trees in the Install Guide page

Update the list of links in top package.json

Update the durations of the development builds

How to build a debug version

Files cache

Run the production builds

Start the self-hosted runners

Check the status of the runners

Publish the helper

Update the dependency to the new helper

Commit the changes to `build-assets/package.json`

Check for disk space

Generate the GitHub workflows

Commit and push

Manually trigger the build GitHub Actions

Durations & resulting binaries

Run the tests

Automated tests

More macOS tests

Manual tests

Publish the release

Create a new GitHub pre-release

Prepare a new blog post

Update the web install page

Check the list of links in top package.json

Update the package.json list of binaries

Publish on the npmjs.com server

Test if the binaries can be installed with xpm

Tag the npm package as `latest`

Build the static website locally

Publish the website

Update the `xpack` branch

Create the final GitHub release

Check SourceForge mirror

Cleanups

Remove the pre-release binaries

Clean the work area

Share on Twitter

Analytics