How to make new xPack GNU bison releases
This page is intended for those who maintain the xPack GNU bison project and documents how to make new releases.
Prerequisites
The build scripts run on GNU/Linux and macOS.
For details on installing the prerequisites, please read the XBB prerequisites page.
Get project sources
The project is hosted on GitHub:
To clone the stable branch (xpack
), run the following commands in a
terminal (on Windows use the Git Bash console):
rm -rf ~/Work/xpack-dev-tools/bison-xpack.git && \
git clone https://github.com/xpack-dev-tools/bison-xpack.git \
~/Work/xpack-dev-tools/bison-xpack.git
For development purposes, clone the xpack-development
branch:
rm -rf ~/Work/xpack-dev-tools/bison-xpack.git && \
mkdir -p ~/Work/xpack-dev-tools && \
git clone \
--branch xpack-development \
https://github.com/xpack-dev-tools/bison-xpack.git \
~/Work/xpack-dev-tools/bison-xpack.git
Or, if the repo was already cloned:
git -C ~/Work/xpack-dev-tools/bison-xpack.git pull
Get the writable helper sources
The project has a dependency to a common helper, that is
normally installed as a read-only dependency; for development
purposes, to be able to make changes to the scripts located inside the helper,
clone the xpack-development
branch and link it to
the user global xPacks store:
rm -rf ~/Work/xpack-dev-tools/xbb-helper-xpack.git && \
mkdir -p ~/Work/xpack-dev-tools && \
git clone \
--branch xpack-development \
https://github.com/xpack-dev-tools/xbb-helper-xpack.git \
~/Work/xpack-dev-tools/xbb-helper-xpack.git && \
xpm link -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git
Or, if the repo was already cloned:
git -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git pull
xpm link -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git
Other repositories in use are:
- none
Release schedule
This distribution generally follows the official GNU bison releases, which are very rare.
Prepare the new releases
Before starting the build, perform some checks and tweaks.
Download the build scripts
The build scripts are available in the build-assets/scripts
folder of the
xpack-dev-tools/bison-xpack
Git repo.
To download them on a new machine, clone the xpack-development
branch,
as seen above.
Check Git
In the xpack-dev-tools/bison-xpack
Git repo:
- switch to the
xpack-development
branch - pull new changes
- if necessary, merge the
website
branch
This is really important, otherwise the next steps will build again the previous release!
The xpack
branch should be unchanged since the previous release
and will be updated when the new release is out.
Update helper & other dependencies
Check the latest versions at https://github.com/xpack-dev-tools/ and
update the dependencies in build-assets/package.json
.
Increase the version and update it in the top package.json
- determine the version (like
3.8.2-1
) - update the version in top
package.json
- use the new version, suffixed by
.pre
, like3.8.2-1.1.pre
.
Update the customFields
in build-assets/package.json
(if necessary)
If the project uses customFields
in build-assets/package.json
,
update them.
Start the local web server
Execute the npm script clean
then start
in the website sub-project,
or run the following in a terminal:
(cd ~/Work/xpack-dev-tools/xbb-helper-xpack.git/website; npm run clean; npm run start)
Navigate to the Maintainer Info page, the Update the version specific code section.
Increase the version and update VERSION
Determine the version (like 3.8.2-1
)
and update the build-assets/scripts/VERSION
file; the format
is 3.8.2-1
.
The fourth number is the xPack release number of this version. A fifth number will be added to the version in `package.json` when the package is published to the `npm` server.
Check the latest upstream release
The latest releases must have been announced in GNU bison releases.
Compare the latest upstream release with the latest xPack release. If necessary, update the triplet, otherwise increase the fourth number, as below.
Update the customFields
(if necessary)
If the project uses customFields
in build-assets/package.json
- update the release specific properties (if any)
- restart the local web server
Update the version specific code
- open the
build-assets/scripts/versioning.sh
file - add a new
if
with the new version before the existing code
Fix possible open issues
Check GitHub issues and pull requests:
and fix them; assign them to a milestone (like 3.8.2-1
).
Update CHANGELOG.md
- open the
CHANGELOG.md
file - check if all previous fixed issues are in
- check the latest commits
xpm run git-log
; if necessary, copy/paste lines, group by dates and edit them using the below regular expressions - add a new entry like * v3.8.2-1 prepared
- commit with a message like prepare v3.8.2-1
To turn the dates into headings, change from:
([0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]) [*]
to:
## $1
*
To remove the rest of the dates, change from:
^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] [*]
to:
*
Push the changes to the xpack-development
branch
Be sure that the latest commit that updates VERSION
is pushed
to the xpack-development
branch!
This is really important, otherwise the next steps will build again the previous release!
Run the development builds
The builds currently run on 5 dedicated machines (x64 GNU/Linux, arm64 GNU/Linux, arm GNU/Linux, x64 macOS, arm64 macOS).
Before the real build, run test/development builds on all platforms.
Visual Studio Code extension
All actions are defined as xPack actions and can be conveniently triggered via the VS Code graphical interface, using the xPack C/C++ Managed Build Tools extension.
Patches
This project uses the upstream sources, without patches.
Restart the docker daemons
If, for any reasons, the docker builds need to be canceled, it is very likely that some processes will continue to run in the background.
To completely terminate them, and ensure that docker runs from a clean slate, restart the docker daemons on all GNU/Linux machines:
sudo systemctl restart docker
Check if both projects are pushed
Check if both the current project and the helper are
on the xpack-development
branch and the latest
commits are pushed to GitHub.
Build the x64 macOS binaries
For x64 macOS, first run the build on the development machine
(wksi
, a recent macOS):
export XBB_ENVIRONMENT_SKIP_CHECKS="y"
rm -f ~/Work/xpack-dev-tools/bison-xpack.git/build-assets/package-lock.json
git -C ~/Work/xpack-dev-tools/bison-xpack.git pull
xpm run install -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
git -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git pull
xpm link -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git
xpm run link-deps -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
xpm run deep-clean --config darwin-x64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm install --config darwin-x64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run build-development --config darwin-x64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
For a debug build:
xpm run build-development-debug --config darwin-x64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
The build takes several minutes to complete.
When functional, push the xpack-development
branch to GitHub.
Run the native build on the production machine
(xbbmi
, an older macOS);
start a VS Code remote session, or connect with a terminal:
caffeinate ssh xbbmi
To check the available space:
xpm run check-space -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
To free the space used by all previous builds of all xpack-dev-tools
packages:
xpm run clear-all-project-builds -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
To update the build scripts and build the development binaries:
rm -f ~/Work/xpack-dev-tools/bison-xpack.git/build-assets/package-lock.json && \
git -C ~/Work/xpack-dev-tools/bison-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
git -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git pull && \
xpm link -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git && \
xpm run link-deps -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
\
xpm run deep-clean --config darwin-x64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm install --config darwin-x64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run build-development --config darwin-x64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
Several minutes later, the output of the build script is a compressed
archive and its SHA signature, created in
the buils-assets/build/darwin-x64/deploy
folder:
-
xpack-bison-3.8.2-1-darwin-x64.tar.gz
-
xpack-bison-3.8.2-1-darwin-x64.tar.gz.sha
To update the build scripts and build the release binaries:
rm -f ~/Work/xpack-dev-tools/bison-xpack.git/package-lock.json && \
git -C ~/Work/xpack-dev-tools/bison-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
\
xpm run deep-clean --config darwin-x64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm install --config darwin-x64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run build --config darwin-x64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
Build the arm64 macOS binaries
Run the native build on the production machine
(xbbma
, an older macOS);
start a VS Code remote session, or connect with a terminal:
caffeinate ssh xbbma
To check the available space:
xpm run check-space -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
To free the space used by all previous builds of all xpack-dev-tools
packages:
xpm run clear-all-project-builds -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
To update the build scripts and build the development binaries:
rm -f ~/Work/xpack-dev-tools/bison-xpack.git/build-assets/package-lock.json && \
git -C ~/Work/xpack-dev-tools/bison-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
git -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git pull && \
xpm link -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git && \
xpm run link-deps -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
\
xpm run deep-clean --config darwin-arm64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm install --config darwin-arm64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run build-development --config darwin-arm64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
Several minutes later, the output of the build script is a compressed
archive and its SHA signature, created in
the buils-assets/build/darwin-arm64/deploy
folder:
-
xpack-bison-3.8.2-1-darwin-arm64.tar.gz
-
xpack-bison-3.8.2-1-darwin-arm64.tar.gz.sha
To update the build scripts and build the release binaries:
rm -f ~/Work/xpack-dev-tools/bison-xpack.git/package-lock.json && \
git -C ~/Work/xpack-dev-tools/bison-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
\
xpm run deep-clean --config darwin-arm64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm install --config darwin-arm64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run build --config darwin-arm64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
Build the x64 GNU/Linux binaries
Run the docker build on the production machine (xbbli
);
start a VS Code remote session, or connect with a terminal:
caffeinate ssh xbbli
To check the available space:
xpm run check-space -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
To free the space used by all previous builds of all xpack-dev-tools
packages:
xpm run clear-all-project-builds -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
To update the build scripts and build the development binaries:
rm -f ~/Work/xpack-dev-tools/bison-xpack.git/build-assets/package-lock.json && \
git -C ~/Work/xpack-dev-tools/bison-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
git -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git pull && \
xpm link -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git && \
xpm run link-deps -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
\
xpm run deep-clean --config linux-x64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run docker-prepare --config linux-x64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run docker-link-deps --config linux-x64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run docker-build-development --config linux-x64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
Several minutes later, the output of the build script is a compressed
archive and its SHA signature, created in
the buils-assets/build/linux-x64/deploy
folder:
-
xpack-bison-3.8.2-1-linux-x64.tar.gz
-
xpack-bison-3.8.2-1-linux-x64.tar.gz.sha
To update the build scripts and build the release binaries:
rm -f ~/Work/xpack-dev-tools/bison-xpack.git/package-lock.json && \
git -C ~/Work/xpack-dev-tools/bison-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
\
xpm run deep-clean --config linux-x64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run docker-prepare --config linux-x64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run docker-build-development --config linux-x64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
Build the arm64 GNU/Linux binaries
Run the docker build on the production machine (xbbla
);
start a VS Code remote session, or connect with a terminal:
caffeinate ssh xbbla
To check the available space:
xpm run check-space -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
To free the space used by all previous builds of all xpack-dev-tools
packages:
xpm run clear-all-project-builds -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
To update the build scripts and build the development binaries:
rm -f ~/Work/xpack-dev-tools/bison-xpack.git/build-assets/package-lock.json && \
git -C ~/Work/xpack-dev-tools/bison-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
git -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git pull && \
xpm link -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git && \
xpm run link-deps -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
\
xpm run deep-clean --config linux-arm64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run docker-prepare --config linux-arm64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run docker-link-deps --config linux-arm64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run docker-build-development --config linux-arm64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
Several minutes later, the output of the build script is a compressed
archive and its SHA signature, created in
the buils-assets/build/linux-arm64/deploy
folder:
-
xpack-bison-3.8.2-1-linux-arm64.tar.gz
-
xpack-bison-3.8.2-1-linux-arm64.tar.gz.sha
To update the build scripts and build the release binaries:
rm -f ~/Work/xpack-dev-tools/bison-xpack.git/package-lock.json && \
git -C ~/Work/xpack-dev-tools/bison-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
\
xpm run deep-clean --config linux-arm64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run docker-prepare --config linux-arm64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run docker-build-development --config linux-arm64 -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
Build the arm GNU/Linux (32-bit) binaries
Run the docker build on the production machine (xbbla32
);
start a VS Code remote session, or connect with a terminal:
caffeinate ssh xbbla32
To check the available space:
xpm run check-space -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
To free the space used by all previous builds of all xpack-dev-tools
packages:
xpm run clear-all-project-builds -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
To update the build scripts and build the development binaries:
rm -f ~/Work/xpack-dev-tools/bison-xpack.git/build-assets/package-lock.json && \
git -C ~/Work/xpack-dev-tools/bison-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
git -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git pull && \
xpm link -C ~/Work/xpack-dev-tools/xbb-helper-xpack.git && \
xpm run link-deps -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
\
xpm run deep-clean --config linux-arm -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run docker-prepare --config linux-arm -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run docker-link-deps --config linux-arm -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run docker-build-development --config linux-arm -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
Several minutes later, the output of the build script is a compressed
archive and its SHA signature, created in
the buils-assets/build/linux-arm/deploy
folder:
-
xpack-bison-3.8.2-1-linux-arm.tar.gz
-
xpack-bison-3.8.2-1-linux-arm.tar.gz.sha
To update the build scripts and build the release binaries:
rm -f ~/Work/xpack-dev-tools/bison-xpack.git/package-lock.json && \
git -C ~/Work/xpack-dev-tools/bison-xpack.git pull && \
xpm run install -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
\
xpm run deep-clean --config linux-arm -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run docker-prepare --config linux-arm -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets && \
xpm run docker-build-development --config linux-arm -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
Update the trees in the Install Guide page
Copy the trees displayed at the end of the test builds and paste to Folder hierarchy section in the web file:
website/docs/install/_miscellaneous.mdx
If present, replace the actual version (like 3.8.2-1) with the following expression:
${customField('xpackVersion')}
macOS does not have a tree
command; either compile it from
sources, or install it
via HomeBrew, and make a link to /usr/local/bin
.
Update the list of links in top package.json
Copy/paste the full list of links displayed at the end of the build, in sequence, for each platform (GNU/Linux, macOS), and check the differences compared to the repository.
Commit if necessary.
Update the durations of the development builds
In website/docs/maintainer/_development-durations.mdx
, update
the durations of the builds.
Commit if necessary.
How to build a debug version
In some cases it is necessary to run a debug session with the binaries.
For these cases, the build script accepts the --debug
options.
There are also xPack actions that use this option (build-development-debug
and docker-build-development-debug
).
Files cache
The XBB build scripts use a local cache such that files are downloaded only during the first run, later runs being able to use the cached files.
However, occasionally some servers may not be available, and the builds may fail.
The workaround is to manually download the files from alternate
locations (like
https://github.com/xpack-dev-tools/files-cache/tree/master/libs),
place them in the XBB cache (Work/cache
) and restart the build.
Run the production builds
The automation is provided by GitHub Actions and three self-hosted runners.
Start the self-hosted runners
- on the development machine (
wksi
) open ssh sessions to the build machines (xbbli
,xbbla
,xbbla32
,xbbmi
,xbbma
):
caffeinate ssh xbbli
caffeinate ssh xbbla
caffeinate ssh xbbla32
caffeinate ssh xbbmi
caffeinate ssh xbbma
Create a screen
session, to provide a persistent standard output for
the runners:
screen -S ga
To quit it, use # Ctrl-a Ctrl-d
.
For xbbli
& xbbla
, which have more memory, start two runners:
~/actions-runners/xpack-dev-tools/1/run.sh &
~/actions-runners/xpack-dev-tools/2/run.sh &
On all other machines start a single runner:
~/actions-runners/xpack-dev-tools/run.sh &
To check the status of the runners locally:
ps -a | grep 'Runner.Listener'
To kill them:
killall Runner.Listener
Check the status of the runners
The status of all self-hosted runners is available at the GitHub Runners page.
Publish the helper
Publish a new release of the helper on npmjs.
Update the dependency to the new helper
In build-assets/package.json
, update the reference
to "@xpack-dev-tools/xbb-helper":
(possibly in all xPack projects).
Commit the changes to build-assets/package.json
Use the message build-assets/package.json: bump deps.
Check for disk space
Check if the build machines have enough free space and eventually do some cleanups:
xpm run check-space -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
This is equivalent to running a df
command.
- macOS
- GNU/Linux
df -gH /
df -BG -H /
To free the space used by all previous builds of all xpack-dev-tools
packages:
xpm run clear-all-project-builds -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets
This is equivalent to running a wide rm
to remove the build
folders:
rm -rf ~/Work/xpack-dev-tools/*/build-assets/build
Generate the GitHub workflows
Run the generate-workflows xPack action to update the GitHub workflow files.
xpm run generate-workflows -C ~/Work/xpack-dev-tools/bison-xpack.git/build-assets