12 KiB
Jenkins builder setup and configuration
How to set up a new jenkins builder
Contact a jenkins admin
Let a jenkins admin know that you’re interested in setting up a jenkins build system.
For a permanent build system, this should generally be a dedicated machine that is not generally being used for other purposes. The coreboot builds are very intensive.
It's also best to be aware that although we don't know of any security issues, the jenkins-node image is run with the privileged flag which gives the container root access to the build machine. See this article about why this is discouraged.
It's recommended that you give an admin root access on your machine so that they can reset it in case of a failure. This is not a requirement, as the system can just be disabled until someone is available to fix any issues.
Currently active Jenkins admins:
- Patrick Georgi:
- Email: patrick@georgi-clan.de
- IRC: pgeorgi
Build Machine requirements
For a builder, we need a fast system with lots of threads and plenty of RAM. The builder builds and stores the git repos and output in tmpfs along with the ccache save area, so if there isn't enough memory, the builds will slow down because of smaller ccache areas and can run into "out of storage space" errors.
Current Build Machines
To give an idea of what a suitable build machine might be, currently the coreboot project has 3 active jenkins build machines.
-
Congenialbuilder - 128 threads, 256GiB RAM
- Fastest Passing coreboot gerrit build: 4 min, 30 sec
- Slowest Passing coreboot gerrit build: 9 min, 56 sec
-
Gleeful builder - 64 thread, 64GiB RAM
- Fastest Passing coreboot gerrit build: 6 min, 6 sec
- Slowest Passing coreboot gerrit build, 34 min
-
Ultron (9elements) - 48 threads, 128GiB RAM
- Fastest Passing coreboot gerrit build: 6 min, 32 sec
- Slowest Passing coreboot gerrit build: 44 min
Jenkins Builds
There are a number of builds handled by the coreboot jenkins builders, for a number of different projects - coreboot, flashrom, memtest86+, em100, etc. Many of these have builders for their current master branch as well as gerrit and coverity builds.
You can see all the builds here: https://qa.coreboot.org/
Most of the time on the builders is taken up by the coreboot master and gerrit builds.
Stress test the machine
Test the machine to make sure that building won't stress the hardware too much. Install stress-ng, then run the stress test for at least an hour.
On a system with 32 cores, it was tested with this command:
$ stress-ng --cpu 20 --io 6 --vm 6 --vm-bytes 1G --verify --metrics-brief -t 60m
You can watch the temperature with the sensors package or with ‘acpi -t’ if your machine supports that.
You can check for thermal throttling by running this command and seeing if the values go down on any of the cores after it's been running for a while.
$ while [ true ]; do clear; cat /proc/cpuinfo | grep 'cpu MHz' ; sleep 1; done
If the machine throttles or resets, you probably need to upgrade the cooling system.
jenkins-server docker installation
Manual Installation
If you’ve met all the above requirements, and an admin has agreed to set up the builder in jenkins, you’re ready to go on to the next steps.
Set up your network so jenkins can talk to the container
Expose a local port through any firewalls you might have on your router. This would generally be in the port forwarding section, and you'd just forward a port (typically 49151) from the internet directly to the builder’s IP address.
You might also want to set up a port to forward to port 22 on your machine and set up openssh so you or the jenkins admins can manage the machine remotely (if you allow them).
Install and set up docker
Install docker by following the directions on the docker site. These instructions keep changing, so just check the latest information.
Set up environment variables
To make configuration and the later commands easier, these should go in your shell's .rc file. Note that you only need to set them if you're using something other than the default.
# Set the port used on your machine to connect to jenkins.
export COREBOOT_JENKINS_PORT=49151
# Set the revision of the container from docker hub
export DOCKER_COMMIT=65718760fa
# Set the location of where the jenkins cache directory will be.
export COREBOOT_JENKINS_CACHE_DIR="/srv/docker/coreboot-builder/cache"
# Set the name of the container
export COREBOOT_JENKINS_CONTAINER="coreboot_jenkins"
Make sure any variables needed are set in your environment before continuing to the next step.
Using the Makefile for docker installation
From the coreboot directory, run
make -C util/docker help
This will show you the available targets and variables needed:
Commands for working with docker images:
coreboot-sdk - Build coreboot-sdk container
upload-coreboot-sdk - Upload coreboot-sdk to hub.docker.com
coreboot-jenkins-node - Build coreboot-jenkins-node container
upload-coreboot-jenkins-node - Upload coreboot-jenkins-node to hub.docker.com
doc.coreboot.org - Build doc.coreboot.org container
clean-coreboot-containers - Remove all docker coreboot containers
clean-coreboot-images - Remove all docker coreboot images
docker-clean - Remove docker coreboot containers & images
Commands for using docker images
docker-build-coreboot - Build coreboot under coreboot-sdk
<BUILD_CMD=target>
docker-abuild - Run abuild under coreboot-sdk
<ABUILD_ARGS='-a -B'>
docker-what-jenkins-does - Run 'what-jenkins-does' target
docker-shell - Bash prompt in coreboot-jenkins-node
<USER=root or USER=coreboot>
docker-jenkins-server - Run coreboot-jenkins-node image (for server)
docker-jenkins-attach - Open shell in running jenkins server
docker-build-docs - Build the documentation
docker-livehtml-docs - Run sphinx-autobuild
Variables:
COREBOOT_JENKINS_PORT=49151
COREBOOT_JENKINS_CACHE_DIR=/srv/docker/coreboot-builder/cache
COREBOOT_JENKINS_CONTAINER=coreboot_jenkins
COREBOOT_IMAGE_TAG=f2741aa632f
DOCKER_COMMIT=65718760fa
Set up the system for the jenkins builder
As a regular user - Not root, run:
sudo mkdir -p ${COREBOOT_JENKINS_CACHE_DIR}
sudo mkdir -p ${COREBOOT_JENKINS_CCACHE_DIR}
sudo chown $(whoami):$(whoami) ${COREBOOT_JENKINS_CCACHE_DIR}
sudo chown $(whoami):$(whoami) ${COREBOOT_JENKINS_CACHE_DIR}
wget http://www.dediprog.com/save/78.rar/to/EM100Pro.rar
mv EM100Pro.rar ${COREBOOT_JENKINS_CACHE_DIR}
Install the coreboot jenkins builder
make -C util/docker docker-jenkins-server
Your installation is complete on your side.
Tell the Admins that the machine is set up
Let the admins know that the builder is set up so they can set up the machine profile on qa.coreboot.org.
They need to know:
- Your external IP address or domain name. If you don’t have a static IP, make sure you have a dynamic dns hostname configured.
- The port on your machine and firewall that’s exposed for jenkins:
$COREBOOT_JENKINS_PORT - The core count of the machine.
- How much memory is available on the machine. This helps determine the amount of memory used for ccache.
First build
On the first build after a machine is reset, it will frequently take 20-25 minutes to do the entire what-jenkins-does build while the ccache is getting filled up and the entire coreboot repo gets downloaded. As the ccache gets populated, the build time will drop.
Additional Information
How to log in to the docker instance for debugging
$ make -C util/docker docker-jenkins-attach
$ su coreboot
$ cd ~/slave-root/workspace
$ bash
WARNING: This should not be used to make changes to the build system, but just to debug issues. Changes to the build system are highly discouraged as it leads to situations where patches can pass the build testing on one builder and fail on another builder. Any changes that are made in the image will be lost on the next update, so if you accidentally change something, you can remove the containers and images and update to get a fresh installation.
How to download containers/images for a fresh installation and remove old containers
To delete the old containers & images:
$ docker stop $COREBOOT_JENKINS_CONTAINER
$ docker rm $COREBOOT_JENKINS_CONTAINER
$ docker images # lists all existing images
$ docker rmi XXXX # Use the image ID found in the above command.
To get and run the new coreboot-jenkins image, change the value in the
DOCKER_COMMIT variable to the new image value.
$ make -C util/docker docker-jenkins-server
Getting ready to push the docker images
Set up an account on hub.docker.com
Get an admin to add the account to the coreboot team on hub.docker.com
https://hub.docker.com/u/coreboot/dashboard/teams/?team=owners
Make sure your credentials are configured on your host machine by running
$ docker login
This will prompt you for your docker username, password, and your email address, and write out to ~/.docker/config.json. Without this file, you won’t be able to push the images.
Updating the Dockerfiles:
The coreboot-sdk Dockerfile will need to be updated when any additional dependencies are added. Both the coreboot-sdk and the coreboot-jenkins-node Dockerfiles will need to be updated to the new version number and git commit id anytime the toolchain is updated. Both files are stored in the coreboot repo under coreboot/util/docker.
Read the dockerfile best practices page before updating the files.
Rebuilding the coreboot-sdk docker image to update the toolchain:
$ make -C util/docker coreboot-sdk
This takes a relatively long time.
Test the coreboot-sdk docker image:
There are two methods of running the docker image - interactively as a shell, or doing the build directly. Running interactively as a shell is useful for early testing, because it allows you to update the image (without any changes getting saved) and re-test builds. This saves the time of having to rebuild the image for every issue you find.
Running the docker image interactively:
Run:
$ make -C util/docker docker-jenkins-server
$ make -C util/docker docker-jenkins-attach
Running the build directly:
From the coreboot directory:
$ make -C util/docker docker-build-coreboot
You’ll also want to test building the other projects and payloads: ChromeEC, flashrom, memtest86+, em100, Grub2, SeaBIOS, iPXE, coreinfo, nvramcui, tint...
Pushing the coreboot-sdk image to hub.docker.com for use:
When you’re satisfied with the testing, push the coreboot-sdk image to the hub.docker.com
$ make -C util/docker upload-coreboot-sdk
Building and pushing the coreboot-jenkins-node docker image:
This docker image is pretty simple, so there’s not really any testing that needs to be done.
$ make -C util/docker coreboot-jenkins-node
$ make -C util/docker upload-coreboot-jenkins-node
Coverity Setup
To run coverity jobs, the builder needs to have the tools available, and to be marked as a coverity builder.
Set up the Coverity tools
Download the Linux-64 coverity build tool and decompress it into your
cache directory as defined by the $COREBOOT_JENKINS_CACHE_DIR variable
https://scan.coverity.com/download
Rename the directory from its original name (cov-analysis-linux64-7.7.0.4) to ‘coverity’, or better, create a symlink:
ln -s cov-analysis-linux64-7.7.0.4 coverity
Let the admins know that the ‘coverity’ label can be added to the builder.