Documentation: Add documentation on jenkins builders
Put this in a new directory called 'infrastructure' and make a link and an index.md file for the directory. Signed-off-by: Martin Roth <martin@coreboot.org> Change-Id: I54a0204e7525a25f2fd717a73007b304aac67396 Reviewed-on: https://review.coreboot.org/c/coreboot/+/43984 Tested-by: build bot (Jenkins) <no-reply@coreboot.org> Reviewed-by: Angel Pons <th3fanbus@gmail.com>
This commit is contained in:
Martin Roth
Patrick Georgi
parent
b339fbb79b
commit
2cee2ff256
3 changed files with 399 additions and 0 deletions
|
|
@ -187,5 +187,6 @@ Contents:
|
|||
* [SuperIO](superio/index.md)
|
||||
* [Vendorcode](vendorcode/index.md)
|
||||
* [Utilities](util.md)
|
||||
* [coreboot infrastructure](infrastructure/index.md)
|
||||
* [Release notes for past releases](releases/index.md)
|
||||
* [Flashing firmware tutorial](flash_tutorial/index.md)
|
||||
|
|
|
|||
392
Documentation/infrastructure/builders.md
Normal file
392
Documentation/infrastructure/builders.md
Normal file
|
|
@ -0,0 +1,392 @@
|
|||
# Jenkins builder setup and configuration
|
||||
|
||||
## How to set up a new jenkins builder
|
||||
|
||||
### Contact a jenkins admin
|
||||
|
||||
Let a jenkins admin know that you’re interested in setting up a jenkins
|
||||
build system.
|
||||
|
||||
For a permanent build system, this should generally be a dedicated
|
||||
machine that is not generally being used for other purposes. The
|
||||
coreboot builds are very intensive.
|
||||
|
||||
It's also best to be aware that although we don't know of any security
|
||||
issues, the jenkins-node image is run with the privileged flag which
|
||||
gives the container root access to the build machine. See
|
||||
[this article](https://blog.trendmicro.com/trendlabs-security-intelligence/why-running-a-privileged-container-in-docker-is-a-bad-idea/)
|
||||
about why this is discouraged.
|
||||
|
||||
It's recommended that you give an admin root access on your machine so
|
||||
that they can reset it in case of a failure. This is not a requirement,
|
||||
as the system can just be disabled until someone is available to fix any
|
||||
issues.
|
||||
|
||||
Currently active Jenkins admins:
|
||||
* Patrick Georgi:
|
||||
* Email: [patrick@georgi-clan.de](mailto:patrick@georgi-clan.de)
|
||||
* IRC: pgeorgi
|
||||
|
||||
|
||||
### Build Machine requirements
|
||||
|
||||
For a builder, we need a fast system with lots of threads and plenty of
|
||||
RAM. The builder builds and stores the git repos and output in tmpfs
|
||||
along with the ccache save area, so if there isn't enough memory, the
|
||||
builds will slow down because of smaller ccache areas and can run into
|
||||
"out of storage space" errors.
|
||||
|
||||
#### Current Build Machines
|
||||
|
||||
To give an idea of what a suitable build machine might be, currently the
|
||||
coreboot project has 3 active jenkins build machines.
|
||||
|
||||
* Congenialbuilder - 128 threads, 256GiB RAM
|
||||
* Fastest Passing coreboot gerrit build: 4 min, 30 sec
|
||||
* Slowest Passing coreboot gerrit build: 9 min, 56 sec
|
||||
|
||||
|
||||
* Gleeful builder - 64 thread, 64GiB RAM
|
||||
* Fastest Passing coreboot gerrit build: 6 min, 6 sec
|
||||
* Slowest Passing coreboot gerrit build, 34 min
|
||||
|
||||
|
||||
* Ultron (9elements) - 48 threads, 128GiB RAM
|
||||
* Fastest Passing coreboot gerrit build: 6 min, 32 sec
|
||||
* Slowest Passing coreboot gerrit build: 44 min
|
||||
|
||||
|
||||
### Jenkins Builds
|
||||
|
||||
There are a number of builds handled by the coreboot jenkins builders,
|
||||
for a number of different projects - coreboot, flashrom, memtest86+,
|
||||
em100, etc. Many of these have builders for their current master branch
|
||||
as well as gerrit and coverity builds.
|
||||
|
||||
You can see all the builds here:
|
||||
[https://qa.coreboot.org/](https://qa.coreboot.org/)
|
||||
|
||||
Most of the time on the builders is taken up by the coreboot master and
|
||||
gerrit builds.
|
||||
|
||||
* [coreboot gerrit build](https://qa.coreboot.org/job/coreboot-gerrit/)
|
||||
([Time trend](https://qa.coreboot.org/job/coreboot-gerrit/buildTimeTrend))
|
||||
|
||||
|
||||
* [coreboot master build](https://qa.coreboot.org/job/coreboot/)
|
||||
([Time trend](https://qa.coreboot.org/job/coreboot/buildTimeTrend))
|
||||
|
||||
|
||||
### Stress test the machine
|
||||
|
||||
Test the machine to make sure that building won't stress the hardware
|
||||
too much. Install stress-ng, then run the stress test for at least an
|
||||
hour.
|
||||
|
||||
On a system with 32 cores, it was tested with this command:
|
||||
|
||||
```
|
||||
$ stress-ng --cpu 20 --io 6 --vm 6 --vm-bytes 1G --verify --metrics-brief -t 60m
|
||||
```
|
||||
|
||||
You can watch the temperature with the sensors package or with ‘acpi -t’
|
||||
if your machine supports that.
|
||||
|
||||
You can check for thermal throttling by running this command and seeing
|
||||
if the values go down on any of the cores after it's been running for a
|
||||
while.
|
||||
|
||||
```
|
||||
$ while [ true ]; do clear; cat /proc/cpuinfo | grep 'cpu MHz' ; sleep 1; done
|
||||
```
|
||||
|
||||
If the machine throttles or resets, you probably need to upgrade the
|
||||
cooling system.
|
||||
|
||||
|
||||
## jenkins-server docker installation
|
||||
|
||||
|
||||
### Manual Installation
|
||||
|
||||
If you’ve met all the above requirements, and an admin has agreed to set
|
||||
up the builder in jenkins, you’re ready to go on to the next steps.
|
||||
|
||||
|
||||
### Set up your network so jenkins can talk to the container
|
||||
|
||||
Expose a local port through any firewalls you might have on your router.
|
||||
This would generally be in the port forwarding section, and you'd just
|
||||
forward a port (typically 49151) from the internet directly to the
|
||||
builder’s IP address.
|
||||
|
||||
You might also want to set up a port to forward to port 22 on your
|
||||
machine and set up openssh so you or the jenkins admins can manage
|
||||
the machine remotely (if you allow them).
|
||||
|
||||
|
||||
### Install and set up docker
|
||||
|
||||
Install docker by following the
|
||||
[directions](https://docs.docker.com/engine/install/) on the docker
|
||||
site. These instructions keep changing, so just check the latest
|
||||
information.
|
||||
|
||||
|
||||
#### Set up environment variables
|
||||
|
||||
To make configuration and the later commands easier, these should go in
|
||||
your shell's .rc file. Note that you only need to set them if you're
|
||||
using something other than the default.
|
||||
|
||||
```
|
||||
# Set the port used on your machine to connect to jenkins.
|
||||
export COREBOOT_JENKINS_PORT=49151
|
||||
|
||||
# Set the revision of the container from docker hub
|
||||
export DOCKER_COMMIT=65718760fa
|
||||
|
||||
# Set the location of where the jenkins cache directory will be.
|
||||
export COREBOOT_JENKINS_CACHE_DIR="/srv/docker/coreboot-builder/cache"
|
||||
|
||||
# Set the name of the container
|
||||
export COREBOOT_JENKINS_CONTAINER="coreboot_jenkins"
|
||||
```
|
||||
|
||||
Make sure any variables needed are set in your environment before
|
||||
continuing to the next step.
|
||||
|
||||
|
||||
### Using the Makefile for docker installation
|
||||
|
||||
From the coreboot directory, run
|
||||
|
||||
```
|
||||
make -C util/docker help
|
||||
```
|
||||
|
||||
This will show you the available targets and variables needed:
|
||||
|
||||
```
|
||||
Commands for working with docker images:
|
||||
coreboot-sdk - Build coreboot-sdk container
|
||||
upload-coreboot-sdk - Upload coreboot-sdk to hub.docker.com
|
||||
coreboot-jenkins-node - Build coreboot-jenkins-node container
|
||||
upload-coreboot-jenkins-node - Upload coreboot-jenkins-node to hub.docker.com
|
||||
doc.coreboot.org - Build doc.coreboot.org container
|
||||
clean-coreboot-containers - Remove all docker coreboot containers
|
||||
clean-coreboot-images - Remove all docker coreboot images
|
||||
docker-clean - Remove docker coreboot containers & images
|
||||
|
||||
Commands for using docker images
|
||||
docker-build-coreboot - Build coreboot under coreboot-sdk
|
||||
<BUILD_CMD=target>
|
||||
docker-abuild - Run abuild under coreboot-sdk
|
||||
<ABUILD_ARGS='-a -B'>
|
||||
docker-what-jenkins-does - Run 'what-jenkins-does' target
|
||||
docker-shell - Bash prompt in coreboot-jenkins-node
|
||||
<USER=root or USER=coreboot>
|
||||
docker-jenkins-server - Run coreboot-jenkins-node image (for server)
|
||||
docker-jenkins-attach - Open shell in running jenkins server
|
||||
docker-build-docs - Build the documentation
|
||||
docker-livehtml-docs - Run sphinx-autobuild
|
||||
|
||||
Variables:
|
||||
COREBOOT_JENKINS_PORT=49151
|
||||
COREBOOT_JENKINS_CACHE_DIR=/srv/docker/coreboot-builder/cache
|
||||
COREBOOT_JENKINS_CONTAINER=coreboot_jenkins
|
||||
COREBOOT_IMAGE_TAG=f2741aa632f
|
||||
DOCKER_COMMIT=65718760fa
|
||||
```
|
||||
|
||||
### Set up the system for the jenkins builder
|
||||
|
||||
As a regular user - *Not root*, run:
|
||||
|
||||
```
|
||||
sudo mkdir -p ${COREBOOT_JENKINS_CACHE_DIR}
|
||||
sudo mkdir -p ${COREBOOT_JENKINS_CCACHE_DIR}
|
||||
sudo chown $(whoami):$(whoami) ${COREBOOT_JENKINS_CCACHE_DIR}
|
||||
sudo chown $(whoami):$(whoami) ${COREBOOT_JENKINS_CACHE_DIR}
|
||||
wget http://www.dediprog.com/save/78.rar/to/EM100Pro.rar
|
||||
mv EM100Pro.rar ${COREBOOT_JENKINS_CACHE_DIR}
|
||||
```
|
||||
|
||||
### Install the coreboot jenkins builder
|
||||
|
||||
```
|
||||
make -C util/docker docker-jenkins-server
|
||||
```
|
||||
|
||||
Your installation is complete on your side.
|
||||
|
||||
### Tell the Admins that the machine is set up
|
||||
Let the admins know that the builder is set up so they can set up the
|
||||
machine profile on qa.coreboot.org.
|
||||
|
||||
They need to know:
|
||||
* Your external IP address or domain name. If you don’t have a static
|
||||
IP, make sure you have a dynamic dns hostname configured.
|
||||
* The port on your machine and firewall that’s exposed for jenkins:
|
||||
`$COREBOOT_JENKINS_PORT`
|
||||
* The core count of the machine.
|
||||
* How much memory is available on the machine. This helps determine
|
||||
the amount of memory used for ccache.
|
||||
|
||||
|
||||
### First build
|
||||
On the first build after a machine is reset, it will frequently take
|
||||
20-25 minutes to do the entire what-jenkins-does build while the ccache
|
||||
is getting filled up and the entire coreboot repo gets downloaded. As
|
||||
the ccache gets populated, the build time will drop.
|
||||
|
||||
|
||||
## Additional Information
|
||||
|
||||
|
||||
### How to log in to the docker instance for debugging
|
||||
```
|
||||
$ make -C util/docker docker-jenkins-attach
|
||||
$ su coreboot
|
||||
$ cd ~/slave-root/workspace
|
||||
$ bash
|
||||
```
|
||||
|
||||
|
||||
WARNING: This should not be used to make changes to the build system,
|
||||
but just to debug issues. Changes to the build system are highly
|
||||
discouraged as it leads to situations where patches can pass the build
|
||||
testing on one builder and fail on another builder. Any changes that are
|
||||
made in the image will be lost on the next update, so if you
|
||||
accidentally change something, you can remove the containers and images
|
||||
and update to get a fresh installation.
|
||||
|
||||
|
||||
### How to download containers/images for a fresh installation and remove old containers
|
||||
|
||||
To delete the old containers & images:
|
||||
|
||||
```
|
||||
$ docker stop $COREBOOT_JENKINS_CONTAINER
|
||||
$ docker rm $COREBOOT_JENKINS_CONTAINER
|
||||
$ docker images # lists all existing images
|
||||
$ docker rmi XXXX # Use the image ID found in the above command.
|
||||
```
|
||||
|
||||
To get and run the new coreboot-jenkins image, change the value in the
|
||||
`DOCKER_COMMIT` variable to the new image value.
|
||||
|
||||
```
|
||||
$ make -C util/docker docker-jenkins-server
|
||||
```
|
||||
|
||||
#### Getting ready to push the docker images
|
||||
|
||||
Set up an account on hub.docker.com
|
||||
|
||||
Get an admin to add the account to the coreboot team on hub.docker.com
|
||||
|
||||
[https://hub.docker.com/u/coreboot/dashboard/teams/?team=owners](https://hub.docker.com/u/coreboot/dashboard/teams/?team=owners)
|
||||
|
||||
Make sure your credentials are configured on your host machine by
|
||||
running
|
||||
|
||||
```
|
||||
$ docker login
|
||||
```
|
||||
|
||||
This will prompt you for your docker username, password, and your email
|
||||
address, and write out to ~/.docker/config.json. Without this file, you
|
||||
won’t be able to push the images.
|
||||
|
||||
#### Updating the Dockerfiles:
|
||||
|
||||
The coreboot-sdk Dockerfile will need to be updated when any additional
|
||||
dependencies are added. Both the coreboot-sdk and the
|
||||
coreboot-jenkins-node Dockerfiles will need to be updated to the new
|
||||
version number and git commit id anytime the toolchain is updated. Both
|
||||
files are stored in the coreboot repo under coreboot/util/docker.
|
||||
|
||||
Read the [dockerfile best practices](https://docs.docker.com/v1.8/articles/dockerfile_best-practices/)
|
||||
page before updating the files.
|
||||
|
||||
#### Rebuilding the coreboot-sdk docker image to update the toolchain:
|
||||
|
||||
```
|
||||
$ make -C util/docker coreboot-sdk
|
||||
```
|
||||
|
||||
This takes a relatively long time.
|
||||
|
||||
#### Test the coreboot-sdk docker image:
|
||||
|
||||
There are two methods of running the docker image - interactively as a
|
||||
shell, or doing the build directly. Running interactively as a shell is
|
||||
useful for early testing, because it allows you to update the image
|
||||
(without any changes getting saved) and re-test builds. This saves the
|
||||
time of having to rebuild the image for every issue you find.
|
||||
|
||||
#### Running the docker image interactively:
|
||||
|
||||
Run:
|
||||
|
||||
```
|
||||
$ make -C util/docker docker-jenkins-server
|
||||
$ make -C util/docker docker-jenkins-attach
|
||||
```
|
||||
|
||||
#### Running the build directly:
|
||||
|
||||
From the coreboot directory:
|
||||
|
||||
```
|
||||
$ make -C util/docker docker-build-coreboot
|
||||
```
|
||||
|
||||
You’ll also want to test building the other projects and payloads:
|
||||
ChromeEC, flashrom, memtest86+, em100, Grub2, SeaBIOS, iPXE, coreinfo,
|
||||
nvramcui, tint...
|
||||
|
||||
#### Pushing the coreboot-sdk image to hub.docker.com for use:
|
||||
|
||||
When you’re satisfied with the testing, push the coreboot-sdk image to
|
||||
the hub.docker.com
|
||||
|
||||
```
|
||||
$ make -C util/docker upload-coreboot-sdk
|
||||
```
|
||||
|
||||
#### Building and pushing the coreboot-jenkins-node docker image:
|
||||
|
||||
This docker image is pretty simple, so there’s not really any testing
|
||||
that needs to be done.
|
||||
|
||||
```
|
||||
$ make -C util/docker coreboot-jenkins-node
|
||||
$ make -C util/docker upload-coreboot-jenkins-node
|
||||
```
|
||||
|
||||
### Coverity Setup
|
||||
|
||||
To run coverity jobs, the builder needs to have the tools available, and
|
||||
to be marked as a coverity builder.
|
||||
|
||||
|
||||
#### Set up the Coverity tools
|
||||
|
||||
Download the Linux-64 coverity build tool and decompress it into your
|
||||
cache directory as defined by the `$COREBOOT_JENKINS_CACHE_DIR` variable
|
||||
|
||||
[https://scan.coverity.com/download](https://scan.coverity.com/download)
|
||||
|
||||
Rename the directory from its original name
|
||||
(cov-analysis-linux64-7.7.0.4) to ‘coverity’, or better, create a
|
||||
symlink:
|
||||
|
||||
```
|
||||
ln -s cov-analysis-linux64-7.7.0.4 coverity
|
||||
```
|
||||
|
||||
|
||||
Let the admins know that the ‘coverity’ label can be added to the
|
||||
builder.
|
||||
6
Documentation/infrastructure/index.md
Normal file
6
Documentation/infrastructure/index.md
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
# coreboot infrastructure
|
||||
|
||||
This section contains documentation about coreboot infrastructure
|
||||
|
||||
## Jenkins builders and builds
|
||||
[Setting up Jenkins build machines](builders.md)
|
||||
Loading…
Reference in a new issue