-noble`).
#
# The job runs as root in the container; git is in the image, so checkout
# clones as usual, and Chromium runs without its sandbox as Playwright does
# for root on its own. Each slice boots its own server on its own SQLite
# file (e2e/server-launch.mjs) and runs the setup project before its share
# of the app flows.
client-e2e:
name: 'Client e2e ${{ matrix.shard }}/2'
needs: changes
if: needs.changes.outputs.code == 'true'
runs-on: ubuntu-24.04
timeout-minutes: 15
container:
image: mcr.microsoft.com/playwright:v1.65.0-noble@sha256:9beb546099e11f52d62e505537914870e01a67e0e6b00ae41da4775324e73dbe
options: --ipc=host
strategy:
fail-fast: false
matrix:
shard: [1, 2]
steps:
- uses: actions/checkout@v6
# The run boots the real server (e2e/server-launch.mjs builds it and
# starts it against a throwaway SQLite file) and the Vite dev server in
# front of it, so it needs every workspace.
- uses: ./.github/actions/setup
with:
scope: full
build-shared: 'true'
# The public and app projects are the regression flows. The screenshot
# and help-media projects make pictures for the wiki and the help center
# and assert nothing, so they stay out of CI.
- name: Run a slice of the end-to-end flows
run: >-
cd client && npx playwright test --project=public --project=app
--shard=${{ matrix.shard }}/2
- name: Upload traces
if: failure()
uses: actions/upload-artifact@v6
with:
name: e2e-results-${{ matrix.shard }}
path: client/test-results/
retention-days: 7
# The image every release ships, built the way docker.yml builds it and
# started the way docker-compose.yml starts it (read-only root, every
# capability dropped but the three the entrypoint needs, bind-mounted data
# and uploads). Nothing else in CI builds the Dockerfile: the unit tests only
# read it as text, which is how a Dockerfile that could not be parsed
# reached dev unseen. It is separate from the vulnerability scan on purpose,
# so a newly published CVE cannot hide a broken build or the other way round.
#
# The Helm chart and the compose files are what self-hosters deploy from,
# and before this check only a release ever touched them. The chart is
# linted, rendered with the defaults and with each secret and persistence
# mode, and the output checked against the Kubernetes API schemas. Both
# tools come from their release pages, pinned by version and checksum.
#
# One job for both: the image steps run when the image area changed, the
# chart and compose steps when the deploy area changed, exactly as the two
# jobs this replaced did. Each step names the step it builds on, so a red
# chart step does not skip the image steps or the other way round.
docker:
name: 'Docker and Helm'
needs: changes
if: needs.changes.outputs.image == 'true' || needs.changes.outputs.deploy == 'true'
runs-on: ubuntu-24.04
timeout-minutes: 15
env:
HELM_VERSION: v3.22.0
HELM_SHA256: 1e4ab49e429626cf6c6958d914248b78c9730803c2751b87627e171dc800e7bb
KUBECONFORM_VERSION: v0.8.0
KUBECONFORM_SHA256: 9bc2bffbf71f261128533edaf912153948b7ff238f9a531ae6d34466ec287883
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Install helm and kubeconform
id: helm-install
if: needs.changes.outputs.deploy == 'true'
run: |
set -euo pipefail
bin="$RUNNER_TEMP/bin"
mkdir -p "$bin"
cd "$RUNNER_TEMP"
# GitHub's release downloads answer 504 now and then: retry, the checksum still decides.
fetch() { curl -sSfL --retry 5 --retry-delay 15 --retry-all-errors -o "$1" "$2"; }
fetch helm.tgz "https://get.helm.sh/helm-$HELM_VERSION-linux-amd64.tar.gz"
echo "$HELM_SHA256 helm.tgz" | sha256sum -c -
tar -xzf helm.tgz --strip-components=1 -C "$bin" linux-amd64/helm
fetch kubeconform.tgz "https://github.com/yannh/kubeconform/releases/download/$KUBECONFORM_VERSION/kubeconform-linux-amd64.tar.gz"
echo "$KUBECONFORM_SHA256 kubeconform.tgz" | sha256sum -c -
tar -xzf kubeconform.tgz -C "$bin" kubeconform
echo "$bin" >> "$GITHUB_PATH"
- name: Lint the chart
if: ${{ !cancelled() && steps.helm-install.outcome == 'success' }}
run: helm lint --strict charts/trek
- name: Render the chart
id: helm-render
if: ${{ !cancelled() && steps.helm-install.outcome == 'success' }}
run: |
set -euo pipefail
out="$RUNNER_TEMP/rendered"
mkdir -p "$out"
helm template trek charts/trek > "$out/defaults.yaml"
helm template trek charts/trek --set existingSecret=trek-secrets > "$out/existing-secret.yaml"
helm template trek charts/trek --set generateEncryptionKey=true > "$out/generated-key.yaml"
helm template trek charts/trek \
--set persistence.enabled=false \
--set ingress.enabled=true \
--set 'ingress.tls[0].secretName=trek-tls' \
--set 'ingress.tls[0].hosts[0]=chart-example.local' \
--set 'extraEnv[0].name=SESSION_DURATION' \
--set 'extraEnv[0].value=7d' \
--set 'extraEnvFrom[0].secretRef.name=trek-smtp' \
--set 'extraVolumes[0].name=kitinerary-extractors' \
--set 'extraVolumes[0].configMap.name=trek-kitinerary-extractors' \
--set 'extraVolumeMounts[0].name=kitinerary-extractors' \
--set 'extraVolumeMounts[0].mountPath=/usr/local/share/kitinerary/extractors' \
--set 'extraVolumeMounts[0].readOnly=true' \
--set probes.liveness=null \
> "$out/variants.yaml"
helm template trek charts/trek \
--set httpRoute.enabled=true \
--set 'httpRoute.parentRefs[0].name=gateway' \
> "$out/http-route.yaml"
helm template trek charts/trek \
--set 'persistence.data.accessModes[0]=ReadWriteMany' \
--set 'persistence.uploads.accessModes[0]=ReadWriteMany' \
> "$out/access-modes.yaml"
helm template trek charts/trek \
--set podSecurityContext.runAsUser=1000 \
--set podSecurityContext.runAsGroup=1000 \
--set podSecurityContext.runAsNonRoot=true \
--set podSecurityContext.seccompProfile.type=RuntimeDefault \
--set securityContext.allowPrivilegeEscalation=false \
--set 'securityContext.capabilities.drop[0]=ALL' \
> "$out/non-root.yaml"
helm template trek charts/trek \
--set ingress.enabled=true \
--set fullnameOverride=custom-trek \
--set service.port=8080 \
--set-json 'httpRoute={"enabled":true,"parentRefs":[{"name":"gateway","namespace":"gateway-system","sectionName":"https"}],"hostnames":["trek.example.com"],"paths":["/","/api"],"annotations":{"example.com/note":"custom route"},"timeouts":{"request":"0s","backendRequest":"0s"}}' \
> "$out/http-route-variants.yaml"
- name: Check the manifests against the Kubernetes schemas
# The pinned Gateway API schema adds HTTPRoute validation without skipping unknown resources.
if: ${{ !cancelled() && steps.helm-render.outcome == 'success' }}
run: >-
kubeconform -strict -summary
-schema-location default
-schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/041baf4c1d3740ca21461cab765e7432ecd073ef/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json'
"$RUNNER_TEMP"/rendered/*.yaml
- name: Check every compose file
if: ${{ !cancelled() && needs.changes.outputs.deploy == 'true' }}
run: |
set -euo pipefail
shopt -s nullglob
files=(docker-compose*.yml docker-compose*.yaml)
[ "${#files[@]}" -gt 0 ] || { echo "::error::No compose file found."; exit 1; }
for f in "${files[@]}"; do
echo "$f"
docker compose -f "$f" config --quiet
done
- name: Set up buildx
id: buildx
if: ${{ !cancelled() && needs.changes.outputs.image == 'true' }}
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
# Only pushes write the layer cache. Pull requests read what the last
# push to their base left, so a stream of PRs cannot evict it and the
# cache stays within the repository's quota.
- name: Build the image
id: image-build
if: ${{ !cancelled() && steps.buildx.outcome == 'success' }}
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
with:
context: .
platforms: linux/amd64
push: false
load: true
tags: trek:ci
build-args: |
APP_VERSION=0.0.0-ci
cache-from: type=gha,scope=trek-image
cache-to: ${{ github.event_name == 'push' && 'type=gha,mode=max,scope=trek-image' || '' }}
- name: Start the container
id: image-start
if: ${{ !cancelled() && steps.image-build.outcome == 'success' }}
run: |
mkdir -p "$RUNNER_TEMP/trek/data" "$RUNNER_TEMP/trek/uploads"
docker run -d --name trek-ci -p 127.0.0.1:3000:3000 \
--read-only --tmpfs /tmp:noexec,nosuid,size=128m \
--security-opt no-new-privileges:true \
--cap-drop ALL --cap-add CHOWN --cap-add SETUID --cap-add SETGID --cap-add KILL \
-v "$RUNNER_TEMP/trek/data:/app/data" \
-v "$RUNNER_TEMP/trek/uploads:/app/uploads" \
-e ENCRYPTION_KEY="$(openssl rand -hex 32)" \
trek:ci
# Ready means migrated, seeded and answering from the database. A first
# boot migrates an empty database, which takes seconds; three minutes
# is the bound, not the expectation.
- name: Wait until the server is ready
id: image-ready
if: ${{ !cancelled() && steps.image-start.outcome == 'success' }}
run: |
for _ in $(seq 1 90); do
if [ "$(docker inspect -f '{{.State.Running}}' trek-ci)" != "true" ]; then
echo "::error::The container exited before it became ready."
exit 1
fi
if curl -sf http://127.0.0.1:3000/api/health/ready; then
echo
exit 0
fi
sleep 2
done
echo "::error::/api/health/ready did not answer within 180 seconds."
exit 1
- name: Check what the image serves
id: image-serves
if: ${{ !cancelled() && steps.image-ready.outcome == 'success' }}
run: |
set -euo pipefail
version="$(docker exec trek-ci cat /app/server/VERSION)"
[ "$version" = "0.0.0-ci" ] || { echo "::error::VERSION holds '$version'"; exit 1; }
curl -sf http://127.0.0.1:3000/ | grep -q '' \
|| { echo "::error::/ does not serve the client"; exit 1; }
# Same capabilities as the shipped compose file: dumb-init has to be able
# to pass the stop signal on to node, or docker stop ends in SIGKILL (137).
- name: Stop gracefully
id: image-stop
if: ${{ !cancelled() && steps.image-serves.outcome == 'success' }}
run: |
set -euo pipefail
docker stop -t 30 trek-ci >/dev/null
code="$(docker inspect -f '{{.State.ExitCode}}' trek-ci)"
[ "$code" = "0" ] || { echo "::error::docker stop ended with exit code $code, not a graceful shutdown"; exit 1; }
docker logs trek-ci 2>&1 | grep 'SIGTERM received' >/dev/null \
|| { echo "::error::the server never saw SIGTERM"; exit 1; }
# The same image on a Postgres database (TREK_DB_URL): an empty database
# gets the baseline schema, the admin is seeded, and the server answers
# ready. The database runs beside it on a private network, never exposed.
# It is the newest supported major, the one the image's client tools are
# for; server-postgres covers the oldest.
- name: Start the container on Postgres
id: pg-start
if: ${{ !cancelled() && steps.image-stop.outcome == 'success' }}
run: |
set -euo pipefail
docker network create trek-ci-pg
docker run -d --name trek-ci-db --network trek-ci-pg \
-e POSTGRES_PASSWORD=ci -e POSTGRES_DB=trek postgres:18
for _ in $(seq 1 30); do
docker exec trek-ci-db pg_isready -U postgres -d trek >/dev/null 2>&1 && break
sleep 2
done
docker exec trek-ci-db pg_isready -U postgres -d trek \
|| { echo "::error::Postgres did not accept connections within 60 seconds."; exit 1; }
mkdir -p "$RUNNER_TEMP/trek-pg/data" "$RUNNER_TEMP/trek-pg/uploads"
docker run -d --name trek-ci-on-pg --network trek-ci-pg -p 127.0.0.1:3001:3000 \
--read-only --tmpfs /tmp:noexec,nosuid,size=128m \
--security-opt no-new-privileges:true \
--cap-drop ALL --cap-add CHOWN --cap-add SETUID --cap-add SETGID --cap-add KILL \
-v "$RUNNER_TEMP/trek-pg/data:/app/data" \
-v "$RUNNER_TEMP/trek-pg/uploads:/app/uploads" \
-e ENCRYPTION_KEY="$(openssl rand -hex 32)" \
-e TREK_DB_URL=postgres://postgres:ci@trek-ci-db:5432/trek \
trek:ci
# Twice: the first start creates the schema in the empty database, the
# restart finds it migrated and must start without touching it.
- name: Wait until the server on Postgres is ready, then again after a restart
id: pg-ready
if: ${{ !cancelled() && steps.pg-start.outcome == 'success' }}
run: |
wait_ready() {
for _ in $(seq 1 90); do
if [ "$(docker inspect -f '{{.State.Running}}' trek-ci-on-pg)" != "true" ]; then
echo "::error::The container on Postgres exited before it became ready."
exit 1
fi
if curl -sf http://127.0.0.1:3001/api/health/ready; then
echo
return 0
fi
sleep 2
done
echo "::error::/api/health/ready on Postgres did not answer within 180 seconds."
exit 1
}
wait_ready
docker restart -t 30 trek-ci-on-pg >/dev/null
wait_ready
- name: Stop the container on Postgres gracefully
if: ${{ !cancelled() && steps.pg-ready.outcome == 'success' }}
run: |
set -euo pipefail
docker stop -t 30 trek-ci-on-pg >/dev/null
code="$(docker inspect -f '{{.State.ExitCode}}' trek-ci-on-pg)"
[ "$code" = "0" ] || { echo "::error::docker stop ended with exit code $code, not a graceful shutdown"; exit 1; }
- name: Container log
if: always() && needs.changes.outputs.image == 'true'
run: docker logs trek-ci 2>&1 || true
- name: Container log on Postgres
if: always() && needs.changes.outputs.image == 'true'
run: |
docker logs trek-ci-on-pg 2>&1 || true
docker logs trek-ci-db 2>&1 | tail -50 || true
scan:
name: 'Sonar'
runs-on: ubuntu-24.04
timeout-minutes: 20
# Fork PRs and Dependabot runs don't receive SONAR_TOKEN, so skip the scan
# rather than fail it. The release pull request mirrors the push run, whose
# scan already covers dev, so it skips the scan too.
if: >-
needs.changes.outputs.mirror != 'true' &&
github.actor != 'dependabot[bot]' &&
(github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository)
# Coverage jobs failing must stop the scan: a missing lcov file would make
# every line of that package report as uncovered, and the dashboard would
# show a collapse that did not happen. The gates run beside the tests, so
# they are listed here too, and a failed gate still stops the scan.
# scripts/ci/workflow-jobs.test.mjs fails while a job that uploads a
# sonar-*-coverage artifact is missing here.
needs:
- changes
- shared
- server-types
- server-lint
- client-types
- client-checks
- client-lint
- server-coverage
- client-coverage
steps:
- uses: actions/checkout@v6
with:
# Sonar reads git blame to date each issue and to decide what counts
# as new code. A shallow clone leaves it blind to both.
fetch-depth: 0
- name: Download server coverage
uses: actions/download-artifact@v7
with:
name: sonar-server-coverage
path: server/coverage/
- name: Download client coverage
uses: actions/download-artifact@v7
with:
name: sonar-client-coverage
path: client/coverage/
- name: Download shared coverage
uses: actions/download-artifact@v7
with:
name: sonar-shared-coverage
path: shared/coverage/
- name: Download plugin-sdk coverage
uses: actions/download-artifact@v7
with:
name: sonar-plugin-sdk-coverage
path: plugin-sdk/coverage/
- name: Scan
uses: SonarSource/sonarqube-scan-action@59202af334b4ba7b5db01b701b8e79f8b3bef0f4 # v8.5.0
env:
# SonarQube Cloud is the scan action's default host, so no
# SONAR_HOST_URL — the token is a sonarcloud.io token for the
# liketrek org.
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
with:
# New Code is configured as "previous version", so this string is what
# draws that line — changing it re-bases every "new code" metric.
# Pinned rather than read from package.json on purpose: package.json is
# still on 3.4.1 while the project has been analysed as 4.0.0 all along,
# and taking the lower number would silently move the baseline. Bump it
# here as part of the release, together with the version bump.
#
# Everything else — sources, tests, exclusions, the documented
# false-positive list — lives in sonar-project.properties so it stays
# reviewable in a PR rather than buried in workflow YAML.
args: -Dsonar.projectVersion=4.0.0
# The single check the branch ruleset requires. It needs every job that
# gates a merge, so listing a job here is what makes it required. The scan
# stays out (see the layout notes at the top).
ci-ok:
needs:
- changes
- shared
- server-types
- server-lint
- server-tests
- server-coverage
- server-postgres
- server-s3
- client-types
- client-checks
- client-lint
- client-tests
- client-coverage
- client-e2e
- docker
# always(): without it a failed dependency would skip this job, and the
# ruleset reads a skipped required check as passed.
if: always()
runs-on: ubuntu-24.04
# 20 rather than 5: on the release pull request the mirror step waits for
# the push run's ci-ok, and gives up two minutes before this kills it.
timeout-minutes: 20
# The mirror reads the push run's jobs; nothing here writes.
permissions:
actions: read
checks: read
contents: read
steps:
- name: Every needed job passed or was skipped
env:
NEEDS: ${{ toJSON(needs) }}
run: |
set -euo pipefail
echo "$NEEDS" | jq -r 'to_entries[] | "\(.value.result)\t\(.key)"' | sort
failed="$(echo "$NEEDS" | jq -r '[to_entries[] | select(.value.result != "success" and .value.result != "skipped") | .key] | join(", ")')"
if [ -n "$failed" ]; then
echo "::error::Not passed: $failed"
exit 1
fi
# On the release pull request every job above skipped, so the step above
# only proves that changes passed. The verdict is the push run's: wait
# for the ci-ok of the push run of this workflow on dev for exactly the
# pull request's head commit, and pass or fail with it. No push run, a
# cancelled one or a timeout fails, with what to do in the log.
- name: Check out the mirror script
if: needs.changes.outputs.mirror == 'true'
uses: actions/checkout@v6
with:
persist-credentials: false
sparse-checkout: scripts/ci
- uses: actions/setup-node@v6
if: needs.changes.outputs.mirror == 'true'
with:
node-version-file: .nvmrc
- name: Mirror the ci-ok of the push run on dev
if: needs.changes.outputs.mirror == 'true'
timeout-minutes: 19
env:
GITHUB_TOKEN: ${{ github.token }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: node scripts/ci/mirror-push-run.mjs