Skip to content

Latest commit

 

History

History
118 lines (86 loc) · 27.7 KB

File metadata and controls

118 lines (86 loc) · 27.7 KB

Artemis CI/CD Restructuring

Date: TBD | Data verified against live GitHub API + workflow files, May 2026.
P1 = this iteration, critical | P2 = this iteration | P3 = next iteration | P4 = backlog


North Star Goal

Goal: from pushed commit to testable feature on a test server — as fast as possible.

Step Today Realistic with optimisation What it takes
Build (WAR + Docker) ~12 min ~4–5 min Faster build runners + warm caches + external_builder to eliminate duplicate Angular compilation
Deployment trigger Manual (minutes to hours) ~0 min Auto-trigger deploy when build succeeds + PR has an active Helios lock
Deploy + server start ~5–6 min ~3–4 min Faster test server hardware
Total ~20 min + human delay ~8–10 min ≤ 5 min requires hot reload

The root problem is the build, not the deployment. Eliminating duplicate Angular compilation and adding Docker layer caching saves ~8 min per push today. The remaining floor is Spring Boot startup: 1–3 min just to start the server. Cold-container ≤5 min total is not achievable with current hardware regardless of deployment automation.

The real decision this meeting needs to make:

Path What it means Realistic cycle time
A — Eliminate build waste + fast hardware Wire external_builder, add Docker layer cache, provision faster runners and test server VMs ~8–10 min
B — Hot reload Spring Boot DevTools / JRebel on test servers — skip full restart, upload only changed classes. Doesn't work for DB migrations or structural changes. ~3–5 min for code changes
C — Accept ~10 min Eliminate the waste, don't invest in hardware beyond current, accept the cold-start floor ~10–12 min

1. Feedback Speed

Topic Problem Options Trade-offs Recommendation Owner By When
P2 — Module-affected test selection get_changed_modules.sh already runs on draft PRs. Full suite (~28 min) runs on every non-draft PR regardless of how many modules changed. A) Module-affected on all PRs, full only on develop push B) Keep current C) Full always A is faster but risks missing cross-module regressions on PRs. B catches more but is slow for small changes. C is slowest, highest confidence. Cross-module regressions are rare in practice and caught on develop push anyway. Recommend A — the cross-module risk is low and caught on develop; the speed gain is real for every PR.
P3 — Draft PR CI policy Draft PRs already skip the full suite and run module-affected tests only. Question is whether drafts should get even less. A) Style/compile only on draft B) Keep current (module-affected on draft already) A gives faster feedback on in-progress work but risks merging to ready-for-review with obvious failures. B is already a reasonable cut. Recommend B — module-affected on draft is a good balance; restricting further risks surprises at review time.
P2 — Docker build duplicates Angular compilation build.yml has two parallel independent jobs: one builds the WAR; one builds the Docker image, recompiling Angular + WAR from source. external_builder Dockerfile stage exists but the Docker job has no needs: build dependency and never uses it. Angular is compiled twice per push. A) Wire external_builder: add needs: build, download WAR artifact, pass --build-arg WAR_FILE_STAGE=external_builder B) Not now A saves ~8 min per push and eliminates duplicate compilation. Downside: requires restructuring the job graph and coordination with ls1intum/.github. Risk of breaking the Docker build if WAR artifact download fails. B has zero risk but wastes time on every push indefinitely. Recommend A — the job graph restructuring is straightforward; the saving compounds across every PR.
P3 — Path-based CI skipping paths-ignore is inconsistent across 26 workflows. build.yml push trigger has no paths-ignore at all — a docs-only commit triggers a full build. A) Audit and align across all 26 workflows Maintenance overhead of keeping path lists aligned across 26 files vs. developers waiting for CI on changes that can't affect the build. Low technical risk. Recommend A — annoying for contributors to wait for a full CI run on a README change; alignment is low effort.
P2 — Docker layer cache No cache-from/cache-to configured anywhere. Every build re-downloads the same base image layers from scratch. Industry: 50–80% build time reduction for unchanged layers. A) Add type=gha cache to the ls1intum/.github reusable Docker workflow B) Keep as-is A is a free speed win with no real downside; requires coordinating a change to the upstream org workflow. B wastes build time on every run indefinitely. Recommend A — no meaningful trade-off; the only cost is the coordination with the org repo.
P3 — E2E waits for build to fully finish E2E triggers via workflow_run on "Build" — cannot start until build.yml completes including Docker push, adding ~14 min of mandatory waiting. A) pull_request trigger with internal-repo guard (runs in parallel with build) B) Keep workflow_run A is significantly faster but pull_request doesn't automatically have secret access for fork PRs — needs a guard (github.event.pull_request.head.repo.full_name == github.repository). B is safe but slow. Recommend A — the internal-repo guard is standard practice; the 14-min saving is significant at scale.
P3 — Gradle cache across runners Dependency cache stored per runner via actions/setup-java. No Gradle remote build cache. If jobs land on different aet-large-ubuntu slots, compiled build outputs are not shared. A) Configure Gradle remote build cache (shared across pool) B) Check runner stickiness first A adds complexity (remote cache server, credentials, invalidation logic) with uncertain benefit if runners are already sticky. B is low-effort investigation that might reveal the local cache already works fine. Recommend B first — measure before adding infrastructure; if runners rotate, then pursue A.

2. Pipeline Reliability

Topic Problem Options Trade-offs Recommendation Owner By When
P1 — Runner pool is too small 73.7% of test.yml runs fail. Both server-tests and client-tests run on aet-large-ubuntu. A confirmed run waited 6 hours in the queue (queued 17:24, started 22:34). Starved runners cause Testcontainers PostgreSQL to fail on startup (PSQLException, SQLSTATE(08006)). Root cause of the failure rate. A) Add static VMs to the pool (near-zero if TUM infra available) B) ARC — Kubernetes elastic scaling C) GitHub-hosted 8-core (~€300/month for test.yml alone) D) Static floor + ARC burst A is fastest and cheapest if TUM has VM capacity — no new infrastructure. B scales to zero when idle but needs K8s and weeks of setup. C is expensive (~€500/month total) and unjustifiable for academic OSS. D is the right long-term answer but not the immediate fix. Recommend A immediately, evaluate B for long-term — fastest path to relief; ARC can come later without disrupting the short-term fix. Bring current pool size and available VM capacity to meeting.
P1 — No way to tell if a failure is infra or code Developers read raw Gradle logs to tell if server-tests failed because of their code or a Testcontainers crash. These look identical in PR status. When CI is red 73% of the time, developers stop trusting it. A) Post-test step greps log for PSQLException/SQLSTATE(08006) and posts a clear annotation B) Keep as-is A is 30 min of work with immediate trust improvement. B means every developer wastes time investigating failures that aren't theirs. No real downside to A. Recommend A — 30 min, 15 lines of bash, immediate improvement in developer confidence.
P2 — @tumaet/apollon async cleanup modeling-assessment.component.spec.ts consistently throws "7 unhandled errors" — Apollon schedules React state updates that outlive the Vitest DOM lifecycle. Confirmed bug, unrelated to code changes. A) Fix the test isolation (assign to modeling assessment maintainer) B) Quarantine with @Disabled A solves the problem permanently. B is safe but kicks the can — the test stays broken and a fix date is easy to miss. Recommend A — assign at this meeting; quarantine is not a plan.
P3 — No server-side flaky test tracking flaky_test_detection.sh exists for E2E. Nothing equivalent for server tests. No way to distinguish a consistently flaky test from a real regression without manually comparing run histories. A) Trunk.io — zero-config, automatic quarantine, OSS free B) BuildPulse — richer analytics, OSS free C) Manual A requires no code changes — just a YAML step. B has better historical analysis but more setup. Both are free for OSS. Manual doesn't scale. Recommend A (Trunk.io) — the team already has capacity issues; zero-config is the right call here.

3. What Guards the Main Branch

Confirmed required checks (GitHub API): client-tests | server-tests | client-style | server-style | Build .war artifactstrict: true, 4 reviews required, enforce_admins: false.

Topic Problem Options Trade-offs Recommendation Owner By When
P1 — Coverage thresholds are never enforced 20 modules have per-module thresholds (64.6%–94.2%) in gradle/jacoco.gradle. JaCoCo verification step has continue-on-error: true (line 138). Coverage can drop below threshold on any module without blocking a merge. A) Remove continue-on-error: true now — confirm develop passes all 20 thresholds first B) Lower thresholds to current passing baseline first, then ratchet up C) Keep advisory A immediately makes coverage real, but if develop is already below threshold it instantly blocks all merges. B is safe regardless — establish a true baseline before enforcing. C means the infrastructure is permanently theatre. Critical pre-meeting action: run ./gradlew test jacocoTestCoverageVerification -x webapp on a clean develop and bring the result. Recommend A if develop is clean, B if it's not — the result of that pre-meeting run determines everything.
P1 — Coverage reporter runs duplicate tests pullrequest-coverage-reporter.yml runs npm run coverage:pr (module-affected server tests for changed modules) on aet-large-ubuntu independently of test.yml. Not a required check. ~23% of all aet-large-ubuntu capacity wasted daily — directly worsens runner starvation. A) Delete it — test.yml step summary already shows per-module coverage B) Download JaCoCo artifact from test.yml and post PR comment without re-running A is immediate and frees capacity; loses the PR comment but the step summary is equivalent. B preserves the comment but costs ~1 day and the comment adds little over the step summary. Recommend A — delete it. The PR comment is redundant; freeing 23% of runner capacity is not.
P2 — E2E tests don't block merges test-e2e.yml posts commit status "End-to-End (E2E) Tests" but confirmed NOT in required_status_checks.contexts. PRs merge with failing E2E today. 26.3% of E2E runs fail on infra. A) Add as required now B) Add after runner pool is stabilised C) Keep advisory A would block 26.3% of PRs on infra failures until D1 is fixed — more pain than it prevents right now. B is the right outcome but blocked on D1. C permanently accepts E2E regression risk. Recommend B — set a concrete trigger: add E2E as required once the E2E failure rate drops below 5%.
P3 — Admins can bypass all checks enforce_admins: false confirmed via API. Admins can merge any PR regardless of CI status and the 4-review requirement. A) Enable enforce_admins: true B) Keep with a documented emergency process A eliminates the bypass entirely. B retains an escape hatch for genuine emergencies (security hotfixes, broken develop) at the cost of governance trust. Recommend B — academic open source projects legitimately need an emergency bypass; document the process explicitly so it's not silently abused.
P3 — Bean instantiation check uses H2 bean-instantiations.yml passes jdbc:h2:mem:mydb as a JVM argument. It is a required check. H2 is not PostgreSQL — Liquibase migrations using PostgreSQL-specific SQL pass this check silently. A) Replace with PostgreSQL Testcontainer (~+2 min overhead) B) Remove as required check C) Keep H2 A gives real confidence at minor overhead cost. B removes false confidence but loses the gate entirely. C is the worst option — false confidence is worse than no check. Recommend A — 2 min overhead is cheap for a required check that actually means something.
P4 — MySQL compatibility test is disabled test-mysql.yml is workflow_dispatch only with comment # TODO: Re-enable once MySQL testcontainers are reliable. No owner. A) Re-enable on nightly schedule B) Delete A restores MySQL testing but risks unstable nightly builds with no owner to fix it. B accepts MySQL compatibility as untested. If Artemis supports MySQL in production, this is a real gap. Recommend B unless MySQL is actively used in production — a disabled test with no owner and no issue is already abandoned; formalise that or assign an owner with a deadline.

4. Test Server and Delivery Pipeline

Root issue: the build takes ~12 min. Everything else here is downstream of that.

With a 12-min build and a 1-hour Helios lock, a developer spends 20% of their slot just waiting for the first deployment — before they've written a single line of feedback. Every subsequent push repeats that wait. The deployment trigger (manual re-trigger via Helios) adds more human delay on top. Fix the build first; then the rest of this section becomes tractable.

Topic Problem Options Trade-offs Recommendation Owner By When
P1 — The build takes 12 min This is the root bottleneck. Duplicate Angular compilation (build.yml Docker job ignores the already-built WAR), no Docker layer cache, no external_builder wiring — all solvable today without new hardware. If waste is eliminated and the build is still too slow, then buy faster runners. A) Fix waste first (Section 1: external_builder + Docker layer cache) → target ~4–5 min B) Buy faster build runners C) Both A costs engineering time, no money. B costs money, minimal engineering time. The right answer depends on what A actually delivers — benchmark after. Start with A — measurable, fixable waste exists today. Eliminate it and measure. Then decide whether hardware is still needed.
P1 — Every deployment requires a manual re-trigger Within a Helios lock, every push requires manually re-triggering the deployment workflow. testserver-deployment.yml already handles the full deploy flow. The only missing piece: trigger it automatically when a build succeeds and the PR has an active Helios lock. A) Auto-trigger testserver-deployment.yml from build.yml on build success, if Helios reports an active lock on this PR B) Keep manual A eliminates human delay entirely. The deployment machinery already exists. The only new requirement is a Helios API endpoint to query whether a PR has an active lock — if it doesn't exist, Helios needs to add it. Recommend A — the workflow exists. The only open question is Helios lock state API. Confirm at this meeting: does Helios expose this?
P2 — 6 VMs for ~80 open PRs At peak, more than 6 developers need a test server simultaneously. No software change fixes the ratio — but the ratio can be improved without adding 6 more VMs. A single-node Artemis instance needs 3 containers and ~8–10 GB RAM (app heap alone is 5 GB). A 32 GB VM can host 2–3 environments; a 64 GB VM can host 4–5. The question is whether Helios supports capacity-aware slot allocation. A) Multiple environments per VM — run N Docker Compose stacks on each VM, isolated by port+subdomain; Traefik routes pr-123.test7.artemis.tum.de → the right container; Helios tracks free slots per VM. Multiplies capacity without new hardware. B) Dynamic VM provisioning — Helios calls Hetzner Cloud API (or equivalent) to spin up a fresh VM on demand when a lock is requested, tears it down when the lock expires. A 16 GB Hetzner VM costs ~€0.07/hour — a 1-hour session is €0.07. Truly elastic, no idle waste. C) Kubernetes with namespace-per-PR — Helm chart per PR; helm install pr-123; DNS via ingress. Fully elastic. Blocked by: no Helm charts exist, LocalCI requires Docker socket (hard in K8s without DinD/Sysbox). Weeks of migration work. D) More static VMs — cheapest and fastest if TUM has spare VM capacity; still has a fixed ceiling. E) Priority queue — PRs in active code review jump the queue; no new hardware needed but doesn't eliminate waits. A multiplies existing capacity with no recurring cost — the main new piece is Helios capacity-aware scheduling + Traefik setup (~1–2 week effort). B is truly elastic but adds cloud spend and Helios needs cloud API integration; also adds ~1–2 min VM startup latency per first deploy. C is the right long-term answer for a large OSS project but the migration is 4–8 weeks and blocked on Docker socket issues for LocalCI. D is the fastest to ship but doesn't scale beyond the ceiling you set. E is a band-aid. Recommend A in the short term, B as the medium-term target. A makes each existing VM work harder. B removes the ceiling entirely. Together they eliminate slot contention without a K8s migration. Pre-meeting data needed: peak simultaneous demand and current VM RAM capacity.

5. Cost and Runner Efficiency

Topic Problem Options Trade-offs Recommendation Owner By When
P1 — Coverage reporter wastes scarce runner capacity pullrequest-coverage-reporter.yml runs module-affected server tests on aet-large-ubuntu (timeout: 80 min). Not a required check. ~23% of all aet-large-ubuntu capacity consumed daily by a duplicate, non-required check. A) Delete B) Reuse JaCoCo artifact from test.yml Same trade-off as Section 3. A frees capacity immediately; B preserves the PR comment. The PR comment is redundant. Recommend A — see Section 3.
P2 — client-tests competes for aet-large-ubuntu Both server-tests AND client-tests run on aet-large-ubuntu. client-style, server-style, client-tests-selected, client-compilation all run on ubuntu-latest (free for public OSS). client-tests has no technical requirement for a self-hosted runner. A) Move client-tests to ubuntu-latest B) Keep on aet-large-ubuntu A immediately frees one aet-large-ubuntu slot per run with no technical downside — client tests are npm/TypeScript, not PostgreSQL. B is the current state. No meaningful argument for B. Recommend A — move it. There is no reason client-tests should be on a self-hosted runner.
P3 — Artifact storage uncapped 90-day default retention on all upload-artifact steps. No retention-days set anywhere across all 26 workflows. WAR ~50 MB × ~6 builds/day × 90 days = ~27 GB uncapped. A) Add retention-days: 14 — brings to ~4 GB steady-state No real trade-off. 14 days covers any realistic debugging window. Recommend A — do it.
P4 — No policy for adding new heavy workflows Any contributor can add a workflow targeting aet-large-ubuntu with no review of cost. Runner consumption grows silently. A) Require a runner-minute estimate in any PR adding a self-hosted workflow B) Keep ad-hoc A adds lightweight process that prevents surprises. B has zero overhead but allows unchecked growth. Recommend A — one sentence in the PR template; low cost, prevents the current situation from recurring.

6. Security and Supply Chain

Topic Problem Options Trade-offs Recommendation Owner By When
P1 — Almost no actions are pinned to an immutable SHA 1 of 28 unique third-party actions is SHA-pinned. Two are branch-pinned: codacy/codacy-coverage-reporter-action@master (has CODACY_PROJECT_TOKEN) and octokit/request-action@v2.x (v2.x is a branch, used in deployment workflows). 25 others are mutable tag-pinned. A) Add pinDigests: true to renovate.json — Renovate will open a PR that SHA-pins all actions and then maintains those pins automatically on every upstream release B) Run npx pin-github-actions once manually, rely on Renovate for future updates A is the right approach: Renovate already runs and covers GitHub Actions under config:recommended; adding pinDigests: true means pinning and updates are both automated with no manual step. B front-loads a large manual diff and duplicates what Renovate can own entirely. Recommend A — one line in renovate.json. Renovate does the rest.
P1 — Every job runs with near-unlimited permissions build.yml, test.yml, test-e2e.yml have no permissions: block. Default: write access to Actions, Checks, Contents, Deployments, Issues, Packages, Pages, Pull Requests on every job. A) Add permissions: contents: read workflow default + per-job overrides B) Keep as-is A eliminates the blast radius of a compromised step. Small risk of breaking jobs that depend on implicit permissions (they'll fail with a clear 403). B leaves every step with a capability it almost certainly doesn't need. Recommend A — fail secure. Any job that breaks will give a clear permission error that's easy to fix.
P3 — Renovate config is nearly empty renovate.json exists and uses config:recommended (covers npm, Gradle, GitHub Actions). But no automerge rules, no grouping, no schedule, and no pinDigests. The bot runs but PRs pile up unreviewed and the SHA-pinning benefit is not realised. A) Add pinDigests: true (see Q1), add automerge for patch-level updates (automerge: true, matchUpdateTypes: ["patch"]), group minor updates by ecosystem to reduce PR noise B) Keep minimal config A makes Renovate actually useful: patches auto-merge without review overhead, and grouping reduces the ~20 weekly PRs that otherwise get ignored. B means Renovate opens PRs that nobody reviews — security updates land but slowly. Recommend A — configure Renovate properly now that it's already running.
P2 — CodeQL scanning the wrong Java version codeql-analysis.yml uses java: 21. All other workflows (test.yml, build.yml, bean-instantiations.yml, test-mysql.yml) use Java 25. Java 25 code in security-sensitive paths is not scanned. A) Change java: 21java: 25 No trade-off. A 5-minute change that closes a real gap. Recommend A — do it.
P2 — Shell injection in deployment workflow testserver-deployment.yml:74: BRANCH_NAME=${{ github.event.inputs.branch_name }} — expression interpolated directly in a run: block. Branch names with shell metacharacters cause unexpected behavior. A) Move to env: block, reference as $BRANCH_NAME Low exploitation risk (org-member only triggers workflow_dispatch). But the pattern is wrong and sets a bad precedent for contributors copying it. Fix is 10 minutes. Recommend A — fix it; the risk is low but the pattern matters.
P3 — Long-lived secrets where OIDC would work VAULT_TOKEN is a long-lived secret in the production deployment workflow. A leaked token is valid indefinitely. HashiCorp Vault natively supports OIDC JWT auth from GitHub Actions. A) Replace VAULT_TOKEN with OIDC in the prod deployment workflow B) Keep stored secret A eliminates a long-lived credential permanently. Requires one-time coordination with Vault admin to configure the JWT auth backend. B has zero setup cost but carries indefinite credential exposure risk. Recommend A — the one-time setup cost is worth permanently eliminating the stored secret.
P4 — No release artifact provenance or SBOM Releases built with no cryptographic provenance. actions/upload-release-asset@v1 (archived 2021) is the current upload mechanism. A) Replace archived upload action + add SLSA Level 1 (actions/attest-build-provenance) + CycloneDX SBOM B) Skip A is increasingly expected for OSS and may become required under EU Cyber Resilience Act. The archived upload action is a separate problem worth fixing regardless. B accepts the gap. Recommend A — low effort, and the archived action needs replacing anyway; combine them.

7. Open Source Contributor Experience

Topic Problem Options Trade-offs Recommendation Owner By When
P3 — Fork PRs have no documented policy 6 workflows use pull_request_target: pullrequest-closed, pullrequest-labeler, pullrequest-opened, pullrequest-ready-to-merge-validator, pullrequest-readyforreview, validate-pr-description. Each runs with base-branch secrets. No fork CI policy documented. A) Audit all 6, confirm no untrusted checkout+execute, add fork skip conditions, document B) Keep ad-hoc A is a security audit that needs to happen regardless of contributor experience concerns — pull_request_target without proper guards is a privilege escalation risk. B leaves the repo potentially exposed and contributors confused. Recommend A — this is a security audit as much as a contributor experience improvement.
P3 — No nightly or scheduled test suite Every workflow triggers on PR push or branch push only. Slow tests (full E2E, MySQL) can't run on every PR without hurting feedback speed. Gradual regressions go undetected between releases. A) Add nightly run on develop: full E2E + MySQL + dependency audit B) No A catches regressions early and enables tests too slow for PRs. Adds another thing that can break and needs ownership. Once D1 (runners) is fixed, the infra cost is low. B accepts the gap. Recommend A, but after D1 — a nightly run on a flaky runner pool just creates nightly noise. Fix the runners first.
P2 — No alerting when develop CI breaks No notification when develop CI turns red. A regression merges; nobody knows until the next PR fails. A) Slack webhook in test.yml and build.yml conditioned on develop failure B) GitHub native watch A is active and reaches the whole team. B requires each person to opt in. 30 min of work for A. Recommend A — opt-in doesn't work in practice; team-wide alerting does.
P3 — CI failure messages are hard to understand Some annotation exists (dorny/test-reporter, ashley-taylor/junit-report-annotations-action) but coverage is inconsistent and infra failures look identical to code failures. A) Standardise annotations across all jobs + add infra failure detection step B) Keep as-is A improves contributor experience significantly but takes time across many jobs. The infra failure annotation (Section 2) is the highest-value piece and should be done first regardless. Recommend A in two steps — ship the infra annotation first (P1, 30 min), then standardise the rest across jobs.
P4 — No CI documentation for contributors CONTRIBUTING.md covers local testing but not CI structure, which checks are required, or why CI is red 73% of the time. A) Add "CI pipeline" section to CONTRIBUTING.md Low effort, straightforwardly useful. Best written after the restructuring is done so it reflects the new state. Recommend A, after restructuring settles — don't document a pipeline you're actively changing.