OntoPortal deployments

Kubernetes-first deployment project for three related targets:

1. Clean OntoPortal: upstream-compatible OntoPortal deployment with no MatPortal runtime patches enabled.
2. AgroPortal-style clean profile: same baseline model, but using AgroPortal-friendly defaults that can be adjusted without changing chart templates.
3. MatPortal: MatPortal-specific profile that enables MatPortal branding, optional Matomo/FAIR/assistant hooks, and explicitly gated runtime patches.

The chart is k3s first and also runs on larger Kubernetes clusters. Docker Compose is generated from the same Helm values profiles. Environment recipes combine profiles, provider overlays, optional source-image build plans, Terraform settings, and Compose output.

Current status

This is a deployment project, not a certified production release. See docs/validation-evidence.md for tested paths and gaps. Before production, create site-specific overlays for secrets, hostnames, image pinning, storage, backup/restore, TLS, autoscaling, and monitoring.

Prerequisites

• python3 with pyyaml for validation and Compose generation.
• node and npm for Playwright UI-test syntax checks and browser smoke tests.
• helm for chart lint/render/install.
• kubectl for live k3s deployment and smoke tests.
• Optional: docker compose for Compose validation/runs, Restic for backups, and Terraform/OpenTofu for terraform/.

Repository layout

chart/ontoportal/                  Helm chart; shared source of truth
values/profiles/                   Clean OntoPortal, AgroPortal, MatPortal, k3s, Compose profiles
values/cloud/                      AWS EKS, Azure AKS, and GKE provider overlays plus RWX examples
values/addons/                     Optional Matomo, FAIR, HPA/KEDA/VPA, monitoring, Trivy, SonarQube values
values/image-builds/               Optional source-repository image build plans for OntoPortal/BioPortal/AgroPortal
environments/                      Modular integrated deployment recipes
compose/generated/                 Rendered Compose files; committed snapshots generated by make compose-all
images/                            Optional in-repository multi-architecture image build definitions
examples/                          External Secrets and observability starter manifests
scripts/                           Deployment, smoke-test, maintenance, Compose/environment rendering
scripts/addons/                    Add-on helpers that do not belong in the app chart
terraform/                         Helm-on-k3s Terraform deployment
terraform/examples/                tfvars examples for clean and MatPortal profiles
docs/                              k3s tutorials, HA, backups, maintenance, security, observability, production, and operations docs

Clone and validate

git clone git@github.com:matportal/ontoportal-deployment.git
cd ontoportal-deployment
python3 -m pip install --user pyyaml
make validate
make validate-generated
make validate-ui-tests
# Optional; requires Docker daemon.
make compose-config
helm lint chart/ontoportal -f values/profiles/ontoportal-clean.yaml -f values/profiles/k3s-local.yaml
helm template ontoportal chart/ontoportal --namespace ontoportal \
  -f values/profiles/ontoportal-clean.yaml \
  -f values/profiles/k3s-local.yaml >/tmp/ontoportal-clean.yaml

compose/generated/* is committed as generated output. Regenerate it with make compose-all whenever chart values or scripts/render-compose.py change, then confirm git diff only contains expected generated updates.

Secret handling

The inline --set secrets.*=... examples below are for disposable development clusters only. For long-lived environments, create or sync a Kubernetes Secret and install with secrets.create=false and secrets.existingSecret=<name> so values do not leak through shell history, process lists, or Helm release history. Terraform set_sensitive_values still stores values in Terraform state; protect state or prefer an external secret manager.

See chart/ontoportal/README.md for the secret key contract.

Recommended k3s tutorial

For a laptop or single server, prefer the full tutorial over the short examples below:

• docs/tutorial-k3s-laptop-server.md: install k3s, create Secrets, configure local/server hostnames, deploy, smoke test, upgrade, and uninstall.

The short commands below are useful for quick validation, but they intentionally omit explanations and some safety checks.

Fast path: clean OntoPortal on k3s

kubectl create namespace ontoportal || true
helm upgrade --install ontoportal chart/ontoportal \
  --namespace ontoportal \
  -f values/profiles/ontoportal-clean.yaml \
  -f values/profiles/k3s-local.yaml \
  --set global.createNamespace=false \
  --set secrets.apiKey='replace-with-random-token' \
  --set secrets.admin.password='replace-with-admin-password' \
  --set secrets.mysqlRootPassword='replace-with-db-password'

scripts/smoke.sh ontoportal ontoportal

The same install can be run through the wrapper:

HELM_EXTRA_ARGS="-f values/profiles/k3s-local.yaml \
  --set secrets.apiKey=replace-with-random-token \
  --set secrets.admin.password=replace-with-admin-password \
  --set secrets.mysqlRootPassword=replace-with-db-password" \
  scripts/deploy.sh ontoportal-clean ontoportal ontoportal

Fast path: MatPortal on k3s

HELM_EXTRA_ARGS="-f values/profiles/k3s-local.yaml \
  -f values/addons/matomo.yaml \
  -f values/addons/fairness.yaml \
  --set secrets.apiKey=replace-with-random-token \
  --set secrets.admin.password=replace-with-admin-password \
  --set secrets.mysqlRootPassword=replace-with-db-password \
  --set secrets.matomoDbPassword=replace-with-matomo-password \
  --set secrets.matomoDbRootPassword=replace-with-matomo-root-password" \
  scripts/deploy.sh matportal matportal matportal

Fast path: Docker Compose

Compose is generated from the same profiles:

make compose-all
cp compose/generated/.env.ontoportal-clean.sample .env
# edit .env
docker compose --env-file .env -f compose/generated/docker-compose.ontoportal-clean.yml up -d

MatPortal Compose:

python3 scripts/render-compose.py \
  -f values/profiles/matportal.yaml \
  -f values/profiles/docker-compose.yaml \
  -f values/addons/matomo.yaml \
  -f values/addons/fairness.yaml
cp compose/generated/.env.matportal.sample .env.matportal
# edit .env.matportal
docker compose --env-file .env.matportal -f compose/generated/docker-compose.matportal.yml up -d

Modular environment recipes

Use environments/*.yaml when you want one integrated target definition rather than a long list of ad hoc commands. Recipes combine a base profile, cloud/provider overlays, optional add-ons, optional source-image build plans, and Terraform/Compose outputs.

make environment ENV=ontoportal-compose-existing
make environment ENV=bioportal-compose-source ENV_RENDER_ARGS="--image-tag sha-validated"
make environment ENV=agroportal-aws-eks ENV_RENDER_ARGS="--image-tag sha-validated"

Rendered bundles are written under dist/environments/<name>/ and include merged Helm values, a build matrix, optional image-values.yaml, terraform.tfvars, and Compose files where requested. See docs/modular-environments.md.

Image source builds

Deployments can continue to use existing images through images.*. For a source-build flow, add a values file from values/image-builds/ or define a site-specific imageBuilds overlay, run the manual build-source-images GitHub workflow, then include the generated or rendered image-values.yaml with the immutable image tag accepted by your release process.

python3 scripts/image-build-matrix.py -f values/image-builds/ontoportal-source.yaml
python3 scripts/render-environment.py environments/bioportal-compose-source.yaml --image-tag sha-validated

This supports the public OntoPortal/BioPortal/AgroPortal pattern where some teams run published images while others mount or build from application repositories during development. Source-build components can publish to GHCR and mirror to Docker Hub in the same workflow run via publishImages. MatPortal deployments may require private or unreleased source repositories/images; the public CI profile validates wiring with substitute images, not the complete private MatPortal stack. See docs/image-builds.md, docs/modular-environments.md, and docs/tutorial-custom-profile-source-builds.md.

Cloud provider overlays

Provider overlays live in values/cloud/:

• aws-eks.yaml: EKS/ALB/gp3 baseline plus optional EFS RWX example.
• azure-aks.yaml: AKS/AGIC/managed-csi baseline plus optional Azure Files RWX example.
• gcp-gke.yaml: GKE Ingress annotation/standard-rwo baseline plus optional Filestore RWX example.

The Terraform module deploys to an existing Kubernetes cluster; it does not create EKS, AKS, or GKE clusters. Use terraform/examples/*-tfvars.example as starting points and keep site hosts, certificate IDs, external secret names, identity bindings, and image pins in private site overlays. See docs/cloud-deployments.md.

Terraform

cd terraform
cp examples/ontoportal-clean-k3s.tfvars.example ontoportal-clean.auto.tfvars
# edit passwords and kubeconfig settings
terraform init
terraform apply

Terraform installs the same Helm chart into an existing k3s cluster. It can also install optional KEDA, VPA, kube-prometheus-stack, Trivy Operator, SonarQube, Loki, Grafana Alloy, cert-manager, External Secrets Operator, Kyverno, and Velero. Validate Terraform locally with make terraform-validate when Terraform is installed; GitHub Actions also runs terraform fmt/init/validate for Terraform changes.

Optional components

Optional components are not part of the clean baseline unless explicitly enabled.

OntoPortal chart overlays:

• Matomo analytics: values/addons/matomo.yaml
• FAIRness service: values/addons/fairness.yaml
• Assistant service: values/addons/assistant.yaml
• Prometheus ServiceMonitor for OntoPortal pods: values/addons/monitoring-servicemonitor.yaml (requires Prometheus Operator CRDs first)
• Native HPA autoscaling overlay for API/UI: values/addons/hpa-autoscaling.yaml (requires metrics-server and RWX/external shared storage; see docs/autoscaling-keda.md)
• KEDA autoscaling overlay for API/UI: values/addons/keda-autoscaling.yaml (requires KEDA CRDs first; see docs/autoscaling-keda.md)
• VPA recommendation overlay: values/addons/vpa-recommendations.yaml (requires VPA CRDs/controller first; starts with updateMode: Off)

Platform add-ons installed as separate Helm releases by Terraform:

• KEDA operator and CRDs: Terraform enable_keda=true
• Vertical Pod Autoscaler operator and CRDs: Terraform enable_vpa=true
• kube-prometheus-stack: Terraform enable_monitoring=true
• Trivy Operator: Terraform enable_trivy_operator=true
• SonarQube: Terraform enable_sonarqube=true
• Grafana Loki: Terraform enable_loki=true
• Grafana Alloy log collector: Terraform enable_grafana_alloy=true
• cert-manager: Terraform enable_cert_manager=true
• External Secrets Operator: Terraform enable_external_secrets=true; see examples/external-secrets/ for a complete k3s rehearsal flow.
• Kyverno: Terraform enable_kyverno=true
• Velero: Terraform enable_velero=true plus velero_values_file=<site-specific values file>

Script-managed add-ons:

• Wazuh: scripts/addons/wazuh-deploy.sh

Autoscaling and UI smoke tests

The chart supports native HPA and KEDA ScaledObject rendering for API, UI, FAIRness, and assistant deployments, plus VPA resources for request right-sizing across the application and bundled data services. Use values/addons/hpa-autoscaling.yaml for metrics-server-backed HPA, values/addons/keda-autoscaling.yaml with Terraform enable_keda=true or another pre-installed KEDA operator, and values/addons/vpa-recommendations.yaml with Terraform enable_vpa=true or another pre-installed VPA controller. API/UI horizontal scaling still requires RWX shared storage or an externalized shared-data design. Start VPA in updateMode: Off, review recommendations, and only then deliberately enable mutating modes for suitable workloads. Validate live autoscaling resources with make keda-check and make vpa-check after deployment. See docs/autoscaling-keda.md.

Browser smoke tests live in tests/ui/ and are run with Playwright:

make validate-ui-tests
npm install
npx playwright install chromium
UI_BASE_URL=http://localhost:3000 API_BASE_URL=http://localhost:9393 npm run test:ui

Against a Kubernetes release, use make ui-test NAMESPACE=ontoportal RELEASE=ontoportal; the script port-forwards the UI/API services before running Playwright.

Validation

make validate
make validate-generated
make validate-ui-tests
# Optional; requires Docker daemon.
make compose-config
helm lint chart/ontoportal -f values/profiles/ontoportal-clean.yaml -f values/profiles/k3s-local.yaml
helm lint chart/ontoportal -f values/profiles/agroportal-clean.yaml -f values/profiles/k3s-local.yaml
helm lint chart/ontoportal -f values/profiles/matportal.yaml -f values/profiles/k3s-local.yaml -f values/addons/matomo.yaml -f values/addons/fairness.yaml
helm template ontoportal-hpa chart/ontoportal --namespace ontoportal -f values/profiles/ontoportal-clean.yaml -f values/profiles/production-recommended.yaml -f values/addons/hpa-autoscaling.yaml >/tmp/ontoportal-hpa.yaml
helm template ontoportal-keda chart/ontoportal --namespace ontoportal -f values/profiles/ontoportal-clean.yaml -f values/profiles/production-recommended.yaml -f values/addons/keda-autoscaling.yaml >/tmp/ontoportal-keda.yaml
helm template ontoportal-vpa chart/ontoportal --namespace ontoportal -f values/profiles/ontoportal-clean.yaml -f values/profiles/production-recommended.yaml -f values/addons/vpa-recommendations.yaml >/tmp/ontoportal-vpa.yaml
make production-check PROFILE=ontoportal-clean
make production-check PROFILE=matportal

make validate runs static checks, renders generated artifacts, and runs production-readiness warnings. The k3s-smoke workflows install disposable k3s clusters and run scripts/smoke.sh for clean OntoPortal and MatPortal profiles.

Before enabling monitoring.serviceMonitor.enabled, verify the cluster has Prometheus Operator CRDs:

kubectl api-resources | grep -i '^servicemonitors' || echo 'install Prometheus Operator CRDs first'

Design rules

• Shared chart templates must not hard-code matportal-* service names.
• MatPortal-specific behavior must live in values/profiles/matportal.yaml, values/addons/*.yaml, or chart/ontoportal/files/matportal/.
• Compose files are generated from Helm values, not maintained by hand.
• Add-ons with independent lifecycle and CRDs are separate Helm/Terraform releases, not embedded into the OntoPortal chart.
• Runtime source-code patches are opt-in and documented as migration bridges, not the clean baseline.

Production and operations docs

• docs/validation-evidence.md: what has actually been tested and what still needs live cluster validation.
• docs/tutorial-k3s-laptop-server.md: laptop/single-server k3s installation tutorial.
• docs/high-availability.md: HA architecture and validation checklist.
• docs/backup-restic.md: Restic backup and restore strategy.
• docs/modular-environments.md: integrated recipes for Compose, k3s, AWS EKS, Azure AKS, GKE, and source-image build flows.
• docs/tutorial-custom-profile-source-builds.md: create a custom profile, build from source, publish to GHCR/Docker Hub, and pin accepted images.
• docs/upstream-environment-map.md: how published OntoPortal/BioPortal/AgroPortal deployment patterns map to reusable layers in this repository.
• docs/cloud-deployments.md: cloud provider overlays, storage/ingress assumptions, and RWX guidance.
• docs/image-builds.md: optional amd64/arm64 image builds to GHCR or Docker Hub.
• docs/github-setup.md: private GitHub repository setup, branch protection, and secret-scanning checklist.
• MAINTAINER_NOTES.md and docs/operations-guide.md: instructions for future maintainers to test, fix, and document changes.
• docs/update-policy.md: required update process for code, values, Terraform, scripts, generated Compose, and documentation.
• docs/release-process.md: tag strategy, tested matrix, image digest capture, upgrade/rollback notes, and release note template.
• docs/production-readiness.md: production checklist and known gaps.
• docs/observability.md: Prometheus/Grafana/Loki/Alloy setup plus examples/observability/ alert and dashboard starters.
• docs/testing-plan.md: static, render, live, Compose, Terraform, and add-on test matrix.

Run a static production-readiness check with:

make production-check PROFILE=ontoportal-clean
make production-check PROFILE=matportal