Skip to content

development.md

Latest commit

 

History

History
85 lines (60 loc) · 3.75 KB

development.md

File metadata and controls

85 lines (60 loc) · 3.75 KB
Raw

Development guide

This guide is for working on the QueryFlux Rust workspace and its local dependencies (Python/sqlglot, Docker).

Prerequisites

  • Rust (stable toolchain), cargo, rustfmt, clippy
  • Python 3.10+ (3.13 is used in some Makefile PYTHONPATH examples; adjust paths if your venv uses another minor version)
  • Docker and Docker Compose (for make dev, make test-e2e, and the full stack)

First-time setup

make setup

This creates .venv/ and installs Python packages from requirements.txt (including sqlglot for dialect translation via PyO3).

Point PyO3 at that interpreter when building or running (required whenever translation is enabled):

export PYO3_PYTHON="$(pwd)/.venv/bin/python3"
# Optional: helps some environments resolve the venv site-packages
export PYTHONPATH="$(pwd)/.venv/lib/python3.13/site-packages"

Adjust PYTHONPATH if your venv’s lib/pythonX.Y differs.

Daily commands

Goal Command
Format (if you use rustfmt manually) cargo fmt --all
Lint + unit tests (no Docker) make check
Release build make build
E2E tests (Docker: Trino, StarRocks, Lakekeeper) make test-e2e
Start backing services (Docker) make dev
Stop Docker stack make stop
Follow container logs make logs
Clean build + Docker volumes make clean

Running QueryFlux locally

make dev brings up dependencies via docker compose (see Makefile for the exact services). After containers are healthy, run the binary from the repo root with your config, for example:

export PYO3_PYTHON="$(pwd)/.venv/bin/python3"
export PYTHONPATH="$(pwd)/.venv/lib/python3.13/site-packages"  # version may vary
cargo run --bin queryflux -- --config config.local.yaml

Use config.local.yaml for the compose-oriented stack, or copy and edit config.example.yaml for your own layout.

Note: If sqlglot is not importable, the process starts but translation is disabled and logs a warning; dialect mismatches may then fail on the backend.

Workspace layout

The Cargo workspace lives under crates/. Common places to touch code:

Crate Responsibility
queryflux Binary entrypoint, wiring config → routers → cluster manager → adapters
queryflux-core Shared types (SessionContext, query IDs, config structs)
queryflux-config Loading YAML into ProxyConfig
queryflux-frontend HTTP/frontends, dispatch, Trino handlers
queryflux-routing RouterTrait implementations and RouterChain
queryflux-cluster-manager Group capacity, strategies, acquire_cluster / release_cluster
queryflux-translation sqlglot / TranslationService
queryflux-engine-adapters Trino, DuckDB, StarRocks, …
queryflux-persistence In-memory and PostgreSQL stores
queryflux-e2e-tests Integration tests behind Docker

Architecture narrative: docs/architecture.md and docs/README.md.

Configuration reference

Authoritative shapes are the serde types in queryflux-core (config.rs) and working examples such as config.local.yaml. The snippet in the root README.md may lag; prefer config.local.yaml and docs/routing-and-clusters.md for group members and top-level clusters.

Troubleshooting

  • PyO3 / Python not found: Set PYO3_PYTHON to the venv’s python3 and ensure make setup completed.
  • Port conflicts: Adjust ports in docker/docker-compose.yml or disable conflicting local services.
  • E2E failures: Bring up docker/docker-compose.test.yml (Trino, StarRocks, Iceberg stack); see make test-e2e.

For contribution expectations (PRs, tests, docs), see contribute.md.