System architecture

QCLM is split into a desktop client and a backend made of small services. The split keeps the instrument PC responsive: heavy processing runs in the backend, which can live on the same machine or on a more powerful server in the lab.

flowchart TB
    subgraph Desktop["Desktop app (JavaFX)"]
        UI[Acquisition window, analysis tabs]
    end

    UI -- "REST /api/v1" --> GW
    GW -- "GET /events (Server-Sent Events)" --> UI

    subgraph Backend["Backend (Docker Compose)"]
        GW[API gateway :8080]
        EXP[Experiment service]
        ACQ[Acquisition service]
        PRO[Processing service]
        INF[Inference service]
        META[Metadata service]
        DATA[Data service]
        K[(Kafka)]
        PG[(PostgreSQL)]
        VOL[(Data volume)]

        GW --> EXP & ACQ & PRO & INF & META & DATA
        EXP --> ACQ & PRO & INF & META
        ACQ -- "qclm.raw.frames" --> K
        K -- "qclm.raw.frames" --> PRO
        PRO -- "qclm.cubes" --> K
        K -- "qclm.cubes" --> INF
        INF -- "qclm.inference.results" --> K
        ACQ & PRO & INF -.-> DATA
        META --> PG
        DATA --> VOL
        K -- "qclm.events" --> GW
    end

Services

Service	Port	Responsibility
API gateway	8080	The only entry point. Routes /api/v1/* to the services, applies rate limiting (50 requests/s, burst 100) and optional JWT authentication, and streams live system events to the desktop app at /events.
Experiment	8086	Turns a request into a scan plan: a hyperspectral sweep (start/end/step) or a sparse wavelength list. Then orchestrates acquisition → processing → inference.
Acquisition	8083	Drives the laser, detector and stage through a LaserController interface (currently SimulatedLaserController). Publishes each frame to Kafka.
Processing	8084	Collects frames into a spectral cube, runs the processing pipeline, parses imported files, and runs batch jobs. Publishes finished cubes.
Inference	8085	Consumes cubes, runs classification models, and serves the built-in spectral index definitions.
Metadata	8081	Stores experiments, runs and their status in PostgreSQL.
Data	8082	Stores frames, cubes and results, on a local volume or in S3-compatible object storage.

Internal service ports are not published outside Docker. The desktop app only ever talks to the gateway.

Data flow during a scan

1. The desktop app asks the experiment service for a scan plan and starts it.
2. The acquisition service tunes the laser to each wavelength in turn, captures a frame and publishes it to the qclm.raw.frames topic.
3. The processing service consumes the frames. When asked to build the cube (automatically or from the app), it stacks them into a (bands, rows, cols) cube, applies calibration and publishes the cube to qclm.cubes.
4. The inference service consumes the cube and publishes classification results to qclm.inference.results.
5. Every service reports progress on qclm.events. The gateway relays those events to the desktop app over Server-Sent Events, which is how the viewer updates live.

Reliability

• Dead-letter topics. If a message can't be processed, it moves to a matching dead-letter topic (qclm.raw.frames.dlq, qclm.cubes.dlq, …) instead of blocking the stream. The Admin page in the desktop app lists dead letters and lets you retry or discard them.
• Correlation IDs. Each request carries a correlation ID through HTTP and Kafka, so you can follow one operation across all the service logs.
• Health and metrics. Every service exposes Spring Boot Actuator health and Prometheus metrics. The Admin page shows them, and the status bar shows gateway latency.

Why Kafka?

Frames arrive faster than they can be processed. Kafka decouples the instrument from the processing: acquisition never waits for processing or inference, and if processing restarts it picks up where it left off. It also lets several consumers, such as processing and live monitoring, read the same stream.