dzoni/ci-observer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: initial commit

Browse Source
This commit is contained in:
nikola
2026-05-19 14:53:37 +02:00
commit ea7e728222
34 changed files with 680 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ci-observer-handover/contexts/CI_OBSERVER_COMPACT_DUMP.md
+59
View File
@@ -0,0 +1,59 @@
# CI Observer Compact Dump
## Current Production (authoritative)
- Server path: `/webapps/ci-observer-dashboard-prod`
- Container: `ci-observer-dashboard-prod`
- Port: `8089 -> 8088`
- Deploy branch: `main`
- Health check: `curl -s http://127.0.0.1:8089/api/config`
## Stack
- FE: HTML + CSS + Vanilla JavaScript
- BE: Python + FastAPI
- FE-BE bridge: REST API (`/api/runs`, `/api/filters`, `/api/config`)
- Data layer: SQLite + GitHub Actions API
## Required Env
- `GITHUB_TOKEN`
- `GH_REPOS`
- `GH_WORKFLOWS` (required for polling in current logic)
- `REFRESH_SECONDS`
- `DASHBOARD_TITLE`
## Key Features in main
- Metadata-rich run cards (event, actor, author, commit SHA, message, date, duration)
- Repo short name display (without org prefix)
- Race-style clock (`HH:MM:SS.cc`) + date in same panel
- Light mode toggle (hanging bulb), default dark, preference in localStorage
- Light mode contrast tuned for pass/fail (medium)
## Recent Important Commits on main
- `689a78c` merge: metadata cards + race clock
- `a719b7d` docs: runbook + architecture + stack overview
- `d33f96e` merge: light mode toggle + contrast tuning
## Standard Production Update
```bash
cd /webapps/ci-observer-dashboard-prod
git checkout main
git pull
docker compose up -d --build
```
## Troubleshooting Fast Path
- UI loads but no runs:
- check `.env` (`GH_WORKFLOWS` not empty)
- `docker compose up -d --build`
- FE change not visible:
- browser hard refresh `Ctrl+Shift+R`
## Test Workflow Pattern
- Create isolated test folder: `/webapps/ci-observer-dashboard-<feature>`
- Use separate port (e.g. 8090)
- Validate, then merge to `main`, deploy prod, cleanup test folder
## Canonical Docs
- `docs/OPERATIONS_RUNBOOK.md`
- `docs/ARCHITECTURE.md`
- `docs/STACK_OVERVIEW.md`
- `ci-observer-handover/AGENTS.md`
ci-observer-handover/contexts/CI_OBSERVER_CONTEXT_LATEST_2026-02-11.md
+78
View File
@@ -0,0 +1,78 @@
# CI Observer - Latest Context (2026-02-11)
## Final Production State
- Production folder on runner: `/webapps/ci-observer-dashboard-prod`
- Production container: `ci-observer-dashboard-prod`
- Production port: `8089 -> 8088`
- Production branch policy: deploy from `main`
- Current app is running and data is visible.
## Repository
- Repo: `Wolkabout/ci-observer-dashboard`
- Local path used for development: `/home/nikola/codex-cli/ci-observer-dashboard`
## Important Commits on main
- `689a78c` merge: GHA metadata cards + race clock improvements
- `a719b7d` docs: runbook + architecture + stack overview
- `d33f96e` merge: light mode bulb toggle + light contrast tuning
## Major Features Implemented
1. Metadata-rich run cards:
- Added event, actor, author, commit short SHA, commit message, date, duration.
2. Repo name simplification:
- Card header shows short repo name (`IoT-Platform-Core`, `argocd`) instead of `Org/repo`.
3. Race clock UI:
- Large 24h digital time in top-right: `HH:MM:SS.cc`
- Seconds and centiseconds visually de-emphasized.
- Subtle pulse each second.
- Date shown with clock (`YYYY-MM-DD`).
4. Light mode + bulb toggle:
- Default: dark mode.
- User toggle: hanging bulb button placed between title and clock.
- Theme saved in browser localStorage.
- Light mode pass/fail contrast tuned (medium level).
5. Documentation added to repo:
- `docs/OPERATIONS_RUNBOOK.md`
- `docs/ARCHITECTURE.md`
- `docs/STACK_OVERVIEW.md`
## Runtime Config Notes
- App currently requires both `GH_REPOS` and `GH_WORKFLOWS`.
- If `GH_WORKFLOWS` is empty, polling does not start (UI loads but no runs).
- Required `.env` keys:
- `GITHUB_TOKEN`
- `GH_REPOS`
- `GH_WORKFLOWS`
- `REFRESH_SECONDS`
- `DASHBOARD_TITLE`
## Current Cleanup Status
- Old test containers and folders were cleaned.
- `/webapps` now keeps production folder only for CI Observer.
## Standard Production Update Command
```bash
cd /webapps/ci-observer-dashboard-prod
git checkout main
git pull
docker compose up -d --build
```
## If Data Disappears But UI Loads
- Check `.env` first (`GH_WORKFLOWS` is usually the cause).
- Then rebuild:
```bash
docker compose up -d --build
```
## Next Session Starting Point
- Start from production folder: `/webapps/ci-observer-dashboard-prod`
- Verify status:
```bash
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" | grep ci-observer-dashboard-prod
curl -s http://127.0.0.1:8089/api/config
```
ci-observer-handover/contexts/CODEX_CONTEXT.md
+55
View File
@@ -0,0 +1,55 @@
# Codex Work Context (Wolkabout GHA Diagram)
## Project
- Repo: `/home/nikola/codex-cli`
- Task: Create/iterate Wolkabout GHA pipeline SVG with 2D and 3D variants.
## Latest User Request
- Keep the current 2D diagram unchanged.
- Produce a separate 3D version where **all elements except text are skewed** (nagnuto), and 3D depth is stronger.
## Key Output Files
- 2D (approved):
- `/home/nikola/codex-cli/Wolkabout-platform-pipeline-GHA-v3.svg`
- `/home/nikola/codex-cli/Wolkabout-platform-pipeline-GHA-v3.png`
- 3D (working version, skewed):
- `/home/nikola/codex-cli/Wolkabout-platform-pipeline-GHA-v3-3d.svg`
- `/home/nikola/codex-cli/Wolkabout-platform-pipeline-GHA-v3-3d.png`
## What Was Done (Summary)
- Built a richer 2D diagram with 3 lanes (IoT, ArgoCD, WTF), per-workflow card details, start/end chevrons, subtle lane backgrounds, and rectangular handoff arrows that route along outer edges.
- Added runner/env/deploy info and detailed step descriptions based on workflow files in:
- `/home/nikola/WA-git/IoT-Platform-Core/.github/workflows/all-in-one.yml`
- `/home/nikola/WA-git/IoT-Platform-Core/.github/workflows/unit-tests.yml`
- `/home/nikola/WA-git/argocd/.github/workflows/deploy-be.yml` (branch: `atest1`)
- `/home/nikola/WA-git/Wolkabout-Test-Framework/.github/workflows/new-parallel-wtf.yml`
- Handoff arrows updated to rectangular paths:
- `M 2530 300 L 2530 440 L 30 440 L 30 590 L 40 590`
- `M 2530 590 L 2530 780 L 30 780 L 30 900 L 40 900`
- Increased lane background visibility and added deeper card shadows + gradient fills.
## 3D Variant Adjustments
- Created a new file: `Wolkabout-platform-pipeline-GHA-v3-3d.svg`.
- Applied stronger depth via:
- Card gradient (`#1a2742` -> `#0a0f18`)
- Deeper shadow filters `card-shadow-deep` and `card-shadow-soft`
- Heavier shadow on start/end tags (`flow-shadow`) with offset
- Applied **global skew** so everything except text tilts:
- Wrapped all content in `<g id="tilt" transform="skewX(-6)"> ... </g>`
- Added `transform="skewX(6)"` to every `<text ...>` to keep text visually upright.
## Notes on Tools/Warnings
- `inkscape` export used to generate PNGs.
- Inkscape warns about unimplemented style property 109 (safe to ignore).
## Next Steps / Open Items
- Validate the 3D skew visual to confirm the tilt is strong enough.
- If needed: increase skew (e.g., `skewX(-8)` or `-10`) and adjust text counter-skew.
- Optional: add more 3D depth (side faces) if requested.
## Location of Current 3D Edits in SVG
- Global skew group added near root:
- `<g id="tilt" transform="skewX(-6)">` wrapping the entire scene
- Closing `</g>` before `</svg>`
- Text counter-skew applied by replacing `<text ...>` with `<text transform="skewX(6)" ...>`.
ci-observer-handover/contexts/CODEX_CONTEXT_COMPOSE_TEST.md
+68
View File
@@ -0,0 +1,68 @@
# Codex Context: Terraform VM + Docker Compose Test (Single Node)
## Goal
Test a single-node Docker Compose deployment on a Terraform-provisioned VM (no k3s), using the same layout as swarm. Ensure backend + FE + proxies come up and healthcheck works. This is a throwaway, clean environment.
## Current State
- VM created via Terraform: `terraform-testvm-1` on Proxmox node `rbmk2`.
- IP: `10.1.50.125/24`, gateway `10.1.50.1`.
- SSH: `ssh devops@10.1.50.125 -p42315` (company policy non-22).
- Docker installed via `get.docker.com` on the VM.
- Compose location on VM: `/webapps/wolkabout`.
- Volume/config location on VM: `/mnt/docker-volumes/wolkabout`.
- HAProxy fixed by ensuring `options.http` ends with a blank line.
## Files on laptop
- New compose file with MySQL + Cassandra: `/home/nikola/codex-cli/docker-compose.with-dbs.yml` (version 2.4).
- This file was copied to the VM as `/webapps/wolkabout/docker-compose.yml`.
## Compose contents (high level)
- Services: `mysql8`, `cassandra`, `cloud-config`, `rabbitmq`, `mosquitto`, `backend`, `frontend`, `frontend-federated`, `http2amqp`, `mqtt2amqp`, `loki`, `haproxy`.
- Ordered startup using `depends_on` + healthchecks (DBs first, then base infra, then app/FE).
- Network name: `core-net` (kept same name).
- MySQL image: `mysql:8.0.32-debian` with:
- root password `password`
- DB `swarm`, user `wolk`, password `Test1234`
- volumes: `/mnt/docker-volumes/wolkabout/mysql8`, init folder `/mnt/docker-volumes/wolkabout/mysql8-init`, config `/mnt/docker-volumes/wolkabout/config/mysql-custom.cnf`
- Cassandra image: `cassandra:4.1.2`, volume `/mnt/docker-volumes/wolkabout/cassandra`, heap env.
## Config changes still needed on VM
Update these to point to local DBs/containers:
- `/mnt/docker-volumes/wolkabout/config/wolk-commons.yml`
- `spring.datasource.url` -> `jdbc:mysql://mysql8:3306/swarm?...`
- `spring.datasource.username` -> `wolk`
- `spring.datasource.password` -> `Test1234`
- `spring.cassandra.contact-points` -> `cassandra`
- `/mnt/docker-volumes/wolkabout/config/mosquitto.conf`
- `auth_opt_host mysql8`
- `auth_opt_dbname swarm`
- `auth_opt_user wolk`
- `auth_opt_pass Test1234`
## MySQL/Cassandra init
No dump yet. We want a clean environment. Create minimal init:
- MySQL init dir: `/mnt/docker-volumes/wolkabout/mysql8-init/init.sql`
- For now can just `CREATE DATABASE IF NOT EXISTS swarm;`.
- Cassandra keyspace: `swarm` (from `wolk-commons.yml`).
Recommended scripts on VM (create in `/webapps/wolkabout`):
- `init-mysql.sh` waits for mysql, applies init.sql if present.
- `init-cassandra.sh` waits for cassandra, creates keyspace `swarm`.
## Docker registry login
ECR images require login on the VM. On swarm1 there is `~/.aws/credentials` and `~/.aws/config` (region `eu-west-1`). Use those creds (copy from swarm1 or your internal source) and run:
- `aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin 629307322585.dkr.ecr.eu-west-1.amazonaws.com`
## Known runtime issue
- `mosquitto` fails if MySQL schema/tables do not exist (`MosquittoUser`, `MosquittoTopic`). Backend may not auto-create them. If needed, we must add minimal SQL schema later.
## Healthcheck target
Backend is considered ready when:
- `https://<domain>.wolkabout.com/api/infos/server` returns JSON.
## Next Actions (expected)
1. Create `/mnt/docker-volumes/wolkabout/config/mysql-custom.cnf` (utf8mb4 + mysql_native_password + max_connections + bind-address).
2. Update `wolk-commons.yml` + `mosquitto.conf` to local DB hostnames.
3. Create init scripts + minimal init.sql.
4. `docker compose down` then `up -d`.
5. Run init scripts and validate with `docker compose ps`.
ci-observer-handover/contexts/CODEX_CONTEXT_PIPELINE.md
+80
View File
@@ -0,0 +1,80 @@
# Codex Work Context (New Pipeline Experiment)
## Goal
Create a simplified, stable pipeline that provisions a dynamic Integration/Atest
environment via Terraform and deploys the platform using Docker Compose on a
single-node VM. This is a PoC to validate the flow end-to-end.
## Guiding Principles
- Keep everything basic and transparent. No hidden logic or magic variables.
- Prefer explicit, readable configuration over abstraction.
- Security hardening is not a priority at this stage.
- The goal is: it works, and it is easy to see why it works or fails.
## Current Plan (High-Level)
1. Terraform provisions a VM from Proxmox template `debian12-tf-template`.
2. Docker Compose is used to deploy the platform on that VM.
3. HAProxy handles routing for API and frontends (TLS enabled).
4. Healthcheck confirms readiness via:
`https://<domain>.wolkabout.com/api/infos/server`
5. Tear down the VM after the run (if/when automated).
## Compose Package Location
Prepared in:
- `/home/nikola/codex-cli/terraform-vm-docker`
Contains:
- `docker-compose.yml`
- `.env`
- `bring-up.md`
- `healthcheck.sh`
- `config/` (wolk config YAMLs + firebase JSON)
- `haproxy_config/` (haproxy.cfg + options.http)
- `ssl/2026-SSL-WA-full.pem`
- `custom-assets/federation.manifest.json`
- `whiteLabeling/` (generated by backend on first successful start)
## Runtime Assumptions (from existing Swarm setup)
Images in use:
- `629307322585.dkr.ecr.eu-west-1.amazonaws.com/mono-web:150`
- `629307322585.dkr.ecr.eu-west-1.amazonaws.com/wolkabout:cloudconfig-22.8.2`
- `629307322585.dkr.ecr.eu-west-1.amazonaws.com/wolkabout:wafels-112.federation-poc-1`
- `629307322585.dkr.ecr.eu-west-1.amazonaws.com/wolkabout:frontend-federated`
- `629307322585.dkr.ecr.eu-west-1.amazonaws.com/utils-http2amqp:151`
- `629307322585.dkr.ecr.eu-west-1.amazonaws.com/utils-mqtt2amqp:151`
- `629307322585.dkr.ecr.eu-west-1.amazonaws.com/wolkabout:mqtt_ws`
- `rabbitmq:4.1-management-alpine`
- `grafana/loki:3.4.1`
- `haproxy:2.6`
Known dependencies:
- MySQL: `10.0.50.157:3306` (schema `swarm`, user `wolk`, pass `Test1234`)
- Cassandra: `10.0.50.163` (keyspace `swarm`)
## Healthcheck
Use `healthcheck.sh` to verify readiness:
- `https://<domain>.wolkabout.com/api/infos/server`
- Expects HTTP 200 and JSON with `organization` key.
## Networking
- Compose network name: `core-net`
- Single-node bridge network (no overlay)
- Ports exposed: `80`, `443`, `2883`, `8883`, `3100`
## Notes
- MQTT/HTTP2AMQP config mounts are present but commented in Compose.
- All secrets are kept inline for now to reduce moving parts.
- Debugging should be obvious from config + logs without additional tooling.
ci-observer-handover/contexts/ci-observer-dashboard-context.md
+51
View File
@@ -0,0 +1,51 @@
# CI Observer Dashboard Context
> Superseded by: `ci-observer-handover/contexts/CI_OBSERVER_CONTEXT_LATEST_2026-02-11.md`
> Use the newer file for current production/runtime state.
## Why this exists
- Need a read-only, always-on CI status wall for CTO/DevOps.
- Must not affect existing CI/CD pipelines or protected branches.
- Quick POC required using local infra and GitHub Actions metadata.
## What we built
- Standalone dashboard service that polls GitHub Actions and renders a live view.
- Runs on the runner VM as a Docker container.
- Uses a classic GitHub PAT (read-only scopes) to read workflow runs.
## Current behavior
- Polls GitHub Actions workflow runs every `REFRESH_SECONDS` (default 5s).
- Shows last 10 runs by updated time.
- Status colors:
- green = pass
- orange = partial (cancelled/skipped/etc.)
- red = fail
- blue = running/info
- Cards show run number, workflow, branch, start time, duration, and a link to the run.
- UI tuned for 23–24" monitor: 5 cards per row.
## Repo and runtime
- Repo: `Wolkabout/ci-observer-dashboard`
- Deploy path on runner VM: `/webapps/ci-observer-dashboard`
- Default port: `8088`
- URL: `http://10.1.50.20:8088`
## How to run (POC)
```
cd /webapps/ci-observer-dashboard
cp .env.example .env
# set GITHUB_TOKEN, GH_REPOS, GH_WORKFLOWS, REFRESH_SECONDS
docker compose up -d --build
```
## Current config (example)
```
GH_REPOS=Wolkabout/IoT-Platform-Core
GH_WORKFLOWS=unit-tests.yml
REFRESH_SECONDS=5
```
## Notes for future Codex sessions
- This is read-only and independent of CI pipelines.
- To add more repos/workflows, extend `GH_REPOS` and `GH_WORKFLOWS` (comma-separated).
- AI summary integration (Ollama) is planned but not implemented yet.