Release v 1.6

README.md

@@ -1,28 +1,166 @@
 # keepup
 
+A lightweight Prometheus exporter that collects infrastructure inventory pushed by remote agents - OS releases, package versions (with end-of-life enrichment), and Kubernetes/Helm deployments - and exposes it as metrics.
+
+`keepup` holds no state of its own: Redis is both the write buffer and the read source. Agents `PUT` JSON, `keepup` validates and stores it with a TTL, and Prometheus scrapes `/metrics` on demand.
+
+## Contents
+
+- [How it works](#how-it-works)
+- [Quick start](#quick-start)
+- [Configuration](#configuration)
+- [API](#api)
+  - [`PUT /os-release`](#put-os-release)
+  - [`PUT /package-version`](#put-package-version)
+  - [`PUT /helm-cluster`](#put-helm-cluster)
+- [Metrics](#metrics)
+- [Testing](#testing)
+- [Deploying with Helm](#deploying-with-helm)
+- [Releasing](#releasing)
+
+## How it works
+
+```
+ agent(s)                    keepup                        Prometheus
+┌─────────┐   PUT + token   ┌───────────────────┐  scrape  ┌────────────┐
+│ os-info │ ───────────────>│ handler ──▶ Redis  │<────────│ /metrics   │
+│ pkg-vers│                 │           (TTL)    │          │            │
+│ helm    │                 └───────────────────┘          └────────────┘
+└─────────┘
+```
+
+Every data domain follows the same shape:
+
+| Domain | Endpoint | Redis key | Metric |
+|---|---|---|---|
+| OS release *(deprecated)* | `PUT /os-release` | SHA1 of `{data_center}-{host_ip}` | `os_release_info` |
+| Package versions | `PUT /package-version` | SHA1 of `{data_center}-{host_ip}-PACKAGE_UUID` | `package_version_info` |
+| Kubernetes / Helm | `PUT /helm-cluster` | SHA1 of `{cluster_name}` | `kubernetes_cluster_info` |
+
+On each scrape, the collector `SCAN`s all Redis keys for the domain, deserializes every entry, and emits one Prometheus metric per entity - there is no in-memory cache, so every scrape hits Redis directly.
+
+**Package EOL enrichment**: every `package-version` push is checked against `endoflife.date`, cached in Redis for 7 days under `eol_cache:all_packages`. Supported packages: `redis`, `memcached`, `mongodb`, `mysql`, `rabbitmq`, `envoy`, `debian`, `postgresql`, `elasticsearch`, `php`. Versions are compared as `major.minor` only (Debian epoch prefixes like `5:7.0.15-1~deb12u1` are stripped down to `7.0`).
+
+## Quick start
+
+Requires Go and a local Redis instance.
+
+```bash
+# run locally - must run from src/ so .env is found
+cd src && go run main.go
+
+# build a binary
+go build -o keepup src/main.go
+
+# build the Docker image
+docker build -f docker/Dockerfile -t keepup .
+```
+
+The server listens on `LISTEN_PORT` (default `9101` in dev) and exposes:
+
+- `PUT`/`GET /os-release`, `/package-version`, `/helm-cluster` - data ingestion & lookup (require `x-api-token`)
+- `GET /metrics` - Prometheus scrape endpoint (no auth)
+- `GET /healthcheck` - liveness probe
+
+## Configuration
+
+Config is loaded from environment variables. If `APP_ENV` is unset, `keepup` loads `src/.env` (development only). **All fields are required** - the app panics at startup if any are missing.
+
+| Variable | Default (`.env`) | Purpose |
+|---|---|---|
+| `APP_ENV` | `dev` | when unset, triggers `.env` loading |
+| `API_TOKEN` | `secret` | value required in the `x-api-token` header on every PUT/GET |
+| `LISTEN_PORT` | `9101` | HTTP listen port |
+| `REDIS_ADDR` | `127.0.0.1` | Redis host |
+| `REDIS_PORT` | `6379` | Redis port |
+| `REDIS_DBNO` | `7` | Redis logical DB number |
+| `TTL_SECONDS` | `300` | expiry for every stored entry |
+
+## API
+
+All data endpoints require an `x-api-token` header matching `API_TOKEN`, and accept both `PUT` (insert) and `GET` (lookup by `id`).
+
+### `PUT /os-release`
+
+> **Deprecated** - kept for backwards compatibility, no longer receiving new fields (e.g. `team`). Do not build new integrations against it.
+
+```jsonc
+{
+  "release": {
+    "os_id": "debian",
+    "version_codename": "bullseye",
+    "version": "11 (bullseye)",
+    "version_id": "11",
+    "data_center": "aaa",
+    "host_ip": "101.122.418.4"
+  }
+}
 ```
-go run main.go
+
+### `PUT /package-version`
+
+```jsonc
+{
+  "packages": {
+    "debian": "11",
+    "mongodb": "7.3",
+    "redis": "5:7.0.15-1~deb12u1",
+    "mysql": "unknown",
+    "host_ip": "101.122.418.4",
+    "data_center": "aaa",
+    "team": "platform"
+  }
+}
 ```
 
+`host_ip`, `data_center`, and `team` are pulled out of the map and stored as entity metadata; every remaining key is treated as a package name -> installed version pair. Each package is enriched with `current_version_eof`, `newest_version`, and `expired` before being persisted.
+
+### `PUT /helm-cluster`
 
+```jsonc
+{
+  "cluster_name": "minikube",
+  "kube_version": "1.29.0",
+  "team": "platform",
+  "helm_charts": [
+    { "chart_name": "redis", "version": "18.1.5", "namespace": "database" },
+    { "chart_name": "keepup", "version": "0.5.0", "namespace": "monitoring" }
+  ]
+}
 ```
-sudo cat /sys/devices/virtual/dmi/id/product_uuid
-0a6e14bf-b17f-4ec5-aec0-35427a1723a1
-# https://puppet.com/docs/puppet/7/core_facts.html#dmi
-# https://tickets.puppetlabs.com/browse/FACT-234
+
+Unlike the other two endpoints, the request body maps directly onto the stored struct (no wrapper key, no field filtering).
+
+## Metrics
+
+| Metric | Labels |
+|---|---|
+| `os_release_info` *(deprecated)* | `id`, `os_id`, `version_codename`, `version`, `version_id`, `data_center`, `host_ip` |
+| `package_version_info` | `id`, `package_name`, `current_version`, `current_version_eof`, `newest_version`, `expired`, `data_center`, `host_ip`, `team` |
+| `kubernetes_cluster_info` | `id`, `cluster_name`, `kube_version`, `chart_name`, `chart_version`, `chart_namespace`, `team` |
+
+## Testing
+
+There are no unit tests - only an end-to-end shell script that exercises all three endpoints against a running server:
+
+```bash
+# start the server first (see Quick start), then:
+cd tests/end-to-end && ./run.sh
 ```
 
+## Deploying with Helm
+
+`charts/keepup/` deploys `keepup` with a Redis sidecar in the same pod (`redis.enabled: true` by default, so no external Redis is required). Key values:
+
+- `apiToken` - auth token agents must send
+- `ttlSeconds` - entry expiry
+- `ingress.*` - expose the API externally
+- `servicemonitor.enabled` - wire up Prometheus scraping automatically
+
+```bash
+helm install keepup charts/keepup --set apiToken=<your-token>
+```
 
+## Releasing
 
-```bash 
-cat /etc/os-release 
-PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
-NAME="Debian GNU/Linux"
-VERSION_ID="11"
-VERSION="11 (bullseye)"
-VERSION_CODENAME=bullseye
-ID=debian
-HOME_URL="https://www.debian.org/"
-SUPPORT_URL="https://www.debian.org/support"
-BUG_REPORT_URL="https://bugs.debian.org/"
-```
+Pushing a git tag triggers `.github/workflows/build-docker-image.yml`, which builds and pushes the image to `ghcr.io/code-tool/keepup`. The build version is injected via `-ldflags "-X main.buildVersion=..."` and logged at startup.

charts/keepup/Chart.yaml

@@ -2,5 +2,5 @@
 apiVersion: v2
 name: keepup
 description: eondoflife.date version tracker service
-version: 1.5.1
-appVersion: 1.5.0
+version: 1.6.0
+appVersion: 1.6.0

charts/keepup/values.yaml

@@ -7,7 +7,7 @@ main:
 
 redis:
   enabled: true
-  image: "redis:8.2.6"
+  image: "redis:8.2.8"
   pullPolicy: "IfNotPresent"
 
 ingress:

docker/Dockerfile

@@ -1,5 +1,5 @@
-FROM golang:1.25.9-trixie AS builder
-ARG BUILD_VERSION='v1.5.0'
+FROM golang:1.25.12-trixie AS builder
+ARG BUILD_VERSION='v1.6.0'
 ENV LISTEN_PORT=9101
 WORKDIR /opt/keepup/
 COPY go.mod ./

src/handler/helmscrape.go

@@ -16,6 +16,7 @@ type KubernetesCluster struct {
 	ID          uuid.UUID       `json:"id"`
 	ClusterName string          `json:"cluster_name"` // Default value from scraper: minikube
 	KubeVersion string          `json:"kube_version"`
+	Team        string          `json:"team"`
 	HelmCharts  []HelmChartData `json:"helm_charts"`
 	UpdatedAt   string          `json:"updated_at"`
 }

src/handler/middleware.go

@@ -173,7 +173,7 @@ func (p *PackageVersionsHandler) handleInsertPackages(w http.ResponseWriter, r *
 
 	cleanedPackages := make(map[string]string)
 	for key, value := range req.Packages {
-		if key != "host_ip" && key != "data_center" {
+		if key != "host_ip" && key != "data_center" && key != "team" {
 			cleanedPackages[key] = value
 		}
 	}
@@ -188,6 +188,7 @@ func (p *PackageVersionsHandler) handleInsertPackages(w http.ResponseWriter, r *
 	pkg := PackageVersions{
 		DataCenterPkg: req.Packages["data_center"],
 		HostIPPkg:     req.Packages["host_ip"],
+		Team:          req.Packages["team"],
 		Packages:      convertedPackages,
 	}
 

src/handler/packageversions.go

@@ -28,6 +28,7 @@ type PackageVersions struct {
 	IDPkg         uuid.UUID                `json:"id"`
 	DataCenterPkg string                   `json:"data_center"`
 	HostIPPkg     string                   `json:"host_ip"`
+	Team          string                   `json:"team"`
 	UpdatedAt     string                   `json:"updated_at"`
 	Packages      map[string]PackageDetail `json:"packages"`
 }
@@ -210,7 +211,7 @@ func updateEOLCache(ctx context.Context, con *redis.Client) error {
 	//TODO: Handle all related packages.
 	//Option 1: Get all data from endoflife and store in redis.
 	//Option 2: Dynamicly resolve pacakge names, but should be checked fro eof api side.
-	supportedPackages := []string{"redis", "memcached", "mongodb", "mysql", "rabbitmq", "envoy", "debian", "postgresql", "elasticsearch", "php"}
+	supportedPackages := []string{"redis", "memcached", "mongodb", "mysql", "rabbitmq", "envoy", "debian", "postgresql", "elasticsearch", "php", "gitlab-runner", "linux"}
 
 	cacheDocument := map[string]interface{}{
 		"package": map[string][]EndOfLifeEntry{},

src/metrics/helmmetrics.go

@@ -14,6 +14,7 @@ var (
 	ChartName              = "chart_name"
 	ChartVersion           = "chart_version"
 	ChartNamespace         = "chart_namespace"
+	Teamcluster            = "team"
 	HelmReleaseMetricValue = float64(1)
 
 	kubernetesClusterMetricDesc = prometheus.NewDesc(
@@ -26,6 +27,7 @@ var (
 			ChartName,
 			ChartVersion,
 			ChartNamespace,
+			Teamcluster,
 		}, nil,
 	)
 )
@@ -55,6 +57,7 @@ func (kc KubernetesClusterCollector) Collect(ch chan<- prometheus.Metric) {
 				chart.ChartName,
 				chart.Version,
 				chart.Namespace,
+				cluster.Team,
 			)
 		}
 	}

src/metrics/pvmetrics.go

@@ -17,6 +17,7 @@ var (
 	Expired           = "expired"
 	DataCenterpkg     = "data_center"
 	HostIPpkg         = "host_ip"
+	Teampkg           = "team"
 
 	packageMetricDesc = prometheus.NewDesc(
 		"package_version_info",
@@ -30,6 +31,7 @@ var (
 			Expired,
 			DataCenterpkg,
 			HostIPpkg,
+			Teampkg,
 		}, nil,
 	)
 )
@@ -63,6 +65,7 @@ func (pc PackageVersionsCollector) Collect(ch chan<- prometheus.Metric) {
 				fmt.Sprintf("%t", details.Expired),
 				pkgs.DataCenterPkg,
 				pkgs.HostIPPkg,
+				pkgs.Team,
 			)
 		}
 	}

tests/end-to-end/example-001.json

@@ -8,6 +8,7 @@
       "memcached": "1.6.18-1",
       "envoy": "unknown",
       "host_ip": "101.122.418.4",
-      "data_center": "aaa"
+      "data_center": "aaa",
+      "team": "platform"
     }
 }

tests/end-to-end/example-003.json

@@ -0,0 +1,17 @@
+{
+  "cluster_name": "minikube",
+  "kube_version": "1.29.0",
+  "team": "platform",
+  "helm_charts": [
+    {
+      "chart_name": "redis",
+      "version": "18.1.5",
+      "namespace": "database"
+    },
+    {
+      "chart_name": "keepup",
+      "version": "0.5.0",
+      "namespace": "monitoring"
+    }
+  ]
+}

tests/end-to-end/run.sh

@@ -6,11 +6,14 @@ echo "=== PUT test data ==="
 
 curl -XPUT -H "x-api-token: secret" http://127.0.0.1:9101/package-version -d @example-001.json
 curl -XPUT -H "x-api-token: secret" http://127.0.0.1:9101/os-release -d @example-002.json
+curl -XPUT -H "x-api-token: secret" http://127.0.0.1:9101/helm-cluster -d @example-003.json
 
 echo "=== GET test data ==="
 curl -X GET -H "x-api-token: secret" -s http://127.0.0.1:9101/package-version -d '{"id":"91015d87-2c51-5601-b337-1414f2b5496a"}' | grep debian
 curl -X GET -H "x-api-token: secret" -s http://127.0.0.1:9101/os-release -d '{"id":"8b00021e-af61-546e-a0c1-1038bc422d39"}' | grep bullseye
+curl -X GET -H "x-api-token: secret" -s http://127.0.0.1:9101/helm-cluster -d '{"id":"688c14fe-9b83-5887-ba6c-f4fa310adc63"}' | grep minikube
 curl -X GET -s http://127.0.0.1:9101/metrics | grep 'os'
 curl -X GET -s http://127.0.0.1:9101/metrics | grep 'package_version'
+curl -X GET -s http://127.0.0.1:9101/metrics | grep 'kubernetes_cluster'
 
 echo "Done"