feat: build against continuum 3.1.0 on Spring Boot 4 and Vert.x 5

.github/workflows/gradle-build.yml

@@ -23,7 +23,9 @@ on:
 
 concurrency:
   group: gradle-build-${{ github.event.pull_request.number || github.ref }}
-  cancel-in-progress: true
+  # A main run publishes immutable release artifacts for ~1 h; cancelling it mid-way
+  # leaves some modules live and the rest unpublished, so main runs queue instead.
+  cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
 
 jobs:
   gradle_build_and_publish:

LOAD_TESTING.md

@@ -0,0 +1,169 @@
+# Load Testing Structures
+
+How Structures is load tested before a release is promoted, what the setup assumes, what each
+test exercises, and what the runs have shown. The most recent run is recorded in
+[docs/performance/LOAD_TEST_3.6.0.md](docs/performance/LOAD_TEST_3.6.0.md); this document is the
+method behind it.
+
+## Goals and expectations
+
+The runs are due diligence, not a benchmark of the ceiling. A release is expected to take a
+sustained, mixed load over both transports for at least ten minutes and show:
+
+- **No failures.** Every operation returns; a single client-visible error is investigated.
+- **No instability.** No pod restarts, no `WARN` or `ERROR` in any server pod's log for the window.
+- **No drift.** Latency in the last 30 s window looks like the second (the first carries warm-up).
+- **Headroom.** CPU and memory stay well below what the machine offers, so the numbers reflect
+  the software and not the host.
+- **Latency in the expected band** for this topology: reads and searches p95 under 50 ms,
+  single saves under 100 ms, 200-document bulk saves under 200 ms. Outliers are read against the
+  window they occur in, not the run total.
+
+Absolute numbers are only comparable between runs on the same hardware and topology; between
+releases, the useful comparison is the shape - same load, same profile of p50/p95/p99, same
+resource footprint.
+
+## Assumptions
+
+- The cluster is the **KinD cluster** `dev-tools/kind` creates: three Structures replicas behind
+  ingress-nginx with TLS, a two-node Elasticsearch, Ignite clustering over Kubernetes discovery.
+  It is a functional stand-in for production topology, not for production hardware: every node is
+  a container on one machine, and the load generators run on that same machine.
+- The generators, the ingress and the pods share the host's CPU. Generator processes are kept
+  light (four Node processes at a combined ~120 req/s cost a fraction of one core) so they do not
+  compete with the system under test; watch `docker stats` if the rates are raised.
+- Traffic goes through the ingress, as it would in production: STOMP over WebSocket at
+  `wss://localhost/v1`, OpenAPI at `https://localhost/api`. Use `localhost`, not
+  `structures.local`: on macOS a `.local` name goes to mDNS before `/etc/hosts` and every new
+  connection waits 5 s for that to time out. The mkcert certificate and the ingress rules cover both.
+- One structure, `load-testing.person` (multi-tenant `SHARED`, client-supplied ids, an address
+  sub-object, a short `age`), one tenant. Data accumulates across runs; the run record notes the
+  starting document count where it matters.
+- Elasticsearch is left at its defaults (1 s refresh). Write-path spikes that hit STOMP and
+  OpenAPI in the same window are usually its refresh or merge activity, not either transport.
+- The server pods run with no CPU or memory limits, so a leak or a runaway shows up as growth in
+  `kubectl top`, not as an OOM kill.
+
+## Hardware and test infrastructure
+
+The 3.6.0 run, and the reference for future comparison:
+
+| | |
+|---|---|
+| Machine | MacBook Pro, Apple M4 Max, 16 cores (12 performance, 4 efficiency), 128 GB, macOS 26.5 |
+| Docker | Docker Desktop, engine 29.2, VM with 16 CPUs / 63 GB |
+| Kubernetes | KinD v0.30, Kubernetes v1.34, one control-plane + three worker nodes (each sees all 16 CPUs) |
+| Ingress | ingress-nginx controller v1.15, on the control-plane node, 2 CPU / 1 GiB limit (`dev-tools/kind/config/ingress-nginx/values.yaml`) |
+| Elasticsearch | 8.19.13, two nodes, 1 CPU / 2 GiB each, 1 GiB heap (`dev-tools/kind/config/elasticsearch/values.yaml`) |
+| Structures | three replicas, no resource limits, image built by CI from the release-candidate commit |
+| Metrics | metrics-server v0.9 (`kubectl top`, sampled every 15 s), Elasticsearch `_nodes/stats`, pod logs |
+| Generators | Node 24, `structures-js/load-generator`, `@kinotic/continuum-client` 3.0.0, on the host |
+
+`./dev-tools/kind/kind-cluster.sh create` installs all of this; `deploy --tag <image tag>` puts the
+candidate image on it.
+
+## The tests
+
+All live in `structures-js/load-generator`. Each is one Node process driving one workload with a
+concurrency cap (`MAX_CONCURRENT_REQUESTS`), a rate cap (`MAX_REQUESTS_PER_SECOND`) and a duration
+(`DURATION_SECONDS`); a run is several of them at once. Every operation is timed in the executor,
+so each process reports count, errors, rate and p50/p90/p95/p99/max per operation every
+`REPORT_INTERVAL_SECONDS` and at the end, and writes the totals as JSON to `REPORT_FILE`.
+
+**STOMP tests** open one WebSocket through the ingress, authenticate as `admin`, and call the
+entity service the way the TypeScript client does; requests are multiplexed on that connection,
+so these measure the server, the event bus and Ignite routing rather than connection setup.
+
+- `sustainedBulkSave` - `bulkSave` of `BATCH_SIZE` generated people (default 200) per request:
+  the ingest path, JSON parsing, id handling and an Elasticsearch bulk index per request.
+- `sustainedSearch` - `search` with a Lucene query (`SEARCH_TEXT`, default `firstName: John`),
+  page `PAGE_SIZE` (100): query parsing, tenant filtering, a search per request.
+- `sustainedFindAll` - `findAll` of the first page of `PAGE_SIZE`: the cheapest read, a match-all
+  with tenant filter, useful as the baseline the others are read against.
+
+**`openApiMixed`** drives the OpenAPI endpoints of the same structure over HTTPS through the
+ingress with keep-alive connections, weighted the way an application would use them, so the HTTP
+router, its JSON handling and authentication are exercised alongside STOMP:
+
+| operation | share | what it exercises |
+|---|---|---|
+| save (one person) | 25 % | single-document ingest, JSON body parsing |
+| bulk save (`BATCH_SIZE`) | 5 % | bulk ingest over HTTP |
+| find by id | 25 % | a get by id; ids come from this process's own saves, so they hit real rows |
+| find all (page) | 15 % | first page of `PAGE_SIZE` |
+| search | 20 % | Lucene query as a `text/plain` body |
+| count | 10 % | `count/all` |
+
+**`createPersonStructure`** creates and publishes `load-testing.person` if it is missing and is
+run once before the first load test against a cluster.
+
+The older count-bounded tests (`bulkLoadSmall`/`Medium`/`Large`, `search`, `findAll`, the
+multi-tenant variants, `generateComplexStructures`) still exist and get the same reporting.
+
+## Running a test
+
+```sh
+cd structures-js/load-generator && pnpm install && pnpm build
+export NODE_EXTRA_CA_CERTS="$(mkcert -CAROOT)/rootCA.pem" NODE_ENV=production OTEL_SDK_DISABLED=true \
+       STRUCTURES_HOST=localhost STRUCTURES_PORT=443 STRUCTURES_USE_SSL=true \
+       STRUCTURES_OPENAPI_BASE_URL=https://localhost/api START_DELAY_SECONDS=0 LOG_TASKS=false
+
+# once per cluster
+TEST_NAME=createPersonStructure MAX_CONCURRENT_REQUESTS=1 node dist/main.mjs
+
+# the standard ten-minute run
+export DURATION_SECONDS=600 REPORT_INTERVAL_SECONDS=30
+TEST_NAME=sustainedBulkSave BATCH_SIZE=200 MAX_CONCURRENT_REQUESTS=2  MAX_REQUESTS_PER_SECOND=5  REPORT_FILE=bulk.json   node dist/main.mjs &
+TEST_NAME=sustainedSearch                  MAX_CONCURRENT_REQUESTS=16 MAX_REQUESTS_PER_SECOND=40 REPORT_FILE=search.json node dist/main.mjs &
+TEST_NAME=sustainedFindAll                 MAX_CONCURRENT_REQUESTS=4  MAX_REQUESTS_PER_SECOND=20 REPORT_FILE=find.json   node dist/main.mjs &
+TEST_NAME=openApiMixed                     MAX_CONCURRENT_REQUESTS=16 MAX_REQUESTS_PER_SECOND=50 REPORT_FILE=api.json    node dist/main.mjs &
+wait
+```
+
+While it runs, sample the cluster every 15 s: `kubectl top pods` for the Structures,
+Elasticsearch and ingress pods, `docker stats` for the KinD nodes, and Elasticsearch
+`_nodes/stats` (indexing and search totals, heap). Afterwards, check restarts
+(`kubectl get pods`) and the logs of every server pod for `WARN` and `ERROR` over the window.
+
+`OTEL_SDK_DISABLED=true` keeps the generator's OpenTelemetry SDK from trying to export to a
+collector that is not there; `START_DELAY_SECONDS` defaults to 60 for the Docker case where the
+server is still starting.
+
+## Outcomes so far
+
+**3.6.0 candidate** ([full record](docs/performance/LOAD_TEST_3.6.0.md)) - Spring Boot 4.1.1,
+Vert.x 5.1.8, Jackson 3, continuum 3.1.0, the four processes above for ten minutes:
+
+- 89,397 operations at ~122 requests/s across both transports, ~3,500 documents/s indexed
+  (2.1 M in the run), **0 failures**, no restarts, no `WARN` or `ERROR` logged.
+- STOMP search p50 10 ms / p95 30 ms / p99 60 ms; STOMP bulk save (200) p50 28 ms / p95 65 ms;
+  OpenAPI save p50 26 ms / p95 76 ms; OpenAPI reads p95 15–19 ms.
+- Structures pods 100–280 m CPU at the median, peaks under 400 m; Elasticsearch 225 m median per
+  node, heap 68 % at most; ingress 29 m at peak.
+- Flat across the run's windows after warm-up. One window carried a single 1.5–1.9 s spike on both
+  write paths at once (Elasticsearch refresh or merge).
+- Open question: pod memory rose about 300 MiB over the ten minutes on two of three pods. A JVM
+  growing into its heap looks the same as a slow leak at this length; a longer soak at the same
+  load would settle it.
+
+**3.6.0 final image** (`3.6.0-pr11.1e30e55`, continuum 3.1.0 release, [same record](docs/performance/LOAD_TEST_3.6.0.md),
+second section), on a cluster recreated from nothing so the migration job ran against an empty
+Elasticsearch, after the Gradle suite (109), e2e native + openapi (55) and k8s (5) on that cluster:
+
+- 95,497 operations at ~155 requests/s, 1,106,420 documents indexed from an empty index,
+  **0 failures**, no restarts, no `WARN` or `ERROR` logged.
+- Same shape as the candidate run: STOMP search p50 9 ms / p95 27 ms; STOMP bulk save p50 27 ms /
+  p95 79 ms; OpenAPI save p50 28 ms / p95 83 ms; OpenAPI reads p95 16-23 ms. Flat after warm-up,
+  one write-path spike window (Elasticsearch refresh or merge), as before.
+- Memory rose 285-478 MiB per pod again over the ten minutes; the soak question stands. One pod
+  carried the three sticky STOMP connections and ran about three times hotter than the others.
+- The generator's nominal rate cap leaks when the queue keeps draining (p-queue's fixed window
+  restarts); the OpenAPI process issued 79 ops/s against a 50/s cap. Compare observed rates, not
+  caps, between runs.
+
+## What a release run should add
+
+- The same four processes, same rates, on the candidate image, compared against the previous
+  record's table.
+- A longer soak (an hour) at least once per major dependency change, for the memory question.
+- Any new transport or hot path gets a workload here before it ships.

README.md

@@ -142,6 +142,7 @@ Ready to dive deeper? Our comprehensive documentation covers everything:
 - **[Decorators Reference](https://structuresframework.org/webdocs/reference/decorators)** - All available decorators
 - **[Multi-tenant Guide](https://structuresframework.org/webdocs/guide/multi-tenant-access)** - Multi-tenancy setup
 - **[Docker Compose Setup](docker-compose/README.md)** - Development environment configuration
+- **[Load Testing](LOAD_TESTING.md)** - How releases are load tested, the setup it assumes, and what the runs have shown
 
 ## 🔧 Development Setup
 

buildSrc/src/main/groovy/org.kinotic.java-application-conventions.gradle

@@ -23,7 +23,7 @@ dependencies {
     implementation "org.kinotic:continuum-core:${continuumVersion}"
     implementation "org.kinotic:continuum-core-vertx:${continuumVersion}"
 
-    implementation "javax.annotation:javax.annotation-api:${javaxAnnotationApi}"
+    implementation 'jakarta.annotation:jakarta.annotation-api'
 
     annotationProcessor 'org.springframework.boot:spring-boot-configuration-processor'
     testImplementation 'org.springframework.boot:spring-boot-starter-test'

buildSrc/src/main/groovy/org.kinotic.java-common-conventions.gradle

@@ -4,7 +4,7 @@ plugins {
     id 'io.spring.dependency-management'
 }
 
-group 'org.kinotic'
+group = 'org.kinotic'
 
 // gradle.properties holds the plain version (e.g. 3.5.8). Development builds get
 // -SNAPSHOT appended automatically; CI passes -Prelease on main to publish the
@@ -16,15 +16,14 @@ if (hasProperty('release')) {
 } else if (!effectiveVersion.contains('-')) {
     effectiveVersion += '-SNAPSHOT'
 }
-version effectiveVersion
-sourceCompatibility = '21'
+version = effectiveVersion
 
 repositories {
     // Use Maven Central for resolving dependencies.
     mavenCentral()
 
-    maven { url 'https://repo.spring.io/milestone' }
-    maven { url 'https://repo.spring.io/snapshot' }
+    maven { url = 'https://repo.spring.io/milestone' }
+    maven { url = 'https://repo.spring.io/snapshot' }
 
     maven {
         name = 'Central Portal Snapshots'
@@ -40,6 +39,16 @@ configurations {
     }
 }
 
+// The Boot BOM manages Netty below what Vert.x was built against (vertx-dependencies <netty.version>),
+// and the gap is a NoSuchMethodError on the event loop the first time a client upgrades to HTTP/2 -
+// which nothing speaking HTTP/1.1 ever sees. Overriding the BOM's property keeps the whole Netty set
+// consistent at the version Vert.x asks for. Pinned by HttpTransportTest.
+//
+// Vert.x's JSON codec has the same shape against Jackson 2 (vertx-dependencies <jackson2.version>
+// against Boot's jackson-2-bom), one patch apart rather than seven; left to the BOM and pinned by
+// VertxJsonCodecTest so a widening gap shows up there and not in production.
+ext['netty.version'] = nettyVersion
+
 dependencyManagement {
     imports {
         mavenBom "org.springframework.boot:spring-boot-dependencies:${springBootVersion}"
@@ -50,52 +59,41 @@ dependencyManagement {
     dependencies {
         dependency "co.elastic.clients:elasticsearch-java:${elasticClientVersion}"
 
-        dependency "com.github.ben-manes.caffeine:caffeine:${caffeineVersion}"
-
+        // Ahead of the Boot BOM: vertx-web-graphql is compiled against a newer graphql-java than Boot
+        // manages, and federation and the extended scalars follow it. Pinned by GraphQlJsonObjectAdapterTest.
         dependency "com.graphql-java:graphql-java:${graphQlJavaVersion}"
+        dependency "com.graphql-java:graphql-java-extended-scalars:${graphQlJavaExtendedScalarsVersion}"
+        dependency "com.apollographql.federation:federation-graphql-java-support:${graphQlFederationJvmVersion}"
+
+        dependency "org.apache.commons:commons-lang3:${apacheCommonsLangVersion}"
+        dependency "org.apache.commons:commons-text:${apacheCommonsTextVersion}"
 
-        dependency "javax.annotation:javax.annotation-api:${javaxAnnotationApi}"
+        dependency "commons-io:commons-io:${commonsIoVersion}"
 
         dependency "io.vertx:vertx-web:${vertxVersion}"
         dependency "io.vertx:vertx-web-graphql:${vertxVersion}"
         dependency "io.vertx:vertx-web-client:${vertxVersion}"
         dependency "io.vertx:vertx-health-check:${vertxVersion}"
 
-        dependency "me.escoffier.vertx:vertx-completable-future:${vertxCompletableFutureVersion}"
-
-        dependency "org.apache.commons:commons-compress:${commonsCompressVersion}"
-
         dependency "org.apache.ignite:ignite-core:${igniteVersion}"
         dependency "org.apache.ignite:ignite-kubernetes:${igniteVersion}"
         dependency "org.apache.ignite:ignite-spring:${igniteVersion}"
 
-        dependency "org.apache.lucene:lucene-analyzers-common:${luceneVersion}"
-        dependency "org.apache.lucene:lucene-backwards-codecs:${luceneVersion}"
-        dependency "org.apache.lucene:lucene-core:${luceneVersion}"
-        dependency "org.apache.lucene:lucene-grouping:${luceneVersion}"
-        dependency "org.apache.lucene:lucene-highlighter:${luceneVersion}"
-        dependency "org.apache.lucene:lucene-join:${luceneVersion}"
-        dependency "org.apache.lucene:lucene-memory:${luceneVersion}"
-        dependency "org.apache.lucene:lucene-misc:${luceneVersion}"
-        dependency "org.apache.lucene:lucene-queries:${luceneVersion}"
-        dependency "org.apache.lucene:lucene-queryparser:${luceneVersion}"
-        dependency "org.apache.lucene:lucene-sandbox:${luceneVersion}"
-        dependency "org.apache.lucene:lucene-spatial:${luceneVersion}"
-        dependency "org.apache.lucene:lucene-suggest:${luceneVersion}"
-
-        dependency "org.elasticsearch:elasticsearch:${elasticClientVersion}"
+
         dependency "org.elasticsearch.client:elasticsearch-rest-client:${elasticClientVersion}"
-        dependency "org.elasticsearch.client:elasticsearch-rest-high-level-client:${elasticClientVersion}"
+
+        // The jakarta flavour: the javax one names javax.validation / javax.xml.bind classes nothing on a
+        // Boot 4 classpath provides. Pinned by SwaggerModelResolverTest.
+        dependency "io.swagger.core.v3:swagger-core-jakarta:${swaggerCoreVersion}"
+        dependency "io.swagger.core.v3:swagger-models-jakarta:${swaggerCoreVersion}"
+
+        dependency "com.github.dasniko:testcontainers-keycloak:${testcontainersKeycloakVersion}"
 
         dependency "org.kinotic:continuum-core:${continuumVersion}"
         dependency "org.kinotic:continuum-core-vertx:${continuumVersion}"
         dependency "org.kinotic:continuum-gateway:${continuumVersion}"
         dependency "org.kinotic:continuum-idl:${continuumVersion}"
 
-
-        // must be overridden for the elastic client
-        dependency 'jakarta.json:jakarta.json-api:2.0.1'
-
         // JWT and OIDC dependencies
         dependency "io.jsonwebtoken:jjwt-api:${jsonwebtokenVersion}"
         dependency "io.jsonwebtoken:jjwt-impl:${jsonwebtokenVersion}"
@@ -105,7 +103,7 @@ dependencyManagement {
 
 dependencies {
     implementation 'org.apache.commons:commons-lang3'
-    implementation "org.apache.commons:commons-text:${apacheCommonsTextVersion}"
+    implementation 'org.apache.commons:commons-text'
 
     implementation 'io.opentelemetry.instrumentation:opentelemetry-instrumentation-annotations'
 

buildSrc/src/main/groovy/org.kinotic.java-library-conventions.gradle

@@ -80,6 +80,12 @@ jreleaser {
                     active = 'RELEASE'
                     url = 'https://central.sonatype.com/api/v1/publisher'
                     applyMavenCentralRules = true
+                    // Each module is its own Central Portal deployment, polled until PUBLISHED. The default
+                    // 100 x 20 s (~33 min) cap was nearly hit by structures-sql on the 3.5.9 release (74 polls,
+                    // ~26 min); a module that overruns it fails while earlier ones are already live and immutable.
+                    // 200 x 30 s (~100 min) matches what JReleaser uses for its own releases.
+                    maxRetries = 200
+                    retryDelay = 30
                     stagingRepository('build/staging-deploy')
                 }
             }