Skip to content

perf(activation): optimize provisioning and reuse verified TLS identity - #3008

Open
Rashika-Rajaraman wants to merge 1 commit into
split/2935-tls-streamingfrom
fix/2935
Open

Rashika-Rajaraman wants to merge 1 commit into
split/2935-tls-streamingfrom
fix/2935

Conversation

@Rashika-Rajaraman

@Rashika-Rajaraman Rashika-Rajaraman commented Nov 5, 2026 •

Copy link
Copy Markdown

PR Checklist

  • Unit tests have been added for new changes.
  • API tests have been updated if applicable.
  • All commented code has been removed.
  • If a dependency has been added, its license is compatible with Apache 2.0 and the dependency is clearly outlined.

What are you changing?

Reduces repeated provisioning work and reuses an eligible existing TLS certificate and key during reconfiguration. Addresses #2935.

Why this reduces provisioning time

Avoids repeated AMT and Vault reads, completed TLS setup, and unnecessary reconnects during cleanup. Eligible identity reuse also skips the TLS rollover wait.

What it does

  • Shares validation results and reuses recent AMT inventory while the tunnel session remains current.
  • Preserves an eligible certificate/key and establishes a fresh authenticated tunnel. Retained identities require peer verification even when amt_post_tls_reject=false.
  • Avoids repeating completed TLS setup and keeps first-time --tls-tunnel routing correct after CCM activation.
  • Preserves working tunnels after expected cleanup responses, such as a 400 for an absent object. Transport failures still trigger recovery.
  • Clears retained approval and falls back to TLS provisioning when reuse checks fail.

When reuse is allowed

The active certificate must match Vault, be signed by the expected MPS root, match the device hostname, and have a valid credential association. The certificate and root must both be valid with more than 30 days remaining.

The control mode, TLS settings and profile must satisfy the reuse policy. Mutual TLS, supplied certificates, unsupported signing authorities, IEEE 802.1X and Wi-Fi profiles use the existing provisioning path. A final live check confirms that the approved identity remains valid after configuration.

CIRA policies and the MPS entry are still removed and recreated. Clock synchronization remains. The configured rollover wait, normally 15 seconds, remains where reuse cannot justify skipping it.

Review guide

Validation and review notes

1,510 tests passed, plus lint, formatting and compilation. Tests cover initial routing, stale sessions, rejected identities, shared Vault reads and changes detected during final validation.

Two review threads remain open: simplifying the final live check and establishing hostname availability earlier. The current checks remain while alternatives are investigated.

AMT 21: measured optimization results

RPS build RPC-go build Activation Reconfiguration Reduction vs baseline: activation / reconfiguration
Main Existing 2.x.x 194.39 s 105.68 s —
Streaming (#3039) Streaming (#1676) 91.13 s 28.17 s 53.12% / 73.34%
Streaming + TLS reuse (#3008) Streaming (#1676) 73.19 s 6.21 s 62.35% / 94.12%

Overall, the optimization changes together reduce activation time by 62.35% and reconfiguration time by 94.12% compared with baseline on AMT 21.

Measured on AMT 21 using the same device and test-profile, without --tls-tunnel. Each build received one fresh activation followed by one reconfiguration. Durations run from RPS receiving the activation request to final provisioning status, excluding RPC startup. Percentages are cumulative reductions against the baseline; these are single-run comparisons.

Tested commits: RPS main bff7f7a, streaming 66563cb, TLS reuse 43ecb50; RPC baseline d92e20f and streaming 70c2781.

Compared with streaming alone, this PR reduced activation by a further 17.94 seconds and reconfiguration by 21.96 seconds. This measures the full TLS reuse/provisioning optimization PR, rather than any one helper or certificate check.

Testing on other AMT versions

These changes were also tested on several other AMT versions. The activation and reconfiguration times are as follows:

Device Activation Reconfiguration
AMT 12.0.95 36.71 s 17.77 s
AMT 15 41.22 s 18.85 s
AMT 16.1.25 39.29 s 19.31 s

These results cover streaming and TLS reuse together; durations are measured by RPS, excluding RPC startup.

AMT 16 build details and coverage · AMT 12/15 results.

Stack and testing

Streaming #3039 → TLS reuse #3008 → Authentication recovery #3040 → Diagnostics #3041.

The PRs are grouped for easier combined testing. Authentication recovery is functionally independent of streaming and TLS reuse; its position in the stack is for testing convenience. Diagnostics is also a separate concern.

Review each PR against its immediate parent. Merge in this order and update each child's base to main after its parent merges, rebasing if needed.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Rejected HTTP responses can retain a closed tunnel when persistence is disabled or the server requests closure, causing subsequent requests to fail or time out.

3 open issues
What changed in this PR

This PR reduces redundant work in RPS activation and reconfiguration for #2935, adding verified TLS identity reuse and transport changes needed ahead of rpc-go#1676.

Changes:

  • Cache recent TLS inventory, revalidate eligible identities for reuse, and avoid redundant provisioning and rollover waits.
  • Update tunnel handling for HTTP errors and streamed TLS 1.3 handshakes, and retry MEBx password authentication challenges.
  • Add transition diagnostics and regression tests for provisioning, reuse, and recovery paths.
File Description
src/​utils/​stateTransitionLog.ts Add state-transition debug logging.
src/​utils/​constants.ts Add a numeric status-code type guard.
src/​TLSTunnelManager.ts Accept encrypted handshake chunks after ServerHello begins.
src/​TLSTunnelManager.injectData.test.ts Test streamed handshake chunk handling.
src/​test/​provisioningAnalysis.test.ts Characterize cleanup calls and activation routing.
src/​test/​helper/​xstate.ts Add a helper for testing machine segments.
src/​test/​helper/​tlsReuseCertificates.ts Generate certificates for reuse tests.
src/​stateMachines/​unconfiguration.ts Skip unused TLS reads when TLS is enforced.
src/​stateMachines/​unconfiguration.test.ts Test optimized cleanup and failure routing.
src/​stateMachines/​tlsReuse.ts Define identity reuse eligibility and trust selection.
src/​stateMachines/​common.tunnelReset.test.ts Test HTTP-error retention and transport resets.
src/​stateMachines/​common.ts Recognize retained identities and revise tunnel resets.
src/​stateMachines/​common.test.ts Test retained trust and status-code detection.
src/​stateMachines/​activation.ts Add inventory caching, identity reuse, and recovery routing.
src/​stateMachines/​activation.test.ts Cover reuse eligibility, invalidation, waits, and routing.
src/​stateMachines/​activation.revalidation.test.ts Test phase re-entry and legacy tunnel routing.
src/​stateMachines/​activation.mebx401.test.ts Test MEBx authentication recovery.
src/​models/​RCS.Config.ts Track a validated existing TLS certificate.
src/​DataProcessor.ts Attach transition tracing to activation actors.

Review effort: Balanced

Comment thread src/stateMachines/common.ts Resolved
Comment thread src/TLSTunnelManager.ts Resolved
Comment thread src/stateMachines/activation.ts Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Identity reuse can retain connections with peer verification disabled, and fragmented stale TLS records can leak into and corrupt a new handshake.

1 open issue
1 previously missed issue

In code that hasn't changed since last review

Medium severity Buffer fragmented TLS records before classifying stale data

src/​TLSTunnelManager.ts:154

If the RPC streaming reader splits a stale application-data record before the server handshake, this guard can forward stale bytes into the new handshake: a one-byte header fragment bypasses classification, while a split immediately after the header drops only the header and forwards the payload. This can corrupt the handshake instead of discarding the stale record. Buffer and classify complete TLS records before the first server handshake, and add tests for fragmented stale records.

3 resolved since last review

Review effort: Balanced

Comment thread src/stateMachines/activation.ts Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The tunnel is not marked for reset before asynchronous EOF draining, allowing the next WS-Man call to reuse the draining manager and lose its request when output is dropped.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread src/DataProcessor.ts Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

TLS identity reuse, activation re-entry, and streamed EOF handling cross firmware and RPC boundaries, while mocked tests leave representative AMT device behavior and before/after provisioning performance unvalidated.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@Rashika-Rajaraman

Copy link
Copy Markdown
Author

Hardware testing results — 11 November 2026

Tested the coordinated builds from RPS c7ecc99 and RPC-go 947d5c2 with TLS tunneling and amt_post_tls_reject=true.

Device First-time activation Reconfiguration
AMT 21.0.2 72.65 seconds 5.99 seconds
AMT 16.1.25 39.29 seconds 19.31 seconds

All four runs completed successfully.

  • AMT 21 used TLS 1.3. Reconfiguration confirmed unchanged TLS identity reuse, skipped the rollover wait, and established a fresh tunnel with certificate verification against the MPS root CA. Obsolete-session closure events were ignored.
  • AMT 16 used TLS 1.2. Both runs completed the legacy port switch and certificate verification. Reconfiguration included the 15-second port-switch wait, which accounts for most of its duration.
  • The final provisioning statuses reported successful activation or already-enabled ACM, wired network configuration, and CIRA configuration. AMT 16 activation also reported TLS configured; its reconfiguration status did not include a separate TLS entry, but the tunnel logs confirmed TLS connection and certificate verification.

Durations are measured from the RPS activation state-machine entry to the final provisioning status, excluding RPC startup. These are single-run measurements of the combined build, not a controlled before/after comparison or proof that CIRA connected to MPS.

Fragmented records, partial-response EOF, queued old-session requests, write/read failures, and replacement-tunnel shutdown are covered by automated regression tests. The normal hardware runs do not establish that every failure race was exercised. Latest CI passes on both PRs, and Copilot reports no open issues; human review is still required.

Related PRs: #3008 and device-management-toolkit/rpc-go#1676. RPS must land and be deployed before RPC-go.

@Rashika-Rajaraman
Rashika-Rajaraman marked this pull request as ready for review November 11, 2026 11:14
@Rashika-Rajaraman

Rashika-Rajaraman commented Nov 17, 2026 •

Copy link
Copy Markdown
Author

Additional testing completed on AMT 12 and AMT 15. All four runs passed.

Device First-time activation Reconfiguration
AMT 12.0.95 36.71 seconds 17.77 seconds
AMT 15 41.22 seconds 18.85 seconds

Timings were measured from RPS entering the activation flow to final provisioning status, excluding RPC startup.

@rsdmike rsdmike left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the testing on 21/16/12/15. I ran base vs this branch on an AMT 18 (RPS 2.48.0 image, same deps). Times are full rpc wall time:

  • first CCM 71s → 62s, with rpc-go#1676 38s
  • CCM reconfig 72s → 41s, with #1676 19s
  • first ACM 81s → 72s, with #1676 39s
  • ACM reconfig 73s → 42s, with #1676 19s
  • old RPS with #1676 still works on 18 (TLS 1.2)

On 18 the win comes from keeping the tunnel on HTTP 400s (1 handshake instead of 5) and not doing the vault leaf check and 15s port switch twice. The identity reuse only runs on 19+, so your AMT 21 run is the only hardware coverage for it.

Reuse makes sense to keep, it matches what we already do on 16-18. It's a lot of code for one decision though. Could it be smaller? Comments inline.

The two inline ones on the --tls-tunnel guard and validatedCertPEM need fixing before merge, the rest is cleanup.

To get the PR smaller overall:

  • drop src/test/provisioningAnalysis.test.ts, it asserts the old graph on a modified machine
  • drop stateTransitionLog.ts, or put it in its own PR
  • MEBx 401 fix in its own PR. Same thing happens on ADMIN_SETUP, UPGRADE_TO_ADMIN_SETUP and CHANGE_AMT_PASSWORD on main today, so it'd be nice to fix them together there
  • streaming changes (record parser, endInput, session id filtering, streamClosed) in their own PR next to rpc-go#1676

Comment thread src/stateMachines/activation.ts Resolved
Comment thread src/stateMachines/activation.ts Resolved
Comment thread src/stateMachines/activation.ts Resolved
Comment thread src/stateMachines/activation.ts Resolved
Comment thread src/stateMachines/activation.ts Resolved
Comment thread src/DataProcessor.ts Resolved
Comment thread src/DataProcessor.ts Resolved
Comment thread src/TLSTunnelManager.ts Resolved
Comment thread src/TLSTunnelManager.ts Resolved
Comment thread src/stateMachines/activation.test.ts Resolved

@rsdmike rsdmike left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for splitting it up, looks good. Couple small things if you get a chance, not blocking: the full reprobe in verifyReusableTlsIdentity could probably just be a TLS settings read, and the hostname == null check can't ever be true.

@Rashika-Rajaraman

Rashika-Rajaraman commented Nov 20, 2026 •

Copy link
Copy Markdown
Author

Thanks for splitting it up, looks good. Couple small things if you get a chance, not blocking: the full reprobe in verifyReusableTlsIdentity could probably just be a TLS settings read, and the hostname == null check can't ever be true.

@rsdmike, as these PRs are already tested, I'll fix these minor NITs in the follow up task. I've created an issue to make TLS certificate renewal threshold configurable. #3044

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants