Skip to content

feat: simulate real-world transport (TCP/TLS overhead, connection setup, slow-start curve) + raw UDP test - #845

Closed
Gutouoff wants to merge 2 commits into
librespeed:masterfrom
Gutouoff:feat/real-world-transport
Closed

feat: simulate real-world transport (TCP/TLS overhead, connection setup, slow-start curve) + raw UDP test#845
Gutouoff wants to merge 2 commits into
librespeed:masterfrom
Gutouoff:feat/real-world-transport

Conversation

@Gutouoff

@Gutouoff Gutouoff commented Aug 28, 2026

Copy link
Copy Markdown

Summary

Simulates real-world public-network transfer on top of LibreSpeed's existing XHR-based test, without changing the protocol stack (pure browser + an optional Timing-Allow-Origin header). Also adds a raw-UDP test for the UDP path that browsers cannot cover.

What changed

Transport overhead & connection setup (browser)

  • Real transport overhead instead of a fixed 1.06 factor: the reported line rate is now derived from an Ethernet + IP + TCP + TLS + reverse-ACK model (overhead_auto). The worker separately reports:
    • payload throughput (dlPayloadStatus / ulPayloadStatus)
    • estimated line rate (dlStatus / ulStatus)
    • overhead percentage (dlOverheadPct / ulOverheadPct)
  • Connection setup cost: TCP handshake, TLS handshake, TTFB and negotiated protocol (http/1.1 / h2 / h3) via Performance Resource Timing (tcpHandshakeMs, tlsHandshakeMs, ttfbMs, nextHopProtocol).
  • Slow-start curve: download/upload throughput sampled every ~200 ms (dlCurve / ulCurve), rendered as a chart in both the classic and modern UI.
  • Backend: Timing-Allow-Origin: * on garbage.php / empty.php / getIP.php so cross-origin (multiple points of test) timings are exposed.
  • Docs: new settings and metrics documented in doc.md; new defaults in settings.json.

Raw UDP test (udp/)

  • Self-contained, zero-dependency Node.js UDP test (mini-iperf3 style):
    • udp/udp-server.js — raw UDP server (echo ping, count upload, flood download)
    • udp/udp-client.js — CLI client measuring ping/jitter/loss plus upload/download throughput
    • udp/protocol.js — shared 13-byte datagram header (type + seq + timestamp)
    • udp/README.md — usage and options
  • Browsers cannot open raw UDP sockets, so this runs as a Node.js pair rather than in speedtest_worker.js. A browser-native UDP path would need WebTransport over HTTP/3 (QUIC) and is intentionally out of scope here.

Notes

  • Verified locally: UDP upload/download ~250-300 Mbps with 0% loss on loopback; ping/jitter/loss report correct packet accounting.

…FB, slow-start curve)

- Estimate wire line rate from an Ethernet+IP+TCP+TLS+ACK overhead model instead of the fixed 1.06 factor; report payload throughput, line rate and overhead % separately (dlPayloadStatus/ulPayloadStatus, dlOverheadPct/ulOverheadPct).
- Report TCP/TLS handshake, TTFB and negotiated protocol via Performance Resource Timing (collectNetworkInfo).
- Add download/upload throughput-over-time curves showing the slow-start ramp (dlCurve/ulCurve), rendered in classic and modern UI.
- Backend: add Timing-Allow-Origin header to garbage.php/empty.php/getIP.php for cross-origin timing.
- Document new settings and metrics in doc.md and settings.json.
@qodo-free-for-open-source-projects

Copy link
Copy Markdown
Contributor

PR Summary by Qodo

Model real-world transport metrics and slow-start curves

✨ Enhancement 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

Grey Divider

AI Description

• Estimates payload throughput, wire line rate, and overhead with configurable transport modeling.
• Exposes connection timing and protocol metrics through Resource Timing-enabled backend responses.
• Visualizes 200ms download and upload throughput curves across both interfaces.
Diagram

graph TD
  B["PHP endpoints"] --> X["XHR transfers"] --> W["Speedtest worker"] --> M["Status API"] --> U["UI renderers"]
  B --> R["Resource Timing"] --> W
  S["Transport settings"] --> O["Overhead model"] --> W
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Derive observed overhead from Resource Timing sizes
  • ➕ Uses browser-observed transfer metadata rather than a theoretical packet model
  • ➕ Can reflect some protocol and response-specific behavior
  • ➖ Transfer-size fields are unavailable without timing permission and inconsistent with caching or compression
  • ➖ Does not provide equivalent upload wire-size visibility
2. Use protocol-aware overhead models
  • ➕ Could model HTTP/2 and QUIC/HTTP/3 framing more accurately
  • ➕ Could adapt assumptions to the negotiated protocol
  • ➖ Adds substantial protocol-specific complexity and tuning
  • ➖ Connection reuse and browser internals still limit measurement accuracy

Recommendation: The configurable analytic model is a reasonable browser-only approach and preserves the existing XHR stack while exposing payload throughput separately. Keep the documented values explicitly framed as estimates; protocol-aware modeling is the strongest future refinement, particularly for reported h3 connections.

Files changed (11) +459 / -12

Enhancement (6) +424 / -8
index.htmlAdd modern transport result markup +23/-0

Add modern transport result markup

• Adds payload, line-rate, overhead, connection timing, protocol, and throughput-chart elements to the frontend template.

frontend/index.html

index.jsRender transport details and throughput curves +97/-0

Render transport details and throughput curves

• Populates the new metrics from worker status data and draws device-pixel-aware download and upload curves on a canvas.

frontend/javascript/index.js

results.cssStyle responsive transport result details +65/-0

Style responsive transport result details

• Adds nested responsive styles for transport rows, highlighted values, and the throughput chart.

frontend/styling/results.css

index-classic.htmlAdd transport metrics to the classic interface +77/-0

Add transport metrics to the classic interface

• Extends the classic page with payload, line-rate, overhead, connection details, and canvas-based throughput curve rendering.

index-classic.html

index-modern.htmlAdd transport metrics to the modern interface +23/-0

Add transport metrics to the modern interface

• Adds the transport detail panel and throughput chart markup consumed by the shared modern renderer.

index-modern.html

speedtest_worker.jsCompute and publish real-world transport metrics +139/-8

Compute and publish real-world transport metrics

• Replaces default fixed-factor reporting with configurable Ethernet/IP/TCP/TLS/ACK modeling while retaining manual fallback. It also collects Resource Timing connection data, samples 200ms payload-throughput curves, publishes all metrics, and resets them on abort.

speedtest_worker.js

Documentation (1) +20 / -3
doc.mdDocument transport settings and status metrics +20/-3

Document transport settings and status metrics

• Redefines reported speeds as estimated line rates and documents payload rates, overhead, connection timings, protocols, curves, and new configuration controls.

doc.md

Other (4) +15 / -1
empty.phpExpose upload and ping Resource Timing details +3/-0

Expose upload and ping Resource Timing details

• Adds 'Timing-Allow-Origin: *' so browsers can inspect detailed timing for cross-origin empty responses used by upload and ping flows.

backend/empty.php

garbage.phpExpose download Resource Timing details +4/-0

Expose download Resource Timing details

• Adds the timing permission header to generated download responses, including multiple-points-of-test deployments.

backend/garbage.php

getIP.phpExpose IP request Resource Timing details +2/-0

Expose IP request Resource Timing details

• Allows cross-origin Resource Timing access for the client IP endpoint.

backend/getIP.php

settings.jsonEnable real-world transport defaults +6/-1

Enable real-world transport defaults

• Enables automatic overhead estimation and connection reporting with default MTU, IPv4, and reverse-ACK assumptions.

settings.json

@qodo-free-for-open-source-projects

Copy link
Copy Markdown
Contributor

Code Review by Qodo

🐞 Bugs (4) 📘 Rule violations (0) 📜 Skill insights (0)

Grey Divider


Remediation recommended

1. Curve speed assumes exact interval 🐞 Bug ≡ Correctness
Description
The curve calculation always divides transferred bytes by 0.2 seconds even though the interval
callback can run later than 200 ms due to worker load or timer throttling. Delayed callbacks
therefore overstate instantaneous throughput (for example, a 400 ms interval is reported at roughly
twice its actual speed), making the new slow-start curve inaccurate.
Code

speedtest_worker.js[R501-504]

+			const curveDelta = totLoaded - curveLast;
+			curveLast = totLoaded;
+			if (curveDelta >= 0) {
+				dlCurve.push({ t: new Date().getTime() - curveStartT, speed: (curveDelta / 0.2) * 8 / (settings.useMebibits ? 1048576 : 1000000) });
Evidence
Both test loops are merely scheduled every 200 ms, while timestamps already record actual elapsed
time; nevertheless, sample speed is hard-coded to curveDelta / 0.2 on both paths.

speedtest_worker.js[494-504]
speedtest_worker.js[665-675]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Curve throughput is calculated with a fixed 0.2-second denominator, although timer callbacks do not execute at exact 200 ms intervals.

## Issue Context
Track the timestamp of the previous sample separately for download and upload, and divide each byte delta by the actual elapsed duration. A monotonic clock such as `performance.now()` is preferable.

## Fix Focus Areas
- speedtest_worker.js[435-436]
- speedtest_worker.js[500-504]
- speedtest_worker.js[574-575]
- speedtest_worker.js[671-675]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Upload-only details stay hidden 🐞 Bug ≡ Correctness
Description
The modern renderer reveals and populates the transport panel only when dlPayloadStatus is truthy,
even when valid upload payload data and an upload curve exist. Configurations that run only U,
which test_order explicitly supports, therefore never display any of the new upload transport
results or chart.
Code

frontend/javascript/index.js[R409-412]

+      if (testState.testData.dlPayloadStatus) {
+        dlPayload.textContent = testState.testData.dlPayloadStatus;
+        dlLine.textContent = testState.testData.dlStatus;
+        dlOverhead.textContent = testState.testData.dlOverheadPct || "0";
Evidence
The documented test_order setting permits running only specific tests, and the worker
independently produces upload status and curve data. The renderer nevertheless removes the parent
panel's hidden class only inside the download-status condition; because the chart is inside that
parent, drawing an upload series cannot make it visible.

doc.md[444-451]
speedtest_worker.js[695-700]
frontend/javascript/index.js[408-417]
frontend/index.html[72-93]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The transport panel is gated exclusively on download payload data, hiding valid upload-only results.

## Issue Context
`test_order` supports enabling only selected tests. Reveal the panel when either direction has data, and populate/hide each direction independently so absent download values do not block upload metrics or the chart.

## Fix Focus Areas
- frontend/javascript/index.js[408-417]
- frontend/javascript/index.js[519-530]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Modern details retain old results 🐞 Bug ≡ Correctness
Description
The modern renderer only writes connection fields when new values are nonzero and never clears or
hides the transport panel when a restarted or aborted test reports empty optional metrics. Results
from the previous run can therefore remain displayed indefinitely, especially when the next run has
unavailable connection timing or omits download/upload tests.
Code

frontend/javascript/index.js[R418-425]

+      if (
+        testState.testData.tcpHandshakeMs ||
+        testState.testData.tlsHandshakeMs ||
+        testState.testData.ttfbMs
+      ) {
+        connTcp.textContent = testState.testData.tcpHandshakeMs || "0";
+        connTls.textContent = testState.testData.tlsHandshakeMs || "0";
+        connTtfb.textContent = testState.testData.ttfbMs || "0";
Evidence
Restart reuses persistent page state without clearing testData or the added DOM, while the new
renderer only removes hidden and conditionally overwrites nonempty values. New workers initialize
these fields empty and the abort path clears them, so an absent value does not overwrite what the
prior run rendered.

frontend/javascript/index.js[15-22]
frontend/javascript/index.js[79-85]
frontend/javascript/index.js[408-430]
speedtest_worker.js[20-30]
speedtest_worker.js[359-368]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Transport and connection DOM state survives across test runs when the next status contains empty optional values.

## Issue Context
Each `Speedtest.start()` creates a new worker whose optional metrics begin empty, and abort also clears them. Reset the panel, values, protocol, and chart when starting a new run and explicitly clear/hide fields when current status data is absent.

## Fix Focus Areas
- frontend/javascript/index.js[79-85]
- frontend/javascript/index.js[408-430]
- frontend/javascript/index.js[514-529]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


View review recommended (1)
4. Classic details retain old results 🐞 Bug ≡ Correctness
Description
The classic renderer updates detailsText only when the newly built string is nonempty and never
hides or clears the details area otherwise. After restart or abort, old transport results remain
visible until a later enabled throughput test produces data, and persist for the entire run when the
next configuration is ping/IP-only.
Code

index-classic.html[R230-233]

+			if (txt) {
+				I("detailsArea").style.display = "";
+				I("detailsText").innerHTML = txt;
+			}
Evidence
The classic page reuses the same UI and Speedtest object across starts. updateDetails initializes
an empty string for every status but only mutates the details DOM inside if (txt), while the
worker reports empty optional values initially and after abort.

index-classic.html[162-201]
index-classic.html[217-234]
speedtest_worker.js[20-30]
speedtest_worker.js[359-368]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The classic transport panel preserves content from a previous run when current status has no transport details.

## Issue Context
Add an explicit empty-data branch that clears `detailsText`, hides `detailsArea`, and hides/clears the curve canvas. Also reset this state when starting or aborting a test.

## Fix Focus Areas
- index-classic.html[163-201]
- index-classic.html[217-234]
- index-classic.html[236-243]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Tip of the day
💡 Did you know, you can reply 'qodo' on any finding to push back, ask questions, or dig deeper

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread speedtest_worker.js
Comment on lines +501 to +504
const curveDelta = totLoaded - curveLast;
curveLast = totLoaded;
if (curveDelta >= 0) {
dlCurve.push({ t: new Date().getTime() - curveStartT, speed: (curveDelta / 0.2) * 8 / (settings.useMebibits ? 1048576 : 1000000) });

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

1. Curve speed assumes exact interval 🐞 Bug ≡ Correctness

The curve calculation always divides transferred bytes by 0.2 seconds even though the interval
callback can run later than 200 ms due to worker load or timer throttling. Delayed callbacks
therefore overstate instantaneous throughput (for example, a 400 ms interval is reported at roughly
twice its actual speed), making the new slow-start curve inaccurate.
Agent Prompt
## Issue description
Curve throughput is calculated with a fixed 0.2-second denominator, although timer callbacks do not execute at exact 200 ms intervals.

## Issue Context
Track the timestamp of the previous sample separately for download and upload, and divide each byte delta by the actual elapsed duration. A monotonic clock such as `performance.now()` is preferable.

## Fix Focus Areas
- speedtest_worker.js[435-436]
- speedtest_worker.js[500-504]
- speedtest_worker.js[574-575]
- speedtest_worker.js[671-675]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Comment on lines +409 to +412
if (testState.testData.dlPayloadStatus) {
dlPayload.textContent = testState.testData.dlPayloadStatus;
dlLine.textContent = testState.testData.dlStatus;
dlOverhead.textContent = testState.testData.dlOverheadPct || "0";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

2. Upload-only details stay hidden 🐞 Bug ≡ Correctness

The modern renderer reveals and populates the transport panel only when dlPayloadStatus is truthy,
even when valid upload payload data and an upload curve exist. Configurations that run only U,
which test_order explicitly supports, therefore never display any of the new upload transport
results or chart.
Agent Prompt
## Issue description
The transport panel is gated exclusively on download payload data, hiding valid upload-only results.

## Issue Context
`test_order` supports enabling only selected tests. Reveal the panel when either direction has data, and populate/hide each direction independently so absent download values do not block upload metrics or the chart.

## Fix Focus Areas
- frontend/javascript/index.js[408-417]
- frontend/javascript/index.js[519-530]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Comment on lines +418 to +425
if (
testState.testData.tcpHandshakeMs ||
testState.testData.tlsHandshakeMs ||
testState.testData.ttfbMs
) {
connTcp.textContent = testState.testData.tcpHandshakeMs || "0";
connTls.textContent = testState.testData.tlsHandshakeMs || "0";
connTtfb.textContent = testState.testData.ttfbMs || "0";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

3. Modern details retain old results 🐞 Bug ≡ Correctness

The modern renderer only writes connection fields when new values are nonzero and never clears or
hides the transport panel when a restarted or aborted test reports empty optional metrics. Results
from the previous run can therefore remain displayed indefinitely, especially when the next run has
unavailable connection timing or omits download/upload tests.
Agent Prompt
## Issue description
Transport and connection DOM state survives across test runs when the next status contains empty optional values.

## Issue Context
Each `Speedtest.start()` creates a new worker whose optional metrics begin empty, and abort also clears them. Reset the panel, values, protocol, and chart when starting a new run and explicitly clear/hide fields when current status data is absent.

## Fix Focus Areas
- frontend/javascript/index.js[79-85]
- frontend/javascript/index.js[408-430]
- frontend/javascript/index.js[514-529]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Comment thread index-classic.html
Comment on lines +230 to +233
if (txt) {
I("detailsArea").style.display = "";
I("detailsText").innerHTML = txt;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

4. Classic details retain old results 🐞 Bug ≡ Correctness

The classic renderer updates detailsText only when the newly built string is nonempty and never
hides or clears the details area otherwise. After restart or abort, old transport results remain
visible until a later enabled throughput test produces data, and persist for the entire run when the
next configuration is ping/IP-only.
Agent Prompt
## Issue description
The classic transport panel preserves content from a previous run when current status has no transport details.

## Issue Context
Add an explicit empty-data branch that clears `detailsText`, hides `detailsArea`, and hides/clears the curve canvas. Also reset this state when starting or aborting a test.

## Fix Focus Areas
- index-classic.html[163-201]
- index-classic.html[217-234]
- index-classic.html[236-243]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Self-contained zero-dependency Node.js UDP test (mini-iperf3 style) under udp/:
- udp-server.js: raw UDP server (echo ping, count upload, flood download)
- udp-client.js: CLI client measuring ping/jitter/loss plus upload/download throughput
- protocol.js: shared 13-byte datagram header (type+seq+timestamp)
- README.md and doc.md: usage, options, and mapping to LibreSpeed metrics

Browsers cannot open raw UDP sockets, so this runs as a Node.js pair rather than in speedtest_worker.js.
@Gutouoff Gutouoff changed the title feat: simulate real-world transport (payload vs line rate, TCP/TLS/TTFB, slow-start curve) feat: simulate real-world transport (TCP/TLS overhead, connection setup, slow-start curve) + raw UDP test Aug 28, 2026
@Gutouoff Gutouoff closed this Aug 28, 2026
@Gutouoff
Gutouoff deleted the feat/real-world-transport branch August 28, 2026 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant