Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,17 @@ performance changes on a real device in a **stable, reproducible** way. Not run

## What it measures

`SentryStartupBenchmark` runs a cold start and reports **`timeToInitialDisplay`**
(`StartupTimingMetric`) per iteration — the whole app cold start, taken from framework trace
events. No trace markers are required in the SDK or the app.

The flip side of marker-free measurement: an SDK change has to be large enough (roughly tens of
milliseconds) to show above cold-start noise. Sub-millisecond changes are not resolvable with
`timeToInitialDisplay` alone; for those, capture a perfetto trace and inspect the relevant slices
directly (each iteration's trace is saved under
`SentryStartupBenchmark` runs a cold start and reports two metrics per iteration:

- **`timeToInitialDisplay`** (`StartupTimingMetric`) — the whole app cold start, taken from
framework trace events. Because it captures the entire start, an SDK change has to be large enough
(roughly tens of milliseconds) to show above cold-start noise.
- **`SentryAndroid.init`** (`TraceSectionMetric`) — the duration of the `SentryAndroid.init`
`android.os.Trace` section the SDK emits, which isolates SDK-init cost from the rest of the start
and resolves changes that `timeToInitialDisplay` would lose in the noise.

For even finer detail (sub-millisecond changes, or cost inside init), capture a perfetto trace and
inspect the relevant slices directly (each iteration's trace is saved under
`build/outputs/connected_android_test_additional_output/`).

`CompilationMode.Full()` pins ART AOT so dexopt state can't drift between runs. `StartupMode.COLD`
Expand Down Expand Up @@ -49,5 +52,6 @@ Results print to the console and are written to
Macrobenchmark measures one build per run, so compare separate runs — but **interleave them**:
running all of variant A followed by all of variant B lets thermal drift systematically penalize
whichever variant runs second. Instead, alternate A/B rounds (build variant A, run, build variant
B, run, repeat 2–3 times), keep each round's `*-benchmarkData.json`, and compare the
`timeToInitialDisplay` values pooled per variant.
B, run, repeat 2–3 times), keep each round's `*-benchmarkData.json`, and compare the values pooled
per variant. Prefer the `SentryAndroid.init` metric for SDK-init changes — it isolates init cost, so
it moves on changes that `timeToInitialDisplay` would bury in cold-start noise.
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
package io.sentry.uitest.android.macrobenchmark

import androidx.benchmark.macro.CompilationMode
import androidx.benchmark.macro.ExperimentalMetricApi
import androidx.benchmark.macro.StartupMode
import androidx.benchmark.macro.StartupTimingMetric
import androidx.benchmark.macro.TraceSectionMetric
import androidx.benchmark.macro.junit4.MacrobenchmarkRule
import androidx.test.ext.junit.runners.AndroidJUnit4
import org.junit.Rule
Expand All @@ -13,17 +15,20 @@ import org.junit.runner.RunWith
* Cold-start benchmark for the sentry-samples-android app, used to evaluate SDK-init changes on a
* real device in a stable, repeatable way.
*
* Reports timeToInitialDisplay ([StartupTimingMetric]) per iteration. This measures the whole app
* cold start from framework trace events, with no trace markers in the SDK or the app — which also
* means SDK changes need to be large enough (roughly tens of milliseconds) to show above cold-start
* noise.
* Reports two metrics per iteration:
* - timeToInitialDisplay ([StartupTimingMetric]) — the whole app cold start from framework trace
* events. Because it captures the entire start, an SDK change has to be large enough (roughly
* tens of milliseconds) to show above cold-start noise.
* - SentryAndroid.init ([TraceSectionMetric]) — the duration of the `SentryAndroid.init`
* [android.os.Trace] section the SDK emits, isolating SDK-init cost from the rest of the start.
*
* [CompilationMode.Full] pins ART AOT compilation so dexopt state does not drift between runs.
* Iterations are capped at 12: on an unthrottled Pixel 3, back-to-back cold starts hit thermal
* throttling after ~14 iterations, which inflates the tail of longer runs. This is NOT a CI test;
* it requires a connected device. To A/B an SDK change, see README.md (build the app twice, once
* per SDK variant, in interleaved rounds).
*/
@OptIn(ExperimentalMetricApi::class)
@RunWith(AndroidJUnit4::class)
class SentryStartupBenchmark {

Expand All @@ -33,7 +38,7 @@ class SentryStartupBenchmark {
fun startupFullCompilation() =
benchmarkRule.measureRepeated(
packageName = TARGET_PACKAGE,
metrics = listOf(StartupTimingMetric()),
metrics = listOf(StartupTimingMetric(), TraceSectionMetric(INIT_TRACE_SECTION)),
compilationMode = CompilationMode.Full(),
startupMode = StartupMode.COLD,
iterations = 12,
Expand All @@ -44,5 +49,8 @@ class SentryStartupBenchmark {

private companion object {
const val TARGET_PACKAGE = "io.sentry.samples.android"

// Matches the android.os.Trace section name in SentryAndroid.init.
const val INIT_TRACE_SECTION = "SentryAndroid.init"
}
}
Loading