Testing Guide

PerlOnJava provides a two-level testing strategy to balance development speed with comprehensive validation.

Interpreting Compatibility Figures

Project-wide compatibility percentages are dated snapshots, not support guarantees. A published figure must identify its corpus, measurement date, passing count, and total count.

The upstream Perl percentage is calculated by dev/tools/perl_test_runner.pl as passing TAP checks divided by the checks in the imported perl5_t compatibility corpus. It describes that imported corpus, not every test distributed with upstream Perl. Skips, TODOs, incomplete files, and errors are reported separately.
The CPAN percentage comes from the generated CPAN compatibility report. Modules are randomly selected from the CPAN index and run with jcpan -t; dependencies encountered during testing are recorded too. A pass means the tested module's complete test suite passed in that run.

See the current project status for the latest published snapshots. For adoption decisions, consult the Feature Matrix and test the exact CPAN distributions and versions required by the application.

Quick Start

Fast Unit Tests (Recommended for Development)

make test-unit

Runs only the fast unit tests from src/test/resources/unit/. These tests:

✅ Run in seconds (not minutes)
✅ Cover core functionality
✅ Use parallel execution (8 jobs)
✅ Provide immediate feedback during development
✅ Output TAP format with detailed statistics

Comprehensive Test Suite

make test-all

Runs all tests including comprehensive module tests. These tests:

🔍 Include Benchmark.pm and other Perl core modules
⏱️ Take longer to complete (minutes)
📊 Generate detailed JSON report (test_results.json)
🎯 Identify high-priority opportunities (incomplete tests)
📈 Provide feature impact analysis

Perl Thread Compatibility

The permanent pull-request gate runs the complete unchanged upstream threads, threads::shared, Thread::Queue, and Thread::Semaphore suites on the JVM and interpreter backends with virtual carriers. It also runs focused lifecycle, signal, stack, condition, timeout, and deadlock tests with platform carriers:

perl dev/import-perl5/update_perl5.pl
make test-threads

The complete same-commit Perl core direct/thread matrix is a separate strict gate while direct language parity is being completed:

make test-threads-core

The update helper clones the gitignored perl5/ tree when absent and fast-forwards its default branch when present. CI uses a current sparse checkout containing the four required distributions and the core test harness; the exact commit is recorded as run provenance, not pinned as a compatibility target. The release-only regex anchors use the imported perl5_t/t/re tree; populate it with perl dev/import-perl5/sync.pl when necessary.

Before a thread/runtime release, extend that gate to the complete four-mode matrix (both execution backends on virtual and platform carriers) and the post-Joni regex-thread anchors:

make test-threads-release

The release target includes the platform-carrier core-wrapper matrix. Windows CI runs the shell-independent focused equivalent with make test-threads-windows.

The distribution matrix uses eight jobs and a hard 300-second timeout for each file. The five regex anchors contain 48 assertions per backend and use a 600-second bound. JSON reports are written under build/reports/threads/. The shorter target is used by Ubuntu pull-request CI and uses the runner's strict exit mode, so any failed, errored, timed-out, or incomplete file fails the Make target. Windows runs the normal Java/unit build plus the focused thread-runtime gate.

For a native callback or ORM release, run the slow ecosystem gate as well:

make test-threads-ecosystem

It runs pinned Test2, Storable, and Moose thread tests, unchanged Net::SSLeay thread tests 61/62, and DBIx::Class through jcpan --jobs 8, with a hard one-hour outer bound. See the Perl threads reference for the compatibility contract and resource policies.

Testing Approaches

1. Perl-Style Testing (Default)

Uses dev/tools/perl_test_runner.pl - a prove-like test harness:

# Fast unit tests
make test-unit

# All tests with JSON report
make test-all

# Custom test run
perl dev/tools/perl_test_runner.pl --jobs 4 --timeout 20 src/test/resources/unit

Features:

TAP (Test Anything Protocol) output
Resource-weighted parallel test execution
Timeout protection
Feature impact analysis
Incomplete test detection
JSON reporting

The runner is a semantic validator, not a performance benchmark harness. --jobs is a scheduling-unit budget shared by the whole run: ordinary files consume one unit, known CPU/memory-heavy semantic fixtures consume three, and any future test requiring demonstrated process isolation can run alone. No current test has an exclusive semantic profile. A heavy file's weight is clamped to the caller's budget, so it still runs with --jobs 1 or --jobs 2. The runner starts known long-running heavy files before ordinary files, preserving input order within each class. Starting the long work early avoids a heavy-test tail; the more uniform ordinary files fill unused scheduling units as heavy tests complete.

Resource profiles are defined in dev/tools/lib/PerlTestRunner/Scheduler.pm. Tune them from semantic stability, peak memory, and orphan-process checks across supported CI platforms. Do not tune them to preserve benchmark ratios or elapsed-time measurements; collect authoritative timings with a separate controlled benchmark procedure.

Options:

--jobs|-j NUM      Total scheduling-unit budget (default: 5)
--timeout SEC      Timeout per test in seconds
--output FILE      Save detailed results to JSON file
--jperl PATH       Path to jperl executable (default: ./jperl)

2. Using jprove (Standard Perl prove)

PerlOnJava includes jprove (Unix) and jprove.bat (Windows), wrappers that run the standard Perl prove test harness with jperl:

# Run tests in a directory
./jprove src/test/resources/unit

# Run with verbose output
./jprove -v t/*.t

# Run specific test files
./jprove t/basic.t t/advanced.t

# Run recursively
./jprove -r t/

# Run with parallel jobs
./jprove -j4 t/

Common Options:

-v, --verbose      Print all test lines
-l, --lib          Add 'lib' to @INC
-r, --recurse      Recursively descend into directories
-j, --jobs N       Run N test jobs in parallel
-q, --quiet        Suppress some test output
--timer            Print elapsed time after each test
--color            Colored test output (default)
--nocolor          Disable colored output

Example Output:

./jprove src/test/resources/unit/array.t
src/test/resources/unit/array.t .. ok
All tests successful.
Files=1, Tests=15, 1 wallclock secs
Result: PASS

jprove is useful when you want standard Perl prove behavior and options, while perl_test_runner.pl provides additional features like JSON reporting and feature impact analysis.

3. JUnit Testing (For CI/CD)

Uses JUnit 5 with tags for test filtering:

# Fast unit tests
make test-gradle-unit

# All tests
make test-gradle-all

Use Cases:

CI/CD pipeline integration
IDE integration (IntelliJ, VSCode)
JUnit test reports
Maven-style testing

Test Organization

src/test/resources/
├── unit/              # Fast unit tests (seconds)
│   ├── array.t
│   ├── hash.t
│   ├── regex/
│   └── ...

Test Categories

(Work in progress)

Category	Location	Speed	Purpose
Unit Tests	`unit/`	Fast (seconds)	Core functionality, operators, syntax
Module Tests	`Benchmark/`, `lib/`, etc.	Slow (minutes)	Perl core modules, CPAN compatibility
Integration Tests	`dist/`, `ext/`	Varies	Package integration, extensions

Development Workflow

During Development (Fast Feedback Loop)

# 1. Make changes
vim src/main/java/org/perlonjava/...

# 2. Build and run the fast unit suite
make

Before Committing (Comprehensive Validation)

# Run full test suite
make test-all

# Review test_results.json for any regressions

Tracking Test Progress Over Time

For long-running development work, track test results over time to monitor progress and catch regressions:

1. Run Tests with Timestamped Logs

# Run with extended timeout and save to dated log
perl dev/tools/perl_test_runner.pl \
  --jobs 10 \
  --timeout 300 \
  --output out.json \
  perl5_t/t \
  > logs/test_$(date +%Y%m%d_%H%M%S).log 2>&1

Workflow details:

Inspect exact process command lines before and after a long run. Never use a broad Java process kill: it can terminate an active build or another user's test. If an abandoned PerlOnJava JVM is proven by its full command line, terminate that exact PID only.
Do not recursively delete wildcard test directories as routine preparation. Remove an exact, verified stale path only when the failing test requires it.
--jobs 10 - Allow ten scheduling units of concurrent test work
--timeout 300 - Allow 5 minutes per test (for slower tests)
logs/test_YYYYMMDD_HHMMSS.log - Timestamped log for tracking history

2. Compare Test Runs

Compare two test runs to see what changed:

perl dev/tools/compare_test_logs.pl \
  logs/test_20260206_090000.log \
  logs/test_20260206_102400.log

Output shows:

Tests that started passing
Tests that started failing
Changes in test counts
New tests added or removed
Summary of improvements or regressions

Use cases:

Before/after implementing a feature
Daily progress tracking on a branch
Identifying when a regression was introduced
Measuring impact of optimizations

3. Log Organization

Suggested logs/ directory structure:

logs/
├── test_20260206_090000.log  # Baseline
├── test_20260206_102400.log  # After Feature A
├── test_20260206_153000.log  # After Bug Fix
└── test_20260207_101500.log  # Latest

Keep baseline logs for major milestones to track long-term progress.

CI/CD Pipeline

# Build and run all tests
make build
make test-gradle-all

Test Output

TAP Output (perl_test_runner.pl)

Finding test files in src/test/resources/unit...
Found 142 test files
Running tests with ./jperl (8 parallel jobs, 10s timeout)
------------------------------------------------------------
[  1/142] unit/array.t                     ... ✓ 15/15 ok (0.23s)
[  2/142] unit/hash.t                      ... ✓ 12/12 ok (0.18s)
[  3/142] unit/regex/basic.t              ... ✓ 25/25 ok (0.31s)
...

TEST SUMMARY:
  Total files: 142
  Passed:      140
  Failed:      2
  Errors:      0
  Timeouts:    0
  Incomplete:  0

  Total tests: 3,456
  OK:          3,421
  Not OK:      35
  Pass rate:   99.0%

JUnit Output (Gradle)

> Task :testUnit

PerlScriptExecutionTest > Unit test: unit/array.t PASSED
PerlScriptExecutionTest > Unit test: unit/hash.t PASSED
...

BUILD SUCCESSFUL in 12s
142 tests completed, 140 succeeded, 2 failed

Advanced Usage

Running Specific Tests

# Single test file
perl dev/tools/perl_test_runner.pl src/test/resources/unit/array.t

# Specific directory
perl dev/tools/perl_test_runner.pl src/test/resources/unit/regex

# With custom settings
perl dev/tools/perl_test_runner.pl --jobs 16 --timeout 60 --output mytest.json t/

Analyzing Test Results

After running make test-all, examine test_results.json:

# View summary
jq '.summary' test_results.json

# Find failing tests
jq '.results | to_entries | map(select(.value.status == "fail")) | .[].key' test_results.json

# Feature impact
jq '.feature_impact' test_results.json

Debugging Failures

# Run single test with full output
./jperl src/test/resources/unit/array.t

# Run with verbose output
./jperl -d src/test/resources/unit/array.t

# Check for syntax errors
./jperl -c src/test/resources/unit/array.t

Best Practices

Run test-unit frequently during development for fast feedback
Run test-all before commits to catch regressions
Set a resource budget (--jobs 8) to control concurrent test work
Set appropriate timeouts - short for unit tests (10s), longer for module tests (30s)
Review incomplete tests - they often indicate bugs that block many tests
Save JSON reports for trend analysis and debugging

Performance Tips

Unit tests: Should complete in < 5 minutes total
All tests: May take 10-30 minutes depending on system
Parallel work: Adjust --jobs based on CPU capacity and available memory; heavy semantic fixtures consume three units each
Timeouts: Increase for slow systems, decrease for fast feedback

Integration with IDEs

IntelliJ IDEA / VSCode

Open project
Right-click on PerlScriptExecutionTest.java
Select "Run tests"
Or run specific tags: @Tag("unit") or @Tag("full")

Command Line

# Run with tags
make test-gradle-unit  # @Tag("unit")
make test-gradle-all   # @Tag("full")

Troubleshooting

Tests timing out

Increase timeout: --timeout 60

Too slow

Reduce parallel jobs on low-memory systems: --jobs 2

Out of memory

Reduce parallel jobs
Increase JVM heap: export JAVA_OPTS="-Xmx4g"

Test hangs

Check for infinite loops, use timeout command:

timeout 30s ./jperl problematic_test.t

Importing Perl5 Test Suite

PerlOnJava can import and run tests from the official Perl5 repository to verify compatibility and behavior.

Setup

To import Perl test files and verify their behavior under PerlOnJava:

Clone the Perl5 repository (if not already done):

rm -rf perl5  # if it exists
git clone https://github.com/Perl/perl5.git

Run the import script to copy tests and apply patches:
```
perl dev/import-perl5/sync.pl
```

This script reads dev/import-perl5/config.yaml and copies configured files from the perl5/ directory (Perl 5 source repository) to perl5_t/ at the project root, creating:

Core tests in perl5_t/t/
Module tests in perl5_t/[Module]/
Test infrastructure (TestInit.pm, MANIFEST)
Supporting files (Porting/ directory)

The script also applies patches from dev/import-perl5/patches/ for PerlOnJava compatibility.

For how this differs from CPAN install-time patches under jcpan, see CPAN Distroprefs for PerlOnJava and dev/design/patch-and-cpan-prefs-layout.md.

Running Imported Tests

To run the imported Perl5 tests:

perl dev/tools/perl_test_runner.pl --output out.json perl5_t/t

See dev/import-perl5/README.md for more details on:

The import system architecture
How to add patches for PerlOnJava compatibility
Managing test expectations

FilesExpand file tree

testing.md

Latest commit

History