Skip to content
SIA-IDEPublic

About

Intelligibility-Driven Real-Time Personalized Speech Enhancement for the Super-Aged via Dual-Objective Decoupling and Progressive Sparsification

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Intelligibility-Driven Real-Time Personalized Speech Enhancement for the Super-Aged

Haotian Peng, Jie Gao, Wei Wang, Qingshi Zhang, and Jinsong Du

Shenyang Institute of Automation, Chinese Academy of Sciences

IEEE TASLP PyTorch code Python 3.8+ FCRN-C checkpoint

Accepted by IEEE Transactions on Audio, Speech, and Language Processing (TASLP)

ID-PSE is a personalized speech-enhancement framework designed for speakers aged 75 and above. It jointly preserves weak phonemic cues, restores aging-degraded harmonic structure, and converts a full-context model into a causal student through Progressive Weights Sparsification.

ID-PSE framework

The ID-PSE framework. Click the figure to open the original PDF.

Highlights

  • Decoupled Dual-Objective Learning. VoiceFixer-restored speech provides perceptual guidance, while the original speech supplies a HuBERT-based phonemic anchor.
  • FiLM-based personalized enhancement. A 192-dimensional ECAPA-TDNN speaker embedding modulates all five encoder stages of a complex convolutional recurrent network.
  • Progressive Weights Sparsification. Future-dependent weights are progressively suppressed before physical pruning and causal adaptation, reducing the full-context FCRN-F to the real-time FCRN-C architecture.

Experiments on SeniorTalk show that ID-PSE provides the strongest overall balance between perceptual quality and phonemic integrity among the evaluated systems, including in simulated home and hospital scenarios.

Qualitative result

ID-PSE qualitative spectrogram comparison

Compared with the strongest personalized baseline, ID-PSE suppresses interfering speech while restoring aging-degraded harmonic structure.

Audio demo

The repository includes two 15-second, single-channel, 16 kHz examples. Click a file to listen to or download it.

Scenario Noisy mixture ID-PSE output Clean reference
Home, SNR/SIR 15 dB Input Enhanced Reference
Hospital, SNR/SIR 5 dB Input Enhanced Reference

Quick start

1. Install

git clone https://github.com/SIA-IDE/ID-PSE.git
cd ID-PSE
pip install -r requirements.txt

Use mutually compatible torch and torchaudio builds. For CUDA or Apple Silicon, install the platform-specific PyTorch packages first and then install the remaining dependencies.

2. Run inference

python demo_3_inference.py

The interactive demo supports CPU, CUDA, and MPS. It provides the two prepared scenarios above and accepts custom noisy audio and a pre-computed 192-dimensional enrollment embedding.

3. Inspect complexity and speed

Script Purpose Command
demo_1_parameter.py Parameters, FLOPs, future-weight loss, and pruning checks python demo_1_parameter.py
demo_2_speed.py Three-frame causal-context forward benchmark python demo_2_speed.py
demo_3_inference.py Interactive enhancement with the released checkpoint python demo_3_inference.py

demo_2_speed.py is a local network-forward benchmark. Its optional torch.compile mode and local hardware results should not be compared directly with the native FP32 timing table in the paper.

Reported results

The following results are reported for the General test scenario in the paper.

Model CER ↓ PCS ↑ STOI ↑ SI-SDR ↑
pDCCRN 0.52 0.55 0.65 -5.24 dB
ID-PSE 0.45 0.63 0.68 -3.85 dB
FCRN-F 0.45 0.65 0.68 -3.50 dB

FCRN-F is the non-causal full-context reference and is excluded from causal-system ranking.

Model Parameters FLOPs per second of audio Causal
ID-PSE / FCRN-C 5.10 M 8.47 G Yes
FCRN-F 7.39 M 14.42 G No

These are manuscript results rather than values regenerated by the demo scripts.

Model configuration

Component Configuration
Audio Single channel, 16 kHz
Time-frequency analysis 512-point STFT, 256-sample hop
Speaker conditioning 192-dimensional enrollment embedding
Encoder and decoder Five convolutional and five transposed-convolutional blocks
Bottleneck Two-layer GRU, hidden size 128
Temporal kernels FCRN-F: 5 frames; FCRN-C: 3 frames
Frequency kernels 5 in the first/last blocks; 3 in intermediate blocks
Output Complex ratio mask plus complex spectral residual

Repository structure

.
├── README.md                         # Project overview, demos, results, and citation
├── .gitignore                        # Local manuscript and generated-file exclusions
├── model.py                          # FCRN-F/FCRN-C definitions and PWS helpers
├── demo_1_parameter.py               # Parameter, FLOPs, and pruning checks
├── demo_2_speed.py                   # Causal-context forward benchmark
├── demo_3_inference.py               # Interactive enhancement demo
├── requirements.txt                  # Python dependencies
├── demo_data/
│   ├── id_pse.pt                     # Pre-trained FCRN-C state dictionary
│   ├── enrollment_embedding.pt       # Pre-computed speaker embedding
│   ├── home_sir_15db_snr_15db.wav    # Home-scenario noisy input
│   ├── hospital_sir_5db_snr_5db.wav  # Hospital-scenario noisy input
│   └── source_speech.wav             # Clean reference speech
├── demo_out/
│   ├── home_out.wav                  # Enhanced home-scenario output
│   └── hospital_out.wav              # Enhanced hospital-scenario output
└── docs/
    └── images/
        ├── README.md                 # Figure asset notes
        ├── fig_framework.pdf         # Original framework figure
        ├── fig_framework.png         # GitHub preview of the framework figure
        ├── fig_result.pdf            # Original qualitative result figure
        └── fig_result.png            # GitHub preview of the result figure

Citation

The bibliographic fields marked TO_BE_ADDED will be updated after the final IEEE record and DOI become available.

@article{pengIDPSE2026,
  author  = {Peng, Haotian and Gao, Jie and Wang, Wei and Zhang, Qingshi and Du, Jinsong},
  title   = {Intelligibility-Driven Real-Time Personalized Speech Enhancement for the Super-Aged via Dual-Objective Decoupling and Progressive Sparsification},
  journal = {IEEE Transactions on Audio, Speech, and Language Processing},
  year    = {2026},
  volume  = {TO_BE_ADDED},
  number  = {TO_BE_ADDED},
  pages   = {TO_BE_ADDED},
  doi     = {TO_BE_ADDED},
  note    = {Accepted; final bibliographic information to be added}
}

About

Intelligibility-Driven Real-Time Personalized Speech Enhancement for the Super-Aged via Dual-Objective Decoupling and Progressive Sparsification

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages