List part files lazily during upload instead of the whole table (#1550)

[Slach]

Fix #1550

Where the memory went

Production case: clickhouse-backup server 2.8.0 at 9.4-10.8 GB RSS during create_remote. Heap profile (api.enable_pprof: true, GET /debug/pprof/heap):

Showing nodes accounting for 4000.68MB, 99.37% of 4025.95MB total
      flat  flat%   sum%        cum   cum%
 3535.59MB 87.82% 87.82%  3535.59MB 87.82%  internal/bytealg.MakeNoZero             <- path strings (strings.Join <- filepath.Join <- filepath.Walk)
  375.09MB  9.32% 97.14%   375.09MB  9.32%  backup.(*Backuper).splitFilesByName.func1  <- the []string slices
      90MB  2.24% 99.37%       90MB  2.24%  transfermanager.newDefaultSlicePool        <- S3 upload buffers, expected
         0     0% 99.37%  3910.68MB 97.14%  backup.(*Backuper).uploadTableData
         0     0% 99.37%  3910.68MB 97.14%  backup.(*Backuper).splitPartFiles
         0     0% 99.37%  3910.68MB 97.14%  backup.(*Backuper).splitFilesByName

go_memstats_heap_alloc_bytes    4.26e9
go_memstats_heap_inuse_bytes    4.91e9
go_memstats_heap_idle_bytes     5.94e9
process_resident_memory_bytes   9.67e9

The database: two tables with 1115 columns, PARTITION BY Date, ~3400 partitions -> ~3900 active Wide parts of ~12 MB each (min_bytes_for_wide_part = 10 MB). ~2500 files per part (<column>.bin, .cmrk2, .sparse.idx.bin). find | wc -l on the backup shadow dir: 9.8M and 10.5M files in the two tables, 21.7M files in the whole backup. Both tables were being uploaded at the same time (upload_concurrency: 2), so ~20 million relative paths were resident.

Two things made that expensive:

1. splitFilesByName walked every part of the table before the first upload goroutine started, and the resulting []metadata.SplitPartFiles stayed referenced until uploadTableData returned. Memory was O(files in the table), while the upload only needs the files of the upload_concurrency parts currently being archived.

2. relativePath := strings.TrimPrefix(filePath, basePath) is a substring of the full path allocated by filepath.Walk, so every stored relative path also pinned the ~110 byte /var/lib/clickhouse/backup/<backup>/shadow/<db>/<table>/<disk>/ prefix. Measured on 1M synthetic paths of that shape:

   TrimPrefix only:        1000000 paths retain 170 MiB (178 B/path)
   TrimPrefix + Clone:     1000000 paths retain  61 MiB ( 64 B/path)
   

   20M × 178 B ≈ 3.4 GB, matching the 3.5 GB of MakeNoZero in the profile.

How we save it

• upload_by_part: true (default): splitFilesByName now returns only the part names. The directory walk moved into walkPartFiles, called inside the per-part upload goroutine right before UploadCompressedStream / UploadPath, after the resumableState.IsAlreadyProcessed check. Consequences:
  • at most upload_concurrency part file lists are alive per table, instead of the whole table: for the case above ~2 × 2500 paths instead of ~10M per table
  • parts already uploaded (--resume) skip the directory walk entirely
  • the walk of part N+1 overlaps with the upload of part N instead of all walks happening up front, so the upload of the first part starts immediately
• upload_by_part: false: the list still has to be built up front to cut archives by max_file_size, but each relative path is strings.Cloned, so the basePath prefix is no longer retained (~3× less per file).
• Behavior is unchanged otherwise: same set of files, same skipProjections filtering (Add option skip_projections with table pattern to allow make backup without projection #861), same Required parts handling, same warning on walk errors.

TestWalkPartFilesMemory (unit) reproduces the shape at small scale: 50 parts × 400 files — whole-table list 2520 KiB vs 104 KiB max for one part at a time.

Verification

• GOFLAGS= make test (vet + unit)
• RUN_TESTS='TestProjections|TestFTP' ./test/integration/run.sh — covers skipProjections and compression_format: none/tar upload paths

🤖 Generated with Claude Code

[coveralls]

Coverage Report for CI Build 34815986991

Warning

Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes.
Quick fix: rebase this PR. Learn more →

Coverage decreased (-0.03%) to 67.04%

Details

• Coverage decreased (-0.03%) from the base build.
• Patch coverage: 1 uncovered change across 1 file (27 of 28 lines covered, 96.43%).
• 41 coverage regressions across 9 files.

Uncovered Changes

File	Changed	Covered	%
pkg/backup/upload.go	28	27	96.43%

Coverage Regressions

41 previously-covered lines in 9 files lost coverage.

File	Lines Losing Coverage	Coverage
pkg/storage/gcs.go	15	54.46%
pkg/clickhouse/clickhouse.go	11	79.66%
pkg/backup/restore.go	4	71.78%
pkg/backup/create.go	2	74.49%
pkg/backup/list.go	2	70.18%
pkg/metadata/backup_metadata.go	2	71.43%
pkg/storage/object_disk/object_disk.go	2	68.63%
pkg/storage/structs.go	2	81.82%
pkg/status/status.go	1	90.78%

---

Coverage Stats

Relevant Lines:	26056
Covered Lines:	17468
Line Coverage:	67.04%
Coverage Strength:	35142.75 hits per line

---

💛 - Coveralls