Constant traffic?!

[tinhoff]

Hi again,

noticing something strange: Updated a few days ago from 1.2.2 to 1.4.2 (currently I'm on 1.4.5); so can not tell exactly, which release started it.

When starting zerofs, without mounting or using anything, I already getting constant traffic from zerofs, ~1mb/s RX, 50kb/s TX. Console logs in zerofs show no activity running at all, starting with RUST_LOG=zerofs=debug shows the same as the web-ui: There is absolutely no activity at all. Had it run the last 24 hours without mounting my zpool (nor with nbdclient connected, not any other usage): According to my beszel monitoring, it was 24 hours constantly 789kb/s to 1.14 mb/s RX, and 40 - 90 kb/s TX for 24H straight. Using tcpdump, yeah, this is traffic to the s3 server.

Is this suppose to happen? It's a pretty small pool I'm using, 500GB nbd with currently roughly 300GB in use on a COW file system. Is this some internal cleanup / garbage collecting, which will stop some day? As, currently, this generates around 80GB of download traffic alone per day.

BR

[Barre]

Hi,

#485

There's a bug in the compactor that makes it timeout too quickly (and retry needlessly, wasting bandwidth), once this PR land, this should fix it for you!

[tinhoff]

Cool, I'll wait for this PR. Thanks for the quick response, yet again.

[Barre]

If you don't mind upgrading, It'll be 2.0.0 :)

25b48fd

[tinhoff]

I don't mind recreating my backup pool, as 'zfs send | zfs recv' ist fast and internal traffic on Hetzner is free of charge. Sounds like a totally interesting change, love it! When will u create a binary for 2.0.0? Never build a go app by myself and currently also using the docker setup by only exposing the nbd port, which scales great for my use-case, as, most trivia remote-backup setup I ever used :-p

[Barre]

I just published it: https://github.com/Barre/ZeroFS/releases/tag/v2.0.0

[tinhoff]

Updated to 2.0.3, send|received ~300GB from old to new pool and scrubbed the new pool once again to be really sure: took way longer as expected on an internal Hetzner cloud server, but, every bit verified successfully. Locally, it performed various GC tasks roughly about 12-13 hours, which downloaded a bit more then 1.1 TB from S3. Uff....

But currently, it still constantly downloads 10~15mb/s while neither web-ui nor gc show any activity:

zerofs_fsn1.1.kzhiddhdm5h0@storeknox    | 2026-07-05T10:42:08.586228Z  INFO zerofs::fs::store::extent: segment GC: 1166 segments, 241.0 GiB appended (240.3 GiB live, ~675.0 MiB reclaimable, 0% dead); idle
zerofs_fsn1.1.kzhiddhdm5h0@storeknox    | 2026-07-05T10:43:16.980432Z  INFO zerofs::fs::store::extent: segment GC: 1166 segments, 241.0 GiB appended (240.3 GiB live, ~675.0 MiB reclaimable, 0% dead); idle
zerofs_fsn1.1.kzhiddhdm5h0@storeknox    | 2026-07-05T10:44:25.866714Z  INFO zerofs::fs::store::extent: segment GC: 1166 segments, 241.0 GiB appended (240.3 GiB live, ~675.0 MiB reclaimable, 0% dead); idle
zerofs_fsn1.1.kzhiddhdm5h0@storeknox    | 2026-07-05T10:45:39.761983Z  INFO zerofs::fs::store::extent: segment GC: 1166 segments, 241.0 GiB appended (240.3 GiB live, ~675.0 MiB reclaimable, 0% dead); idle
zerofs_fsn1.1.kzhiddhdm5h0@storeknox    | 2026-07-05T10:46:55.055176Z  INFO zerofs::fs::store::extent: segment GC: 1166 segments, 241.0 GiB appended (240.3 GiB live, ~675.0 MiB reclaimable, 0% dead); idle
zerofs_fsn1.1.kzhiddhdm5h0@storeknox    | 2026-07-05T10:48:07.245214Z  INFO zerofs::fs::store::extent: segment GC: 1166 segments, 241.0 GiB appended (240.3 GiB live, ~675.0 MiB reclaimable, 0% dead); idle
zerofs_fsn1.1.kzhiddhdm5h0@storeknox    | 2026-07-05T10:49:17.644179Z  INFO zerofs::fs::store::extent: segment GC: 1166 segments, 241.0 GiB appended (240.3 GiB live, ~675.0 MiB reclaimable, 0% dead); idle

At least I interpret "idle" as: okay, I'm currently doing nothing. But, why it is constantly drawing 10-15mb (byte, not bit!) without claiming to do something? Its nearly a TB per day (where 1TB download/month is the free traffic tear on Hetzner for 1TB of data)

[tinhoff]

Looks like 111981 objects

[tinhoff]

biggest 6.6kb from yesterday, smallest 42bytes 3 days old

[Barre]

To me it most definitely looks like you have stale ZeroFS instance running on the same store. Can you double check that?

[tinhoff]

nope - for sure. the virtual server on hetzner I used to clone the zpool is shutdown since yesterday and subscription is even cancelled, in my homelab, no other instance is running. currently none at all, as, I temporary shut it down. as soon as I spin it up again, the issue exists.

[Barre]

I'd like to rule the way Hetzner's handles if-match / if-none-match. Can you install a local redis instances (apt install redis / yum install redis). Then, in the [aws] section in config, add conditional_put = "redis://localhost:6379" and see if this settles to a good state.

[tinhoff]

yeah, this was fun. Thanks for your great support.
Quite sure, redis fixes the situation.

After start, it literally started instantly to delete ~110.000 .manifest files (which takes pretty long, btw.), and now its deleteing /compacted/*.sst files (which yet again takes pretty long), but I see, that download traffic is going down slowly over time. Had a total of 380.000 files on S3 there, which felt dubious right away, but thought: hey, new storage layout is a new storage layout....

So the traffic I saw/see was/is the constant directory listings of hetzner_pool/manifests/* and hetzner_pool/compacted/* and compaction? It least my monitoring shows, that, as more files gets deleted, as less download-traffic is visible.

Btw.: Would be cool if some day, otrace is also visible in the web ui.

Yet again - thank you so much !!!

[tinhoff]

everything back to normal -> epic!
merci !

[Barre]

Great to hear!

So the traffic I saw/see was/is the constant directory listings of hetzner_pool/manifests/* and hetzner_pool/compacted/* and compaction?

Yes, indeed.

Btw.: Would be cool if someday otrace were also visible in the web UI.

Yeah, that could indeed be useful, feel free to open an issue.

I'll add more startup checks (verifying that the target object store supports the feature set we need) so that others don't get bitten by this.

[Barre]

https://github.com/Barre/ZeroFS/releases/tag/v2.0.5 is now checking for this at startup.