Skip to content

Implement asynchronous SFTP file transfer (version 3) - #948

Draft
webknjaz wants to merge 15 commits into
ansible:develfrom
webknjaz:features/sftp-pipelined-remote-file
Draft

webknjaz wants to merge 15 commits into
ansible:develfrom
webknjaz:features/sftp-pipelined-remote-file

Conversation

@webknjaz

@webknjaz webknjaz commented Nov 9, 2026 •

Copy link
Copy Markdown
Member
SUMMARY

Another version of #641, stacked on #944: it contains #944's commits as they are, plus one commit fixing how _PipelinedRemoteFile handles its requests.

  • libssh caps each request to the server limit and returns the capped length. Implement asynchronous SFTP file transfer (version 2) #944 treats that as an error, so with a server advertising a limit below the 32 KiB chunk floor, every pipelined transfer would fail. Writes now queue the rest of the buffer, and reads accept the shorter chunk.
  • Reads no longer assume that a short read means EOF. Requests issued past it are discarded and the file seeks back to the delivered offset.
  • Requests still in flight are waited for on close instead of being freed, since libssh keeps each reply until it is waited for. Write errors are raised from flush(), which also runs when the context manager exits normally.

The last two address both Copilot findings on #944.

🤖 Produced by a probabilistic agent with a token generator using an LLM to sound convincing.

ISSUE TYPE
  • Feature Pull Request
ADDITIONAL INFORMATION

Trade-off: a download costs one extra round trip at the end of the file, because a short read restarts the read-ahead.

Change size under src/:

#944 compared with devel:  4 files changed, 411 insertions(+), 27 deletions(-)
this PR on top of #944:    3 files changed, 151 insertions(+), 94 deletions(-)

Copilot AI balanced review requested due to automatic review settings November 9, 2026 22:49
@pyba-chronographer pyba-chronographer Bot added the bot:chronographer:provided There is a change note present in this PR label Nov 9, 2026
webknjaz added a commit to webknjaz/ansible--pylibssh that referenced this pull request Nov 9, 2026
They credit this version alongside the original feature note.

Assisted-by: LLM
@packit-as-a-service

Copy link
Copy Markdown

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

  • sudo dnf install -y 'dnf*-command(copr)'
  • dnf copr enable packit/ansible-pylibssh-948
  • And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

1 similar comment
@packit-as-a-service

Copy link
Copy Markdown

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

  • sudo dnf install -y 'dnf*-command(copr)'
  • dnf copr enable packit/ansible-pylibssh-948
  • And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

@webknjaz
webknjaz marked this pull request as draft November 9, 2026 22:56

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Pipelined file destruction can crash the interpreter, allocation-failure cleanup can abandon unread replies, and short non-EOF responses lack regression coverage.

3 open issues
What changed in this PR

Adds pipelined SFTP transfers to improve throughput on high-latency connections. Following the approach discussed in #944, pipelining stays inside _PipelinedRemoteFile, preserving the existing get() and put() loops.

Changes:

  • Adds asynchronous request queuing, server-limit handling, and draining of pending requests.
  • Preserves synchronous transfers for libssh versions before 0.11 through a compatibility shim.
  • Expands transfer and context-manager tests and adds a changelog entry.
File Description
tests/​unit/​sftp_test.py Adds large-payload and pipelined transfer tests.
tests/​unit/​sftp_remote_file_cm_test.py Tests asynchronous reads, writes, errors, cleanup, and compatibility.
src/​pylibsshext/​sftp.pyx Implements pipelined remote files and selects the transfer implementation.
src/​pylibsshext/​sftp.pxd Declares pipeline state and native request wrappers.
src/​pylibsshext/​includes/​sftp.pxd Exposes libssh asynchronous transfer and limits APIs.
src/​pylibsshext/​includes/​sftp_compat.h Provides compatibility stubs for older libssh versions.
docs/​changelog-fragments/​641.feature.rst Announces asynchronous SFTP support.
.codespellrc Adds deque to the spelling allowlist.

Review effort: Balanced

Comment thread src/pylibsshext/sftp.pyx Outdated
finally:
self._close()

cdef void _close(self) noexcept:
Comment thread src/pylibsshext/sftp.pyx
Comment on lines +453 to +454
if <size_t>bytes_read < aio.length:
self._restart_reads()
Comment thread src/pylibsshext/sftp.pyx Resolved
@codecov

codecov Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.30435% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.05%. Comparing base (b36c332) to head (9fb2030).

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

webknjaz added a commit to webknjaz/ansible--pylibssh that referenced this pull request Nov 11, 2026
They credit this version alongside the original feature note.

Assisted-by: LLM
webknjaz added a commit to webknjaz/ansible--pylibssh that referenced this pull request Nov 11, 2026
The pipelining now moves into the remote file in PR ansible#944 already, so
the note describes the request handling fixes added on top instead.

Assisted-by: LLM
@webknjaz
webknjaz force-pushed the features/sftp-pipelined-remote-file branch from ebf26d8 to 7f3d504 Compare November 11, 2026 14:33

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

non-contrib?

Comment on lines +86 to +87
int sftp_seek64(sftp_file file, stdint.uint64_t new_offset)
stdint.uint64_t sftp_tell64(sftp_file file)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

diff

@webknjaz webknjaz Nov 12, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 This comment was generated by a probabilistic agent with a token generator using an LLM. It was not written by @webknjaz.

These declarations exist for short reads. _PipelinedRemoteFile.readinto() keeps up to SFTP_MAX_REQUESTS reads in flight at consecutive offsets, each as long as sftp_aio_begin_read() allowed after capping it to the server limit. When a reply comes back shorter than its request, the requests issued after it no longer line up with the data. Rather than treating every short read as EOF, as #944 does, the remaining requests are waited for and discarded, the file seeks back to _read_offset (the bytes actually delivered, from sftp_tell64() plus what was read since), and the read-ahead restarts from there. At a real EOF this costs one extra round trip.

The SFTP v3 spec that OpenSSH implements, draft-ietf-secsh-filexfer-02, only guarantees full reads for regular files. §6.4 on SSH_FXP_READ:

For normal disk files, it is guaranteed that this will read the specified number of bytes, or up to end of file. For e.g. device files this may return fewer bytes than requested.

§7 on SSH_FXP_DATA repeats that the data "may also be shorter if end of file is reached or if the read is from something other than a regular file".

So treating a short read as EOF would let a pipelined get() silently truncate device files, pipes and the like, which the synchronous loop on devel reads correctly until it gets 0 bytes. That's why the seek and restart stay, along with the stdint cimport for _read_offset.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Jakuje I couldn't understand why my agent piled additional complexity on top of your PR initially. I think I'll ask it to drop it for now. But what do you think about this potential corner case?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't dropped it so you can look at that mechanic as it seems to cover the corner case that's in the spec. Meanwhile, I'm going away for a week.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading and writing device files, pipes and stuff like that does not make much sense over SFTP IMHO. I recall writing some code to require the file to be a regular file, but I can not find the change now. Supporting anything else adds huge amount of complexity for no good reason.

Comment thread src/pylibsshext/sftp.pxd
# License along with this library; if not, see file LICENSE.rst in this
# repository.
#
from libc cimport stdint

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

diff

Comment thread src/pylibsshext/sftp.pxd Outdated
cdef public object _open_remote_file_cm

cdef class _RemoteFile:
cdef class _BaseRemoteFile:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can this just be

Suggested change
cdef class _BaseRemoteFile:
cdef class _RemoteFile:

w/o the base-layer?

Comment thread src/pylibsshext/sftp.pyx Outdated

from posix.fcntl cimport O_CREAT, O_RDONLY, O_TRUNC, O_WRONLY

from libc.stdlib cimport free, malloc

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

diff

Jakuje and others added 11 commits November 12, 2026 12:33
This pulls libssh 0.11.0 APIs into python and implements
fallback shim header to build also with the old libssh versions.

Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Refactor remote file handling by introducing `_BaseRemoteFile`
and add `_PipelinedRemoteFile` to support asynchronous SFTP
operations available in libssh 0.11.0+.

This implements non-blocking read and write methods using
libssh's SFTP AIO interface, managing in-flight requests
with an internal queue to enable pipelining and improve
transfer performance.

Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Add unit tests for SFTP get and put operations to verify that chunk
pipelining, ramp-up, and draining work properly across multiple chunks
with libssh 0.11.0 and newer. A fixture is introduced to reduce SFTP
chunk size limits, allowing the pipelining logic to be tested without
requiring large payload transfers.

Assisted-by: LLM
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Passing more than SFTP_MAX_CHUNK_LIMIT to the write API made the whole
SFTP server hard-fail, due to reaching hardcoded SFTP packet size limit.

Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
The `deque` is a standard python structure used in the new
SFTP async IO.

Signed-off-by: Jakub Jelen <jjelen@redhat.com>
libssh caps each asynchronous request to the server limit and reports
the capped length rather than failing, which used to be treated as an
error. Since chunks are never sized below 32 KiB, a server advertising
a lower limit would fail every pipelined transfer. Writes now keep
queueing the rest of the buffer and reads accept the shorter chunk.
Requests issued past a short read are discarded and the file seeks
back to the delivered offset, so the result no longer depends on short
reads only happening at EOF.

Requests still in flight are now waited for on close instead of being
freed, since libssh keeps each reply until it is waited for. Write
errors that surface after the data was queued are raised from
`flush()`, which also runs when the context manager exits normally.

Co-Authored-By: Jakub Jelen <jjelen@redhat.com>
Assisted-by: LLM
They credit this version alongside the original feature note.

Assisted-by: LLM
It credits the previous version of this feature too.

Assisted-by: LLM
The pipelining now moves into the remote file in PR ansible#944 already, so
the note describes the request handling fixes added on top instead.

Assisted-by: LLM
@webknjaz
webknjaz force-pushed the features/sftp-pipelined-remote-file branch from 7f3d504 to 6ea2f11 Compare November 12, 2026 15:59
Comment thread src/pylibsshext/sftp.pyx Outdated
Implements a standard context manager interface for
convenient access lifetime management.
"""

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

drop formatting

The pipelined transfers were never released, so the fixes on top of
them don't need a note of their own.

Assisted-by: LLM
@webknjaz
webknjaz requested review from Jakuje and a max review from Copilot November 14, 2026 03:31

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Destruction with pending requests can crash because cleanup accesses cleared fields, and skipped flushing on normal exit can conceal rejected writes.

6 open issues
1 resolved since last review

Review effort: Max

Comment thread src/pylibsshext/sftp.pyx

self._discard_in_flight()
sftp.sftp_close(self._fd)
self._fd = NULL
Comment thread src/pylibsshext/sftp.pyx
exc_tb: _tps.TracebackType | None,
) -> None:
try:
if exc_type is None and self._is_writer:
Comment thread src/pylibsshext/sftp.pyx
Comment on lines +458 to +459
if requested > 0:
return _C_AIO.from_ptr(raw_aio, requested, True)
Comment thread tests/unit/sftp_test.py
sftp_session.get(str(src_path), dst_path)
assert dst_path.read_bytes() == transmit_payload
assert (
_RemoteFileCounterProxy.instances[0].io_calls > PIPELINED_CHUNKS_COUNT

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:provided There is a change note present in this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants