Repository navigation
Conversation
They credit this version alongside the original feature note. Assisted-by: LLM
|
Congratulations! One of the builds has completed. 🍾 You can install the built RPMs by following these steps:
Please note that the RPMs should be used only in a testing environment. |
1 similar comment
|
Congratulations! One of the builds has completed. 🍾 You can install the built RPMs by following these steps:
Please note that the RPMs should be used only in a testing environment. |
There was a problem hiding this comment.
🟡 Changes recommended
Pipelined file destruction can crash the interpreter, allocation-failure cleanup can abandon unread replies, and short non-EOF responses lack regression coverage.
3 open issues
What changed in this PR
Adds pipelined SFTP transfers to improve throughput on high-latency connections. Following the approach discussed in #944, pipelining stays inside _PipelinedRemoteFile, preserving the existing get() and put() loops.
Changes:
- Adds asynchronous request queuing, server-limit handling, and draining of pending requests.
- Preserves synchronous transfers for libssh versions before 0.11 through a compatibility shim.
- Expands transfer and context-manager tests and adds a changelog entry.
| File | Description |
|---|---|
tests/unit/sftp_test.py |
Adds large-payload and pipelined transfer tests. |
tests/unit/sftp_remote_file_cm_test.py |
Tests asynchronous reads, writes, errors, cleanup, and compatibility. |
src/pylibsshext/sftp.pyx |
Implements pipelined remote files and selects the transfer implementation. |
src/pylibsshext/sftp.pxd |
Declares pipeline state and native request wrappers. |
src/pylibsshext/includes/sftp.pxd |
Exposes libssh asynchronous transfer and limits APIs. |
src/pylibsshext/includes/sftp_compat.h |
Provides compatibility stubs for older libssh versions. |
docs/changelog-fragments/641.feature.rst |
Announces asynchronous SFTP support. |
.codespellrc |
Adds deque to the spelling allowlist. |
Review effort: Balanced
| finally: | ||
| self._close() | ||
|
|
||
| cdef void _close(self) noexcept: |
| if <size_t>bytes_read < aio.length: | ||
| self._restart_reads() |
Codecov Report❌ Patch coverage is 🚀 New features to boost your workflow:
|
They credit this version alongside the original feature note. Assisted-by: LLM
The pipelining now moves into the remote file in PR ansible#944 already, so the note describes the request handling fixes added on top instead. Assisted-by: LLM
ebf26d8 to
7f3d504
Compare
| int sftp_seek64(sftp_file file, stdint.uint64_t new_offset) | ||
| stdint.uint64_t sftp_tell64(sftp_file file) |
There was a problem hiding this comment.
Note
🤖 This comment was generated by a probabilistic agent with a token generator using an LLM. It was not written by @webknjaz.
These declarations exist for short reads. _PipelinedRemoteFile.readinto() keeps up to SFTP_MAX_REQUESTS reads in flight at consecutive offsets, each as long as sftp_aio_begin_read() allowed after capping it to the server limit. When a reply comes back shorter than its request, the requests issued after it no longer line up with the data. Rather than treating every short read as EOF, as #944 does, the remaining requests are waited for and discarded, the file seeks back to _read_offset (the bytes actually delivered, from sftp_tell64() plus what was read since), and the read-ahead restarts from there. At a real EOF this costs one extra round trip.
The SFTP v3 spec that OpenSSH implements, draft-ietf-secsh-filexfer-02, only guarantees full reads for regular files. §6.4 on SSH_FXP_READ:
For normal disk files, it is guaranteed that this will read the specified number of bytes, or up to end of file. For e.g. device files this may return fewer bytes than requested.
§7 on SSH_FXP_DATA repeats that the data "may also be shorter if end of file is reached or if the read is from something other than a regular file".
So treating a short read as EOF would let a pipelined get() silently truncate device files, pipes and the like, which the synchronous loop on devel reads correctly until it gets 0 bytes. That's why the seek and restart stay, along with the stdint cimport for _read_offset.
There was a problem hiding this comment.
@Jakuje I couldn't understand why my agent piled additional complexity on top of your PR initially. I think I'll ask it to drop it for now. But what do you think about this potential corner case?
There was a problem hiding this comment.
I haven't dropped it so you can look at that mechanic as it seems to cover the corner case that's in the spec. Meanwhile, I'm going away for a week.
There was a problem hiding this comment.
Reading and writing device files, pipes and stuff like that does not make much sense over SFTP IMHO. I recall writing some code to require the file to be a regular file, but I can not find the change now. Supporting anything else adds huge amount of complexity for no good reason.
| # License along with this library; if not, see file LICENSE.rst in this | ||
| # repository. | ||
| # | ||
| from libc cimport stdint |
| cdef public object _open_remote_file_cm | ||
|
|
||
| cdef class _RemoteFile: | ||
| cdef class _BaseRemoteFile: |
There was a problem hiding this comment.
can this just be
| cdef class _BaseRemoteFile: | |
| cdef class _RemoteFile: |
w/o the base-layer?
|
|
||
| from posix.fcntl cimport O_CREAT, O_RDONLY, O_TRUNC, O_WRONLY | ||
|
|
||
| from libc.stdlib cimport free, malloc |
This pulls libssh 0.11.0 APIs into python and implements fallback shim header to build also with the old libssh versions. Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Refactor remote file handling by introducing `_BaseRemoteFile` and add `_PipelinedRemoteFile` to support asynchronous SFTP operations available in libssh 0.11.0+. This implements non-blocking read and write methods using libssh's SFTP AIO interface, managing in-flight requests with an internal queue to enable pipelining and improve transfer performance. Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Add unit tests for SFTP get and put operations to verify that chunk pipelining, ramp-up, and draining work properly across multiple chunks with libssh 0.11.0 and newer. A fixture is introduced to reduce SFTP chunk size limits, allowing the pipelining logic to be tested without requiring large payload transfers. Assisted-by: LLM Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Passing more than SFTP_MAX_CHUNK_LIMIT to the write API made the whole SFTP server hard-fail, due to reaching hardcoded SFTP packet size limit. Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
The `deque` is a standard python structure used in the new SFTP async IO. Signed-off-by: Jakub Jelen <jjelen@redhat.com>
libssh caps each asynchronous request to the server limit and reports the capped length rather than failing, which used to be treated as an error. Since chunks are never sized below 32 KiB, a server advertising a lower limit would fail every pipelined transfer. Writes now keep queueing the rest of the buffer and reads accept the shorter chunk. Requests issued past a short read are discarded and the file seeks back to the delivered offset, so the result no longer depends on short reads only happening at EOF. Requests still in flight are now waited for on close instead of being freed, since libssh keeps each reply until it is waited for. Write errors that surface after the data was queued are raised from `flush()`, which also runs when the context manager exits normally. Co-Authored-By: Jakub Jelen <jjelen@redhat.com> Assisted-by: LLM
They credit this version alongside the original feature note. Assisted-by: LLM
It credits the previous version of this feature too. Assisted-by: LLM
The pipelining now moves into the remote file in PR ansible#944 already, so the note describes the request handling fixes added on top instead. Assisted-by: LLM
7f3d504 to
6ea2f11
Compare
| Implements a standard context manager interface for | ||
| convenient access lifetime management. | ||
| """ | ||
|
|
The pipelined transfers were never released, so the fixes on top of them don't need a note of their own. Assisted-by: LLM
There was a problem hiding this comment.
🟡 Changes recommended
Destruction with pending requests can crash because cleanup accesses cleared fields, and skipped flushing on normal exit can conceal rejected writes.
6 open issues
Drain pending replies before deallocation clears fields · New Close pipelined files before Cython clears the queue Flush queued writes on every exception-free exit · New Test short non-EOF responses preserve the full download Test reads with buffers larger than the server limit · New Verify overlapping requests at the server boundary · New
1 resolved since last review
Review effort: Max
|
|
||
| self._discard_in_flight() | ||
| sftp.sftp_close(self._fd) | ||
| self._fd = NULL |
| exc_tb: _tps.TracebackType | None, | ||
| ) -> None: | ||
| try: | ||
| if exc_type is None and self._is_writer: |
| if requested > 0: | ||
| return _C_AIO.from_ptr(raw_aio, requested, True) |
| sftp_session.get(str(src_path), dst_path) | ||
| assert dst_path.read_bytes() == transmit_payload | ||
| assert ( | ||
| _RemoteFileCounterProxy.instances[0].io_calls > PIPELINED_CHUNKS_COUNT |



SUMMARY
Another version of #641, stacked on #944: it contains #944's commits as they are, plus one commit fixing how
_PipelinedRemoteFilehandles its requests.flush(), which also runs when the context manager exits normally.The last two address both Copilot findings on #944.
🤖 Produced by a probabilistic agent with a token generator using an LLM to sound convincing.
ISSUE TYPE
ADDITIONAL INFORMATION
Trade-off: a download costs one extra round trip at the end of the file, because a short read restarts the read-ahead.
Change size under
src/: