feat(podman): export driver traces over OTLP - #2782
Conversation
Mirror the VM driver tracing setup for Podman. Export standalone driver spans through OTLP/gRPC as the distinct openshell-driver-podman service, propagate W3C context across ComputeDriver RPCs, record bounded RPC names and failures, and flush buffered spans during graceful shutdown. Podman still runs in-process when selected as a built-in gateway driver. Add a temporary tracing shim that partitions gateway and Podman spans by target into separate tracer providers while preserving their shared trace and parentage. The shim also emits the same ComputeDriver server boundary that the tonic layer emits out of process, keeping the observable trace shape stable when Podman is eventually extracted. Trace container create preparation, image and storage setup, lifecycle operations, and cleanup. Document the service boundary and cover it with isolated and repeated tracing tests. Signed-off-by: Kris Hicks <khicks@nvidia.com>
pimlock
left a comment
pimlock
left a comment
There was a problem hiding this comment.
Few comments after exploring this with an agent.
| .on_response(()) | ||
| .on_body_chunk(()) | ||
| .on_eos(()) |
There was a problem hiding this comment.
I think we should also handle on_response/on_eos to record a status_code, otherwise it's going to be empty.
Could be achieved by adding a RecordGrpcStatus and updating to:
TraceLayer::new_for_grpc()
.on_response(RecordGrpcStatus)
.on_eos(RecordGrpcStatus)
.on_failure(RecordGrpcFailure)
| // 1a. Pull the supervisor image if needed. The supervisor binary | ||
| // is shipped in a standalone OCI image and mounted into sandbox | ||
| // containers via Podman's type=image mount. Refresh mutable tags | ||
| // like latest/dev, but avoid registry checks for pinned images. |
There was a problem hiding this comment.
nit: the different phases of creating a sandbox have numbered comments, but now the 1 is missing
| .instrument(tracing::info_span!( | ||
| "podman.start", | ||
| otel.name = "podman.start", | ||
| otel.status_code = tracing::field::Empty, |
There was a problem hiding this comment.
Consider renaming this to podman.start_container, there is another podman.start here and it's consistent with prepare_container above.
|
|
||
| /// Create a sandbox container. | ||
| #[tracing::instrument( | ||
| name = "podman.create", |
There was a problem hiding this comment.
Curious about naming here, should we go with the full podman.create_sandbox? Is there a reason to keep them short?
I'm also wondering about the prefix, since we are already emitting these in the context of the podman driver.
| if let Poll::Ready(Some(Err(status))) = &result { | ||
| openshell_otel::mark_error(&self.span); | ||
| self.span | ||
| .record("rpc.grpc.status_code", status.code() as i32); | ||
| } |
There was a problem hiding this comment.
Could we also record Code::Ok when the stream returns Poll::Ready(None)? Successful WatchSandboxes spans currently export without rpc.grpc.status_code.
2 participants
Summary
Mirror the VM driver tracing setup for Podman. Export standalone driver spans through OTLP/gRPC as the distinct openshell-driver-podman service, propagate W3C context across ComputeDriver RPCs, record bounded RPC names and failures, and flush buffered spans during graceful shutdown.
Podman still runs in-process when selected as a built-in gateway driver. Add a temporary tracing shim that partitions gateway and Podman spans by target into separate tracer providers while preserving their shared trace and parentage. The shim also emits the same ComputeDriver server boundary that the tonic layer emits out of process, keeping the observable trace shape stable when Podman is eventually extracted.
Trace container create preparation, image and storage setup, lifecycle operations, and cleanup. Document the service boundary and cover it with isolated and repeated tracing tests.
Related Issue
#1055
#2507
Changes
Testing
mise run pre-commitpassesChecklist