Control Coordinator Refactor

[TomCC7]

General Direction

Instead of owning the hardware connection through adapter, align with the connection module design, which provides a single connection for a dedicated hardware, and control coordinator uses a unified high-performance transport to send high-frequency control command (like shm).

Pain Point/Wishlist

1. hide robot_id concept from higher level (manipulation module)
   this makes manipulation module semantic much cleaner.
2. rephrase joint semantic to more generic command_interface
   This is useful consider that you have a robot arm/simulation which only provides high-level interface for control (for example eef-pose). I think we already 'hacked' the interface for go2 control.
3. control coordinator related configs spam global_config. the current control coordinator owning hardware design is hacking the dimos core concept and makes config passing hard. Once the refactor is done we should be able to pass in hardware config through module cli args easily.

[TomCC7]

https://github.com/dimensionalOS/dimos/blob/1593c3c17af62ae91daf8bab71dce2444ca5875c/openspec/changes/refactor-control-coordinator-connections/design.md

[mustafab0]

Discussion

The contract

Four things cross the line between the coordinator and a connection.

What	Direction	When
description	connection → coordinator	once, at startup
state	connection → coordinator	every cycle
commands	coordinator → connection	every cycle
lifecycle	both ways	arming, stopping, faults

What's actually in a frame

Every state and command frame is an envelope wrapped around a payload.

Part	Field	Why it exists
payload	interface_names[]	which values these are: left/j1/position, left/j1/velocity
payload	values[]	the numbers, one per name
envelope	source	which connection sent it. All connections share one bus, so the frame has to say who it's from
envelope	source_ts	when the device sampled it
envelope	epoch	which arm generation. Any stop bumps it, so a command from before the stop is refused instead of restarting motion
envelope	seq	rejects repeated and out-of-order frames

Proposal A — define our own

flowchart TB
  subgraph CV["ControlValues · proposed"]
    direction TB
    F1["source<br/>which connection, or the coordinator"]:::ours
    F2["source_ts<br/>producer sample time"]:::ours
    F3["epoch<br/>arm epoch, gates late frames"]:::ours
    F4["seq<br/>rejects repeats and reorders"]:::ours
    F5["string[] interface_names<br/>keys like left/j1/position"]:::ours
    F6["float64[] values<br/>parallel to names"]:::ours
  end
  classDef ours fill:#F6E8D8,stroke:#A8621B,color:#1A1D1C
  classDef ros fill:#DDEEF1,stroke:#0E6B7A,color:#1A1D1C

Loading

Proposal B — borrow the payload, keep the rest ours

• Buys: ROS, rosbag and Foxglove interop for free, and ros2_control's docs explain the half we share.
• Costs: one extra nesting level, and we follow ROS's whims

flowchart TB
  subgraph CV["ControlValues · proposed"]
    direction TB
    F1["source<br/>which connection, or the coordinator"]:::ours
    F2["source_ts<br/>producer sample time"]:::ours
    F3["epoch<br/>arm epoch, gates late frames"]:::ours
    F4["seq<br/>rejects repeats and reorders"]:::ours
    subgraph IV["control_msgs/InterfaceValue · standard"]
      direction TB
      F5["string[] interface_names<br/>keys like left/j1/position"]:::ros
      F6["float64[] values<br/>parallel to names"]:::ros
    end
  end
  classDef ours fill:#F6E8D8,stroke:#A8621B,color:#1A1D1C
  classDef ros fill:#DDEEF1,stroke:#0E6B7A,color:#1A1D1C

Loading

Proposal C — adopt ros2_control's model as well ( stay with the current control coordinator architecture)

• Buys: the most prior art, and the least to explain to anyone who knows ROS.
• Costs: Back to square 1

flowchart TB
  subgraph CV["control_msgs/DynamicInterfaceValues · standard"]
    direction TB
    F0["std_msgs/Header<br/>stamp and frame_id"]:::ros
    subgraph IV["control_msgs/InterfaceValue · standard"]
      direction TB
      F5["string[] interface_names<br/>keys like j1/position"]:::ros
      F6["float64[] values<br/>parallel to names"]:::ros
    end
  end
  GONE["no source · no epoch · no seq<br/>ros2_control shares memory inside one process,<br/>so it never needed them"]:::gone
  classDef ours fill:#F6E8D8,stroke:#A8621B,color:#1A1D1C
  classDef ros fill:#DDEEF1,stroke:#0E6B7A,color:#1A1D1C
  classDef gone fill:#EDEFE8,stroke:#B9C1BD,color:#5F6763,stroke-dasharray:4 3

Loading

orange is ours to define, blue is taken from ROS.

[mustafab0]

This is now resolved we decided to go with Proposal A

[mustafab0]

Goal

Every robot is a connection module in its own process. The control coordinator talks to all of them with one message type, decides which controller drives which part of which robot, and owns no hardware.

flowchart LR
  subgraph CC["ControlCoordinator"]
    T["tasks: trajectory, velocity, teleop, policies"] --> A["arbitration: one owner per resource"]
  end
  IN["nav, planners"] <-- "Twist, JointState (standard)" --> CC  
  TN["teleop"] -- "Twist, JointState (standard)" --> CC  
  M["manipulation, viz, recorders"] <-- "joint_state · JointState (standard)" --> CC
  A -- "control_command · ControlValues" --> X["XArmConnectionModule"]
  A -- "control_command" --> G["G1ConnectionModule"]
  X -- "control_state · ControlValues" --> CC
  G -- "control_state" --> CC

Loading

Standard messages stay on the software side. Only the link between the coordinator and the hardware uses ControlValues.

1. The wire: ControlValues

source, source_ts, epoch, sequence, and paired interface_names / values (float64). Two shared streams:

• control_state: each connection publishes everything it measures.
• control_command: the coordinator publishes a complete frame per connection, every tick.

Names are keys of the form source/resource/interface, e.g. arm/joint1/position, chassis/base/vx, g1/imu/qw. sequence orders frames. epoch stays 0 until the deferred safety work.

2. Describing a robot operating behaviour

A ControlDescription has five fields:

Field	Meaning
source	the robot's name; the first segment of all its keys
resources	its parts: name, kind (joint, base, sensor), what each reports, what each accepts, units
limits	per command key; a command past a limit is refused
state_rate_hz	how often it reports
deadman_timeout_s	how long without a command before it halts itself

• Sensors in a description are readings the robot reports in its own control packet, alongside the joints( example G1's IMU in LowState)
• Presets build a description in one call: manipulator_description, pd_joint_description, twist_base_description. Given the robot's RobotModel, the arm and body presets read joint names, and limits from it. `ControlDescription.

3. A standard way to define a robot ConnectionModule

One small base class. A driver is one file:

class XArmConnectionModule(ConnectionModule):
    def describe(self):              # from self.config, no hardware
        return manipulator_description("arm", JOINTS, model=self.config.model)
    def connect(self): ...           # open the SDK
    def read_state(self) -> dict: ...  # called at the state rate
    def write(self, values): ...     # validated values; switch SDK mode here if needed
    def halt(self): ...              # stop moving; idempotent

The base class does the rest:

• On start: describe() → validate_description (fails loudly) → connect() → publish state at the state rate.
• On each command frame: keep this module's keys, run validate_command ), then write(). Limits are checked at the connection, so every path to hardware gets the same checks, whoever sent the command.
• Deadman: once commands have started, if none arrives within deadman_timeout_s, halt().
• write() and halt() never run at the same time. If write() raises: halt(), and the error shows in status().
• RPCs: describe_control() (the description, from config) and status().
• It works alone: no coordinator is needed to start, publish state, or test.

4. How does Control Coordinator know how to control a Robot

• Finding connections. The coordinator implements on_system_modules(modules), the hook McpServer uses to find skills. The coordinator asks every module that has a describe_control RPC for its description.

• Tick loop : read state → compute tasks → arbitrate per resource → write. The write goes to control_command, as a complete frame per connection.

• Standard messages on the software side: the coordinator keeps its Twist / JointState inputs and publishes joint_state. Tasks get joints and sensor readings in one per-tick snapshot (JointStateSnapshot, SensorStateSnapshot).

• Estop : stop publishing and call every connection's halt(). If the coordinator itself dies, every connection's deadman halts it.

5. "Why we removed standardized abstraction, and we are rebuilding it internally only in this module?"

Standard messages describe what kind of command something is (a velocity, a joint target). They don't say which motor it is for or who controls that motor right now, and arbitration needs both.

With a typed port per command type on each robot:

• One part, two kinds of command: a planner sends an arm positions while teleop sends it velocities. That's two ports into the same joints, and something has to know that to pick a winner.
• Moves that must happen together: an R1 Pro's base, torso and arms in one step. Separate topics arrive separately, and stopping one doesn't stop the others.
• Accidental second senders: autoconnect wires every cmd_vel: Out[Twist] to a robot's cmd_vel: In[Twist] without complaint, bypassing arbitration.
• Safety checks repeated per message type in every driver.

ros2_control made the same split: hardware exposes named values (joint1/position), a manager decides who owns them, and standard messages go into controllers.

For full context check this comment:
#3390 (comment)

[spomichter]

@sea-bass any feedback here

[mustafab0]

Follow up discussion link: #4427

[sea-bass]

Oh man, there's a lot here, but I'm understanding this as ros2_control without the ROS part. This has come up before as someone who is making a ROS-free alternative to the planning layer above the control.

The main thing I'm not clear on is if, unlike ros2_control, you are planning on only handling the robot driver stuff (aka "hardware interfaces") or also doing controller implementations (aka "controller interfaces").

Some things to look out for:

• A single ros2_control node contains all ite hardware drivers in the same process, so that you can read/write all the hardware states/commands in the same real-time loop. This is an advantage of the framework being done in a system language (C++ in this case). How does dimOS being Python account for this? A connection model splits things into separate processes but also sacrifices some of these "real-time" claims by doing IPC to each separate chunk of drivers, right?
• One good thing that ros2_control does is prevent you from activating multiple controllers that claim competing command interfaces. For example, you can't have 2 separate controllers at the same time sending position commands to the same actuator. And same goes with position + velocity + torque, let's say. Definitely keep that in mind.
• Streaming single values is among one of the most primitive controllers in ros2_control. However, they also have controllers that can accept entire trajectories (or in ML speak, action chunks lol), which do their own interpolation/blending. Is this within scope or do you want to keep that separate? Pros and cons tbh.
• A related question is again something in ros2_control: do you plan to handle cases where e.g. you are sending position commands but your hardware drivers must apply things like PID/gravity compensation to convert these to torque commands? Or again out of scope and this is more exclusively a communication layer?
• Don't feel limited to float data types for all this. ros2_control ended up later adding boolean types to represent hardware interfaces for things like tactile switches, GPIO interfaces, etc. Of course, in C++ this was a colossal effort (and in the limit becomes template hell), but in Python it's much easier.
• This one is more of a question: Do you plan on representing teleop devices also using this control coordinator? I'm thinking of how LeRobot has a clear separation between those 2, when for example technically you can represent a leader arm with the same drivers as a follower arm, just that you're reading the states from one and piping them into.the other. This would be a cool use case to prove out, especially as it affords you the abiility to later do fancier things like add gravity compensation to your leader arms.
• Which brings me to the final remark on "how is this similar to ros2_control": Make sure your control processes, if applicable, have access to the URDF / a kinematic/dynamic model of the robot to let controller implementations use the information here. For example if you (or your users doing fancy things) need any inverse dynamics, jacobians, collision checks, etc. at this level.

[mustafab0]

The main thing I'm not clear on is if, unlike ros2_control, you are planning on only handling the robot driver stuff (aka "hardware interfaces") or also doing controller implementations (aka "controller interfaces").

Yes we do do this currently. The Control Coordinator in dimensional, manages tasks (controller interfaces) and robot driver (hardware interface) together.

A single ros2_control node contains all ite hardware drivers in the same process, so that you can read/write all the hardware states/commands in the same real-time loop

This is the exact approach which I started with. The control coordinator owned the hardware interfaces as adapters. I did this for the very same reason you mention it as an advantage, for in-process realtime loop.

However, dimensional is designed to work otherwise. Where each robot lives in its own connection Module and we interface it over the wire (Transport). It has been a concern of mine that we lose "real-time" behavior when working over a transport. but I have been assured this won't be a big concern for us.

One good thing that ros2_control does is prevent you from activating multiple controllers that claim competing command interfaces.

I have a different approach here, where in control coordinator, multiple controllers or tasks can claim the same robot interface, but we have set priorities to these controllers, which help us arbitrate, and at the end only one controller wins.

Streaming single values is among one of the most primitive controllers in ros2_control
Nothing really stops us doing this currently. For example our grippers are essentially just a float value. But we can make things more abstract too. as you said.

This one is more of a question: Do you plan on representing teleop devices also using this control coordinator?
Yes essentially, control coordinator in my mind is the gateway to all hardware in dimensional. (except for sensors)

Which brings me to the final remark on "how is this similar to ros2_control": Make sure your control processes, if applicable, have access to the URDF / a kinematic/dynamic model of the robot to let controller implementations use the information here. For example if you (or your users doing fancy things) need any inverse dynamics, jacobians, collision checks, etc. at this level.

We are similar to ros2_control in the same sense that we manage the controllers and the hardware interfaces in one place. Other than that the implementation differs quite a bit.

We do have access to URDF and robot description models at this layer to. For example in our simple teleop task, you can give it access to the robot URDF and it can do IK to command the robot with just End-effector poses.

Tasks can be as simple as simply streaming commands in to commands out, but it could also be a ONNX policy for locomotion. or a ACT policy for commanding joint positions.

[leshy]

control_command · ControlValues

can you give examples of how this message looks like? let's say controlling twist of go2 or robot arm poses

plz no AI generated text for clarity for readers

[mustafab0]

This is example of Control coordinator sending commands to the Go2

ControlValues(
    source="ControlCoordinator", sequence=1042,
    interface_names=["go2/base/vx", "go2/base/vy", "go2/base/wz"],
    values=         [0.5,           0.0,           0.2],
)

The following is example of Go2 sending its velocity and odom reading to the control coordinator

ControlValues(
    source="ControlCoordinator", sequence=1042,
    interface_names=["go2/base/vx", "go2/base/vy", "go2/base/wz"],
    values=         [0.5,           0.0,           0.2],
)

And you would describe the contract with:

twist_base_description("go2", limits=..., odometry=True, measured_velocity=True)
# produces:
Resource(name="base", kind=BASE,
    state_interfaces   = ("vx", "vy", "wz", "x", "y", "yaw"),   # what it reports
    command_interfaces = ("vx", "vy", "wz"),                    # what it accepts
    units = {"vx": "m/s", "vy": "m/s", "wz": "rad/s", "x": "m", "y": "m", "yaw": "rad"})

[mustafab0]

It is quite useful for Manipulators that have different modes:

A xarm7 moving in position mode, and gripper moves in position mode:

ControlValues(
    source="ControlCoordinator",
    source_ts=1759550000.120,
    epoch=0,
    sequence=20417,
    interface_names=[
        "arm/joint1/position", "arm/joint2/position", "arm/joint3/position", "arm/joint4/position", "arm/joint5/position", arm/joint6/position", "arm/joint7/position", "arm/gripper/position",
    ],
    values=[0.10, -0.35, 0.02, 0.78, -0.05, 1.12, -0.42, 0.040],
)

A xarm7 moving in velocity mode, but gripper moves in position mode:

ControlValues(
    source="ControlCoordinator",
    source_ts=1759550000.130,
    epoch=0,
    sequence=20418,
    interface_names=[
        "arm/joint1/velocity", "arm/joint2/velocity", "arm/joint3/velocity", "arm/joint4/velocity", "arm/joint5/velocity", arm/joint6/velocity", "arm/joint7/velocity", "arm/gripper/position",
    ],
    values=[0.20, -0.10, 0.05, 0.0, 0.0, 0.0, 0.0, 0.040],
)

[leshy]

main issue, custom message, rewriting all drivers

I believe you are leaking an abstraction that belongs internally into a controlcoordinator, requiring a rewrite of all drivers, and causing incompatibility with any other system, to add a custom message that adds no new information, and removes individual topics, types etc that other systems understand. you are ignoring topics and frames, using a single one in dimos, then re-implementing topics and frames in your new custom message

I would say you can do all above that you'd like but keep those messages internal to controlcoordinator and translate into standards externally

your ros2_control example is actually good, it's a library used within ros nodes (this makes sense) but it still respects messages like twist etc for external system interaction. it never breaks any nodes external interface

answers/questions

from discord:

Firstly, standard messages lack all the information needed to command all robots.
For e.g I am already using a custom MotorCommand msg to interface with g1, go2 for low level control.

in a rare case something needs a custom control message, we can add it, but in most cases these are standard messages. custom messages don't justify removing all standard messages - but adding new custom messages

All your examples above carry the same (or less!) information then standard messages so I don't see why you are inventing a new type?

Secondly, as mentioned earlier, muxing or arbitrating between different msg types is difficult. As we start doing complex motions with mobile manipulation we need to arbitrate Twist and JointState messages somehow.

Yes but that's exactly why controlcoordinator exists? I send twist to controlcoordinator, and it controls driver modules with whatever language they understand, those SDKs and drivers don't have to understand global complex motion or msg type conversions

You would have to break down the abstraction and rebuild it in some module, case by case basis

Idea is not to do this at all. you keep your abstractions internal to controlcoordinator, and drivers do driver things. for example go2 understands twist, arm understands jointstates, there is never need internally for go2 driver to understand global coordination happening, the same way that go2 twist doesn't understand navigation or trajectory control but is able to navigate.

5. "Why we removed standardized abstraction, and we are rebuilding it internally only in this module?"
   Standard messages describe what kind of command something is (a velocity, a joint target).
   They don't say which motor it is for or who controls that motor right now, and arbitration needs both.

they do describe motor/entity by frame_id or topic (you actually invented new topic/frame abstraction in your message examples, just use topics or frame ids?)

One part, two kinds of command: a planner sends an arm positions while teleop sends it velocities. That's two ports into the same joints, and something has to know that to pick a winner.

yeah, controlcoordinator, above whole section is supposed to be answering my question on "why new topics" but is justifying existance of controlcoordinator, why? we agree there should be a controlcoordinator.

Moves that must happen together: an R1 Pro's base, torso and arms in one step. Separate topics arrive separately, and stopping one doesn't stop the others.

but you have external modules, so your topics are still separate, this doesn't help the problem but imo also it's not a problem, our pubsub is low latency, your proposal uses separate topics, my proposal uses separate topics, and that's ok

Safety checks repeated per message type in every driver.

depends on the safety layer and driver responsibilities, go2 twist should not do colision checks for example, that's a higher order operation, but it can make sure to stop if twist input stops, which is a low level driver responsibility and will stay there, low level driver should never blindly respect twist sent 5 minutes ago, even if controlcoordinator crashed. so there are different layers of safety on different layers of abstractions

general on topics/frames etc

it's possible that something is lacking and inconvinient here in terms of existing topics frames etc (we'll nail it down now) but then we should fix topics, frames etc instead of reimplement

[leshy]

custom topic issue discussion moved to #4427