Hi — I maintain EvalPort, an open specification for portable LLM evaluation test suites, test cases, and result sets (JSON Schemas + Python/TypeScript SDKs), aimed at letting eval datasets move between frameworks without losing semantic fidelity.
Concrete precedent first: OpenAI's own SDK team is currently reviewing a community PR that adds native to_openeval()/from_openeval() dataset conversion helpers directly into openai-python, in src/openai/types/evals/openeval.py:
openai/openai-python#3619 (fixes openai/openai-python#3549)
To be accurate about status: it is open, not merged — 6 commits in, review requested, real back-and-forth with an OpenAI maintainer. But it's a signal that a major provider SDK team is willing to carry this conversion logic in-tree rather than push it to a third-party adapter.
I'd like to propose the same for cohere-python, grounded in your actual generated types. From src/cohere/types/chat_message_v2.py, the v2 chat message union is ChatMessageV2 = Union[UserChatMessageV2, AssistantChatMessageV2, SystemChatMessageV2, ToolChatMessageV2], discriminated on role, with AssistantChatMessageV2 also carrying tool_calls: Optional[List[ToolCallV2]] and tool_plan. A rough sketch:
# Since this SDK is Fern-generated, per CONTRIBUTING.md this would live in a path
# listed in .fernignore to survive regeneration.
from cohere.types import ChatMessageV2
def to_openeval(messages: list[ChatMessageV2], *, id: str) -> dict:
"""Each variant already carries role as its discriminant (user/assistant/
system/tool), so mapping onto an EvalPort test_case's turns is close to 1:1."""
...
def from_openeval(item: dict) -> list[ChatMessageV2]:
...Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
No pressure at all if this isn't a priority right now — just wanted the precedent and a concrete, type-grounded sketch on record. Happy to answer questions or take a first pass if there's interest.
Hi — I maintain EvalPort, an open specification for portable LLM evaluation test suites, test cases, and result sets (JSON Schemas + Python/TypeScript SDKs), aimed at letting eval datasets move between frameworks without losing semantic fidelity.
Concrete precedent first: OpenAI's own SDK team is currently reviewing a community PR that adds native
to_openeval()/from_openeval()dataset conversion helpers directly intoopenai-python, insrc/openai/types/evals/openeval.py:openai/openai-python#3619 (fixes openai/openai-python#3549)
To be accurate about status: it is open, not merged — 6 commits in, review requested, real back-and-forth with an OpenAI maintainer. But it's a signal that a major provider SDK team is willing to carry this conversion logic in-tree rather than push it to a third-party adapter.
I'd like to propose the same for
cohere-python, grounded in your actual generated types. Fromsrc/cohere/types/chat_message_v2.py, the v2 chat message union isChatMessageV2 = Union[UserChatMessageV2, AssistantChatMessageV2, SystemChatMessageV2, ToolChatMessageV2], discriminated onrole, withAssistantChatMessageV2also carryingtool_calls: Optional[List[ToolCallV2]]andtool_plan. A rough sketch:Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
No pressure at all if this isn't a priority right now — just wanted the precedent and a concrete, type-grounded sketch on record. Happy to answer questions or take a first pass if there's interest.