Skip to content

Decouple gRPC execution from workflow contexts - #1842

Open
geomagilles wants to merge 2 commits into
open-workflow-specification:mainfrom
geomagilles:codex/decouple-grpc-execution
Open

geomagilles wants to merge 2 commits into
open-workflow-specification:mainfrom
geomagilles:codex/decouple-grpc-execution

Conversation

@geomagilles

@geomagilles geomagilles commented Nov 17, 2026 •

Copy link
Copy Markdown

Why

Other workflow engines need to reuse dynamic gRPC execution without constructing SDK workflow applications, definitions, contexts or models. This extracts a reusable component while keeping the reference engine's workflow policy in its adapter.

Approach

The gRPC extraction follows the shared-components direction discussed in #1844 and incorporates the SDK namespace migration from #1858.

  • Move the reusable client to common/grpc, with artifact cloud.openworkflow.sdk:openworkflow-common-grpc and the openworkflow-common Maven parent.
  • Keep dependencies directed from impl/grpc to common. The common artifact uses grpc-java and Protobuf and has no workflow-engine dependency. Applications select their gRPC transport provider.
  • Use WorkflowGrpcCall and WorkflowProtobufJson as public helper names. Reuse native Channel, descriptors, Message and CallOptions; add no request/response DTO hierarchy or conversion framework.
  • Keep existing Protobuf runtime/JSON-library alignment and move the existing Gson version management to the root so common retains the repository's intended version.
  • Update the root README's three SDK usages, the implementation README link and the compiled example. Describe external engines generically.

Since #1844 is still open, this PR introduces the minimal common parent with the gRPC module only. The shared parent/reactor and root README will need reconciliation when both PRs merge. This PR contains no HTTP implementation changes or HTTP prototype commits.

How it works

WorkflowGrpcCall builds native method descriptors and marshallers from resolved Protobuf descriptors. Unary, server-streaming, client-streaming and bidirectional calls return CompletableFuture<List<Message>> after response completion. Built messages are reused directly; list snapshots copy references, not payloads. JSON conversion through WorkflowProtobufJson is explicit.

Callers may supply a configured Channel for TLS, credentials, interceptors, metadata and connection reuse. That channel is never closed. The address/port convenience overload creates a plaintext channel per invocation and initiates its shutdown when the RPC ends, or forceful shutdown when delivery is abandoned. The library does not block awaiting termination.

The original future propagates cancellation to ClientCall.cancel. Local orTimeout/completeOnTimeout and explicit asynchronous completion also stop an unfinished RPC. Cleanup starts before synchronous completion handlers run. Atomic terminal claims preserve the winning outcome during competing cancellation/completion, including the SDK bridge. Supplier-thrown cancellation remains a failed async computation rather than marking the result future cancelled, matching CompletableFuture semantics.

CallOptions are passed unchanged. Standalone defaults remain fail-fast with no deadline; wait-for-ready is opt-in. Native status, cause and trailers are preserved. The SDK keeps its application's executor, wait-for-ready, workflow argument conversion and positioned errors. GrpcExecutor, its public constructor, builder and GRPC_CHANNEL_PROVIDER remain available, with unary and streaming result shapes preserved. Descriptor/resource loading and protoc fallback remain in the implementation module.

Validation

After the namespace update (db9a1d5), Spotless, Checkstyle and the full JDK 21 clean Maven build passed locally. Of 832 tests, 815 passed and 17 were skipped. All 44 focused gRPC cases passed.

The original extraction (163750c) was validated locally with Java 21 on macOS ARM64 and Docker:

  • 44 focused standalone/workflow cases passed, no failures/errors/skips. Includes all RPC modes, JSON compatibility, engine-absence checks, metadata/credentials, status/cause/trailers, deadlines, cancellation, synchronous completion handlers, competing completions, local/async completion, caller-channel reuse and owned transport termination.
  • Full clean reactor: 41 modules succeeded; 832 cases, 815 passed, 17 existing skips, no failures/errors. Includes the existing container, HTTP/authentication, scheduler, persistence and Mermaid tests.
  • Repository-wide Spotless and Checkstyle pass. Common gRPC Javadocs pass.
  • Runtime dependency trees and jdeps show no SDK engine dependency. Protobuf runtime and JSON utility resolve consistently, and Gson retains the repository's existing managed version. The generated SDK BOM includes common gRPC.
  • The README example compiles with javac --release 21 using only the common gRPC JAR and its non-engine runtime dependencies.
  • Structured source and Maven integration reviews finish with no accepted/actionable findings. Complete logs and class reports are retained locally.
mvn -B -pl impl/test -am verify \
  -Dtest=WorkflowGrpcCallTest,WorkflowProtobufJsonTest,GrpcUnaryTest,GrpcUnaryArgsExprTest,GrpcServerStreamingTest,GrpcClientStreamingTest,GrpcBiDirectionalStreamingTest \
  -Dsurefire.failIfNoSpecifiedTests=false
mvn -B -fae clean verify
mvn -B -DskipTests spotless:check checkstyle:check
mvn -B -pl common/grpc -am -DskipTests package javadoc:javadoc

The Maven-provided protoc 3.25.8 macOS ARM64 executable was placed on PATH for the existing fallback. No test exclusions, timeout relaxations, failure-ignore or retry settings were added. Intermediate full runs hit timing-sensitive assertions in unchanged scheduler/container tests; those tests pass on untouched main, and the final full clean reactor passes. The 17 skips are existing annotations, including 12 shell cases disabled on macOS.

Trade-offs and limits

Requests are finite buffered lists. Responses accumulate without a size cap and are returned when the stream ends. There is no incremental consumer, application backpressure, durable stream storage or restart/resumption. The SDK still converts collected Protobuf responses after transport completion, which increases peak memory during conversion; its result future releases the upstream reference after adaptation. A per-message mapper is deliberately deferred.

Cancelling a derived future does not automatically cancel its upstream future; coroutine bridges must cancel the original returned future. Cancellation cannot undo server effects already performed. Owned channel shutdown is asynchronous. Prefer caller-managed channels for connection reuse and TLS; TLS negotiation itself is not covered by the plaintext local-server tests.

The existing map-entry convention for client/bidirectional DSL arguments is preserved. Remote Java causes are not serialized by gRPC; locally supplied causes and trailers remain available. The manually triggered integration-tests profile was not run. Local macOS/GraalVM validation does not replace the Ubuntu/Temurin CI result.

@geomagilles
geomagilles force-pushed the codex/decouple-grpc-execution branch from ba13c03 to 4fbfa56 Compare November 17, 2026 20:33
@geomagilles
geomagilles marked this pull request as ready for review November 17, 2026 20:36
Copilot AI balanced review requested due to automatic review settings November 17, 2026 20:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the extraction of reusable gRPC execution into a standalone client.

No open issues 🎉

What changed in this PR

Extracts dynamic gRPC execution into a standalone client for use by other workflow engines, while retaining SDK-specific behavior in the existing adapter. This prepares Lemline integration without implementing it or changing HTTP support.

Changes:

  • Adds reusable asynchronous gRPC calls and Protobuf JSON conversion without workflow-engine dependencies.
  • Refactors the workflow adapter to delegate transport execution while preserving result shapes, channel providers and positioned errors.
  • Adds transport lifecycle and conversion tests, module wiring and usage documentation.
File Description
impl/​test/​src/​test/​java/​org/​openworkflow/​sdk/​impl/​test/​grpc/​GrpcUnaryTest.java Tests channel-provider reuse and positioned transport errors.
impl/​README.md Links to standalone client documentation.
impl/​pom.xml Registers and manages the new client module.
impl/​grpc/​src/​main/​java/​org/​openworkflow/​sdk/​impl/​executors/​grpc/​ProtobufMessageUtils.java Delegates Protobuf JSON conversion to the shared utility.
impl/​grpc/​src/​main/​java/​org/​openworkflow/​sdk/​impl/​executors/​grpc/​ModelStreamObserver.java Removes the workflow-bound observer base.
impl/​grpc/​src/​main/​java/​org/​openworkflow/​sdk/​impl/​executors/​grpc/​ItemStreamObserver.java Removes the superseded unary result observer.
impl/​grpc/​src/​main/​java/​org/​openworkflow/​sdk/​impl/​executors/​grpc/​GrpcExecutor.java Delegates RPC execution and adapts workflow inputs, results and errors.
impl/​grpc/​src/​main/​java/​org/​openworkflow/​sdk/​impl/​executors/​grpc/​CollectionStreamObserver.java Removes the superseded streaming collection observer.
impl/​grpc/​pom.xml Adds the standalone client dependency.
impl/​grpc-client/​src/​test/​java/​org/​openworkflow/​sdk/​grpc/​ProtobufJsonTest.java Tests JSON conventions and unknown-field rejection.
impl/​grpc-client/​src/​test/​java/​org/​openworkflow/​sdk/​grpc/​DynamicGrpcCallTest.java Tests RPC modes, options, errors, cancellation and channel ownership.
impl/​grpc-client/​src/​main/​java/​org/​openworkflow/​sdk/​grpc/​ProtobufJson.java Adds workflow-independent Protobuf JSON conversion.
impl/​grpc-client/​src/​main/​java/​org/​openworkflow/​sdk/​grpc/​DynamicGrpcCall.java Implements reusable dynamic RPCs with buffered responses.
impl/​grpc-client/​README.md Documents usage, ownership, options and streaming limits.
impl/​grpc-client/​pom.xml Defines client dependencies and aligns the Protobuf runtime.

Review effort: Balanced

Rebuild the standalone gRPC extraction on current main under common/grpc and publish openworkflow-common-grpc through the common parent and root dependency management. Retain the existing Gson and Protobuf runtime alignment when leaving the implementation parent.

Expose WorkflowGrpcCall and WorkflowProtobufJson using native grpc-java and Protobuf values. Keep workflow resources, expressions, model conversion, positioned errors and channel-provider policy in the existing GrpcExecutor adapter. Preserve its public constructor and result shapes, including the existing client/bidi argument convention. Add no request/response DTO hierarchy or payload cloning.

Coordinate terminal delivery before cancellation callbacks, stop unfinished RPCs on local future timeouts or asynchronous completion, and release owned channels. Preserve supplier failure semantics and release raw transport references after SDK adaptation. Cover synchronous handlers, competing completions, native cancellation, deadlines, borrowed-channel reuse, status/cause/trailers and owned transport termination.

Document reusable execution components as the third SDK usage in the root README, update the compiled common gRPC example and use generic engine descriptions. Keep per-message conversion, backpressure and stream resumption for separate work. The minimal common parent can be reconciled with the HTTP/OpenAPI reorganization when those changes merge.

Validate 44 focused gRPC cases and the full reactor: 41 successful modules, 815 passing cases and 17 existing skips. Check repository formatting/style, common Javadocs, native dependency boundaries, BOM publication and the standalone README example. Retain validation and review logs, including intermediate timing-sensitive failures in unchanged tests.

Signed-off-by: Gilles Barbier <gilles@infinitic.io>
@geomagilles
geomagilles force-pushed the codex/decouple-grpc-execution branch from 4fbfa56 to 163750c Compare November 20, 2026 11:06
Copilot AI balanced review requested due to automatic review settings November 20, 2026 11:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the reusable gRPC extraction and workflow-adapter refactor.

No open issues 🎉

Review effort: Balanced

Merge the upstream SDK package rename into the gRPC extraction. Update the standalone client, workflow adapter, tests, documentation, and Maven coordinates to use cloud.openworkflow.sdk.
Copilot AI balanced review requested due to automatic review settings November 20, 2026 14:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the extraction of reusable gRPC functionality into the common component.

No open issues 🎉

Review effort: Balanced

@geomagilles

geomagilles commented Nov 20, 2026 •

Copy link
Copy Markdown
Author

@fjtirado I’ve updated this gRPC PR to follow the same approach we discussed for HTTP and OpenAPI in #1844. The reusable gRPC client now lives in common/grpc, so other workflow engines can use it directly. The reference engine’s workflow-specific logic stays in impl/grpc, which depends on the shared client.

I’ve also applied the namespace change from #1858. The Maven coordinate for the shared client is now cloud.openworkflow.sdk:openworkflow-common-grpc.

The full Maven build passed locally on JDK 21, including all 44 focused gRPC tests. Thanks for the feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants