Skip to content

Commit f35584a

Browse files
WouldYouKindlyaidandjclaude
authored
Add flags to generate only sync or only async stubs (#694)
* Add flags to generate only sync or only async stubs * Specify type parameters for context in async grpc methods * Ignore pyright errors in sync/async only stubs * Update readme with sync_only/async_only option * Start adding tests Rename folders Update pyproject.toml so pyright works on test code for sync/async only Signed-off-by: Aidan Jensen <aidandj.github@gmail.com> * add constructor typing Signed-off-by: Aidan Jensen <aidandj.github@gmail.com> * run black * isort: skip generated grpc files * Add init files to generated sync/async * Add init files to generated sync/async * Add missing enum import * Fix linting error and remove generated sync/async only files - Fix F402 linting error where 'enum' loop variable was shadowing the enum module import. Renamed loop variable to 'enum_proto'. - Stage deletion of test/generated_sync_only and test/generated_async_only directories - these will be regenerated during test runs. - Update generated_concrete files with latest output. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Update Pyright config to exclude sync/async only test directories - Remove executionEnvironments entries for test/generated_sync_only and test/generated_async_only since these directories are being removed - Exclude test/async_only and test/sync_only directories from Pyright checking since they depend on generated files that are no longer committed to git 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Create sync/async only directories before generating files Add mkdir -p commands to create test/generated_sync_only and test/generated_async_only directories before protoc tries to write files into them. These directories are no longer committed to git and need to be created during test runs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Add --report-deprecated-as-note to sync/async only mypy checks The sync_only and async_only mypy checks were missing the --report-deprecated-as-note flag, causing deprecation warnings to be treated as errors. This flag is already used for the concrete module checks and should also be used for these tests. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Fix async_only test to use DummyServiceAsyncStub When generating with the only_async flag, the stub class is named DummyServiceAsyncStub, not DummyServiceStub. Update the test to use the correct async stub class name. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Address review comments: error handling and simplify server type logic - Add error checking when both only_sync and only_async options are specified - Simplify make_server_type() logic using direct comparisons instead of if/elif chain - Check in generated sync_only and async_only test files so CI can run tests - Add __init__.py files to generated sync/async directories for proper package structure Typed constructors were already present in the generated code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Improve tests and add pyright support for sync/async only - Add comprehensive tests for sync_only and async_only that actually run servers and clients, equivalent to test_grpc_usage.py and test_grpc_async_usage.py - Restore pyright executionEnvironments for generated_sync_only and generated_async_only - Add executionEnvironments for test/async_only and test/sync_only with proper extraPaths - Clean generated_sync_only and generated_async_only directories in test script 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Fix async_only test context types to match generated stubs Update ServicerContext type parameters to match the exact types in the generated async_only stubs. The context type parameters should match the method return types (Awaitable or AsyncIterator). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Fix async_only test to use DummyServiceStub at runtime The generated _pb2_grpc.py file only exports DummyServiceStub, while the .pyi file has DummyServiceAsyncStub as a type alias. This matches the pattern in test_grpc_async_usage.py. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Add type alias for Stub in async_only mode In async_only mode, the .pyi file now defines both DummyServiceAsyncStub (the actual class) and DummyServiceStub (a type alias). This matches the runtime behavior where grpc_tools.protoc generates DummyServiceStub in the .py file. Fixes type checking errors where mypy couldn't find DummyServiceStub. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Use Stub naming in async_only mode (not AsyncStub) In async_only mode, generate class DummyServiceStub instead of DummyServiceAsyncStub with a type alias. Since there's only one stub type, it should use the standard Stub naming that matches the runtime .py file generated by grpc_tools.protoc. This addresses review feedback from @aidandj. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Update changelog --------- Signed-off-by: Aidan Jensen <aidandj.github@gmail.com> Co-authored-by: Aidan Jensen <aidandj.github@gmail.com> Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
1 parent 38362fb commit f35584a

File tree

87 files changed

+9467
-80
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

87 files changed

+9467
-80
lines changed

.gitignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,10 @@ __pycache__/
1010
!/test/generated/**/__init__.py
1111
/test/generated_concrete/**/*.py
1212
!/test/generated_concrete/**/__init__.py
13+
/test/generated_async_only/**/*.py
14+
!/test/generated_async_only/**/__init__.py
15+
/test/generated_sync_only/**/*.py
16+
!/test/generated_sync_only/**/__init__.py
1317
.pytest_cache
1418
/build/
1519
/dist/

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
- Protobuf <6.32 still had the edition enums and field options, so it *should* still work. But is untested
88
- Add support for editions (up to 2024)
99
- Add `generate_concrete_servicer_stubs` option to generate concrete instead of abstract servicer stubs
10+
- Add `sync_only`/`async_only` options to generate only sync or async version of GRPC stubs
1011
- Switch to types-grpcio instead of no longer maintained grpc-stubs
1112
- Add `_HasFieldArgType` and `_ClearFieldArgType` aliases to allow for typing field manipulation functions
1213
- Add `_WhichOneofArgType_<oneof_name>` and `_WhichOneofReturnType_<oneof_name>` type aliases

README.md

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -348,6 +348,21 @@ By default mypy-protobuf will output servicer stubs with abstract methods. To ou
348348
protoc --python_out=output/location --mypy_grpc_out=generate_concrete_servicer_stubs:output/location
349349
```
350350

351+
### `sync_only/async_only`
352+
353+
By default, generated GRPC stubs are compatible with both sync and async variants. If you only
354+
want sync or async GRPC stubs, use this option:
355+
356+
```
357+
protoc --python_out=output/location --mypy_grpc_out=sync_only:output/location
358+
```
359+
360+
or
361+
362+
```
363+
protoc --python_out=output/location --mypy_grpc_out=async_only:output/location
364+
```
365+
351366
### Output suppression
352367

353368
To suppress output, you can run

mypy_protobuf/main.py

Lines changed: 179 additions & 76 deletions
Large diffs are not rendered by default.

pyproject.toml

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ line-length = 10000
1313
[tool.isort]
1414
profile = "black"
1515
skip_gitignore = true
16-
extend_skip_glob = ["*_pb2.py"]
16+
extend_skip_glob = ["*_pb2.py", "*_pb2_grpc.py"]
1717

1818
[tool.mypy]
1919
strict = true
@@ -32,12 +32,16 @@ include = [
3232
exclude = [
3333
"**/*_pb2.py",
3434
"**/*_pb2_grpc.py",
35-
"test/test_concrete.py"
35+
"test/test_concrete.py",
3636
]
3737

3838
executionEnvironments = [
3939
# Due to how upb is typed, we need to disable incompatible variable override checks
4040
{ root = "test/generated", extraPaths = ["./"], reportIncompatibleVariableOverride = "none" },
4141
{ root = "test/generated_concrete", extraPaths = ["./"], reportIncompatibleVariableOverride = "none" },
42+
{ root = "test/generated_sync_only", extraPaths = ["./"], reportIncompatibleVariableOverride = "none" },
43+
{ root = "test/generated_async_only", extraPaths = ["./"], reportIncompatibleVariableOverride = "none" },
4244
{ root = "mypy_protobuf/extensions_pb2.pyi", reportIncompatibleVariableOverride = "none" },
45+
{ root = "test/async_only", extraPaths = ["test/generated_async_only"] },
46+
{ root = "test/sync_only", extraPaths = ["test/generated_sync_only"] },
4347
]

run_test.sh

Lines changed: 21 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -108,8 +108,10 @@ MYPY_PROTOBUF_VENV=venv_$PY_VER_MYPY_PROTOBUF
108108

109109
# CI Check to make sure generated files are committed
110110
SHA_BEFORE=$(find test/generated -name "*.pyi" -print0 | xargs -0 sha1sum)
111-
# Clean out generated/ directory - except for __init__.py
111+
# Clean out generated/ directories - except for __init__.py
112112
find test/generated -type f -not -name "__init__.py" -delete
113+
find test/generated_sync_only -type f -not -name "__init__.py" -delete
114+
find test/generated_async_only -type f -not -name "__init__.py" -delete
113115

114116
# Compile protoc -> python
115117
find proto -name "*.proto" -print0 | xargs -0 "$PROTOC" "${PROTOC_ARGS[@]}" --python_out=test/generated
@@ -135,6 +137,13 @@ MYPY_PROTOBUF_VENV=venv_$PY_VER_MYPY_PROTOBUF
135137
find proto -name "*.proto" -print0 | xargs -0 "$PROTOC" "${PROTOC_ARGS[@]}" --mypy_out=generate_concrete_servicer_stubs:test/generated_concrete
136138
find proto/testproto/grpc -name "*.proto" -print0 | xargs -0 "$PROTOC" "${PROTOC_ARGS[@]}" --mypy_grpc_out=generate_concrete_servicer_stubs:test/generated_concrete
137139

140+
# Generate with sync_only stubs for testing
141+
mkdir -p test/generated_sync_only
142+
find proto/testproto -name "*.proto" -print0 | xargs -0 "$PROTOC" "${PROTOC_ARGS[@]}" --mypy_grpc_out=only_sync:test/generated_sync_only --mypy_out=test/generated_sync_only --python_out=test/generated_sync_only
143+
144+
# Generate with async_only stubs for testing
145+
mkdir -p test/generated_async_only
146+
find proto/testproto -name "*.proto" -print0 | xargs -0 "$PROTOC" "${PROTOC_ARGS[@]}" --mypy_grpc_out=only_async:test/generated_async_only --mypy_out=test/generated_async_only --python_out=test/generated_async_only
138147

139148
if [[ -n $VALIDATE ]] && ! diff <(echo "$SHA_BEFORE") <(find test/generated -name "*.pyi" -print0 | xargs -0 sha1sum); then
140149
echo -e "${RED}Some .pyi files did not match. Please commit those files${NC}"
@@ -153,6 +162,8 @@ for PY_VER in $PY_VER_UNIT_TESTS; do
153162
(
154163
source "$UNIT_TESTS_VENV"/bin/activate
155164
find proto/testproto/grpc -name "*.proto" -print0 | xargs -0 python -m grpc_tools.protoc "${PROTOC_ARGS[@]}" --grpc_python_out=test/generated
165+
find proto/testproto/grpc -name "*.proto" -print0 | xargs -0 python -m grpc_tools.protoc "${PROTOC_ARGS[@]}" --grpc_python_out=test/generated_sync_only
166+
find proto/testproto/grpc -name "*.proto" -print0 | xargs -0 python -m grpc_tools.protoc "${PROTOC_ARGS[@]}" --grpc_python_out=test/generated_async_only
156167
)
157168

158169
# Run mypy on unit tests / generated output
@@ -162,6 +173,14 @@ for PY_VER in $PY_VER_UNIT_TESTS; do
162173
CONCRETE_MODULES=( -m test.test_concrete )
163174
MYPYPATH=$MYPYPATH:test/generated_concrete mypy ${CUSTOM_TYPESHED_DIR_ARG:+"$CUSTOM_TYPESHED_DIR_ARG"} --report-deprecated-as-note --no-incremental --python-executable="$UNIT_TESTS_VENV"/bin/python --python-version="$PY_VER_MYPY_TARGET" "${CONCRETE_MODULES[@]}"
164175

176+
# Run sync_only mypy
177+
SYNC_ONLY_MODULES=( -m test.sync_only.test_sync_only )
178+
MYPYPATH=$MYPYPATH:test/generated_sync_only mypy ${CUSTOM_TYPESHED_DIR_ARG:+"$CUSTOM_TYPESHED_DIR_ARG"} --report-deprecated-as-note --python-executable="$UNIT_TESTS_VENV"/bin/python --python-version="$PY_VER_MYPY_TARGET" "${SYNC_ONLY_MODULES[@]}"
179+
180+
# Run async_only mypy
181+
ASYNC_ONLY_MODULES=( -m test.async_only.test_async_only )
182+
MYPYPATH=$MYPYPATH:test/generated_async_only mypy ${CUSTOM_TYPESHED_DIR_ARG:+"$CUSTOM_TYPESHED_DIR_ARG"} --report-deprecated-as-note --python-executable="$UNIT_TESTS_VENV"/bin/python --python-version="$PY_VER_MYPY_TARGET" "${ASYNC_ONLY_MODULES[@]}"
183+
165184
export MYPYPATH=$MYPYPATH:test/generated
166185

167186
# Run mypy
@@ -210,7 +229,7 @@ for PY_VER in $PY_VER_UNIT_TESTS; do
210229
(
211230
# Run unit tests.
212231
source "$UNIT_TESTS_VENV"/bin/activate
213-
PYTHONPATH=test/generated py.test --ignore=test/generated -v
232+
PYTHONPATH=test/generated py.test --ignore=test/generated --ignore=test/generated_sync_only --ignore=test/generated_async_only -v
214233
)
215234
done
216235

test/async_only/test_async_only.py

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
"""
2+
Type-checking and runtime test for async_only GRPC stubs.
3+
4+
This module validates that stubs generated with the only_async flag have the correct types:
5+
- Stub class (not AsyncStub) that only accepts grpc.aio.Channel
6+
- Servicer methods use AsyncIterator for client streaming (not _MaybeAsyncIterator)
7+
- add_XXXServicer_to_server accepts grpc.aio.Server
8+
"""
9+
10+
import grpc.aio
11+
import pytest
12+
import typing_extensions as typing
13+
from testproto.grpc import dummy_pb2, dummy_pb2_grpc
14+
15+
ADDRESS = "localhost:22225"
16+
17+
18+
class Servicer(dummy_pb2_grpc.DummyServiceServicer):
19+
async def UnaryUnary(
20+
self,
21+
request: dummy_pb2.DummyRequest,
22+
context: grpc.aio.ServicerContext[dummy_pb2.DummyRequest, typing.Awaitable[dummy_pb2.DummyReply]],
23+
) -> dummy_pb2.DummyReply:
24+
return dummy_pb2.DummyReply(value=request.value[::-1])
25+
26+
async def UnaryStream(
27+
self,
28+
request: dummy_pb2.DummyRequest,
29+
context: grpc.aio.ServicerContext[dummy_pb2.DummyRequest, typing.AsyncIterator[dummy_pb2.DummyReply]],
30+
) -> typing.AsyncIterator[dummy_pb2.DummyReply]:
31+
for char in request.value:
32+
yield dummy_pb2.DummyReply(value=char)
33+
34+
async def StreamUnary(
35+
self,
36+
request_iterator: typing.AsyncIterator[dummy_pb2.DummyRequest],
37+
context: grpc.aio.ServicerContext[typing.AsyncIterator[dummy_pb2.DummyRequest], typing.Awaitable[dummy_pb2.DummyReply]],
38+
) -> dummy_pb2.DummyReply:
39+
values = [data.value async for data in request_iterator]
40+
return dummy_pb2.DummyReply(value="".join(values))
41+
42+
async def StreamStream(
43+
self,
44+
request_iterator: typing.AsyncIterator[dummy_pb2.DummyRequest],
45+
context: grpc.aio.ServicerContext[typing.AsyncIterator[dummy_pb2.DummyRequest], typing.AsyncIterator[dummy_pb2.DummyReply]],
46+
) -> typing.AsyncIterator[dummy_pb2.DummyReply]:
47+
async for data in request_iterator:
48+
yield dummy_pb2.DummyReply(value=data.value.upper())
49+
50+
51+
def make_server() -> grpc.aio.Server:
52+
server = grpc.aio.server()
53+
servicer = Servicer()
54+
server.add_insecure_port(ADDRESS)
55+
dummy_pb2_grpc.add_DummyServiceServicer_to_server(servicer, server)
56+
return server
57+
58+
59+
@pytest.mark.asyncio
60+
async def test_async_only_grpc() -> None:
61+
server = make_server()
62+
await server.start()
63+
async with grpc.aio.insecure_channel(ADDRESS) as channel:
64+
client = dummy_pb2_grpc.DummyServiceStub(channel)
65+
request = dummy_pb2.DummyRequest(value="cprg")
66+
result1 = await client.UnaryUnary(request)
67+
result2 = client.UnaryStream(dummy_pb2.DummyRequest(value=result1.value))
68+
result2_list = [r async for r in result2]
69+
assert len(result2_list) == 4
70+
result3 = client.StreamStream(dummy_pb2.DummyRequest(value=part.value) for part in result2_list)
71+
result3_list = [r async for r in result3]
72+
assert len(result3_list) == 4
73+
result4 = await client.StreamUnary(dummy_pb2.DummyRequest(value=part.value) for part in result3_list)
74+
assert result4.value == "GRPC"
75+
76+
await server.stop(None)
77+
78+
class TestAttribute:
79+
stub: "dummy_pb2_grpc.DummyServiceStub"
80+
81+
def __init__(self) -> None:
82+
self.stub = dummy_pb2_grpc.DummyServiceStub(grpc.aio.insecure_channel(ADDRESS))
83+
84+
async def test(self) -> None:
85+
val = await self.stub.UnaryUnary(dummy_pb2.DummyRequest(value="test"))
86+
typing.assert_type(val, dummy_pb2.DummyReply)
Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
"""
2+
@generated by mypy-protobuf. Do not edit manually!
3+
isort:skip_file
4+
"""
5+
6+
import builtins
7+
import google.protobuf.descriptor
8+
import google.protobuf.message
9+
import sys
10+
import typing
11+
12+
if sys.version_info >= (3, 10):
13+
import typing as typing_extensions
14+
else:
15+
import typing_extensions
16+
17+
DESCRIPTOR: google.protobuf.descriptor.FileDescriptor
18+
19+
@typing.final
20+
class lower(google.protobuf.message.Message):
21+
DESCRIPTOR: google.protobuf.descriptor.Descriptor
22+
23+
A_FIELD_NUMBER: builtins.int
24+
a: builtins.int
25+
def __init__(
26+
self,
27+
*,
28+
a: builtins.int = ...,
29+
) -> None: ...
30+
_ClearFieldArgType: typing_extensions.TypeAlias = typing.Literal["a", b"a"]
31+
def ClearField(self, field_name: _ClearFieldArgType) -> None: ...
32+
33+
Global___lower: typing_extensions.TypeAlias = lower
34+
35+
@typing.final
36+
class Upper(google.protobuf.message.Message):
37+
DESCRIPTOR: google.protobuf.descriptor.Descriptor
38+
39+
LOWER_FIELD_NUMBER: builtins.int
40+
@property
41+
def Lower(self) -> Global___lower: ...
42+
def __init__(
43+
self,
44+
*,
45+
Lower: Global___lower | None = ...,
46+
) -> None: ...
47+
_HasFieldArgType: typing_extensions.TypeAlias = typing.Literal["Lower", b"Lower"]
48+
def HasField(self, field_name: _HasFieldArgType) -> builtins.bool: ...
49+
_ClearFieldArgType: typing_extensions.TypeAlias = typing.Literal["Lower", b"Lower"]
50+
def ClearField(self, field_name: _ClearFieldArgType) -> None: ...
51+
52+
Global___Upper: typing_extensions.TypeAlias = Upper
53+
54+
@typing.final
55+
class lower2(google.protobuf.message.Message):
56+
DESCRIPTOR: google.protobuf.descriptor.Descriptor
57+
58+
UPPER_FIELD_NUMBER: builtins.int
59+
@property
60+
def upper(self) -> Global___Upper: ...
61+
def __init__(
62+
self,
63+
*,
64+
upper: Global___Upper | None = ...,
65+
) -> None: ...
66+
_HasFieldArgType: typing_extensions.TypeAlias = typing.Literal["upper", b"upper"]
67+
def HasField(self, field_name: _HasFieldArgType) -> builtins.bool: ...
68+
_ClearFieldArgType: typing_extensions.TypeAlias = typing.Literal["upper", b"upper"]
69+
def ClearField(self, field_name: _ClearFieldArgType) -> None: ...
70+
71+
Global___lower2: typing_extensions.TypeAlias = lower2
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
"""
2+
@generated by mypy-protobuf. Do not edit manually!
3+
isort:skip_file
4+
"""
5+
6+
import collections.abc
7+
8+
9+
GRPC_GENERATED_VERSION: str
10+
GRPC_VERSION: str

test/generated_async_only/testproto/Capitalized/__init__.py

Whitespace-only changes.

0 commit comments

Comments
 (0)