Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,9 @@ submitqueue/ # repo root (Go module github.com/uber/submi
├── runway/ # Runway domain (single service — the domain *is* the service)
│ └── controller/ # Runway service controllers (consumes the merge queues; no gateway/orchestrator split)
├── tool/ # Development and CI tooling
├── example/
├── service/ # Runnable server/client wiring (entry points + Docker Compose)
│ ├── submitqueue/ # Runnable SubmitQueue servers/clients + Docker Compose
│ ├── stovepipe/ # Runnable Stovepipe servers/clients
│ ├── stovepipe/ # Runnable Stovepipe server/client + Docker Compose
│ └── runway/ # Runnable Runway server/client + Docker Compose
├── test/
│ ├── e2e/submitqueue/ # End-to-end tests (full stack)
Expand Down Expand Up @@ -132,7 +132,7 @@ Vendor-agnostic, pluggable interfaces with implementations in subdirectories:

A `{domain}/extension/{ext}` or `platform/extension/{ext}` package contains the behavioral interface, its `Config`, the `Factory` *interface*, and impl constructors `New(...)` that return the interface. It must **not** contain `Factory` *implementations* (`NewFactory()` constructors or factory structs) or any queue-selection logic.

Why: an impl package (e.g. `scorer/heuristic`) can't know the queue topology or the other impls, so a "which impl for which queue" decision doesn't belong there. Per-queue routing — and the small adapters that wrap a `New(...)` impl in the `Factory` interface — live in the wiring layer (e.g. `example/{domain}/{service}/server/main.go`), the one place that knows the full queue set. That's where you route on `Config.QueueName`.
Why: an impl package (e.g. `scorer/heuristic`) can't know the queue topology or the other impls, so a "which impl for which queue" decision doesn't belong there. Per-queue routing — and the small adapters that wrap a `New(...)` impl in the `Factory` interface — live in the wiring layer (e.g. `service/{domain}/{service}/server/main.go`), the one place that knows the full queue set. That's where you route on `Config.QueueName`.

Rule of thumb: if you're about to add a `NewFactory()` or a `map[queue]impl` under `{domain}/extension/` or `platform/extension/`, it belongs in the wiring layer instead.

Expand Down Expand Up @@ -239,11 +239,11 @@ make clean # Clean Bazel cache
**Add new RPC method:**
1. Edit `api/{domain}/{service}/proto/*.proto` → `make proto`
2. Add controller in `{domain}/{service}/controller/`
3. Wire up in `example/{domain}/{service}/server/main.go`
3. Wire up in `service/{domain}/{service}/server/main.go`

**Add new queue message controller:**
1. Create `{domain}/{service}/controller/{step}/` implementing `consumer.Controller`
2. Wire up in `example/{domain}/{service}/server/main.go`
2. Wire up in `service/{domain}/{service}/server/main.go`

**Add new extension:**
1. Create the extension under `{domain}/extension/{ext}/{impl}/` (domain-specific, e.g. `submitqueue/extension/...`) or `platform/extension/{ext}/{impl}/` (shared across domains) with factory and interfaces
Expand Down
44 changes: 22 additions & 22 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -5,21 +5,21 @@ BAZEL = ./tool/bazel
COMPOSE = docker-compose

# SubmitQueue compose files
COMPOSE_FILE = example/submitqueue/docker-compose.yml
GATEWAY_COMPOSE_FILE = example/submitqueue/gateway/server/docker-compose.yml
ORCHESTRATOR_COMPOSE_FILE = example/submitqueue/orchestrator/server/docker-compose.yml
COMPOSE_FILE = service/submitqueue/docker-compose.yml
GATEWAY_COMPOSE_FILE = service/submitqueue/gateway/server/docker-compose.yml
ORCHESTRATOR_COMPOSE_FILE = service/submitqueue/orchestrator/server/docker-compose.yml

# Fixed project name for local manual testing (tests use unique random names)
SUBMITQUEUE_LOCAL_PROJECT = submitqueue

# Stovepipe compose file (single Ping-only service)
STOVEPIPE_COMPOSE_FILE = example/stovepipe/docker-compose.yml
STOVEPIPE_COMPOSE_FILE = service/stovepipe/docker-compose.yml

# Fixed project name for local manual testing (tests use unique random names)
STOVEPIPE_LOCAL_PROJECT = stovepipe

# Runway compose files
RUNWAY_COMPOSE_FILE = example/runway/server/docker-compose.yml
RUNWAY_COMPOSE_FILE = service/runway/server/docker-compose.yml

# Fixed project name for local manual testing (tests use unique random names)
RUNWAY_LOCAL_PROJECT = runway
Expand Down Expand Up @@ -65,34 +65,34 @@ build-all-linux: build-submitqueue-gateway-linux build-submitqueue-orchestrator-

build-runway-linux: ## Build Runway Linux binary for Docker
@echo "Building Runway Linux binary for Docker..."
@$(BAZEL) build --platforms=@rules_go//go/toolchain:linux_amd64 //example/runway/server:runway
@$(BAZEL) build --platforms=@rules_go//go/toolchain:linux_amd64 //service/runway/server:runway
@mkdir -p .docker-bin
@cp -f bazel-bin/example/runway/server/runway_/runway .docker-bin/runway 2>/dev/null || \
cp -f bazel-bin/example/runway/server/runway .docker-bin/runway
@cp -f bazel-bin/service/runway/server/runway_/runway .docker-bin/runway 2>/dev/null || \
cp -f bazel-bin/service/runway/server/runway .docker-bin/runway
@echo "Runway Linux binary ready at .docker-bin/runway"

build-submitqueue-gateway-linux: ## Build Gateway Linux binary for Docker
@echo "Building Gateway Linux binary for Docker..."
@$(BAZEL) build --platforms=@rules_go//go/toolchain:linux_amd64 //example/submitqueue/gateway/server:gateway
@$(BAZEL) build --platforms=@rules_go//go/toolchain:linux_amd64 //service/submitqueue/gateway/server:gateway
@mkdir -p .docker-bin
@cp -f bazel-bin/example/submitqueue/gateway/server/gateway_/gateway .docker-bin/gateway 2>/dev/null || \
cp -f bazel-bin/example/submitqueue/gateway/server/gateway .docker-bin/gateway
@cp -f bazel-bin/service/submitqueue/gateway/server/gateway_/gateway .docker-bin/gateway 2>/dev/null || \
cp -f bazel-bin/service/submitqueue/gateway/server/gateway .docker-bin/gateway
@echo "Gateway Linux binary ready at .docker-bin/gateway"

build-submitqueue-orchestrator-linux: ## Build Orchestrator Linux binary for Docker
@echo "Building Orchestrator Linux binary for Docker..."
@$(BAZEL) build --platforms=@rules_go//go/toolchain:linux_amd64 //example/submitqueue/orchestrator/server:orchestrator
@$(BAZEL) build --platforms=@rules_go//go/toolchain:linux_amd64 //service/submitqueue/orchestrator/server:orchestrator
@mkdir -p .docker-bin
@cp -f bazel-bin/example/submitqueue/orchestrator/server/orchestrator_/orchestrator .docker-bin/orchestrator 2>/dev/null || \
cp -f bazel-bin/example/submitqueue/orchestrator/server/orchestrator .docker-bin/orchestrator
@cp -f bazel-bin/service/submitqueue/orchestrator/server/orchestrator_/orchestrator .docker-bin/orchestrator 2>/dev/null || \
cp -f bazel-bin/service/submitqueue/orchestrator/server/orchestrator .docker-bin/orchestrator
@echo "Orchestrator Linux binary ready at .docker-bin/orchestrator"

build-stovepipe-linux: ## Build Stovepipe Linux binary for Docker
@echo "Building Stovepipe Linux binary for Docker..."
@$(BAZEL) build --platforms=@rules_go//go/toolchain:linux_amd64 //example/stovepipe/server:stovepipe
@$(BAZEL) build --platforms=@rules_go//go/toolchain:linux_amd64 //service/stovepipe/server:stovepipe
@mkdir -p .docker-bin
@cp -f bazel-bin/example/stovepipe/server/stovepipe_/stovepipe .docker-bin/stovepipe 2>/dev/null || \
cp -f bazel-bin/example/stovepipe/server/stovepipe .docker-bin/stovepipe
@cp -f bazel-bin/service/stovepipe/server/stovepipe_/stovepipe .docker-bin/stovepipe 2>/dev/null || \
cp -f bazel-bin/service/stovepipe/server/stovepipe .docker-bin/stovepipe
@echo "Stovepipe Linux binary ready at .docker-bin/stovepipe"

check-gazelle: ## Check BUILD.bazel files are up to date
Expand Down Expand Up @@ -361,26 +361,26 @@ proto: ## Generate protobuf files from .proto definitions

# Bazel query helpers
query-deps:
@$(BAZEL) query 'deps(//example/submitqueue/gateway/server:gateway)'
@$(BAZEL) query 'deps(//service/submitqueue/gateway/server:gateway)'

query-targets:
@$(BAZEL) query //...

# Run gateway client (connects to any running gateway service)
run-client-submitqueue-gateway:
@$(BAZEL) run //example/submitqueue/gateway/client:gateway -- -addr $(or $(SERVER_ADDR),localhost:8081) -message "$(or $(MESSAGE),ping)"
@$(BAZEL) run //service/submitqueue/gateway/client:gateway -- -addr $(or $(SERVER_ADDR),localhost:8081) -message "$(or $(MESSAGE),ping)"

# Run orchestrator client (connects to any running orchestrator service)
run-client-submitqueue-orchestrator:
@$(BAZEL) run //example/submitqueue/orchestrator/client:orchestrator -- -addr $(or $(SERVER_ADDR),localhost:8082) -message "$(or $(MESSAGE),ping)"
@$(BAZEL) run //service/submitqueue/orchestrator/client:orchestrator -- -addr $(or $(SERVER_ADDR),localhost:8082) -message "$(or $(MESSAGE),ping)"

# Run stovepipe client (connects to any running stovepipe service)
run-client-stovepipe:
@$(BAZEL) run //example/stovepipe/client:stovepipe -- -addr $(or $(SERVER_ADDR),localhost:8083) -message "$(or $(MESSAGE),ping)"
@$(BAZEL) run //service/stovepipe/client:stovepipe -- -addr $(or $(SERVER_ADDR),localhost:8083) -message "$(or $(MESSAGE),ping)"

# Run runway client (connects to any running runway service)
run-client-runway:
@$(BAZEL) run //example/runway/client:runway -- -addr $(or $(SERVER_ADDR),localhost:8086) -message "$(or $(MESSAGE),ping)"
@$(BAZEL) run //service/runway/client:runway -- -addr $(or $(SERVER_ADDR),localhost:8086) -message "$(or $(MESSAGE),ping)"

run-queue-admin: ## Run queue-admin CLI (use ARGS to pass arguments, e.g. make run-queue-admin ARGS="list-topics")
@$(BAZEL) run //platform/extension/messagequeue/mysql/ctl -- $(ARGS)
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ grpcurl -plaintext -d '{"message": "hello"}' localhost:8081 uber.submitqueue.gat
make local-stop
```

See [example/README.md](example/README.md) for more examples including running individual services and clients.
See [service/README.md](service/README.md) for more examples including running individual services and clients.

## Documentation

Expand All @@ -43,7 +43,7 @@ See [example/README.md](example/README.md) for more examples including running i
| [Contributing](CONTRIBUTING.md) | How to contribute, workflow, guidelines |
| [Testing Guide](doc/howto/TESTING.md) | Unit, integration, and E2E testing patterns |
| [Architecture Guide](CLAUDE.md) | Project layout, patterns, conventions |
| [Examples](example/README.md) | Running services, clients, API reference |
| [Examples](service/README.md) | Running services, clients, API reference |
| [RFCs](doc/rfc/index.md) | Design documents and proposals |

## Project Status
Expand Down
6 changes: 3 additions & 3 deletions doc/howto/TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -388,6 +388,6 @@ assert.Equal(s.T(), "expected", resp.Value)
## See Also

- [CLAUDE.md](../../CLAUDE.md) - Development guidelines and project structure
- [example/submitqueue/docker-compose.yml](../../example/submitqueue/docker-compose.yml) - Full stack service definitions
- [example/submitqueue/gateway/server/docker-compose.yml](../../example/submitqueue/gateway/server/docker-compose.yml) - Gateway isolation
- [example/submitqueue/orchestrator/server/docker-compose.yml](../../example/submitqueue/orchestrator/server/docker-compose.yml) - Orchestrator isolation
- [service/submitqueue/docker-compose.yml](../../service/submitqueue/docker-compose.yml) - Full stack service definitions
- [service/submitqueue/gateway/server/docker-compose.yml](../../service/submitqueue/gateway/server/docker-compose.yml) - Gateway isolation
- [service/submitqueue/orchestrator/server/docker-compose.yml](../../service/submitqueue/orchestrator/server/docker-compose.yml) - Orchestrator isolation
2 changes: 1 addition & 1 deletion doc/rfc/submitqueue/extension-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ Non-obvious points:

## Mechanism

Dependencies are injected per-extension at the existing `Factory.For` (wiring: `example/submitqueue/orchestrator/server/main.go`) — only the handles a contract justifies, never the whole storage aggregator. The repeated batch→changes walk becomes one shared resolver (today's duplicated `collectChanges`, consolidated, and preserving the batch boundaries build's copy flattens). Controllers shrink to passing the identity entity they already load.
Dependencies are injected per-extension at the existing `Factory.For` (wiring: `service/submitqueue/orchestrator/server/main.go`) — only the handles a contract justifies, never the whole storage aggregator. The repeated batch→changes walk becomes one shared resolver (today's duplicated `collectChanges`, consolidated, and preserving the batch boundaries build's copy flattens). Controllers shrink to passing the identity entity they already load.

`entity.BatchChanges` is kept, not removed — it becomes the shared resolver's *detailed output* (URIs + provider details for a batch, what the scorer consumes) rather than a value the score controller assembles and passes in. Its line/file helpers move with it; only its producer changes.

Expand Down
153 changes: 0 additions & 153 deletions example/README.md

This file was deleted.

Loading
Loading