Skip to content

Remote strategy over loopback TCP: bars in, empty orders out (#703) - #710

Open
rustyeddy wants to merge 4 commits into
mainfrom
feat/703-remote-onbars-tcp
Open

rustyeddy wants to merge 4 commits into
mainfrom
feat/703-remote-onbars-tcp

Conversation

@rustyeddy

@rustyeddy rustyeddy commented Nov 20, 2026 •

Copy link
Copy Markdown
Owner

Closes #703. Part of #690.

What

  • strategy/ (new package, does not import trader; a test checks that):
    • strategy.go: package doc and the two message types, {"bars":[marketdata.Bar…]} and {"orders":[]}.
    • frame.go: standard encoding/json with a 4-byte big-endian length prefix. A short write is io.ErrShortWrite; a message over 16 MiB is refused before it is allocated.
    • client.go: Dial (loopback addresses only) and Client.OnBars, which waits at most Client.Timeout (30 s by default, or the context deadline if sooner) and returns at once when the context is cancelled. An answer with orders is an error.
    • server.go: the worker's end. Serve(conn, onBars) answers each group with no orders until the trader closes the connection; ListenAndServe(addr, ready, onBars) listens on a loopback address, accepts one connection and serves it. ready (may be nil) is told the bound address.
  • Root strategy_remote.go: a small adapter that implements the engine's private strategy interface and delegates to strategy.Client. The root strategy.go, EMAX, order.go and the engine are not moved, and there is no new order type.
  • examples/remote-strategy: a standalone worker built on strategy.ListenAndServe.
  • The backtest loop is unchanged: the same runSeries and trader.onFrame drive the remote strategy.

The one interface change

The engine's private interface is now onBars(ctx, bars) ([]*order, error). trader.onFrame checks the error before it looks at any order and stops the run. eachBar returns a nil error, so EMAX and the other bar-by-bar strategies only changed their signatures. There is no second failure mechanism (failer, failed and err() are gone). The six-summary md5 is unchanged (195e457a…).

Behaviour

  • The trader dials the worker; the worker listens. No listener, registration or RunID.
  • A disconnect, a silent worker (timeout), a cancelled context or an answer containing orders each fail the run.
  • No warm-up.

Run the example

go run ./examples/remote-strategy -addr 127.0.0.1:9100   # prints: listening on 127.0.0.1:9100

Then strategy.Dial("127.0.0.1:9100") and run a backtest with it as the trader's strategy, as the tests do. There is no CLI flag yet.

Tests (go test -race -count=1 . ./strategy/ ./examples/... ./cmd/...)

  • strategy: ListenAndServe (bound address, nil ready, non-loopback and in-use addresses refused); framing round-trip, short write, oversize, truncated and non-JSON messages; loopback-only dial; Client against a silent worker (timeout), a cancelled context, a shorter context deadline, a hang-up and an answer with orders; Serve; Client against Serve over TCP.
  • Root: the adapter returns errors; trader.onFrame stops on a strategy error and uses none of its orders; a backtest over TCP delivers the same groups in the same order as an in-process recorder, with no trades; a worker that hangs up mid-run fails the run; the real example binary, built and run as a separate process, exits cleanly.

Also run on ., ./strategy, ./examples/..., ./cmd/...: golangci-lint (0 issues) and go mod tidy -diff (clean). make check passes locally on the full module.

🤖 Generated with Claude Code

…#703)

A remoteStrategy satisfies the existing strategy interface by sending each
frame to a worker as length-prefixed encoding/json and reading its answer. The
trader stops the run when a strategy reports a failure (failer), so a dead
worker ends the backtest instead of hanging it.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
@rustyeddy rustyeddy linked an issue Nov 20, 2026 that may be closed by this pull request
6 tasks
Copilot AI balanced review requested due to automatic review settings November 20, 2026 15:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@rustyeddy rustyeddy left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of 1438b74: Good simplification—existing runSeries and trader.onFrame are reused, JSON is standard-library, and the remote/in-process bar parity test is useful. Before merging, please address:

  1. Blocking: silent worker can hang forever. remoteStrategy.onBars ignores its ctx, and the connection has no read/write deadline. A worker that stays connected but does not respond blocks the backtest indefinitely (including cancellation). Add a modest response deadline and/or cancellation-triggered connection deadline; test a worker that accepts bars then remains silent. Keep this small—no new session framework.
  2. Blocking: TCP writes need full-frame handling. writeMessage and example worker write assume one io.Writer.Write sends the complete length-prefixed frame. Handle n < len(buf) with io.ErrShortWrite (or write the remainder) and test with a short-writing writer. readMessage already uses io.ReadFull correctly. Also cap the example worker's inbound frame length before allocation (currently an arbitrary uint32 can allocate up to ~4 GiB).
  3. Package ownership (Rusty's preference): Move transport/framing and the remote socket client into a small strategy subpackage, not the root trading domain, if it can be done without making a second Order model or import cycle. The existing root strategy interface returns private *order and has private onBars, so a full move of that interface requires a separately agreed public API refactor. For this no-orders increment, a tiny root adapter can remain; don't force a broad refactor in #710.

The optional failer.err() works as a temporary bridge, but should not become a permanent interface. Eventually OnBars(ctx, bars) (orders, error) is cleaner. I have not independently run Go tests. Please keep changes within #703's first empty-orders increment; no registration/journal framework.

Copy link
Copy Markdown
Owner Author

Owner decision on package layout for this PR: option 1. Put the minimal TCP/JSON transport in a new strategy/ directory, with strategy/remote.go for Trader-side dialing/exchange and strategy/server.go for the independently running example worker-side listener/serve logic (or the smallest equivalent layout that avoids duplication). Do not move the existing root strategy.go, EMAX, order.go, or trading engine now.

The root trader package can keep a tiny adapter implementing its existing private strategy interface and delegating to the new package. The new strategy package must not import trader, because that would create an import cycle; this first increment returns only empty orders, so no shared order type needs to be introduced. Keep this a focused move, not a new framework. Please also fix the two correctness findings in my review: silent worker timeout/cancellation and full-frame writes/worker inbound size bound. Open a revised PR update for review before merging.

…e deadline (#703)

strategy.Dial/Remote is the trader's end and strategy.Serve the worker's. OnBars
waits at most Timeout (30 s by default) and returns when its context ends, so a
silent worker cannot hang a run. writeMessage reports a short write; the worker
reads through the same size-capped readMessage. The root keeps a tiny adapter.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
@rustyeddy

Copy link
Copy Markdown
Owner Author

Revised in 5db2903 (not merged; waiting for your review).

Package layout (option 1)

  • New strategy/: strategy.go (package doc and the two message types), frame.go (length-prefixed JSON), remote.go (Dial, Remote.OnBars, Close), server.go (Serve). It does not import trader; strategy_test.go checks that.
  • The root keeps a small adapter (remote.go) that implements its private strategy interface and delegates to the new package, imported as remotestrat because the root already has a strategy type. strategy.go, EMAX, order.go and the engine are not moved. No order type was introduced.
  • The example worker now uses strategy.Serve, so the framing is not duplicated.

Review findings

  1. Silent worker: OnBars sets a read/write deadline (Remote.Timeout, default 30 s, or the context deadline if sooner) and cancelling the context expires it at once. Tests: silent worker times out, cancel returns context.Canceled, a short context deadline wins.
  2. Short writes: writeMessage returns io.ErrShortWrite when Write sends less than the frame (tested with a short-writing writer). The example worker reads through the same readMessage, so its inbound frame is capped at 16 MiB before allocation.
  3. failer stays as the temporary bridge and its comment says it goes away once onBars can return an error.

Checks on ., ./strategy, ./examples/..., ./cmd/...: race tests pass, golangci-lint 0 issues, go mod tidy -diff clean, EMAX md5 unchanged. I ran these in a separate worktree. The local strategy/ stub in the main checkout is untracked and not committed. It will collide with the new strategy/strategy.go when that checkout pulls the branch.

@rustyeddy rustyeddy left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed updated commit 5db2903 against the minimal scope for #703.

Previous blockers resolved:

  • Remote exchange now has a response timeout and honors context cancellation/deadlines; tests cover a silent worker.
  • Shared framing detects short writes and caps incoming message size before allocating; tests cover both.
  • TCP/JSON framing and worker server are in the sibling strategy/ package. Root retains only the small adapter and existing engine integration—no registry or extra lifecycle machinery.

Repeated OnBars, empty orders, disconnect handling, and backtest integration are covered. Looks ready to merge from my review, assuming normal CI is green. Minor non-blocking housekeeping: PR description still says there is no reply deadline and describes the old file layout; update to reflect the latest implementation. Registration, RunID, and actual order replies should stay outside this increment.

(GitHub does not permit this account to formally approve its own PR, so posting the recommendation as a comment review.)

Copy link
Copy Markdown
Owner Author

Follow-up for Claude on PR #710. The rename strategy/remote.go → strategy/client.go is a good clarification. Please make these focused changes before our next review:

  1. Return errors directly from onBars. Change the root private strategy interface to onBars(ctx, bars) ([]*order, error); adapt EMAX/other local strategies, the remote adapter, tests, and trader.onFrame. Check the error before processing any returned orders. Delete failer, remoteStrategy.failed, and err(). No second failure-reporting mechanism.
  2. Rename the root adapter file remote.go → strategy_remote.go (and the corresponding test file if convenient). Keep it small and in the root trader package; do not move the engine/order types.
  3. Expose a minimal Go worker SDK. github.com/rustyeddy/trader/strategy already provides the shared protocol and Serve(conn, onBars); retain that. A small ListenAndServe(addr, callback) wrapper is worthwhile if it replaces the listener/accept boilerplate in examples/remote-strategy. Keep this simple and loopback-default—no SDK module split, registry, registration, RunID, or order-response design here.

Please update the PR description to match the current strategy/client.go layout, timeout behavior and revised interface, and rerun relevant unit/race/backtest tests. #710 should remain the smallest useful TCP/JSON OnBars increment; richer order replies come later.

…703)

The engine's strategy interface is now onBars(ctx, bars) ([]*order, error) and
trader.onFrame stops on the error before using any order, so failer, the
adapter's failed field and err() are gone. The root adapter moves to
strategy_remote.go. strategy.ListenAndServe replaces the listener/accept
boilerplate in the example worker.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Strategy Protocol: minimal external OnBars TCP/JSON backtest (empty orders)

2 participants