Skip to content

docs(rfc): Add pause and unpause RFC - #760

Open
manjari25 wants to merge 1 commit into
mainfrom
manjari/queue-pause-rfc
Open

manjari25 wants to merge 1 commit into
mainfrom
manjari/queue-pause-rfc

Conversation

@manjari25

Copy link
Copy Markdown
Contributor

Why?

We need a mechanism to pause and unpause queues during incidents, maintenance windows etc.

What?

This PR adds an RFC for pausing and unpausing a queue at runtime, comparing pause state on queue config against a separate pause-state extension.

Test Plan

RFC only; code change in next PR.

Issue


## Open questions

- **Fail open or closed on freeze.** Reject fails closed when pause state cannot be read, because `Land()` returns the error. However, the consumer treats a gate read error as open, so an adapter that returns the store's error would silently unfreeze a queue during a store outage. The adapter can instead return a blocked entry on error to fail closed. Which posture should freeze take?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fail close

## Open questions

- **Fail open or closed on freeze.** Reject fails closed when pause state cannot be read, because `Land()` returns the error. However, the consumer treats a gate read error as open, so an adapter that returns the store's error would silently unfreeze a queue during a store outage. The adapter can instead return a blocked entry on error to fail closed. Which posture should freeze take?
- **Cancel during freeze.** Should cancel take effect while processing is frozen, as an incident workflow of freeze, cancel, unfreeze would need? If so, freeze becomes no new effects rather than nothing runs, and needs the topic on the gate key ([Cancellation](#cancellation)).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

marking the request as cancelled is probably fine, the processing to resume on unfreeze

- **Fail open or closed on freeze.** Reject fails closed when pause state cannot be read, because `Land()` returns the error. However, the consumer treats a gate read error as open, so an adapter that returns the store's error would silently unfreeze a queue during a store outage. The adapter can instead return a blocked entry on error to fail closed. Which posture should freeze take?
- **Cancel during freeze.** Should cancel take effect while processing is frozen, as an incident workflow of freeze, cancel, unfreeze would need? If so, freeze becomes no new effects rather than nothing runs, and needs the topic on the gate key ([Cancellation](#cancellation)).
- **OSS backing.** `file` is proposed first because it needs no schema and works with a mounted config file. A MySQL-backed store would serve multi-host OSS deployments, and needs a way for an operator to write it.
- **Read cost.** The `file` store reads on every `Land()` and on every gated delivery. A short cache may be needed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

peanuts.

}
```

The orchestrator wires a gate over the same store:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why this cannot be integrated with consumergate? Altering consumergate to satisfy both scenarios is a possibility, was it considered?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you clarify what both scenarios you mean here?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think that could work here, we can just block on consumer gate to stop consuming things from the queue, we just need change where we read the gating information i guess

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would mean there is no "reject" mode. Just accept + hold for processing. Is that what you mean?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants