Skip to content

MMCore: (more) proper support for multi-channel cameras - #936

Draft
marktsuchida wants to merge 17 commits into
micro-manager:mainfrom
marktsuchida:seq-acq-tracking
Draft

marktsuchida wants to merge 17 commits into
micro-manager:mainfrom
marktsuchida:seq-acq-tracking

Conversation

@marktsuchida

@marktsuchida marktsuchida commented May 2, 2026 •

Copy link
Copy Markdown
Member

This is a DIV-bumping change; it might become the main change for DIV 76 (#886).

  • Introduce internal SequenceAcquisition object to explicitly represent and track the lifetime of each sequence acquisition, replacing ad-hoc state scattered across CMMCore
  • Add proper support for multi-channel (composite) camera sequence acquisition, with MMCore managing channel tags and participant tracking instead of mutating physical camera objects
  • Make autoshutter open/close thread-safe
  • Document the device-side sequence acquisition contract in MMDevice

Device interface changes

Bumps the device interface version:

  • MM::Camera: add GetChannelCameraPtr() for MMCore to query participating physical cameras
  • Remove the AddTag/RemoveTag mechanism (tags are now generated by MMCore during InsertImage())

Updated device adapters: MultiCamera, ArduinoCounter, CameraPulser, TwoPhoton. Regular (physical, single-channel) cameras don't need to change.


Notable behavioral changes

  • Physical cameras of a composite camera no longer emit channel tags when running solo
  • Composite cameras open the shutter once (not per-physical-camera) and close it only after all physical cameras finish
  • PrepareForAcq/InsertImage/AcqFinished callbacks now reject callers not participating in the active acquisition
  • SequenceAcquisitionStarted/Stopped notifications fire once per composite camera, not per physical camera
  • Intrinsic multi-channel cameras must provide a valid in-range CameraChannelIndex; CameraChannelName is now added by MMCore
  • Nested multi-channel cameras are detected and disallowed
  • If autoshutter was opened but camera failed to start, close the shutter (also issue Stopped notification)
  • Concurrent sequence acquisitions involving the same primary/physical cameras is explicitly prohibited
  • Equal width/height/bpp/nComponents is enforced upfront for composite camera channels

Things not covered here

  • This opens up a path toward per-channel sequence buffers, but that is for another PR.
  • We don't (yet) make an effort to make multiple concurrent sequence acquisitions usable. That probably also needs per-acquisition sequence buffers. Can also support unequal image size/format in this case.
  • The multi-channel-related tags still have the primary camera's label prefixed, to preserve behavior.
  • We don't freeze the autoshutter state and shutter to use at the start of acquisition. I'm not sure if that would be better or worse.

@jacopoabramo jacopoabramo left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marktsuchida howdy, as we chatted on Zulip some time ago I'm making some comments on the PR.

Honestly I'm pretty sure that either these are already stale or you're already aware. I have a more detailed document produced by an Opus agent but it's as jargon as it gets. Hopefully it gets the engines back online on this work.

Please let me know if I can do anything else to help. Thanks in advance

Comment on lines +181 to +186
bool SequenceAcquisition::IsComplete() const noexcept
{
std::lock_guard<std::mutex> g(mu_);
return stopRequested_ &&
finishedParticipants_.size() == expectedParticipants_.size();
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An agent identified that certain cameras, i.e. TsiCam, doesn't call PrepareForAcq() or AcqFinished() (at least, in respect to the current commit this PR points in main, which is far behind as far as I can tell).

After a start and a stop, its entry stays in the map with one participant not finished. This causes the following:

  • the next startSequenceAcquisition() throws "Sequence acquisition on '...' is already running";
  • initializeCircularBuffer() and setCircularBufferMemoryFootprint() throw MMERR_NotAllowedDuringSequenceAcquisition;
  • isSequenceRunning() returns true, because findAcquisitionInvolvingCamera() returns any entrythat is not finished, before it asks the camera.

All three stay that way until a device is unloaded.

A fix suggested by would be to threat any camera that reports IsCapturing() == false as finished, even if they didn't call AcqFinished(), close the autoshutter and post SequenceAcquisitionStopped.

I'm guessing this is the least breaking change the model felt capable of suggesting. It also suggest a whole big change to apply to both MMCore and SequenceAcquisition but it feels noise right now.

I'll provide the full document if you'd like to have a view yourself.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not calling PrepareForAcq() and AcqFinished() is a straight camera bug, so this makes it easier to find those bugs, but of course I'd rather not break things that currently "work" (even partially: broken autoshutter); finding non-compliant adapters is better done through tools like #872.

Treating IsCapturing() == false as finished (probably when isSequenceRunning() is called?) would revert the "camera is not finished until the whole acquisition it's participating in is finished" semantics added in this PR. Maybe some more conditional application of that strategy could work (primary camera only?). Will think about it a bit.

Either way, we should probably (also) force-finish the acquisition after stopSequenceAcquisition().

I'm not sure about closing the autoshutter in the recovery case. The shutter would remain open before this PR, and it doesn't seem worth further hiding the adapter bug. Plus it only helps if AcqFinished() is skipped but not PrepareForAcq(), which is extra buggy.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either way, we should probably (also) force-finish the acquisition after stopSequenceAcquisition()

The agent suggested an API for this point:

// MMCore/SequenceAcquisition.h:

   // Records every participant that has not called AcqFinished() as
   // finished. Returns true iff this call finished the last one.
   bool FinishRemainingParticipants();

// MMCore/SequenceAcquisition.cpp

bool SequenceAcquisition::FinishRemainingParticipants()
{
   std::lock_guard<std::mutex> g(mu_);
   if (finishedParticipants_.size() == expectedParticipants_.size())
      return false;
   finishedParticipants_ = expectedParticipants_;
   return true;
}

// MMCore/MMCore.cpp, stopSequenceAcquisitionImpl(), first statement inside
// `if (sa) {` (line 3462), before `if (sa->TakeDeferredShutterClose())`
// (line 3463). It is only reached when camera->StopSequenceAcquisition()
// (line 3453) returned DEVICE_OK, because the error path throws first.
// The existing `if (sa->IsComplete())` at line 3466 then erases the entry.

      if (sa->FinishRemainingParticipants()) {
         LOG_WARNING(coreLogger_) << "Camera '" << sa->CameraLabel() <<
            "' stopped without every participant calling AcqFinished()";
         postNotification(
            notif::SequenceAcquisitionStopped{sa->CameraLabel()});
      }

Not sure how well it applies, just giving it as input for the future, hoping it's not noisy.

Comment thread MMCore/MMCore.cpp
Comment on lines +959 to +961
for (const auto& kv : acquisitions_) {
if (kv.second->HasParticipant(caller))
return kv.second;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for (const auto& kv : acquisitions_) {
if (kv.second->HasParticipant(caller))
return kv.second;
for (const auto& kv : acquisitions_) {
if (kv.second->AllParticipantsFinished())
continue;
if (kv.second->HasParticipant(caller))
return kv.second;

This is a tricky one to explain, I'll simply report the theorized sequence of operations the agent suggested.

  1. startSequenceAcquisition("Camera", 10, 0, true) runs to the end; its entry stays in the map with all participants finished.
  2. The user starts Multi Camera, with Camera as a physical camera. The conflict check at MMCore.cpp:3289 skips finished entries, so the start is accepted.
  3. Camera calls PrepareForAcq(). The map is ordered by label and "Camera" sorts before "MultiCamera", so the old entry is returned. Its BeginPrepare() reports AlreadyPrepared and the shutter is not opened.
  4. InsertImage() adds no channel tags, because the old entry treats Camera as a plain single-channel camera.
  5. AcqFinished() calls RecordFinish() on the old entry, which returns false for a repeat call. The Multi Camera entry never completes: no SequenceAcquisitionStopped, the shutter state is wrong, and Multi Camera cannot be started again.

The fix is to do the same skip that findAcquisitionInvolvingCamera() does.

Comment thread MMCore/MMCore.cpp Resolved
Comment thread MMCore/CoreCallback.cpp
Comment on lines +567 to +573
if (mtx.try_lock()) {
std::lock_guard<std::recursive_mutex> g(mtx, std::adopt_lock);
sret = shutter->SetOpen(true);
} else {
sa->DeferShutterOpen();
deferred = true;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When PrepareForAcq() cannot take the module lock, the shutter opening is deferred.

There are three ways to hang>

  • a camera calls PrepareForAcq() from its worker thread; try_lock() fails because the start thread still holds the lock. The starting thread releases the lock and calls TakeDeferredShutterOpen(), which is false, before the worker reaches DeferShutterOpen() and the worker waits forever
  • PrepareForAcq() runs after startSequenceAcquisition() has returned, while an unrelated thread holds the module lock (i.e. a property read on the same adapter). It defers and nobody runs the deferred open.
  • with a composite camera, physical camera 1 defers from its worker thread; physical camera 2 calls PrepareForAcq() on the start thread, gets WaitForOpener and blocks while holding the module lock, causing a deadlock because the start thread never reaches the deferred open

Again, there are suggested fixes for all 3 cases but I'm deferring them until they're acknowledged as possibilities

Comment thread MMCore/MMCore.cpp
Comment on lines +3487 to +3492
std::lock_guard<std::mutex> aqg(acquisitionsMutex_);
if (hasInFlightAcquisitionLocked())
throw CMMError(
getCoreErrorText(MMERR_NotAllowedDuringSequenceAcquisition).c_str(),
MMERR_NotAllowedDuringSequenceAcquisition);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

startSequenceAcquisitionImpl() takes the camera module lock and then acquisitionsMutex_.

initializeCircularBuffer() and setCircularBufferMemoryFootprint() take acquisitionsMutex_, and then the camera module lock.

If a GUI thread calls one of them while another thread starts an acquisition each holds the lock the other needs.

The fix would be in both initializeCircularBuffer() and setCircularBufferMemoryFootprint() to take the camera module lock first

Comment thread MMCore/MMCore.cpp
Comment on lines +3384 to +3388
if (nRet != DEVICE_OK) {
logError(getDeviceName(camera).c_str(),
getDeviceErrorText(nRet, camera).c_str());
throw CMMError(getDeviceErrorText(nRet, camera).c_str(),
MMERR_DEVICE_GENERIC);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When this call returns an error, stopSequenceAcquisitionImpl() throws after MarkStopRequested().

If the camera thread calls AcqFinished() and the try_lock() fails, SpawnOrDeferShutterClose() sees the stop request and defers the close. The code running the deferred close is after the throw and the shutter stays open (and the entry stays in the map) until someone calls stop again.

The fix would be to keep the error code and move this check after the if (sa) block below

Comment thread MMCore/MMCore.cpp
Comment on lines +3255 to +3266
if (!cbuf_->Initialize(
static_cast<std::size_t>(camera->GetImageWidth()) *
camera->GetImageHeight() *
camera->GetImageBytesPerPixel()))
{
logError(getDeviceName(camera).c_str(),
getCoreErrorText(MMERR_CircularBufferFailedToInitialize).c_str());
throw CMMError(
getCoreErrorText(MMERR_CircularBufferFailedToInitialize).c_str(),
MMERR_CircularBufferFailedToInitialize);
}
cbuf_->Clear();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shared sequence buffer is set up and emptied before the checks against running acquisitions. Consequences:

  • A labelled start of an unrelated camera while another acquisition runs is accepted, and it discards that acquisition's queued images and may change the frame size under its inserting thread. The PR description says concurrent acquisitions are not made usable yet, but they are not refused either.
  • A start that the conflict checks then refuse has already emptied the buffer. The camera->IsCapturing() check before it catches the common case (MultiCamera::IsCapturing() asks everyphysical camera), so this mainly matters when a camera has stopped capturing but its entry has notfinished, as in finding 1.

The fix is to refuse a start while any other acquisition is in progress and set up the buffer only after all the other checks have passed. This is already applied to initializeCircularBuffer().

Comment thread MMCore/MMCore.cpp
}

sa = newSa;
acquisitions_[camera->GetLabel()] = std::move(newSa);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair warning this is weird to me, I'm still dumping it because it might be clearer for you.

~SequenceAcquisition() joins the deferred shutter-close worker.

Suppose an acquisition ended on its ownwhile another thread held the module lock. AcqFinished() then spawned a worker that waits on DeviceModuleLockGuard(shutter), which only happens when the shutter is in the same adapter module asthe camera. If the user starts again before the worker gets the lock, the start thread holds that lock, theassignment acquisitions_[label] = std::move(newSa) destroys the old entry, and its destructor joins aworker that is waiting for that lock.

The suggested fix is to move the eold entry into a local declared before the module lock guard, so it's destroyed after both locks are released.

Suggested change
acquisitions_[camera->GetLabel()] = std::move(newSa);
// at the beginning of startSequenceAcquisitionImpl(...):
// std::shared_ptr<mmcore::internal::SequenceAcquisition> previous;
auto& slot = acquisitions_[camera->GetLabel()];
previous = std::move(slot);
slot = std::move(newSa);

Comment thread MMCore/CoreCallback.cpp
Comment on lines +319 to +326
const std::string prefix = sa->CameraLabel() + '-';
const std::string prefixedIndexKey =
prefix + MM::g_Keyword_CameraChannelIndex;
const std::string prefixedNameKey =
prefix + MM::g_Keyword_CameraChannelName;

if (pi.kind == Kind::CompositeChannel) {
md.AddTag(prefixedIndexKey, std::to_string(pi.index));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the agent this is called frequently, several thousand times for fast cameras (not sure where its getting this assumption, pretty curious but it makes sense in the end). Basically these strings are reconstructed everytime. It suggests building the keys and the indext text once, at acquisition creation.

Comment thread MMDevice/MMDevice.h
Comment on lines +398 to +401
* Required: 0 <= n < GetNumberOfChannels(); behavior outside that range
* is undefined except that the default CCameraBase implementation
* returns nullptr for any in-range n and logs+returns nullptr for
* out-of-range n.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently it complains that this docstring is incorrect since GetNumberOfChannels() doesn-t actually call any log. Just a nitpick it seems to me, and most likely given that it's still in draft not worth the attention. I'm just flagging for posterity in case it gets forgotten since technically a wrong documentation is effectively a bug...

@marktsuchida

Copy link
Copy Markdown
Member Author

Thanks, @jacopoabramo! I should be able to put some work into this next week (traveling right now). The first step will indeed be to rebase onto the current main.

SequenceAcquisition is created for each new sequence acquisition and is
cleaned up (for now) lazily. Better cleanup will be possible once we
start tracking cameras participating in composite cameras (like Multi
Camera).

This is a start toward more directly managing sequence acquisitions by
the Core, so that multiple cameras/acquisitions can be better supported
and thread safety can be improved. For now, it does little more than
just create the SequenceAcquisition object.

(Assisted by Claude Code; any errors are mine.)
Simplify the way in which frame tags are generated for multi-channel
cameras. Especially for composite cameras (e.g. Multi Camera), avoid
mutating the physical camera objects (which could not be cleanly undone
at the right timing) and instead map physical cameras to tags in the
Core.

Known behavioral changes:

1. Physical cameras associated with Multi Camera (or similar composite
   cameras) no longer incorrectly generate camera channel name/index
   tags when running sequence acquisition as an individual camera.
2. When autoshutter is enabled, composite cameras no longer cause the
   shutter to be opened/closed multiple times (even though it was
   idempotent). None of the physical cameras start acquiring until the
   shutter finishes opening, and the shutter is not closed until all
   physical cameras finish/stop acquiring.
3. If opening the shutter fails, `PrepareForAcq()` now returns an error.
4. `PrepareForAcq/InsertImage/AcqFinished` callbacks will fail if not
   issued from a camera currently participating in a sequence
   acquisition (previously any device could theoretically call these at
   any time and send images to the buffer).
5. The `SequenceAcquisitionStarted/Stopped` notifications now use the
   primary camera label and are issued once, instead of for each
   physical camera, in the case of a composite multi-channel camera.
6. Intrinsic multi-channel cameras must tag images with a valid
   CameraChannelIndex that is in range; this is now verified.
   CameraChannelName is now added by MMCore so does not need to be added
   by the camera.
7. Nested multi-channel cameras are explicitly detected and disallowed.

Bumps the device interface version due to the following changes:

- In `MM::Camera`, add `GetChannelCameraPtr()` so that MMCore can query
  the primary camera for all participating (physical) cameras.

- Remove the `AddTag/RemoveTag` mechanism; instead, let the
  `SequenceAcquisition` object manage all participating cameras in a
  composite multi-channel camera acquisition, and generate tags
  (`CameraChannelIndex`, `CameraChannelName`) during `InsertImage()` in
  MMCore, where possible (`CameraChannelIndex` is still required from
  intrinsic multi-channel cameras, because they come from the same
  source).

Note: the CameraChannelIndex and CameraChannelName tags are still
prefixed with `primariCameraLabel-` for now, to preserve existing
behavior.

Composite multi-channel cameras MultiCamera, ArduinoCounter, and
CameraPulser are updated to remove the physical camera mutations
(`AddTag`/`RemoveTag`) and implement `GetChannelCameraPtr()`.

The only intrinsic multi-channel camera, TwoPhoton, is updated to remove
the tags that are no longer necessary.

(Assisted by Claude Code; any errors are mine.)
Finally.

(Assisted by Claude Code; any errors are mine.)
Factor out the common code and remove minor differences (which were
presumably not intentional).

Differences resolved:

- Camera adapter lock is acquired _before_ initializing the sequence
  buffer (because the latter accesses camera parameters).

- Don't catch std::bad_alloc (CircularBuffer::Initialize() already
  does for the main buffer memory).

(Assisted by Claude Code; any errors are mine.)
(Assisted by Claude Code; any errors are mine.)
Remove the unsafe hacks; when shutter is in the same adapter as the
_primary_ camera (whose adapter we _may_ hold the lock for, in the same
or different thread), fall back to closing the shutter afterwards on the
thread calling stopSequenceAcquisition().

But always use try_lock() so that we start closing the shutter asap when
we are able to -- so that the specimen is not exposed to illumination
while the camera tears down the sequence acquisition (which for some
cameras might be slow).

Real (physical) cameras don't typically have a shutter in the same
adapter. Demo/simulated cameras often do, but then we don't care if the
shutter closure is ordered after the return from
StopSequenceAcquisition(). Multi Camera (and similar) does not call
AcqFinished() (only its physical cameras do), so even if the shutter is
from the same adapter as the composite camera, we don't need to defer
shutter closure.

In no case do we call the shutter without holding the device module lock
any more.

(Assisted by Claude Code; any errors are mine.)
Another long-standing bug we can now easily fix.

(Assisted by Claude Code; any errors are mine.)
That is, disallow starting a sequence acquisition where any of the
participants (primary + physical) overlap with an ongoing acquisition.

(Assisted by Claude Code; any errors are mine.)
It will fail anyway on InsertImage(), so fail fast.

(Assisted by Claude Code; any errors are mine.)
This would always cause problems or errors; now catch upfront.

For now, no acquisition can be running because we use a single, global
buffer.

(Assisted by Claude Code; any errors are mine.)
If the shutter is in the same adapter as the camera _and_ the camera
calls PrepareForAcq() from a thread other than the one calling into
StartSequenceAcquisition(), defer the shutter opening so that we don't
deadlock.

This is probably not an issue today, but would become one if, say, a
MultiCamera-like device decides to start physical cameras in parallel
(which has its own set of problems if they come from the same adapter,
so perhaps is unlikely to happen).

Even with this, there will be a deadlock if the camera's
StartSequenceAcquisition() waits for PrepareForAcq() to return; that is
forbidden and is now documented (there's no reasonable way to work
around that situation, and fortunately we don't face it in any known
existing camera).

(Assisted by Claude Code; any errors are mine.)
Avoid waiting forever when shutter opening/waiting failed. Also, avoid
leaking exceptions back to caller of PrepareForAcq().

(Assisted by Claude Code; any errors are mine.)
When the autoshutter is in the same adapter module as the camera and
AcqFinished's try_lock on the module lock fails, the old code
unconditionally set shutterCloseDeferred_ and relied on
stopSequenceAcquisition to drain it. That broke natural completion —
finite-frame acquisitions that complete without anyone calling
stopSequenceAcquisition would leave the autoshutter open forever,
because the deferred-close was only drained from the stop path.

The new design splits that path atomically (under
SequenceAcquisition::mu_):

- If stopRequested_ is set → defer (stopper will drain via
  TakeDeferredShutterClose).
- Otherwise → spawn a worker thread that takes the module lock
  blockingly and closes the shutter itself, then posts
  ShutterOpenChanged. The thread is owned by the SA
  (shutterCloseWorker_) and joined either by stopSequenceAcquisition or
  by ~SequenceAcquisition.

To make the atomicity work, CMMCore::stopSequenceAcquisition now calls
MarkStopRequested() before taking the module lock. That ordering is
important: it guarantees that any AcqFinished racing with the stopper
observes stopRequested_=true and takes the defer branch instead of
spawning a worker the stopper would have to chase.

(Assisted by Claude Code; any errors are mine.)
If the camera is a participant in a composite camera acquisition, it
remains "running" until that acquisition as a whole finishes.

We keep the camera IsCapturing() check, too, because some cameras may
briefly return IsCapturing() == true after calling AcqFinished().

(Assisted by Claude Code; any errors are mine.)
DRY.

(Assisted by Claude Code; any errors are mine.)
Reject attempts to stop a non-primary participant in a composite camera
acquisition.

(Previous behavior was to roguely stop just that physical camera, which
doesn't really make sense.)

(Assisted by Claude Code; any errors are mine.)
@marktsuchida

Copy link
Copy Markdown
Member Author

Force-pushed after rebasing onto main.

@jacopoabramo all your comments still apply and I'll get to them soon!

@jacopoabramo

Copy link
Copy Markdown

@marktsuchida thanks, I'll be happy to re-review afterwards

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

When sequence acquisition fails to start, Core should close autoshutter

2 participants