Repository navigation
MMCore: (more) proper support for multi-channel cameras - #936
marktsuchida wants to merge 17 commits into
Conversation
1356217 to
dade037
Compare
jacopoabramo
left a comment
There was a problem hiding this comment.
@marktsuchida howdy, as we chatted on Zulip some time ago I'm making some comments on the PR.
Honestly I'm pretty sure that either these are already stale or you're already aware. I have a more detailed document produced by an Opus agent but it's as jargon as it gets. Hopefully it gets the engines back online on this work.
Please let me know if I can do anything else to help. Thanks in advance
| bool SequenceAcquisition::IsComplete() const noexcept | ||
| { | ||
| std::lock_guard<std::mutex> g(mu_); | ||
| return stopRequested_ && | ||
| finishedParticipants_.size() == expectedParticipants_.size(); | ||
| } |
There was a problem hiding this comment.
An agent identified that certain cameras, i.e. TsiCam, doesn't call PrepareForAcq() or AcqFinished() (at least, in respect to the current commit this PR points in main, which is far behind as far as I can tell).
After a start and a stop, its entry stays in the map with one participant not finished. This causes the following:
- the next
startSequenceAcquisition()throws "Sequence acquisition on '...' is already running"; initializeCircularBuffer()andsetCircularBufferMemoryFootprint()throwMMERR_NotAllowedDuringSequenceAcquisition;isSequenceRunning()returns true, becausefindAcquisitionInvolvingCamera()returns any entrythat is not finished, before it asks the camera.
All three stay that way until a device is unloaded.
A fix suggested by would be to threat any camera that reports IsCapturing() == false as finished, even if they didn't call AcqFinished(), close the autoshutter and post SequenceAcquisitionStopped.
I'm guessing this is the least breaking change the model felt capable of suggesting. It also suggest a whole big change to apply to both MMCore and SequenceAcquisition but it feels noise right now.
I'll provide the full document if you'd like to have a view yourself.
There was a problem hiding this comment.
Not calling PrepareForAcq() and AcqFinished() is a straight camera bug, so this makes it easier to find those bugs, but of course I'd rather not break things that currently "work" (even partially: broken autoshutter); finding non-compliant adapters is better done through tools like #872.
Treating IsCapturing() == false as finished (probably when isSequenceRunning() is called?) would revert the "camera is not finished until the whole acquisition it's participating in is finished" semantics added in this PR. Maybe some more conditional application of that strategy could work (primary camera only?). Will think about it a bit.
Either way, we should probably (also) force-finish the acquisition after stopSequenceAcquisition().
I'm not sure about closing the autoshutter in the recovery case. The shutter would remain open before this PR, and it doesn't seem worth further hiding the adapter bug. Plus it only helps if AcqFinished() is skipped but not PrepareForAcq(), which is extra buggy.
There was a problem hiding this comment.
Either way, we should probably (also) force-finish the acquisition after
stopSequenceAcquisition()
The agent suggested an API for this point:
// MMCore/SequenceAcquisition.h:
// Records every participant that has not called AcqFinished() as
// finished. Returns true iff this call finished the last one.
bool FinishRemainingParticipants();
// MMCore/SequenceAcquisition.cpp
bool SequenceAcquisition::FinishRemainingParticipants()
{
std::lock_guard<std::mutex> g(mu_);
if (finishedParticipants_.size() == expectedParticipants_.size())
return false;
finishedParticipants_ = expectedParticipants_;
return true;
}
// MMCore/MMCore.cpp, stopSequenceAcquisitionImpl(), first statement inside
// `if (sa) {` (line 3462), before `if (sa->TakeDeferredShutterClose())`
// (line 3463). It is only reached when camera->StopSequenceAcquisition()
// (line 3453) returned DEVICE_OK, because the error path throws first.
// The existing `if (sa->IsComplete())` at line 3466 then erases the entry.
if (sa->FinishRemainingParticipants()) {
LOG_WARNING(coreLogger_) << "Camera '" << sa->CameraLabel() <<
"' stopped without every participant calling AcqFinished()";
postNotification(
notif::SequenceAcquisitionStopped{sa->CameraLabel()});
}Not sure how well it applies, just giving it as input for the future, hoping it's not noisy.
| for (const auto& kv : acquisitions_) { | ||
| if (kv.second->HasParticipant(caller)) | ||
| return kv.second; |
There was a problem hiding this comment.
| for (const auto& kv : acquisitions_) { | |
| if (kv.second->HasParticipant(caller)) | |
| return kv.second; | |
| for (const auto& kv : acquisitions_) { | |
| if (kv.second->AllParticipantsFinished()) | |
| continue; | |
| if (kv.second->HasParticipant(caller)) | |
| return kv.second; |
This is a tricky one to explain, I'll simply report the theorized sequence of operations the agent suggested.
startSequenceAcquisition("Camera", 10, 0, true)runs to the end; its entry stays in the map with all participants finished.- The user starts
Multi Camera, withCameraas a physical camera. The conflict check atMMCore.cpp:3289skips finished entries, so the start is accepted. CameracallsPrepareForAcq(). The map is ordered by label and"Camera"sorts before"MultiCamera", so the old entry is returned. ItsBeginPrepare()reportsAlreadyPreparedand the shutter is not opened.InsertImage()adds no channel tags, because the old entry treatsCameraas a plain single-channel camera.AcqFinished()callsRecordFinish()on the old entry, which returns false for a repeat call. TheMulti Cameraentry never completes: noSequenceAcquisitionStopped, the shutter state is wrong, andMulti Cameracannot be started again.
The fix is to do the same skip that findAcquisitionInvolvingCamera() does.
| if (mtx.try_lock()) { | ||
| std::lock_guard<std::recursive_mutex> g(mtx, std::adopt_lock); | ||
| sret = shutter->SetOpen(true); | ||
| } else { | ||
| sa->DeferShutterOpen(); | ||
| deferred = true; | ||
| } |
There was a problem hiding this comment.
When PrepareForAcq() cannot take the module lock, the shutter opening is deferred.
There are three ways to hang>
- a camera calls
PrepareForAcq()from its worker thread;try_lock()fails because the start thread still holds the lock. The starting thread releases the lock and callsTakeDeferredShutterOpen(), which is false, before the worker reachesDeferShutterOpen()and the worker waits forever PrepareForAcq()runs afterstartSequenceAcquisition()has returned, while an unrelated thread holds the module lock (i.e. a property read on the same adapter). It defers and nobody runs the deferred open.- with a composite camera, physical camera 1 defers from its worker thread; physical camera 2 calls
PrepareForAcq()on the start thread, getsWaitForOpenerand blocks while holding the module lock, causing a deadlock because the start thread never reaches the deferred open
Again, there are suggested fixes for all 3 cases but I'm deferring them until they're acknowledged as possibilities
| std::lock_guard<std::mutex> aqg(acquisitionsMutex_); | ||
| if (hasInFlightAcquisitionLocked()) | ||
| throw CMMError( | ||
| getCoreErrorText(MMERR_NotAllowedDuringSequenceAcquisition).c_str(), | ||
| MMERR_NotAllowedDuringSequenceAcquisition); | ||
|
|
There was a problem hiding this comment.
startSequenceAcquisitionImpl() takes the camera module lock and then acquisitionsMutex_.
initializeCircularBuffer() and setCircularBufferMemoryFootprint() take acquisitionsMutex_, and then the camera module lock.
If a GUI thread calls one of them while another thread starts an acquisition each holds the lock the other needs.
The fix would be in both initializeCircularBuffer() and setCircularBufferMemoryFootprint() to take the camera module lock first
| if (nRet != DEVICE_OK) { | ||
| logError(getDeviceName(camera).c_str(), | ||
| getDeviceErrorText(nRet, camera).c_str()); | ||
| throw CMMError(getDeviceErrorText(nRet, camera).c_str(), | ||
| MMERR_DEVICE_GENERIC); |
There was a problem hiding this comment.
When this call returns an error, stopSequenceAcquisitionImpl() throws after MarkStopRequested().
If the camera thread calls AcqFinished() and the try_lock() fails, SpawnOrDeferShutterClose() sees the stop request and defers the close. The code running the deferred close is after the throw and the shutter stays open (and the entry stays in the map) until someone calls stop again.
The fix would be to keep the error code and move this check after the if (sa) block below
| if (!cbuf_->Initialize( | ||
| static_cast<std::size_t>(camera->GetImageWidth()) * | ||
| camera->GetImageHeight() * | ||
| camera->GetImageBytesPerPixel())) | ||
| { | ||
| logError(getDeviceName(camera).c_str(), | ||
| getCoreErrorText(MMERR_CircularBufferFailedToInitialize).c_str()); | ||
| throw CMMError( | ||
| getCoreErrorText(MMERR_CircularBufferFailedToInitialize).c_str(), | ||
| MMERR_CircularBufferFailedToInitialize); | ||
| } | ||
| cbuf_->Clear(); |
There was a problem hiding this comment.
The shared sequence buffer is set up and emptied before the checks against running acquisitions. Consequences:
- A labelled start of an unrelated camera while another acquisition runs is accepted, and it discards that acquisition's queued images and may change the frame size under its inserting thread. The PR description says concurrent acquisitions are not made usable yet, but they are not refused either.
- A start that the conflict checks then refuse has already emptied the buffer. The
camera->IsCapturing()check before it catches the common case (MultiCamera::IsCapturing()asks everyphysical camera), so this mainly matters when a camera has stopped capturing but its entry has notfinished, as in finding 1.
The fix is to refuse a start while any other acquisition is in progress and set up the buffer only after all the other checks have passed. This is already applied to initializeCircularBuffer().
| } | ||
|
|
||
| sa = newSa; | ||
| acquisitions_[camera->GetLabel()] = std::move(newSa); |
There was a problem hiding this comment.
Fair warning this is weird to me, I'm still dumping it because it might be clearer for you.
~SequenceAcquisition() joins the deferred shutter-close worker.
Suppose an acquisition ended on its ownwhile another thread held the module lock. AcqFinished() then spawned a worker that waits on DeviceModuleLockGuard(shutter), which only happens when the shutter is in the same adapter module asthe camera. If the user starts again before the worker gets the lock, the start thread holds that lock, theassignment acquisitions_[label] = std::move(newSa) destroys the old entry, and its destructor joins aworker that is waiting for that lock.
The suggested fix is to move the eold entry into a local declared before the module lock guard, so it's destroyed after both locks are released.
| acquisitions_[camera->GetLabel()] = std::move(newSa); | |
| // at the beginning of startSequenceAcquisitionImpl(...): | |
| // std::shared_ptr<mmcore::internal::SequenceAcquisition> previous; | |
| auto& slot = acquisitions_[camera->GetLabel()]; | |
| previous = std::move(slot); | |
| slot = std::move(newSa); |
| const std::string prefix = sa->CameraLabel() + '-'; | ||
| const std::string prefixedIndexKey = | ||
| prefix + MM::g_Keyword_CameraChannelIndex; | ||
| const std::string prefixedNameKey = | ||
| prefix + MM::g_Keyword_CameraChannelName; | ||
|
|
||
| if (pi.kind == Kind::CompositeChannel) { | ||
| md.AddTag(prefixedIndexKey, std::to_string(pi.index)); |
There was a problem hiding this comment.
According to the agent this is called frequently, several thousand times for fast cameras (not sure where its getting this assumption, pretty curious but it makes sense in the end). Basically these strings are reconstructed everytime. It suggests building the keys and the indext text once, at acquisition creation.
| * Required: 0 <= n < GetNumberOfChannels(); behavior outside that range | ||
| * is undefined except that the default CCameraBase implementation | ||
| * returns nullptr for any in-range n and logs+returns nullptr for | ||
| * out-of-range n. |
There was a problem hiding this comment.
Apparently it complains that this docstring is incorrect since GetNumberOfChannels() doesn-t actually call any log. Just a nitpick it seems to me, and most likely given that it's still in draft not worth the attention. I'm just flagging for posterity in case it gets forgotten since technically a wrong documentation is effectively a bug...
|
Thanks, @jacopoabramo! I should be able to put some work into this next week (traveling right now). The first step will indeed be to rebase onto the current |
SequenceAcquisition is created for each new sequence acquisition and is cleaned up (for now) lazily. Better cleanup will be possible once we start tracking cameras participating in composite cameras (like Multi Camera). This is a start toward more directly managing sequence acquisitions by the Core, so that multiple cameras/acquisitions can be better supported and thread safety can be improved. For now, it does little more than just create the SequenceAcquisition object. (Assisted by Claude Code; any errors are mine.)
Simplify the way in which frame tags are generated for multi-channel cameras. Especially for composite cameras (e.g. Multi Camera), avoid mutating the physical camera objects (which could not be cleanly undone at the right timing) and instead map physical cameras to tags in the Core. Known behavioral changes: 1. Physical cameras associated with Multi Camera (or similar composite cameras) no longer incorrectly generate camera channel name/index tags when running sequence acquisition as an individual camera. 2. When autoshutter is enabled, composite cameras no longer cause the shutter to be opened/closed multiple times (even though it was idempotent). None of the physical cameras start acquiring until the shutter finishes opening, and the shutter is not closed until all physical cameras finish/stop acquiring. 3. If opening the shutter fails, `PrepareForAcq()` now returns an error. 4. `PrepareForAcq/InsertImage/AcqFinished` callbacks will fail if not issued from a camera currently participating in a sequence acquisition (previously any device could theoretically call these at any time and send images to the buffer). 5. The `SequenceAcquisitionStarted/Stopped` notifications now use the primary camera label and are issued once, instead of for each physical camera, in the case of a composite multi-channel camera. 6. Intrinsic multi-channel cameras must tag images with a valid CameraChannelIndex that is in range; this is now verified. CameraChannelName is now added by MMCore so does not need to be added by the camera. 7. Nested multi-channel cameras are explicitly detected and disallowed. Bumps the device interface version due to the following changes: - In `MM::Camera`, add `GetChannelCameraPtr()` so that MMCore can query the primary camera for all participating (physical) cameras. - Remove the `AddTag/RemoveTag` mechanism; instead, let the `SequenceAcquisition` object manage all participating cameras in a composite multi-channel camera acquisition, and generate tags (`CameraChannelIndex`, `CameraChannelName`) during `InsertImage()` in MMCore, where possible (`CameraChannelIndex` is still required from intrinsic multi-channel cameras, because they come from the same source). Note: the CameraChannelIndex and CameraChannelName tags are still prefixed with `primariCameraLabel-` for now, to preserve existing behavior. Composite multi-channel cameras MultiCamera, ArduinoCounter, and CameraPulser are updated to remove the physical camera mutations (`AddTag`/`RemoveTag`) and implement `GetChannelCameraPtr()`. The only intrinsic multi-channel camera, TwoPhoton, is updated to remove the tags that are no longer necessary. (Assisted by Claude Code; any errors are mine.)
Finally. (Assisted by Claude Code; any errors are mine.)
Factor out the common code and remove minor differences (which were presumably not intentional). Differences resolved: - Camera adapter lock is acquired _before_ initializing the sequence buffer (because the latter accesses camera parameters). - Don't catch std::bad_alloc (CircularBuffer::Initialize() already does for the main buffer memory). (Assisted by Claude Code; any errors are mine.)
(Assisted by Claude Code; any errors are mine.)
Remove the unsafe hacks; when shutter is in the same adapter as the _primary_ camera (whose adapter we _may_ hold the lock for, in the same or different thread), fall back to closing the shutter afterwards on the thread calling stopSequenceAcquisition(). But always use try_lock() so that we start closing the shutter asap when we are able to -- so that the specimen is not exposed to illumination while the camera tears down the sequence acquisition (which for some cameras might be slow). Real (physical) cameras don't typically have a shutter in the same adapter. Demo/simulated cameras often do, but then we don't care if the shutter closure is ordered after the return from StopSequenceAcquisition(). Multi Camera (and similar) does not call AcqFinished() (only its physical cameras do), so even if the shutter is from the same adapter as the composite camera, we don't need to defer shutter closure. In no case do we call the shutter without holding the device module lock any more. (Assisted by Claude Code; any errors are mine.)
Another long-standing bug we can now easily fix. (Assisted by Claude Code; any errors are mine.)
That is, disallow starting a sequence acquisition where any of the participants (primary + physical) overlap with an ongoing acquisition. (Assisted by Claude Code; any errors are mine.)
It will fail anyway on InsertImage(), so fail fast. (Assisted by Claude Code; any errors are mine.)
This would always cause problems or errors; now catch upfront. For now, no acquisition can be running because we use a single, global buffer. (Assisted by Claude Code; any errors are mine.)
If the shutter is in the same adapter as the camera _and_ the camera calls PrepareForAcq() from a thread other than the one calling into StartSequenceAcquisition(), defer the shutter opening so that we don't deadlock. This is probably not an issue today, but would become one if, say, a MultiCamera-like device decides to start physical cameras in parallel (which has its own set of problems if they come from the same adapter, so perhaps is unlikely to happen). Even with this, there will be a deadlock if the camera's StartSequenceAcquisition() waits for PrepareForAcq() to return; that is forbidden and is now documented (there's no reasonable way to work around that situation, and fortunately we don't face it in any known existing camera). (Assisted by Claude Code; any errors are mine.)
Avoid waiting forever when shutter opening/waiting failed. Also, avoid leaking exceptions back to caller of PrepareForAcq(). (Assisted by Claude Code; any errors are mine.)
When the autoshutter is in the same adapter module as the camera and AcqFinished's try_lock on the module lock fails, the old code unconditionally set shutterCloseDeferred_ and relied on stopSequenceAcquisition to drain it. That broke natural completion — finite-frame acquisitions that complete without anyone calling stopSequenceAcquisition would leave the autoshutter open forever, because the deferred-close was only drained from the stop path. The new design splits that path atomically (under SequenceAcquisition::mu_): - If stopRequested_ is set → defer (stopper will drain via TakeDeferredShutterClose). - Otherwise → spawn a worker thread that takes the module lock blockingly and closes the shutter itself, then posts ShutterOpenChanged. The thread is owned by the SA (shutterCloseWorker_) and joined either by stopSequenceAcquisition or by ~SequenceAcquisition. To make the atomicity work, CMMCore::stopSequenceAcquisition now calls MarkStopRequested() before taking the module lock. That ordering is important: it guarantees that any AcqFinished racing with the stopper observes stopRequested_=true and takes the defer branch instead of spawning a worker the stopper would have to chase. (Assisted by Claude Code; any errors are mine.)
If the camera is a participant in a composite camera acquisition, it remains "running" until that acquisition as a whole finishes. We keep the camera IsCapturing() check, too, because some cameras may briefly return IsCapturing() == true after calling AcqFinished(). (Assisted by Claude Code; any errors are mine.)
DRY. (Assisted by Claude Code; any errors are mine.)
Reject attempts to stop a non-primary participant in a composite camera acquisition. (Previous behavior was to roguely stop just that physical camera, which doesn't really make sense.) (Assisted by Claude Code; any errors are mine.)
62293a9 to
b9f6c78
Compare
|
Force-pushed after rebasing onto @jacopoabramo all your comments still apply and I'll get to them soon! |
|
@marktsuchida thanks, I'll be happy to re-review afterwards |
This is a DIV-bumping change; it might become the main change for DIV 76 (#886).
SequenceAcquisitionobject to explicitly represent and track the lifetime of each sequence acquisition, replacing ad-hoc state scattered acrossCMMCoreDevice interface changes
Bumps the device interface version:
MM::Camera: addGetChannelCameraPtr()for MMCore to query participating physical camerasAddTag/RemoveTagmechanism (tags are now generated by MMCore duringInsertImage())Updated device adapters: MultiCamera, ArduinoCounter, CameraPulser, TwoPhoton. Regular (physical, single-channel) cameras don't need to change.
Notable behavioral changes
PrepareForAcq/InsertImage/AcqFinishedcallbacks now reject callers not participating in the active acquisitionSequenceAcquisitionStarted/Stoppednotifications fire once per composite camera, not per physical cameraCameraChannelIndex;CameraChannelNameis now added by MMCoreThings not covered here