Skip to content

code will get stuck when using ddp #1921

Closed
Closed
code will get stuck when using ddp#1921

Description

@cunangjiang
cunangjiang
opened on Jan 16, 2025

When using ddp for diff_model_train.py, the code will get stuck at a certain epoch. When I reduce the dataset, the model runs in more epochs. How can i solve this problem? For example, as shown in the figure, the code will remain stuck here without any errors.

Activity

  1. cunangjiang commented on Jan 16, 2025

    @cunangjiang
    cunangjiang
    on Jan 16, 2025
    Author

    屏幕截图 2025-01-16 195037

  2. cunangjiang commented on Jan 16, 2025

    @cunangjiang
    cunangjiang
    on Jan 16, 2025
    Author

    The GPU memory has remaining space, but the GPU utilization is 0.

    Image

  3. KumoLiu commented on Jan 17, 2025

    @KumoLiu
    KumoLiu
    on Jan 17, 2025
    Contributor

    Hi @cunangjiang, did you check the RAM?

  4. Project-MONAI
    locked and limited conversation to collaborators on Jan 17, 2025
  5. KumoLiu
    converted this issue into a discussion #1922 on Jan 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions