Auto-cancel jobs stuck as RUNNABLE

[victorlin]

This PR closes #546 by implementing solutions 2 and 3 described in the issue.

Scenario 1: User stays attached

Job gets auto-canceled when AWS Batch updates the status reason as "misconfiguration". Example output:

$ nextstrain build --aws-batch --aws-batch-queue nextstrain-job-queue-test --cpus 100 .
…
Job now RUNNABLE
Job is stuck in RUNNABLE. Reason: MISCONFIGURATION:JOB_RESOURCE_REQUIREMENT - The job resource requirement (vCPU/memory/GPU) is higher than that can be met by the CE(s) attached to the job queue.
Automatically canceling stuck job…
Job FAILED after 3.3 minutes (Canceled by Nextstrain CLI due to reason: MISCONFIGURATION:JOB_RESOURCE_REQUIREMENT - The job resource requirement (vCPU/memory/GPU) is higher than that can be met by the CE(s) attached to the job queue.)

Requested resources of 100 CPUs and/or 7400 MiB memory are likely beyond the limits of the largest instance size allowed by the job queue.

Try adjusting your requested resources or the job queue in AWS.

Scenario 2: User detaches after submitting

Job queue has jobStateTimeLimitActions configured so it auto-cancels after 10 minutes. Upon reattaching, we see the helpful message. Example:

$ nextstrain build --aws-batch --attach d0e071b3-c262-4f32-a7b8-a82d8c2eac99 --no-download .
Attaching to Nextstrain AWS Batch Job ID: d0e071b3-c262-4f32-a7b8-a82d8c2eac99
Job is FAILED
Job FAILED after 18.6 minutes (Canceled by JobStateTimeLimit action due to reason: MISCONFIGURATION:JOB_RESOURCE_REQUIREMENT)

Requested resources of 100 CPUs and/or 7400 MiB memory are likely beyond the limits of the largest instance size allowed by the job queue.

Try adjusting your requested resources or the job queue in AWS.

Checklist

• Checks pass
• Update changelog
• Post-merge: Add job state limits to all job queues infra#92

[victorlin]

I'll plan to merge and release this tomorrow.