enh: account for timeouts during job status checks

[mgxd]

Summary

Fixes #2766 .

List of changes proposed in this PR (pull-request)

• Avoid raising exception when checking job status if select timeout messages appear.

Acknowledgment

• (Mandatory) I acknowledge that this contribution will be available under the Apache 2 license.

cc @agt24 - could you test out this branch and see if you still run into problems?

[effigies]

I think it'd be a good idea to add a log entry for this case.

[codecov-io]

Codecov Report

Merging #2767 into master will decrease coverage by 4.7%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master   #2767      +/-   ##
=========================================
- Coverage    67.5%   62.8%   -4.71%     
=========================================
  Files         340     337       -3     
  Lines       43238   43105     -133     
  Branches     5362    5343      -19     
=========================================
- Hits        29187   27071    -2116     
- Misses      13351   14985    +1634     
- Partials      700    1049     +349

Flag	Coverage Δ
#smoketests	?
#unittests	62.8% <ø> (-2.09%)	⬇️

Impacted Files	Coverage Δ
nipype/pipeline/plugins/semaphore_singleton.py	0% <0%> (-100%)	⬇️
nipype/pipeline/plugins/linear.py	22.85% <0%> (-65.72%)	⬇️
nipype/pipeline/plugins/pbsgraph.py	29.41% <0%> (-64.71%)	⬇️
nipype/pipeline/plugins/multiproc.py	20.25% <0%> (-64.56%)	⬇️
nipype/pipeline/plugins/tools.py	21.91% <0%> (-63.02%)	⬇️
nipype/interfaces/nilearn.py	40% <0%> (-56.67%)	⬇️
nipype/utils/spm_docs.py	25.92% <0%> (-44.45%)	⬇️
nipype/pipeline/plugins/debug.py	43.75% <0%> (-43.75%)	⬇️
nipype/pipeline/plugins/legacymultiproc.py	19.41% <0%> (-42.36%)	⬇️
nipype/pipeline/plugins/oar.py	20.45% <0%> (-34.1%)	⬇️
... and 52 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 50862c1...912dde8. Read the comment docs.

[effigies]

Merge master to fix tests?

[agt24]

Thanks for addressing this so quickly. @dalejn is going to test and report back.

[effigies]

@dalejn Any updates?

[dalejn]

Yes, thank you for getting to this so promptly!
It seems to be working as intended--rather than crashing, I now receive the logger warning:
181107-12:32:07,621 nipype.workflow WARNING:
SLURM timeout encountered while checking job status, treating job 12849880 as pending

[mgxd]

great - let us know if anything come up