Skip to content

Fix checksum verification timing out on large OS images - #411

Merged
jtdub merged 4 commits into
networktocode:developfrom
michalis1:michalis1-fix-410
Aug 4, 2026
Merged

Fix checksum verification timing out on large OS images#411
jtdub merged 4 commits into
networktocode:developfrom
michalis1:michalis1-fix-410

Conversation

@michalis1

Copy link
Copy Markdown
Contributor

Fixes #410

Problem

get_remote_checksum hardcodes read_timeout=300 for the checksum command
(verify /md5 on IOS/ASA, run md5sum on IOS-XR), and nothing in the
verify_file -> compare_file_checksum -> get_remote_checksum chain accepts a
timeout. On Catalyst 9k hardware verify /md5 hashes at under 4 MB/s, so a
~1.2 GB image already exceeds 300s and current 2+ GB images need 500-700+
seconds. The result: file_copy / remote_file_copy transfers that completed
successfully on the device get reported as failures when the post-transfer
verification hits netmiko's absolute read timer. Retried jobs can also fail in
the pre-transfer verification, since the file is already on flash by then.

match[1],
)
return match[1]
return match[1]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extra indent expected?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The return match[1] was sitting outside the if match := re.search(...) block but inside the for loop, so it executed unconditionally on the first iteration. When the first pattern didn't match — which is exactly what happens when the verify output is truncated by a timeout — match was None and this raised TypeError: 'NoneType' object is not subscriptable. The second pattern was never tried, and the intended CommandError below the loop was unreachable.

Moving the return inside the if restores the obvious intent: try each pattern, return on the first match, and fall through to CommandError if none match. No behavior change on the happy path — pattern 1 matching still returns immediately.

return False

def verify_file(self, checksum, filename, hashing_algorithm="md5", file_system=None):
def verify_file(self, checksum, filename, hashing_algorithm="md5", file_system=None, read_timeout=900):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is 900 a good sane default?

@michalis1 michalis1 Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The old 300s default can't verify current images: we measured verify /md5 on Cat9200 hardware hashing at under 4 MB/s, so a ~1.2 GB image already exceeds 300s, and today's 2+ GB images need 500–700s+. 600 would be marginal; 900 covers them with headroom and matches the existing FileCopyModel.timeout default.

@jeffkala jeffkala left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, I'd like @jtdub or @gsnider2195 to review as well since they've been so in depth with this lib lately.

Comment thread pyntc/devices/ios_device.py
Comment thread pyntc/devices/ios_device.py
Comment thread pyntc/devices/base_device.py
Comment thread pyntc/devices/ios_device.py
Comment thread pyntc/devices/ios_device.py
Comment thread pyntc/devices/ios_device.py
Comment thread pyntc/devices/ios_device.py
Comment thread pyntc/devices/ios_device.py
@michalis1
michalis1 requested a review from jtdub August 4, 2026 15:29
@jtdub
jtdub merged commit 1e18b84 into networktocode:develop Aug 4, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Verify /md5 times out on large OS images in file_copy and remote_file_copy

3 participants