remove LFS files from testutil

[alexeiser]

As number of clones has exceed 2k each time a new change to this library is made - remove the LFS files and related tests.

[swami-m]

Looks like some checks where not successful, I'm not sure what that means.

[jcharum]

I fixed some things, so if you merge or rebase master, the CI should succeed.

However, these files are used in apparently real, useful tests. This package also seems to be used by github.com/grailbio/base/crypto/encryption which is in turn used by various internal packages. I would think that we'd want to keep these tests in some reasonably easy-to-run form. Perhaps we can put the tests behind a flag and have the test fetch the files on demand from S3 instead?

[alexeiser]

However, these files are used in apparently real, useful tests. This package also seems to be used by github.com/grailbio/base/crypto/encryption which is in turn used by various internal packages. I would think that we'd want to keep these tests in some reasonably easy-to-run form. Perhaps we can put the tests behind a flag and have the test fetch the files on demand from S3 instead?

The issue is that the LFS file is downloaded independent of the tests. We can put the tests under a flag - and ask people to manually download the lfs file?

[jcharum]

However, these files are used in apparently real, useful tests. This package also seems to be used by github.com/grailbio/base/crypto/encryption which is in turn used by various internal packages. I would think that we'd want to keep these tests in some reasonably easy-to-run form. Perhaps we can put the tests behind a flag and have the test fetch the files on demand from S3 instead?

The issue is that the LFS file is downloaded independent of the tests. We can put the tests under a flag - and ask people to manually download the lfs file?

I'm suggesting that we avoid LFS by hosting the large files on S3, putting the test behind a flag, and having the test download from the S3 HTTP URL. Then, if people want to run the large-file-using test, they'd just need to run tests with the flag -- no explicit manual downloading step needed.

(It's not great to have the hosted file dependency, but it seems better than removing crypto-related tests. Maybe @siddharthab has more insight into the usage of this code and value of the tests).

[siddharthab]

I agree with Jerry that we can just put them in a public S3 bucket and put these tests behind a flag. The flag can be enabled in Github Actions CI, but defaults to false.

As for how important these tests are, these are testing the robustness of the encryption library that we use for encrypting the data we store on-site.