Dynamic generation of detector matrix results in CI not failing when it should

[mariakzurek]

While adding a new detector to the repository, I've encountered an issue that causes the CI jobs to be skipped, i.e. the tests will pass even if the code should cause them to fail.

• In the first commit that I pushed I added the new pcal detector

• A failure occurred in the generate-matrix preliminary job, so no matrix was generated. However, the job itself succeeded, and, since the downstream jobs were empty, all workflows succeeded as well
  

• When I pushed again, without any changes, the generate-matrix job worked as expected, and since the commit contained no changes, also in this case no further jobs were created and all the workflows succeeded
  

• Only when I artificially changed something in one of the special files that cause the complete job matrix to be created I was finally able to see the real results of the CI checks, i.e. failure (because of a real bug in the code)
  

• I think that, in most cases, the dynamic job matrix generation works OK; however, I'm really concerned that, especially at this stage of active development across multiple systems (and repositories), the risk of the CI "failing to fail" is worse than the benefits

• Since the number of systems under test is not excessive for now, we could for now consider using workflows with fixed job matrices that always run

• As I recall one of the motivations for the dynamic job matrix was the limitations of available container image pulls from the DockerHub registry, so I'd be happy to help looking into alternatives e.g. using GitHub's own image registry that shouldn't have these limitations

[maureeungaro]

In this case things didn't work because pcal was not present in the whatToTest script. Similar problems in the future can be addressed by adding all clas12 systems to whatToTest (let me do that).