Heap Memory error while generating EvaluatedModel Report for large projects

[losmateusz1]

Description:
When attempting to generate an EvaluatedModel report for larger analyzer result files (~55MB), ORT encounters a heap memory error, causing the process to fail.
Details:
The error appears to occur in the valueToTree method, located in the EvaluatedModel reporter module:
EvaluatedModel.kt#L174.
This issue arises while reading the object into a tree using Jackson's ObjectMapper at the following line:
ObjectMapper.java#L3151.
The current implementation struggles to handle large data sizes, leading to excessive memory consumption.

Questions and Suggestions:

1. Are there potential optimizations in this part of the code to address the memory limitations?
2. Would it be feasible to introduce a mechanism to process large analyzer results incrementally or in smaller chunks?
3. Could alternative serialization methods or configurations within Jackson help mitigate this issue?

[mnonnenmacher]

You could try to enable the deduplicateDependencyTree option of the EvaluatedModelReporter, it significantly reduces the memory consumption. You can enable it by passing -O EvaluatedModel=deduplicateDependencyTree=true to the report command.

[sschuberth]

FYI @Etsija, this might also be an option for us.

[Etsija]

It was indeed the answer - now the Reporter runs happily on my Analyzer+Scanner+Evaluator result file on ORT core.

[losmateusz1]

Thank you very much for the answer, we tested with this option and it works now.

[heliocastro]

One information on top, we used to push this in one single run, - f WebApp,EvaluateModel but then we couldn't pass the options at same time for both.
We explicitly separate that now in two runs.

[sschuberth]

but then we couldn't pass the options at same time for both.

Just curious: What syntax did you try @heliocastro? As @mnonnenmacher explained, it is possible.

[heliocastro]

but then we couldn't pass the options at same time for both.

Just curious: What syntax did you try @heliocastro? As @mnonnenmacher explained, it is possible.

This failed:
-f WebApp,EvaluatedModel -O WebApp=deduplicateDependencyTree=true -O EvaluatedModel=deduplicateDependencyTree=true

[sschuberth]

@heliocastro can you be more specific than "this failed"? I just tried, and it works with for me, as expected.

[heliocastro]

Ok, i tried to clean up the evaluate-result here, but is too big to clean up all company references.
To try explain better and see results you do need a huge result that will explode with memory issues. A regular result will hide the side effect.

With the line above, only one of the options are applied and the other operation crashes, reverse EvaluatedModel,WebApp will do the same for the other operation.

I need to find a huge project without company references to maybe add as a test.

[sschuberth]

With the line above, only one of the options are applied

For the record, that does not match what I saw in the debugger. I could verify that both options are passed correctly if both are specified at the same time.

[Etsija]

@mnonnenmacher is deduplicateDependencyTree option reporter specific? WebApp runs happily now with it, but CycloneDX doesn't, so is the option missing from it? What about SPDXDocument?

Trying with this, using 16GB for my Java:

ort report -i "$EVALUATE_RESULT" -o "$OUTPUT_DIR" -f CycloneDX -O CycloneDX=deduplicateDependencyTree=true

[Etsija]

Answering to myself: from looking at CycloneDXReporter.kt and SPDXDocumentReporter.kt, they don't support the said option yet.

[mnonnenmacher]

Correct, the option only exists for the EvaluatedModel and WebApp reporters (note that the WebApp reporter internally uses the EvaluatedModel reporter). The reason is that when the option was implemented only those two reporters suffered from the memory issues, the other reporters worked fine.

[Etsija]

OK would a ticket be OK to create for those two reporters as well?

[mnonnenmacher]

Sure, if it is required for more reports maybe it should become a global option, but let's discuss that in the ticket you create.

[Etsija]

I created #9889.