fix: remove metric series of deleted profiles and ClusterSummaries

[Kitio-Tek]

Fixes #2029

The series labeled by profile and by ClusterSummary are only ever set or incremented, so they outlive the objects they describe. After a ClusterProfile is deleted, projectsveltos_matching_clusters keeps its last count. A deleted ClusterSummary keeps its reconcile_consecutive_failures and reconcile_last_success_timestamp_seconds, and when its cluster is gone the undeploy is skipped, so a feature that was failing keeps reporting failures that never clear.

This adds clearProfileMetrics and clearClusterSummaryMetrics to metrics.go:

• The ClusterProfile and Profile reconcilers call clearProfileMetrics once reconcileDeleteCommon succeeded, next to cleanMaps. It is the counterpart of the trackMatchingClusters call in reconcileNormal.
• postFinalizerCleanup calls clearClusterSummaryMetrics, so it runs once the finalizer is removed. The undeploy records a last Removed outcome before that, and a ClusterSummary whose deletion is blocked (cluster paused or not ready) keeps its series.

clearClusterSummaryMetrics gets the profile labels from getProfileLabels, like the recording side, and deletes the series of each feature, and of both outcomes in reconcile_outcome_total, by their exact labels. The series of the same profile on other clusters, and of other profiles on the same cluster, stay.

I left reconcile_duration_seconds and reconcile_operations_total alone. They are not labeled by profile and are shared by all the ClusterSummaries of a cluster, so they would have to go with the cluster, and an undeploy still running when the cluster goes could record them again. @gianlucam76 do you want them removed too, in a follow-up? I also didn't touch the shard handover: when a ClusterSummary moves to another shard, the old shard's addon-controller keeps its series. outdated_helm_chart is left to its checker, which has its own gap: #2030.

Tests, next to the existing reconcileDelete specs:

• A ClusterSummary's series, for every feature and both outcomes, are gone from all four metrics once its finalizer is removed, while those of a Profile with the same name on the same cluster, and of the same ClusterProfile on another cluster, stay.
• A ClusterSummary whose finalizer cannot be removed (cluster present but not ready) keeps its series.
• The matching_clusters series of a ClusterProfile, and of a Profile, is gone once the finalizer is removed, while the series of the other kind with the same name, and of a Profile with the same name in another namespace, stay.

The three removal specs fail on main and pass here. make test and make lint pass. On kind, make fv and make new-fv pass, and so does make fv-agentless on a fresh cluster. With the reproduction steps from the issue, /metrics has no series left for the deleted profiles and ClusterSummaries. Only reconcile_duration_seconds and reconcile_operations_total of the deleted SveltosCluster remain, as said above.

Nothing changes while the objects exist: same metrics, labels and values.

[Kitio-Tek]

@gianlucam76 could you review this when you have a moment? It does for ClusterProfiles, Profiles and ClusterSummaries what deleteClusterMetrics from projectsveltos/sveltoscluster-manager#372 does for a deleted SveltosCluster.

[gianlucam76]

Thank you @Kitio-Tek

Will review it tomorrow morning. thank you again for pushing this!

[gianlucam76]

Thank you @Kitio-Tek so much for detecting it, reporting it and fixing it.

I added two small comments. Let me know what you think. Thank you

[gianlucam76]

LGTM. Thank you