scaleSetLabels in gha-runner-scale-set is not reflecting in github runner

[shan100github]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.14.2

GitHub Enterprise Server

3.17.15

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Upgraded gha-runner-scale-set-controller to 0.14.2 through helm chart
2. Deployed gha-runner-scale-set with scaleSetLabels: [ubuntu24] and scaleSetLabels: ubuntu-scale-set for gha-runner-scale-set deployment

3. After deployment ~~runner~~ runner-scale-set showed up in the github with `ubuntu-scale-set` as label but `ubuntu24` was ignored.

Describe the bug

I am trying to migrate from existing RunnerDeployment using actions.summerwind.dev/v1alpha1 to gha-runner-scale-set.
In RunnerDeployment I have specified labels

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: gh-action-ubuntu
  namespace: gh-action-runner
spec:
  replicas: 1
     ....
      labels:
      - ubuntu24

I have specified scaleSetLabels: [ubuntu24] along with runnerScaleSetName: ubuntu-scale-set as override values for gha-runner-scale-set,
Corresponding runners runner-scale-sets are created in github and only runnerScaleSetName set as labels and labels passed to scaleSetLabels are ignored.

corresponding EphemeralRunnerSet created is

apiVersion: actions.github.com/v1alpha1
kind: EphemeralRunnerSet
metadata:
  annotations:
    actions.github.com/runner-group-name: Default
    actions.github.com/runner-scale-set-name: gh-action-ubuntu
  labels:
    actions.github.com/scale-set-name: gh-action-ubuntu
    actions.github.com/scale-set-namespace: ghe-action-runner
    app.kubernetes.io/component: runner-set
    app.kubernetes.io/instance: gh-action-ubuntu
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: gh-action-ubuntu
    app.kubernetes.io/part-of: gha-runner-scale-set
    app.kubernetes.io/version: 0.14.2
    helm.sh/chart: gha-rs-0.14.2

Describe the expected behavior

Expecting runner runner-scale-sets created in github through gha-runner-scale-set helm chart with scaleSetLabels parameters are inserted as labels,

Additional Context

Current environment deployed with helm chart
- actions-runner-controller 0.23.7 (trying to migrate from this state)
- gha-runner-scale-set-controller-0.14.2
- gha-runner-scale-set-0.14.2

Values.yaml

runnerScaleSetName: gh-action-ubuntu
scaleSetLabels:
  - ubuntu24


Relevant to 
- https://github.com/actions/actions-runner-controller/pull/4408 
- https://github.com/actions/actions-runner-controller/issues/3327

Controller Logs

-

Runner Pod Logs

-

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[nikola-jokic]

Hey @shan100github,

The runners don't have these labels applied to them. These labels belong to the scale set. The runner also belongs to the scale set. The job assignment is associated with the scale set, that is how the job lands on the runner belonging to the scale set.

[shan100github]

hey @nikola-jokic

sorry for causing trouble confusing terminology runner-scale-sets vs runner.

I see the runner-scale-set deployed in https://ghe.org.com/organizations/mg-te/settings/actions/runner-scale-sets/3, and based on above deployment only if the workflow with runs-on: [gh-action-ubuntu] which is same as runnerScaleSetName, workflow is getting executed and pod created through EphemeralRunnerSet.
But if i set the workflow with runs-on: [ubuntu24] which is same as scaleSetLabels, workflow is just waiting and on github runner-scale-sets Total assigned jobs is showing 0.

since i see only parameter gh-action-ubuntu for runnerScaleSetName is set as label in https://ghe.org.com/organizations/mg-te/settings/actions/runner-scale-sets/3.

[jan-mrm]

sorry to chime in - I'm having trouble with propagating changes and passed by this issue here.
have you created the 'gha-runner-scale-set' with the 'scaleSetLabels' already set or have you added the 'scaleSetLabels' later as an update @shan100github ?
For me 'scaleSetLabels' do work and are set at in GH at the runner but I have an issue with updates to the 'gha-runner-scale-set'.
When I add a label later, then I need to delete the 'AutoscalingRunnerSet' resource so that it is recreated and gets the new config set. Otherwise the runner is never updated with the newly added label. I wanted to open an issue about that (I think I had the same issue with another config already).

[shan100github]

While deploying 'gha-runner-scale-set' for the first time I have added 'scaleSetLabels'.
Also tried uninstall/install/upgrade multiple times through argocd, still no luck.

[jan-mrm]

okay.
your wording makes me think... "with scaleSetLabels: [ubuntu24] and scaleSetLabels: ubuntu-scale-set".
You deploy with helm? and do you set the key "scaleSetLabels" once with 2 elements in the array or twice with one each?
I would expect your config to look like this?:

scaleSetLabels:
- ubuntu24
- ubuntu-scale-set

If you set the same key twice I think the second one wins and is set.

[shan100github]

hi @jan-mrm
As mentioned in Additional Context following parameters are passed for helm/argo (tried both) 'gha-runner-scale-set'

Values.yaml

runnerScaleSetName: gh-action-ubuntu
scaleSetLabels:
  - ubuntu24

runner scale set gh-action-ubuntu created with label gh-action-ubuntu

[jan-mrm]

@shan100github sorry, my bad. I totally missed that and only saw the "To Reproduce" wording.

hm, last idea / check (since it works fine for me): does your AutoscalingRunnerSet resource has the label 'ubuntu24' set in spec.runnerScaleSetLabels? just checking if its templated properly.

[shan100github]

yes, its available in AutoscalingRunnerSet, but i don't see those in EphemeralRunnerSet

  runnerScaleSetLabels:
  - ubuntu24
  runnerScaleSetName: gh-action-ubuntu

[jan-mrm]

its the same for me, so that should be fine, I think - EphemeralRunnerSet does not have it.
I'd assume the controller creates the runner when the AutoscalingRunnerSet is created and (should) adds the labels.

[shan100github]

Does it have dependency with "GitHub Enterprise Server 3.17.15" ?