[Bug]: CrashLoopBackOff on fresh PVC — 20-use-templates-initially.sh: No such file or directory

[dwjvaughan]

Is this a critical security issue?

• This is not a security issue.

Describe the Bug

On a fresh install (new PVC with no prior data), the puppetserver pod enters
CrashLoopBackOff immediately. The container entrypoint script
/container-entrypoint.d/20-use-templates-initially.sh fails at line 10 with:

/container-entrypoint.d/20-use-templates-initially.sh: line 10: cd: /opt/puppetlabs/server/data/puppetserver/.puppetlabs/etc/puppet: No such file or directory

The script attempts to cd into that path, but the perms-and-dirs init container
does not create it on a fresh PVC, so the directory does not exist when the main
container starts.

Expected Behavior

The perms-and-dirs init container (or equivalent) should create
/opt/puppetlabs/server/data/puppetserver/.puppetlabs/etc/puppet so the
20-use-templates-initially.sh script can succeed on first boot.

Steps to Reproduce

1. Fresh cluster with no existing PVCs for this release
2. helm install (or helm upgrade --install) with default values and
   postgresql.enabled: false (external DB)
3. Watch the puppetserver pod — it enters CrashLoopBackOff within ~30 seconds
4. kubectl logs <pod> --previous shows the error above

Environment

Openvox Helm Chart: 10.1.1
Kubernetes: EKS 1.32
Storage: AWS EBS CSI (gp3), fresh PVC

Additional Context

This is reproducible with a completely stock helm install — no custom values
beyond disabling postgresql. The 20-use-templates-initially.sh script in the
image assumes the directory exists but neither the image nor the chart init
container creates it on a fresh mount.

Relevant log output

[negast]

I'm also trying to get this chart working I fixed this by setting this in values.yaml

puppetserver:
    extraInitArgs: |
      mkdir -p /opt/puppetlabs/server/data/puppetserver/.puppetlabs/etc/puppet;
      chown -R puppet:puppet /opt/puppetlabs/server/data/puppetserver/.puppetlabs;

But then I bumped into another problem
By default it seems the init container runs as root and the main container as puppet which is all good because we need to set permissions and such but then
i get this error
/container-entrypoint.d/55-set-masterport.sh: line 6: cd: /etc/puppetlabs/puppetserver/conf.d/: Permission denied

the entrypoint script will always try this if masterport variable is set so you can't turn of confd because you will get the same error as the dir is not created. Ok but by default it is turned on and such and it creates the directory like this

{{- if eq .Values.puppetserver.persistence.confd.enabled true }}
              cp -rp /etc/puppetlabs/puppetserver/conf.d/* /conf.d/;
              {{- end}}

but it copies it as root then so you still get the same permissions errors.

You can't fix this on with extra init args because extra args is somewhere in the middle of the code block.
It would be better if the extra init args is at the end so we can fix stuff if needed. I'm willing to create a branch for this

[anthonysomerset]

this is possibly related to the change in UID/GID in the 8.12 > 8.13 updates also and possibly the unification of the ubuntu image to match the alpine image

this doesn't just impact new deployments but upgrades

[MirandaStreeter]

Yes, the impact is somewhat widespread. I've opened a conversation at OpenVoxProject/container-openvoxserver#181, any feedback would be appreciated. The sooner we resolve this, the sooner we can update the image version value.

[anthonysomerset]

so if i attempt to override the runAsUser and Group to 64604 - i get slightly further but it bails here:

Defaulted container "puppetserver" out of: puppetserver, perms-and-dirs (init)
Entrypoint PID 7
DEPRECATED: Use /container-custom-entrypoint.d/ instead of /docker-custom-entrypoint.d/
Running /container-entrypoint.d/20-use-templates-initially.sh
Running /container-entrypoint.d/30-ensure-config.sh
Error: Permission denied @ apply2files - /etc/puppetlabs/puppet/puppet.conf
Error: Try 'puppet help config set' for usage

[anders-larsson]

Hi,

I was having the same issue. This was the changes I had to make to make it work if I remember correctly.

global:
  runAsNonRoot: false
  securityContext:
    #runAsUser: 999 # "puppet" UID
    #runAsGroup: 999 # "puppet" GID
    #fsGroup: 999 # "puppet" GID
    runAsUser: 64604 # "puppet" UID
    runAsGroup: 64604  # "puppet" GID
    fsGroup: 64604 # "puppet" GID
...
  persistence:
...
    data:
      enabled: false
      existingClaim: openvoxserver-puppetserver-serverdata-pvc

Not all directories (mounted PVs) get their owner/group updated. Fix manually by changing uid/gid for all files to 64604 (except puppetserver.persistence.code PV). After this the puppetserver container will work but the r10k will fail due to permission errors. This is because it is still running as uid 999 and not 64604 (see voxpupuli/container-r10k#61). Files in puppetserver.persistence.code PV needs to be owned by 999:999 (environments, r10k_cache).

Next error will be that it will be unable to find puppet gem (and dependencies). This appears to be because puppetserver.persistence.data PV is mounted over the directory where gems are stored. I simply removed it from being used since nothing really valuable seemed to be stored there anyway.

I hope I didn't miss something.

EDIT: Missed the linked thread. I believe that was spawned from the discussions we had on IRC.

[anthonysomerset]

I basically went down the same route as you and nearly there - the issue with r10k is that the image is baked with a standard assumption of a fixed UID/GID and the home directory uses that UID during image build, however when you run r10k from the helm chart it also starts running as 64604 so has no permissions to the home directory etc

even then you get a similar error when trying to run r10k manually inside teh container if you manage to get it running:

ERROR    -> Unable to determine current branches for Git source 'puppet_repo' (/etc/puppetlabs/code/environments)
Original exception:
Command exited with non-zero exit code:
Command: git --git-dir /etc/puppetlabs/code/r10k_cache/git@github.com-org-manifests.git fetch origin --prune
Stderr:
No user exists for uid 64604
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Exit code: 128
command terminated with exit code 1

the fix there should be trivial and bring all the containers into alignment but is ultimately on the r10k container side

[anthonysomerset]

you can temporarily work around this also by targeting the 8.12.x releases of openvoxdb and openvoxserver container versions and you don't need the run asuser changes

we have one outstanding issue with the R10k container that prevents updating at this time, waiting for an appropriate fix to land there to push an update here

[anders-larsson]

Hm, interesting.

I am running r10k version 5.0.3-latest successfully with the setup mentioned above. It is still running as uid/gid 999 and ensuring that files in /etc/puppetlabs/code is owned by 999:999 it successfully pulls and updated sources in r10k. Doesn't appear global.securityContext.runAs* gets applied to it.

Downgrading is indeed also working.

[anthonysomerset]

Hm, interesting.

I am running r10k version 5.0.3-latest successfully with the setup mentioned above. It is still running as uid/gid 999 and ensuring that files in /etc/puppetlabs/code is owned by 999:999 it successfully pulls and updated sources in r10k. Doesn't appear global.securityContext.runAs* gets applied to it.

Downgrading is indeed also working.

yeah that might work also - however i'm nervous that this might bring other permissions issues down the line due to the mismatch of permissions - I guess if the files are globally readable its not an issue but that feels like a gamble. my code fixes for permissions to support 8.13+ without much user intervention are going to update runas so that as much as possible links to the values defined including setup scripts to make things simpler to maintain in future