Director workers under BPM cannot read CPI plugin config from sibling job directories

[colins]

Summary

When director workers run under BPM, each worker's mount namespace exposes only the
CPI job directory. A CPI that reads files from any other job directory therefore
fails with No such file or directory, even though the file exists on the director VM.

This bit us with a CPI plugin whose configuration is rendered by a separate colocated
job at /var/vcap/jobs/<plugin>/config/<plugin>.yml. The plugin is loaded by the CPI
and reads that file during a create_vm hook, so every create_vm call failed —
including compilation VMs — and the error surfaced only after the VM had already been
created in the IaaS.

Affected versions

Workers began running under BPM in v283.0.0 (make bosh release compatible with Resolute, #2755 — "workers now use BPM!") and were on by default through v283.1.1.

Release	director.use_bpm_for_workers
v283.0.0	property absent, BPM unconditional
v283.1.0	property absent, BPM unconditional
v283.1.1	added by #2771, default true
v283.1.2	default false (#2795)

Mechanism

jobs/director/templates/bpm.yml grants each worker process only the CPI job's
directories:

cpi_volume_entry = cpi_job.to_s.empty? ? nil : {
  "path" => "/var/vcap/jobs/#{cpi_job}", "allow_executions" => true
}

plus /var/vcap/sys/log/#{cpi_job}, attached via unsafe.unrestricted_volumes.
Sibling job directories are not mounted.

Why the symptom is misleading

Package directories are available inside the namespace, so the plugin's code
loads from /var/vcap/data/packages/<plugin>/... without complaint and only the
config read fails. From the outside this looks identical to an unrendered or
missing job template, which sends you looking at the director manifest, the job
colocation and the property values — all of which are correct. The file is present
on the VM and readable from an SSH session; it is only invisible inside the worker's
namespace.

Diagnosing this took a while precisely because every check outside the namespace
passes.

Workarounds

• v283.1.2 and later: director.use_bpm_for_workers defaults to false.

• v283.1.1: set director.use_bpm_for_workers: false.

• Keeping workers under BPM: grant the paths explicitly —

  properties:
    director:
      cpi_additional_volumes:
      - path: /var/vcap/jobs/<plugin>

• v283.0.0 and v283.1.0: no opt-out property exists, so upgrading is the only route.