This framework is used for building customized third-party packages and public Delphix packages for the Ubuntu-based Delphix Appliance. It also has the functionality to automatically sync third-party packages with the upstream projects.
- System Requirements
- Getting Started
- Project Summary
- Scripts
- Environment Variables
- Package Definition
- Adding New Packages
- Testing your changes
- Package Lists
- Versions and Branches
- Contributing
- Statement of Support
- License
Package builds run inside a Docker container image tagged
linux-pkg-build:$UBUNTU_DISTRIBUTION. That image's root filesystem is
bootstrapped with debootstrap --variant=buildd from the Ubuntu suite named by
UBUNTU_DISTRIBUTION (see Environment Variables), so
a build runs against a rootfs for the suite being targeted rather than against
whatever happens to be installed on the host. Because of that, the host's own
Ubuntu codename no longer needs to match the suite being built; a noble host
can build packages for resolute, or any other suite the package mirror
serves.
The host itself needs:
- a reachable docker daemon (
docker infomust succeed, and your user must be able to talk to it, typically by being in thedockergroup) debootstrapinstalled- passwordless sudo, since bootstrapping and importing the container's root filesystem requires root
Note also that kernel headers and debug symbols for every flavor in
TARGET_KERNEL_FLAVORS now land under the docker root directory rather than
directly in the root filesystem. On a buildserver that is the same ZFS pool, so
there is no new capacity requirement, but a full disk surfaces as a docker layer
write failure rather than as an out-of-space error from the build.
Every host-side invocation of any script, not just the one that happens to
build the container image, checks that the host is a Delphix buildserver image
before doing anything else, and refuses to proceed otherwise. It asks the image
what it is, via get-appliance-platform and get-appliance-variant, and
requires aws and internal-buildserver respectively. The check runs every
time because the host phase mutates the host itself whenever it does run, for
instance by teaching debootstrap about a new suite and creating
/var/lib/linux-pkg, and it exists as a safety precaution to keep developers
from accidentally running it on their personal machines. To bypass it, run
the following command before running any script:
export DISABLE_SYSTEM_CHECK=true
On any host that is not a buildserver image that has to be set for every invocation, not just the first.
Because the build runs in a container, only what is passed explicitly reaches
it. Environment variables cross through an allowlist in lib/container.sh
(CONTAINER_ENV_ALLOWLIST, plus the AWS_*, SECRET_DB_*, *_S3_URL and
per-package <PKG>_GIT_URL / <PKG>_GIT_BRANCH / <PKG>_REVISION patterns).
Files do not cross at all unless they are bind-mounted, and four are: the
checkout itself, ~/.aws when it exists (read-only, so an AWS profile or
credentials file keeps working alongside credentials passed in AWS_*
variables), the key named by SECRET_DB_JUMP_BOX_PRIVATE_KEY, and the host's
docker socket for a package that declares PACKAGE_NEEDS_DOCKER.
If you need to skip the container entirely, for example to debug the container
mechanism itself, set LINUX_PKG_NO_CONTAINER=true and the build will run in
place on the host using whatever is already installed there. This is an escape
hatch rather than the normal path, and the buildserver-image safety check above
still applies to it, on every invocation. So does the check that the host's own
Ubuntu codename matches UBUNTU_DISTRIBUTION: without a container, setup.sh
rewrites this host's /etc/apt/sources.list and installs from it, so the host
has to be running the suite being built.
This quick tutorial shows how to build the packages managed by this framework.
Clone the dlpx-internal-buildserver-develop group on DCoA. The safety check
described above requires a buildserver image, so that is the supported path.
Which Ubuntu release the buildserver itself runs no longer has to match the
suite being built, since builds run against a rootfs bootstrapped for the target
suite rather than against the host's own packages. Any other host needs
DISABLE_SYSTEM_CHECK=true on every invocation.
Clone this repository on the build VM. In order to clone this repository to the build VM, you may need to specify a personal access token for authentication and authorization.
git clone https://<token>@github.com/delphix/linux-pkg.git
We can now build an arbitrary package. Any package in the
packages directory would do. Let's pick cloud-init as an
example:
./buildpkg.sh cloud-init
Build artifacts will be stored in directory
packages/cloud-init/tmp/artifacts/.
There are two main tasks that are performed by this framework: building packages and keeping each package up-to-date with its upstream project by updating the appropriate git branches.
This task is relatively straight forward. What linux-pkg calls a "package" is really a project (usually a git project) that has a build recipe and that produces one or more debian packages and some other metadata files.
See Scripts > updatelist.sh below.
The idea behind this task is to reduce the amount of effort required to maintain third-party packages and keep them up-to-date. Note that this task does not apply to packages created and maintained by Delphix, but only to third-party packages that Delphix modifies. Instead of following a more conventional approach of using tarballs and patches with all its drawbacks, we've decided to leverage the advantages offered by revision control. As such, we've adopted a well defined branching model for each third-party package.
First of all, we have a Delphix repository on github for each third-party
package that we build. Each repository has at least 2 branches: develop and
upstreams/develop. The develop branch of the package is the one we build,
and contains Delphix changes. The upstreams/develop branch is used to track
the upstream version of the package. For packages that are not provided by
Ubuntu but are available on git, the upstreams/develop branch usually just
tracks the develop branch of the project. For packages that are provided by
Ubuntu, the upstreams/develop branch instead tracks the source package that
is maintained by Ubuntu (i.e. the branch contains the files obtained from
apt-get source <source-package>). This offers the advantage of using a version
of the package tuned to work with our Ubuntu distribution.
When updating a package, we first check if the upstreams/develop branch is up-to-date, by fetching the latest version of the upstream git repository or the Ubuntu source package. If changes are detected, we update the upstreams/develop branch and push the changes to GitHub.
The second step is to check if the develop branch is up-to-date with upstreams/develop. If it is already up-to-date, then we are done. If not, then we attempt merging upstreams/develop into develop.
If the merge is successful, then we push the changes to a staging branch on GitHub, called projects/auto-update/develop/merging. The intent is for a different system to fetch those changes, build them, and then launch tests.
See Scripts > sync-with-upstream.sh below.
Once the merge has been tested, Scripts > push-merge.sh is called on the original VM to push the changes to the develop branch on GitHub.
Note that the example above targets the develop branch, but the same workflow could apply to other branches.
A set of scripts were created in this repository to allow easily building and updating packages both manually and through automation (e.g. Jenkins).
This script can be called on most unix-based systems to query metadata on the packages built by linux-pkg. This script does not install anything on the system, so it can be run anywhere without any side effects.
Installs dependencies for the build framework. Needs to be run once to configure the system, before any other scripts (except query-packages.sh).
Builds a single package. Package name must match a directory under packages/.
./buildpkg.sh <package>
The build will look at packages/<package>/config.sh for instructions on where
to fetch the package from and how to build it. The build will be performed in
packages/<package>/tmp/, and build artifacts for this package will be stored
in the artifacts sub-directory.
Note that if the build of the package depends on build artifacts from another linux-pkg package, those will be fetched from a predetermined S3 location.
Pass -S instead of building to open an interactive shell in a build
container, with the same mounts and environment a build of that package would
get:
./buildpkg.sh -S <package>
This is useful for reproducing a build environment or debugging a failure by
hand. Note that this shell does not run setup.sh, so it starts without the
build tooling setup.sh installs; run setup.sh yourself inside the shell
first if you need it.
Usage:
./checkupdates.sh <package>
This checks if a package has updates in the upstream project that haven't been pulled into the upstreams/6.0/stage branch, or if the upstreams/6.0/stage branch has commits that haven't been merged into the 6.0/stage branch.
If updates are available, the file <WORKDIR>/update-available will be created.
The intention of this script is to inform the caller whether an update job should be called for the given package.
Usage:
./sync-with-upstream.sh <package>
This script has 2 tasks:
- Check if the upstream project has updates that are not pulled into the upstreams/develop branch of the package, and if so then update that branch and push changes to GitHub.
- Merge upstreams/develop into develop and push the changes to a staging branch on GitHub, called projects/auto-update/develop/merging. Another system should use that branch to build the package, and then run the appropriate integration tests.
After testing has been completed, push-merge.sh <package> should be called on
the same system to push the merge to the develop branch.
Note that the DRYRUN environment variable must be set when running this script. If DRYRUN is set to "true", then changes are not pushed to GitHub in step 1, and staged changes are pushed to projects/auto-update/develop/merging-dryrun in step 2 instead of the non-dryrun branch. The intention is that when testing changes to the logic we want to be able to run most of the logic, but without affecting the production branches.
Usage:
./push-merge.sh <package>
This must be called on a system that has previously called
sync-with-upstream.sh for the same package. It will push the merge that was
previously prepared by sync-with-upstream.sh to the production develop
branch, after checking that the develop branch hasn't been modified since
sync-with-upstream.sh was called.
Like for sync-with-upstream.sh, the DRYRUN environment variable must be set
to run this script. However, the script will fail unless DRYRUN is set to
"false" given that there is not much that can be tested in dry-run mode.
There's a set of environment variables that can be set to modify the operation of some of the scripts defined above.
-
DISABLE_SYSTEM_CHECK: Set to "true" to disable the safety check that restricts every host-side invocation of any script to Delphix buildserver images (
get-appliance-platformreportingawsandget-appliance-variantreportinginternal-buildserver). That check runs on every invocation, not just the one that happens to build the container image, so on any other host this needs to be set every time, not just once. That includes a plain Ubuntu AWS instance, which is not a buildserver image and so does not satisfy the check on its own. Affects all scripts. This variable is read on the host only; it is not passed into the build container, and so has no effect on the codename check made there, which compares the container's rootfs againstUBUNTU_DISTRIBUTIONand holds by construction. On theLINUX_PKG_NO_CONTAINERpath that same codename check runs on the host instead, where it is the only thing verifying that the host runs the suite whose apt sourcessetup.shis about to write, and there this variable does disable it. Use it with care on that path. -
UBUNTU_DISTRIBUTION: The Ubuntu suite (codename) that packages are built for, such as "noble" or "resolute". Defaults to the value hardcoded for the current branch in
lib/common.sh, but can be overridden in the environment to build for a different suite than the branch's default. Determines the tag of the build container image (linux-pkg-build:$UBUNTU_DISTRIBUTION) and the suitedebootstrapbootstraps that image's rootfs from. -
LINUX_PKG_NO_CONTAINER: Set to "true" to run in place on the host instead of inside the build container. This is an escape hatch for when the container mechanism itself needs to be bypassed or debugged; the buildserver-image safety check above still applies to it. Affects every script that re-execs itself inside the container via
container_reexec()(buildpkg.sh, checkupdates.sh, sync-with-upstream.sh, push-merge.sh), and setup.sh, which checks it directly. -
LINUX_PKG_REBUILD_IMAGE: Set to "true" to force the build container image to be rebuilt from a freshly bootstrapped rootfs, even if an image tagged
linux-pkg-build:$UBUNTU_DISTRIBUTIONalready exists. Setting this is not needed for the three cases where a reused image would be wrong, since those rebuild it on their own: the image records the mirror snapshot its rootfs was bootstrapped from, the uid/gid it was built for, and a hash of theDockerfilethat produced it, and a build whose resolved snapshot, invoking user orDockerfilediffers rebuilds rather than reuses it. Otherwise a long-lived buildserver would keep compiling against adebootstraptoolchain frozen at the image's build time while apt is repointed at the current snapshot on every build, a second user on a shared host would run as a uid the image has no/etc/passwdentry for, and an edit to theDockerfilewould never reach an existing image, since this check returns beforedocker buildis reached and docker's own layer cache never gets a say. -
LINUX_PKG_BUILD_ROOT_DIR: Directory where the build container's rootfs is bootstrapped before being imported into docker. Defaults to
/var/lib/linux-pkg. Must point at a filesystem that is not mountednodevornoexec, sincedebootstraprefuses to install into either. -
DRYRUN: Must be set to either "true" of "false" when running script sync-with-upstream.sh, and to "false" when running script push-merge.sh.
-
PUSH_GIT_USER, PUSH_GIT_PASSWORD: Set to the git credentials used to push updates to package repositories. Affects scripts sync-with-upstream.sh and push-merge.sh.
-
DEFAULT_REVISION: Default revision to use for packages that do not have a revision defined. If not set, it will be auto-generated from the timestamp. Applies to buildpkg.sh.
-
DEFAULT_GIT_BRANCH: The product branch that is being built or updated and must be set via DEFAULT_GIT_BRANCH. It can either be set to a development branch, such as "develop", or a release tag, such as "release/6.0.6.0". The product branch is used in multiple instances. When running setup.sh, it will determine what linux-package-mirror link to use when fetching packages from apt (although those links can be overridden via DELPHIX_PACKAGE_MIRROR_MAIN and DELPHIX_PACKAGE_MIRROR_SECONDARY). When running buildpkg.sh, it will determine which branch to fetch from the package's repository, unless it is overridden via
-b; if the package has build-dependencies on other linux-pkg packages, those dependencies will be fetched from an S3 url that is versioned based on the product branch (although the package dependencies URLs can be overridden via package_S3_URL variables). Finally, when running sync-with-upstream.sh or push-merge.sh it defines what branch of the package is being updated. -
TARGET_KERNEL_FLAVOURS: Some packages have build dependencies on the linux kernel. Those packages have
PACKAGE_DEPENDENCIES="@linux-kernel"in theirconfig.sh. By default, those packages are built for all the supported kernel flavours (see SUPPORTED_KERNEL_FLAVORS incommon.sh), however it is possible to restrict which kernel flavours those packages are built for. -
package_GIT_URL, package_GIT_BRANCH, package_VERSION, package_REVISION: Can be used to override defaults for a given package.
packageis the package name in upper case with-converted to_. For instanceCLOUD_INIT_GIT_BRANCH=feature1would set the branch to fetch packagecloud-initfrom tofeature1. This is useful when runningbuildlist.shto override defaults for multiple packages. Applies to bothbuildlist.shandbuildpkg.sh. -
package_S3_URL: Similar to the package_VAR variables above. This is used to override the default S3 location for where package build-dependencies are fetched for a given linux-pkg package.
-
DELPHIX_PACKAGE_MIRROR_MAIN, DELPHIX_PACKAGE_MIRROR_SECONDARY: When the setup.sh script is run, it will configure the apt sources to point to versioned delphix mirrors of the Ubuntu archive (MAIN mirror) and of some auxiliary archives (SECONDARY mirror). Delphix has many snapshots of those mirrors at different points in time, and if you want to use a custom snapshot, you can pass it in those environment variables.
DELPHIX_PACKAGE_MIRROR_SECONDARY also accepts the literal value
none, which leaves the secondary mirror out of the apt configuration entirely. The secondary mirror carries the PPAs and is Delphix's own, so it has no upstream equivalent, and naming one is impossible when building against the public Ubuntu archive rather than a Delphix mirror -- which is the situation at the start of an Ubuntu LTS upgrade, before the new suite's mirror sync has published anything. Without this,apt-get updatefails on the secondary mirror's missing suite and no package can be built at all. Packages that build-depend on something only the PPAs carry will not resolve, so this is for manual builds against upstream Ubuntu; a real build wants the pinned snapshot. -
JENKINS_OPS_DIR: When fetching artifacts from other linux-pkg packages that are marked as dependencies of a package, by default we look for a specific S3 path that contains production package artifacts generated by post-push jobs of the ops Jenkins agent. The production ops Jenkins agent stores artifacts in the special
jenkins-opssub-directory. When using a developer ops Jenkins agent, it stores build artifacts in a different S3 sub-directory:jenkins-ops.<developer>. By setting JENKINS_OPS_DIR to that sub-directory you can instruct linux-pkg to fetch artifacts of build dependencies produced by the developer Jenkins instance instead of the production one. -
DEPENDENCIES_BASE_URL: When fetching artifacts from other linux-pkg packages that are marked as dependencies of a package, we look for a specific s3 path based on what product branch or version we are building for, which is defined by DEFAULT_GIT_BRANCH. If DEPENDENCIES_BASE_URL is left unset, then the path will be determined automatically. DEPENDENCIES_BASE_URL is most useful when set to the input-artifacts of a previous appliance-build run, i.e. "s3://.../input-artifacts/combined-packages/packages".
For each package built by this framework, there must be a file named
packages/<package>/config.sh. It defines some default variables and various
hooks for building the package. When buildpkg.sh is invoked for building a
package, it calls load_package_config(), which sources the appropriate
config.sh file and then executes the various hooks defined for the package.
The bash library lib/common.sh contains various functions that can be called
from the hooks or the various scripts.
Here is a list of variables that can be defined for a package:
-
DEFAULT_PACKAGE_GIT_URL: (Mandatory) Git repository to fetch the package source code from. This is also the repository that is used when pushing changes with the
push-updates.shscript. Note that this must be anhttps://URL. One exception is if the source of the package being built isn't fetched from git. In this case, set this to "none". -
PACKAGE_DEPENDENCIES: (Optional) If the build of this package requires fetching artifacts from other linux-pkg packages, those should be specified in PACKAGE_DEPENDENCIES, as a space-separated list. The dependencies will be fetched in the
fetch_dependencies()step into<WORKDIR>/<dep>/where "dep" is the dependency's name. A special value can be passed for packages that target all the supported flavours of the linux-kernel:@linux-kernel. -
UPSTREAM_SOURCE_PACKAGE: (Optional) Third-party packages that have an update_upstream hook and are updated from an Ubuntu source package should set this to the name of the source package.
-
UPSTREAM_GIT_URL, UPSTREAM_GIT_BRANCH: (Optional) Third-party packages that have an update_upstream hook and are updated from a git repository should set this to the upstream git url and branch.
-
FORCE_PUSH_ON_UPDATE: (Optional) This applies to some third-party packages that have an update_upstream hook. Most third-party packages are synced with upstream by performing a simple "git-merge" command, so when the merge is pushed it can be done with "git push". However some packages, like the linux-kernel ones, perform a rebase instead, and so the merge must be force-pushed instead. If you want to use force push to push an auto-merge, set FORCE_PUSH_ON_UPDATE to "true". Note that a safety check is always performed prior to doing the push to make that the target branch has not changed since the auto-merge commit was generated, however disabling force-push by default is an extra precaution.
-
SKIP_COPYRIGHTS_CHECK: (Optional) By default, at the end of a package's build we check that each produced deb contains a copyright file, unless SKIP_COPYRIGHTS_CHECK is set to "true".
When operations are performed on a package by build or auto-update scripts,
such as buildpkg.sh or
sync-with-upstream.sh, those operations are usually
split into high-level tasks called "stages". Some of those stages can be
modified or must be defined in a package's config file, so we refer to them
here as "hooks". Hooks that have a default definition are stored in
the default-package-config.sh file.
Other "stages" are not meant to be modified and aren't functionally different
from regular function calls, we want to give them more visibility in the build
process as they are deemed as important high-levels tasks, so they are called
via the stage() helper function.
The fetch() hook is optional, as a default is provided and should be used. It
is called when fetching the source code of the package to build or to update.
The repository is cloned into <WORKDIR>/repo and checked out as
branch repo-HEAD. If we are performing a package update, then we also
fetch the upstreams/develop branch into upstream-HEAD. The default
should only be overridden when not fetching the package source from git.
The prepare() hook is optional. It is called before calling the build hook and
normally installs the build dependencies for the package.
The build() hook is mandatory. It is responsible for building the package and
storing the build products into packages/<package>/tmp/artifacts/.
The update_upstream() hook should only be defined for third party packages
that can be auto-updated. It is responsible for fetching the latest upstream
source code on top of branch upstream-HEAD of our fetched repository in
<WORKDIR>/repo. Note that any changes should be rebased on top of
the upstreams/develop branch. If changes are detected, file
<WORKDIR>/upstream-updated should be created.
The merge_with_upstream() hook is called after the update_upstream() hook
when a package is updated via sync-with-upstream.sh.
Whereas update_upstream() updates the upstream-HEAD branch,
merge_with_upstream then merges the upstream-HEAD branch into the
repo-HEAD branch. For most third-party packages this can be left unset as
the default will be used. For packages that have a more complex merge strategy,
such as the linux-kernel packages, this hook can be used.
The checkstyle() hook is optional. It is called before building the package if
-c is provided to buildpkg.sh. Note that this hook isn't currently used by
our build automation and is more of a prototype for an idea.
fetch_dependencies is an immutable stage. It is called for fetching build
artifacts from other linux-pkg packages that are required for performing the
build. See the PACKAGE_DEPENDENCIES package variable for mroe info.
store_build_info() is an immutable stage. It is called after the build()
stage. It is responsible for storing some build info / metadata, such as the
git hash used to perform the build. Some of the build info that is stored is
used by build automation, so care must be exercised when modifying it.
post_build_checks() is an immutable stage. It is responsible for performing
post-build checks that are common to all packages.
One of the checks verifies that each debian package produced has a copyright
file associated with it in the right location. This file is used elsewhere in
the product to generate the license information for the appliance. This check
can be skipped for a package by defining SKIP_COPYRIGHTS_CHECK=true in its
config file.
In addition to any variables defined by the package itself, a few environment variables are set-up by the framework. Here is a quick list:
-
PACKAGE: The name of the package being built.
-
PACKAGE_GIT_URL, PACKAGE_GIT_BRANCH, PACKAGE_VERSION, PACKAGE_REVISION: Those variables are set by the framework depending on the corresponding
DEFAULT_{*}variables defined in the package'sconfig.shand on other environment variables that are passed to the framework. For more details, refer toget_package_config_from_env()in lib/config.sh. -
WORKDIR: Directory where the package is fetched, built, etc. See Package WORKDIR.
-
KERNEL_VERSIONS: Space separated list of kernel versions that the package should be built for.
determine_kernel_versions()must be called before using this variable.
Each package is being fetched, built and updated in directory
linux-pkg/packages/<package>/tmp/, referred to as WORKDIR. Whenever a
script is called to operate a package, the WORKDIR directory is recreated and
a linux-pkg/workdir symlink is created that points to this WORKDIR.
The following sub-directories are created in WORKDIR:
-
repo: where the repository is fetched and built.
-
artifacts: where the build artifacts are stored.
-
source: where the source package is fetched when updating upstream from a source package.
The following files are created in WORKDIR:
- upstream_tag: During a package's auto-update, we may wish to also push
a tag fetched from the upstream repository for informational purposes. If so,
the
upstream_tagfile should be created and contain the name of the tag that needs to be pushed.
The following files are used as status indicators in WORKDIR:
-
upstream-updated: created if upstream-HEAD has updates that should be pushed.
-
repo-updated: created if repo-HEAD has updates that should be pushed, following a merge.
When considering adding a new package, the workflow will depend on whether the package is a third-party package or in-house package.
Note:: If you are thinking of adding a new package to this framework, you should first read the Delphix Open-Source Policy.
If the package is already provided by Ubuntu, it's recommended to use the source package as the package name. You can get the source package name for a given package by running:
sudo apt update
sudo apt show <package name> | grep Source
It is possible that the source package is not provided and so the command above
will not return anything, in which case you can use <package name> as the name
of the package.
Once you've decided on a package name (we shall refer to it as <package>),
create a directory for it: packages/<package>/.
Next step is to create a new file: packages/<package>/config.sh. You can copy
the template from template/config.sh. To get started, all
we need to provide is info on where to fetch the upstream source code from.
If you are using an Ubuntu source package, you'll only need to specify the name of the source package:
UPSTREAM_SOURCE_PACKAGE="<source package name>"
If the upstream source code is instead to be retrieved from a git repository, then you need to provide the git details:
UPSTREAM_GIT_URL="<git url>"
UPSTREAM_GIT_BRANCH="<git branch>"
Note that steps 3 to 5 are most useful when getting a third party package from
an Ubuntu source package. When the third party package is fetched from git,
you may simply fork the upstream repository and add an upstreams/develop
branch that points to the develop branch; you can then update
DEFAULT_PACKAGE_GIT_URL in config.sh to your forked git repository and skip
to step 6.
You can fetch the upstream source code from an Ubuntu source package by running:
cd packages/<package>/tmp/
mkdir source
cd source
apt-get source <upstream-source-package>
cd ..
mv source/"<upstream-source-package>"*/ repo
cd repo
git init
git checkout -b repo-HEAD
git add -f .
git commit -m '<insert commit message here>'
TODO: create a command that will run the steps above. It used to be done by
buildpkg.sh -i, but this logic has been removed.
The next steps will require you to provide a git repository for your local
version of the package. For development purposes you should create an empty
repository on github, and then put the url into config.sh. Note that the URL
should start with https://.
e.g.
DEFAULT_PACKAGE_GIT_URL="https://github.com/<developer>/<package>"
Next step is to push the upstream code to the newly created repository to your developer repository. You should push the initial commit to both the develop branch and the upstreams/develop branch.
In this step you'll need to define a few hooks in config.sh. In the hooks you
can leverage convenience functions provided by lib/common.sh.
To build the package you'll most likely need to install some build
dependencies. If that is the case, you should add a prepare() hook
that will install those build dependencies. For an Ubuntu source package, those
dependencies can be installed by calling
install_build_deps_from_control_file().
For other packages, you can usually find the build dependencies in the project's
README. It is recommended to edit the debian/control file of the package
to list the required build dependencies, so that
install_build_deps_from_control_file() can be used. Otherwise, you can also
use the install_pkgs() lib function to install packages.
Next step is to add a build() hook. It is recommended to use the
dpkg_buildpackage_default() function.
Note that if you are using an Ubuntu source package, you should now be ready to build the package.
For a package that doesn't have a debian metadata directory already defined in
its source tree, you'll need to create it, and push the changes to the
develop branch of your developer repository. See
Common Steps > Creating debian metadirectory
for more details.
Once this is all ready, you can try building the package by running:
./buildpkg.sh <package>
If you want the package to be automatically updated with upstream (strongly
recommended), you'll need to add the update_upstream()
hook to config.sh. You should use the following functions provided by
lib/common.sh:
-
update_upstream_from_source_package()ifUPSTREAM_SOURCE_PACKAGEis set. -
update_upstream_from_git()ifUPSTREAM_GIT_URL&UPSTREAM_GIT_BRANCHare set.
See Common Steps > Make the package official.
Once you verify that the package can be built, submit a pull request with only the
/packages/<package> chages. Once merged, this change will create a Jenkins job
(/linux-pkg/develop/build-package/<package>/pre-push) to
build the new package as part of the automation pipeline. Without merging these changes
first, we will not be able to test the automated build and integration into the appliance.
In a separate change, add the package to package-lists. See Common Steps > Add package to package-lists.
See section Testing Your Changes.
IMPORTANT: This is the step which will trigger integration of this package into the appliance.
Steps for adding an in-house package are slightly different than for a third-party package.
This example assumes that the source code for the project is already present in
a git repository and contains a Makefile with instructions to compile the
project. If the debian metadata directory is not in the source tree, see
Common Steps > Creating debian metadirectory.
We will refer to the name you picked for your package as <package>. Make sure
the name doesn't conflict with an existing Ubuntu package.
You'll need to create a new directory: packages/<package>/ and add a new
config.sh file in it. You can copy the template from
template/config.sh.
In config.sh, you'll need to define two variables:
DEFAULT_PACKAGE_GIT_URL: thehttps://git URL for the source code of the package.
e.g.:
DEFAULT_PACKAGE_GIT_URL="https://github.com/delphix/<package>"
If your package needs some build dependencies, you'll want to add a
prepare() hook to config.sh which will install those build
dependencies. It is recommended to use the install_pkgs() function provided by
lib/common.sh.
Next step is to add a build() hook. It is recommended to use the
dpkg_buildpackage_default() function provided by
lib/common.sh.
Once those hooks are set-up, you can try building your package by running:
./buildpkg.sh <package>
See Common Steps > Make the package official
Those steps apply to both third-party and in-house packages.
You can refer to the Debian Maintainer Guide
here.
Note that packages built by gradle, such as the delphix-sso-app, do not
require a debian metadirectory.
See the Package Lists section for more info.
Once your new package builds and has been tested in the product, the next step is to create an official repository for it.
-
First, you should read Delphix Open-Source Policy if you haven't already, and provide the necessary info so that a
github.com/delphix/<package>repository can be created for it. You'll need to push the develop branch from your developer repository, as well as the upstreams/develop branch if it is a third-party package. Note that if you have modified develop (i.e. it diverges from upstreams/develop), you should submit your changes for review before pushing them. -
If this is a third-party package that is to be auto-updated by Delphix automation, you should also make sure the
github.com/delphix-devops-botuser is added as a collaborator to the repository. -
Update
DEFAULT_PACKAGE_GIT_URLinpackages/<package>/config.shto the official repository.
If you are not making any changes to linux-pkg, only changes to a given package managed by linux-pkg:
- Run
git-ab-pre-pushfrom your package's repository.
TODO: complete section
If you are testing a newly added package to linux-pkg:
- Run
git-ab-pre-push -b <package>fromlinux-pkg.
Note that this package must already have been added to /packages/ in the
develop branch, so that its specific Jenkins build job exists.
Package lists are basically just lists of packages defined in linux-pkg. They are mainly consumed by the Jenkins build infrastructure by calling the ./query-packages.sh utility. Jenkins needs to know which packages to build and include for a given version of the Delphix appliance.
Package lists are stored under ./package-lists, in two
sub-directories: build and update. The build directory contains packages
that are built and consumed by the Delphix Appliance, while the update
directory contains a list of packages that are automatically synced with
the upstream projects.
There are two physical build lists:
-
main.pkgs: this is the default list for packages that are to be added to the Delphix Appliance. -
kernel-modules.pkgs: this list is similar to themainlist but contains packages that have a dependency on the multiple flavours of the linux kernel that are supported by the Delphix Appliance.
There's also a virtual build list, called "linux-kernel", which lists all the linux kernel packages built by linux-pkg (one for each supported flavour of the linux kernel). You can list the contents of the virtual list by running:
./query-packages.sh list linux-kernel
There is a single update list called main.pkgs, which contains all the
packages that are auto-updated nightly by Jenkins. Note that zfs is not in
that list as it has a dedicated Jenkins job that tracks the upstream
repository and launches as soon as there are new changes.
Most third-party packages should have an update_upstream() hook defined and
be added to that list.
The framework is designed in a way to allow easy integration with the Delphix
release process. The idea is that both the package build artifacts (.debs
and .ddebs) and package source code should be available for each Delphix
release. This should hold for both in-house and third-party packages.
Regarding the build artifacts, those should be taken care of by the existing
Delphix build artifacts storage policy, available
here.
The relevant code for managing the build artifacts is outside of the scope of
this project and lies in the devops-gate.
Regarding the source code, we expect that each package repository and the
linux-pkg repository itself follows the Delphix branching policy outlined
here.
When creating a new branch or release for the Delphix Appliance, an external
script should create the relevant branch or tag for each repository. The
branch or tag should then be passed to the build in the DEFAULT_GIT_BRANCH
environment variable.
When building packages for an older version of the Delphix Appliance, the build
image will need to be picked accordingly. We are currently using
dlpx-internal-buildserver-develop, but this will not be the case anymore once we switch to a
newer Ubuntu distribution.
All contributors are required to sign the Delphix Contributor Agreement prior to contributing code to an open source repository. This process is handled automatically by cla-assistant. Simply open a pull request and a bot will automatically check to see if you have signed the latest agreement. If not, you will be prompted to do so as part of the pull request process.
This project operates under the Delphix Code of Conduct. By participating in this project you agree to abide by its terms.
This software is provided as-is, without warranty of any kind or commercial support through Delphix. See the associated license for additional details. Questions, issues, feature requests, and contributions should be directed to the community as outlined in the Delphix Community Guidelines.
This is code is licensed under the Apache License 2.0. Full license is available here.