Repository navigation
pkg_rpm: reserve local resources for threaded payload compression - #1094
chouquette wants to merge 1 commit into
Conversation
### What does this PR do? Sets the rpm payload compression in `dd_pkg_rpm` to xz: level 9 when `//:is_release` is set and level 5 otherwise. Packages can no longer override it. Also enable multi threaded compression. ### Motivation The omnibus rpms are xz-compressed (they require `rpmlib(PayloadIsXz)`), unlike the Bazel ones. The package parity check (#58991) reports the difference. The levels mirror omnibus, using the release flag as the replacement for its deploy detection. This PR is stacked on the release opt-in PR (#59217), so that regular builds default to level 5. ### Describe how you validated your changes Built a small rpm through `dd_pkg_rpm`: xz level 5 by default and level 9 with `--config=release`. Both require `rpmlib(PayloadIsXz)`. ### Additional notes bazelbuild/rules_pkg#1094 is related so that we can avoid overcommiting on resources. In our case, since packaging are usually the last actions, we shouldn't notice much difference though. Co-authored-by: hugo.beauzee <[email protected]>
cgrindel
left a comment
There was a problem hiding this comment.
Can we add some tests for this?
aiuto
left a comment
There was a problem hiding this comment.
I would really rather we fork this internally at datadog and see how it plays out over the next few months before we pick the threads->memory size ratios.
| def _xz_high_threads_16(_os, _inputs_size): | ||
| return {"cpu": 16, "memory": 16 * _XZ_HIGH_LEVEL_MEMORY_PER_THREAD_MB} | ||
|
|
||
| _PAYLOAD_RESOURCE_SETS = { |
There was a problem hiding this comment.
This isn't particularly easy to read.
And it looks like a coding agent went wild on it.
It looks like it could be replaces with 3 functions, which just multiply the number of CPUs by different constants.
| for non-test actions. Using threaded compression may result in | ||
| overcommitting your system. | ||
| When the value requests an explicit number of threads (for example | ||
| `w6T8.xzdio`), the action reserves that many CPUs from Bazel's local |
There was a problem hiding this comment.
Is there a documented semantics for this compression or are we making up the syntax?
| return {"cpu": 16, "memory": 16 * _XZ_HIGH_LEVEL_MEMORY_PER_THREAD_MB} | ||
|
|
||
| _PAYLOAD_RESOURCE_SETS = { | ||
| "default": {2: _threads_2, 4: _threads_4, 8: _threads_8, 16: _threads_16}, |
There was a problem hiding this comment.
Does this match the default someone would get today if they just set the level without threadas?
### What does this PR do? Switches the rules_pkg `archive_override` to the `datadog-patchset` branch of my personal fork (pinned at `12239bd1bd`). That branch carries these pending rules_pkg patches on top of upstream main: - pkg_rpm: add `rpm_own_tree_dirs` attribute (bazelbuild/rules_pkg#1089, already used here) - Keep every tree artifact mapped to a shared destination (bazelbuild/rules_pkg#1091). Previously, when several tree artifacts were installed to the same directory, only the last one was kept. This lets the Python integrations be installed in the embedded directory alongside the OpenSSL files. - pkg_rpm: reserve local resources for threaded payload compression (bazelbuild/rules_pkg#1094) Full diff: bazelbuild/rules_pkg@main...chouquette:rules_pkg:datadog-patchset ### Motivation The rules_pkg maintainers asked for the compression change to be tested in datadog-agent before it gets merged. The other two patches are still in upstream review, so carrying them on the same branch lets us use them now instead of waiting. We'll switch back to upstream once they all land. ### Describe how you validated your changes Built `//packages/agent/linux:debian` and `//packages/agent/linux:rpm` locally. Co-authored-by: hugo.beauzee <[email protected]>
binary_payload_compressionaccepts rpm's threaded payload modes such asw9T8.xzdio, butMakeRpmalways runs with Bazel's default reservation of one CPU, which the attribute documentation warns about.This derives a
resource_setforMakeRpmfrom the compression value when it requests an explicit thread count:Values without an explicit thread count keep the current behavior.
resource_setcallbacks have to be top-level functions and cannot be parameterized, which is why the supported thread counts are enumerated.Tested by building two
w6T8.xzdiorpms with--local_resources=cpu=8: the secondMakeRpmwaits for the first one, while two single-threaded rpms still run in parallel.