Skip to content

Add kernel-next and supporting packages - #621

Merged
vigh-m merged 6 commits into
bottlerocket-os:developfrom
vigh-m:64kb-kernel/kernel-kit
Nov 13, 2026
Merged

vigh-m merged 6 commits into
bottlerocket-os:developfrom
vigh-m:64kb-kernel/kernel-kit

Conversation

@vigh-m

@vigh-m vigh-m commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Issue:

Related: #624

Description of changes:

This PR adds kernel-next, a kernel that follows the Amazon Linux minor updates, two kmod packages for it, and a change to the full-config tool. It also includes two small fixes to the existing NVIDIA drivers.

kernel-next is not an LTS kernel. The package name stays the same when the kernel moves to a new minor version, for example from 7.2 to 7.3. The build uses the Amazon Linux 2027 kernel-7.2.13-17.148 for now.

What's new in kernel-next:

  • On aarch64, the kernel uses 64K pages. The base config is config-aarch64-64k from the SRPM. On x86_64, the kernel uses 4K pages.
  • The SRPM has no Lustre client and no aws_fwflash driver, so this kernel cannot mount FSx for Lustre file systems.
  • The kernel has no FIPS certification, so it conflicts with the fips image feature.
  • The kernel has no bare-metal support, no squashfs, no runtime kernel-devel, no legacy iptables, and no Neuron drivers.

What's new in kmod-next-nvidia-r620:

  • The package has only the open GPU kernel modules. It has no GRID subpackage and no list of GPUs that stay on the proprietary driver.
  • Fabric manager and IMEX come from the 2.0 -direct packages. The fabric manager binaries are in the -tesla subpackage, and the topologies are in /usr/share/nvidia/nvswitch.

Testing done:

  • make full-config generated the four kernel-next config-full files, two on an x86_64 host and two on an aarch64 host, with SDK v0.82.0.
  • Built and tested kernel-next and the kmods in a custom AMI. Verified on actual hardware that:
    • the Nvidia drivers come up correctly and can see all the GPUs
    • Basic workloads pass
    • NVLink setup between multiple instances has consistent bandwidth
    • MIG can be configured correctly

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@vigh-m
vigh-m force-pushed the 64kb-kernel/kernel-kit branch 2 times, most recently from 43d1142 to 764f99f Compare November 10, 2026 21:55
@vigh-m vigh-m changed the title 64kb kernel/kernel kit Add kernel-next and supporting packages Nov 10, 2026
@vigh-m
vigh-m force-pushed the 64kb-kernel/kernel-kit branch 3 times, most recently from 31efe97 to cace382 Compare November 11, 2026 17:51
@ginglis13

ginglis13 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

You'll need to carry the patch added in #623 for kernel-next

@vigh-m
vigh-m marked this pull request as ready for review November 11, 2026 21:58

[Service]
Type=oneshot
ExecStart=/usr/bin/driverdog --modules-set nvidia-lts-open-gpu load-modules

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need ExecCondition=/usr/bin/ghostdog match-nvidia-driver open-gpu here similar to how we have it in the 580 nvidia packages?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's only open-gpu in r620 so we don't need to match the driver to the hardware.


# Disable DAMON subsystem. We currently do not have a good use-case for DAMON.
# CONFIG_DAMON is not set

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to set a maximum for CMA_AREAS similar to how we have it here?

# Pin the maximum number of CMA (Contiguous Memory Allocator) areas to 20 on all
# architectures. Upstream mm/Kconfig defaults CMA_AREAS to 20 when NUMA is set
# (both arches have CONFIG_NUMA=y). Amazon Linux ships no CMA_AREAS value for
# x86_64, so it resolves to that NUMA default of 20; but its aarch64 config pins
# a stale CONFIG_CMA_AREAS=7 (a pre-v6.11 default) that olddefconfig preserves
# instead of recomputing. Set it explicitly so aarch64 matches the modern NUMA
# default that x86_64 already gets.
CONFIG_CMA_AREAS=20

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is now the default upstream configuration. The config-full- files have CONFIG_CMA_AREAS=20

ABORT_CUDA_JOBS_ON_FM_EXIT=1

# Path to topology and database files.
TOPOLOGY_FILE_PATH=/usr/share/nvidia/nvswitch

@maherthomsi maherthomsi Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other files have this set to TOPOLOGY_FILE_PATH=/usr/share/nvidia/tesla/nvswitch, looking to make sure this is what were expecting

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. /usr/share/nvidia/nvswitch is the default installation path. I chose to keep it that way for the new 620 driver.

Use "LicenseRef-", not "LicenceRef-".
Use the same fabric manager and IMEX URLs as Cargo.toml, and the same
IMEX file names.
Find the config-full file names from the files in the kernel package.
Find the SRPM spec name from the kernel version, and stop if the SRPM
does not have that spec. Give the aarch64 base config to the merge
functions as an argument.
@vigh-m
vigh-m force-pushed the 64kb-kernel/kernel-kit branch from cace382 to 099f12d Compare November 12, 2026 19:33
@vigh-m

vigh-m commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Update to enable CI and add missing patch

@vigh-m
vigh-m force-pushed the 64kb-kernel/kernel-kit branch from 099f12d to 316b034 Compare November 12, 2026 20:45
Comment thread packages/kernel-next/kernel-next.spec Resolved

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this and the {-aarch64, -x86_64} custom configurations are identical to the 6.18 kernel's custom configurations that are now in the kernel-6.18-shared-configs pkg

Not a blocking comment for this PR but maybe a follow up to rearchitect that shared pkg and reuse it for the -next kernels, too. If we anticipate that 7.x kernel configs will diverge from the base 6.18 configs, then may not be necessary.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. I debated doing this. I didn't mostly to leave the door open for 7.x custom configurations once we have a stable kernel. We'll have to do a full evaluation of newer features and see if there are any that Bottlerocket would benefit from

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should merge 6.18 with 7.x configs. We should just keep shared configs along the same kernel family - not across.

7.x and newer kernels will introduce configurations that don't exist on other kernels that we might want to enable. Or we might want to try something new in 7.x that in 6.18 we can't because might impact existing users.

Comment thread packages/kernel-next/kernel-next.spec Resolved
Comment thread packages/kernel-next/kernel-next.spec Resolved
Comment thread packages/kernel-next/kernel-next.spec Resolved
Summary: NVIDIA %{tesla_major} GRID driver
Version: %{tesla_ver}
License: MIT AND GPL-2.0-only AND LicenceRef-NVIDIA-GRID-AWS-EULA
License: MIT AND GPL-2.0-only AND LicenseRef-NVIDIA-GRID-AWS-EULA

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a typo/sp fix, see 485ecaa

@vigh-m
vigh-m force-pushed the 64kb-kernel/kernel-kit branch 3 times, most recently from 8a89cc2 to 250dc28 Compare November 13, 2026 01:17
@vigh-m

vigh-m commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Push to address comments

@vigh-m
vigh-m force-pushed the 64kb-kernel/kernel-kit branch 2 times, most recently from 2c17d75 to 005216a Compare November 13, 2026 04:05
This kernel follows the Amazon Linux minor updates. The build uses the
Amazon Linux 2027 kernel7.2 SRPM. On aarch64, the kernel uses 64K
pages.

tools/latest-kernel-full-config.sh uses config-aarch64-64k and
config-microcode-6-18 for kernel-next.
Build the EFA driver for kernel-next from the EFA installer, as
kmod-6.18-efa does. kernel-next requires this package.
Add the NVIDIA R620 driver for kernel-next as the lts branch. This
package builds only the open GPU kernel modules. The units load them
on every boot, with no ghostdog check.
@vigh-m
vigh-m force-pushed the 64kb-kernel/kernel-kit branch from 005216a to 9644a9e Compare November 13, 2026 17:58
@vigh-m
vigh-m merged commit a52bdb7 into bottlerocket-os:develop Nov 13, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants