Skip to content

chore: remove shelljs from automation scripts - #2502

Open
r0b1n wants to merge 4 commits into
mainfrom
chore/automation-utils-drop-shelljs
Open

r0b1n wants to merge 4 commits into
mainfrom
chore/automation-utils-drop-shelljs

Conversation

@r0b1n

@r0b1n r0b1n commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Pull request type

Refactoring (e.g. file rename, variable rename, etc.)


Description

Removes shelljs from the repo's own scripts (follow-up to #2484), as four commits that can be reviewed separately. Replacing exec (execa v5) with zx is the next step.

1. Trim shell exports and pass cwd instead of pushd (4b8d0be)

  • src/shell.ts exported 29 shelljs functions; only the ones in use are kept.
  • pushUpdateToTestProject (steps.ts) and updateChangelogsAndCreatePR (changelog.ts) run their git commands with execa's cwd option instead of pushd/popd.

This fixes two cases where the process was left in a different directory:

  • pushUpdateToTestProject called pushd and then returned early on "Nothing to commit" without popd.
  • zip() only called popd on success, so a failed zip left the process inside the zipped folder.

2. Replace shelljs file helpers with node:fs (e2f4c2f)

  • rm -rf, mkdir -p and cp call sites now use node:fs/promises, in automation/utils and in the charts-web and data-widgets scripts. shelljs is removed from automation/utils/package.json.
  • shell.ts keeps small helpers only where node:fs has no direct equivalent:
    • findFiles(...patterns): fs.globSync that throws when a pattern matches nothing. shelljs find with fatal failed the same way, so a missing mpk still stops the release instead of being skipped silently.
    • move(source, dest): rename with a copy and remove fallback when crossing file systems, like shelljs mv. This replaces the cp + rm workaround in moveModuleToDist for mv silently swallows errors shelljs/shelljs#878.
    • makeWritableRecursive(dir): the chmod -R a+rw used before re-zipping mpks.
  • ensureFileExists uses fs.access and is now async.
  • zip() switches directories with process.chdir; cross-zip has no cwd option.
  • rui-oss-clearance imports node:fs/promises statically instead of the inline await import(...) calls.

Copying a file into a directory now passes the full destination path, since fs.cp doesn't copy into an existing directory the way shelljs cp does.

pnpm-lock.yaml: removes shelljs 0.8.5.

3. Replace shelljs in run-e2e with node:fs (d363d85)

  • ls, rm, mkdir, mv, cp, test and cat in automation/run-e2e/lib now use node:fs; globs (tests/testProjects-*, *.mpr, dist/<version>/*.mpk) use fs.globSync.
  • gh auth token (dev mode) and the themesource chmod -R +w use spawnSync.
  • shelljs is removed from automation/run-e2e/package.json and from the lockfile.

Behaviour change: shelljs was not in fatal mode in run-e2e, so a failed cp/mv only printed an error and continued. These now throw. The Atlas theme releases (atlasui-theme-files-2024-01-25, -2025-10-08) contain web and native, and the Atlas core releases (3.17.0, 4.5.0) contain themesource, so the copies succeed. A missing results/runtime.log is still reported instead of thrown. A missing widget mpk after the release script throws, but updateTestProject already catches and logs errors from that step.

4. Drop the unused shx dependency of pluggable-widgets-tools (c7e0f34)

@mendix/pluggable-widgets-tools depends on shx, which brings in shelljs 0.9.2 (and an old execa@1 chain), but neither the tools nor this repo call it. It is removed with a "@mendix/pluggable-widgets-tools>shx": "-" override, like the other unused tools dependencies in the root package.json.

The only shelljs left in the lockfile is 0.10.0, used by the tools' rollup config to copy XML, icons, LICENSE and the mpk. The upstream fix is merged but not yet released; the override and that dependency can go once we upgrade the tools.

What should be covered while testing?

Verified locally:

  • automation/utils type-checks and lints; script lints pass for all modules and charts-web.

  • Helper smoke test: zip/unzip, zip restoring the directory on success and failure, ensureFileExists throwing ENOENT, findFiles matching and throwing, makeWritableRecursive (600 → 666), move.

  • pnpm turbo build --filter=@mendix/charts-web: Charts.mpk contains all 10 widgets merged under com/ plus package.xml and License.txt.

  • pnpm turbo build --filter=@mendix/badge-web --force and its unit tests pass without shx installed.

  • run-e2e: setupTestProject (twice, covering the cleanup branch) and updateTestProject for badge-web: the mpr, Atlas theme/web and theme/native, themesource/atlas_core and Badge.mpk are in place, and the themesource files are writable.

  • build:module for data-widgets (all 9 widget mpks copied, themesource, .version, LICENSE) and web-actions (themesource identical to source).

  • Module release pipeline for data-widgets, run locally with CI unset: the steps from scripts/release.ts, with only the clone swapped for an anonymous one because cloneTestProject needs GH_PAT. That covers createModuleMpk in Docker (mxbuild 10.24.0.73019), addWidgetsToMpk (9 widgets added) and moveModuleToDist. The resulting dist/3.11.8/DataWidgets.mpk has the same file list and the same package.xml entries as the published data-widgets-v3.11.8 asset. The only difference is the READMEOSS HTML, which OSS clearance adds after release. No temp folders were left behind.

Not run locally, worth watching on the next module release:

  • Release-only paths: the OSS clearance scripts and rui-create-gh-release, which talk to GitHub releases.
  • pushUpdateToTestProject and the changelog flow, which push to remotes.
  • The EXDEV fallback in move.
  • The Docker parts of run-e2e (ci.mjs, docker-utils.mjs); the E2E jobs exercise them.

…ushd

First step towards dropping shelljs. Export only the shelljs helpers that are
used, and run git commands with execa's cwd option instead of pushd/popd.

This also fixes two cases where the process directory was left changed:
pushUpdateToTestProject returned early on "Nothing to commit" without popd,
and zip() skipped popd when zipping failed.
@r0b1n
r0b1n requested a review from a team as a code owner October 29, 2026 13:04
@github-actions

This comment has been minimized.

Use node:fs/promises for rm, mkdir and cp in automation/utils and the
charts-web and data-widgets scripts, and remove shelljs from automation/utils.

shell.ts keeps small helpers where node:fs has no direct equivalent:
- findFiles: glob that fails when a pattern matches nothing, like shelljs find
- move: rename with a copy and remove fallback across file systems, like mv
- makeWritableRecursive: chmod -R a+rw
zip() now switches directories with process.chdir instead of pushd/popd.
@r0b1n r0b1n changed the title chore(automation-utils): trim shell exports and pass cwd instead of pushd chore(automation-utils): remove shelljs from automation utils Oct 29, 2026
@github-actions

This comment has been minimized.

Use node:fs for ls, rm, mkdir, mv, cp, test and cat in run-e2e, spawnSync for
`gh auth token` and the themesource chmod, and remove shelljs from run-e2e.

shelljs was not in fatal mode here, so a failed cp or mv only printed an
error. They now throw; the Atlas theme and core releases in use contain the
copied folders. A missing runtime.log is still reported instead of thrown.
@r0b1n r0b1n changed the title chore(automation-utils): remove shelljs from automation utils chore: remove shelljs from automation scripts Oct 29, 2026
@github-actions

This comment has been minimized.

pluggable-widgets-tools depends on shx, which pulls in shelljs 0.9.2, but
neither the tools nor this repo call it. Remove it with a pnpm override, like
the other unused tools dependencies. The tools still use shelljs 0.10.0 in
their rollup config until a release without it is available.
@github-actions

Copy link
Copy Markdown
Contributor

AI Code Review

✅ Approved — no issues found


What was reviewed

File Change
automation/utils/src/shell.ts The 29 shelljs re-exports are replaced by findFiles, move, makeWritableRecursive, an async ensureFileExists, and a zip that restores the working directory on failure.
automation/utils/src/{steps,changelog,monorepo,mpk,oss-clearance}.ts, bin/* shelljs calls are replaced with node:fs/promises. pushd/popd becomes execa cwd.
automation/run-e2e/lib/*.mjs, package.json shelljs is replaced with node:fs (globSync, cpSync, rmSync and similar) and spawnSync.
packages/modules/data-widgets/scripts/start.ts, packages/pluggableWidgets/charts-web/scripts/build.mts These scripts now use node:fs/promises.
package.json, pnpm-lock.yaml New override @mendix/pluggable-widgets-tools>shx: "-", and shelljs entries removed from the lockfile.

Skipped (out of scope): pnpm-lock.yaml (generated)

CI: pending: Run code quality check, Plan E2E matrix, sigridci · unrelated failures: security/snyk (external scanner)


Questions

  • None blocking. The release-only paths (createModuleMpk in Docker, the EXDEV fallback in move, pushUpdateToTestProject) were not run, as the description says. Please watch the next module release.

Positives

  • pushUpdateToTestProject no longer leaves the process in another directory on the early "Nothing to commit" return. zip() now restores the working directory on failure too.
  • findFiles keeps the old fail-fast behaviour when a glob matches nothing, so a missing mpk still stops a release.
  • Call sites that relied on shelljs copying into a directory now pass the full destination path to cp. I checked steps.ts, monorepo.ts, charts-web and oss-clearance and found none missed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant