[tnazarew]

The spark and hive Dataproc workflows clean up their cluster with a workaround:

• the producer step runs with continue-on-error
• the cluster is terminated in an always() step
• a separate "Fail if … jobs failed" step fails the job afterwards

All this because composite actions can't have post steps, as the workflow's own comment explains. This replaces that with a small JavaScript action that has a post step.

.github/actions/dataproc_cluster

• main: saves the cluster name, project and region, then runs create-command. Saving first means the post step also deletes a cluster left behind by a failed create.
• post (post-if: always()): runs after all other steps of the job, whatever their result. If the cluster exists, it deletes it with gcloud dataproc clusters delete. Post steps run in reverse order, so it runs before the GCP auth step removes its credentials.
• No dependencies: it uses only Node built-ins, so there's nothing to install or bundle.

Usage:

- uses: ./.github/actions/dataproc_cluster
  with:
    cluster-name: my-cluster
    project-id: gcp-open-lineage-testing
    region: us-west1
    create-command: |
      python producer/spark_dataproc/runner/dataproc_workflow.py create-cluster ...

Workflow changes

• producer_hive_dataproc.yml: "Start producer" now uses the action, with the create command unchanged. Removed continue-on-error, "Terminate producer cluster" and "Fail if hive jobs failed". A failing producer step now fails the job directly, and the cluster is still deleted.
• producer_spark_dataproc.yml (as reworked in spark_dataproc: one cluster per Spark version for all OpenLineage versions #344): "Start producer" now uses the action, and the always() "Terminate producer cluster" step is removed. spark_dataproc: one cluster per Spark version for all OpenLineage versions #344 had already dropped continue-on-error and the "Fail if" step there.

Relation to other PRs

• Create GH action with post action step #203: takes the same post-step approach, but as a general pre/main/post wrapper with a 19,860-line esbuild bundle and npm files, and with the scenario loop moved inside the action. This one is limited to the cluster lifecycle and needs no build step. If this one goes in, Create GH action with post action step #203 could be closed.
• spark_dataproc: one cluster per Spark version for all OpenLineage versions #344: merged; this PR is rebased onto it.

Testing

• Local simulation (main and post run with Node against a cluster that doesn't exist):
  • a failed create fails the step and still saves the state
  • a successful create passes
  • post reports "does not exist" and skips the delete
• Not yet exercised: the delete itself, which needs a real cluster. CI on this PR does not run the Dataproc workflows: main_pr.yml starts them only when a path under producer/spark_dataproc/ or producer/hive_dataproc/ changes, and this PR changes only .github/.
• actionlint 1.7.12: passes, apart from an existing issue on main: producer_hive_dataproc.yml builds the report name from inputs.hive_release, which isn't an input of that workflow. It's not changed here.

🤖 Generated with Claude Code