[sweb]

Description

Closes #774

Adds the field job.config.ttlAfterFinished to set how long the spark-submit Job is kept around. This is useful, because the operator now moves the ownership of the driver pod, the serviceaccount and config maps that were created for this SparkApplication to the Job, once the driver pod finishes (either with Succeeded or Failed).

apiVersion: spark.stackable.tech/v1alpha1
kind: SparkApplication
metadata:
  name: example
spec:
  job:
    config:
      ttlAfterFinished: 1h

There are three possible values for ttlAfterFinished:

• never: The Job never gets deleted and sticks around until the SparkApplication (its owner) gets deleted.
• a duration like 10m (which is the default): The job sticks around for the specified duration and then gets deleted - this then also deletes the mentioned resources that have the job as owner.
• 0s: Or any other variant that results in a duration of 0s - this deletes the Job and the other resources right away.

There is a special case to consider: In case of a fixed pod name via spark.kubernetes.driver.pod.name with retryOnFailureCount > 0 the ttlAfterFinished is set to 0 and the driver pod gets deleted right away so that the retry can happen as it is going to use the same driver pod name.

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Author

• Changes are OpenShift compatible
• CRD changes approved
• CRD documentation for all fields, following the style guide.
• Helm chart can be installed and deployed operator works
• Integration tests passed (for non trivial changes)
• Changes need to be "offline" compatible
• Links to generated (nightly) docs added
• Release note snippet added

Reviewer

• Code contains useful comments
• Code contains useful logging statements
• (Integration-)Test cases added
• Documentation added or updated. Follows the style guide.
• Changelog updated
• Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

• Feature Tracker has been updated
• Proper release label has been added
• Links to generated (nightly) docs added
• Release note snippet added
• Add type/deprecation label & add to the deprecation schedule
• Add type/experimental label & add to the experimental features tracker