[WIP] feat(api): Replace PodTemplateOverrides with TemplateOverrides

[andreyvelich]

This BREAKING CHANGE will replace PodTemplateOverride with TemplateOverrides API.

We would like to group overrides by manager for clear ownership boundaries.

This PR updates KEP, APIs, and implementation.

TemplateOverrides will have:

• PodTemplateOverrides - for Pod overrides
• JobTemplateOverrides - for JobSet/Job overrides

Related: #3020

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from andreyvelich. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Pull request overview

This PR introduces a breaking API change to replace PodTemplateOverrides with manager-scoped TemplateOverrides, aiming to group override ownership boundaries more clearly across controllers/users.

Changes:

• Replaces TrainJobSpec.PodTemplateOverrides with TrainJobSpec.TemplateOverrides keyed by manager.
• Introduces new API types for TemplateOverride, including job-level and pod-level override histories.
• Updates the v2 proposal/KEP documentation to describe the new API shape and examples.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.

File	Description
pkg/apis/trainer/v1alpha1/trainjob_types.go	Updates TrainJob API types to add manager-keyed TemplateOverrides and new override structs.
docs/proposals/2170-kubeflow-trainer-v2/README.md	Updates the proposal to document TemplateOverrides, including rationale and YAML examples.

[Copilot]

In the docs code snippet, Time is marked as +required and uses a non-pointer metav1.Time, which doesn’t match the actual API types in pkg/apis/trainer/v1alpha1/trainjob_types.go where Time is optional (*metav1.Time).

[Copilot]

JobTemplateOverrides/PodTemplateOverrides are declared as +listType=map with +listMapKey=time, but the keyed field Time is optional (*metav1.Time), which makes the map key potentially unset and breaks map-list semantics (unique/stable keys for merge/validation).

Suggested change

	// +listType=map
	// +listMapKey=time
	JobTemplateOverrides []JobTemplateOverride `json:"job,omitempty"`
	// podTemplateOverrides defines overrides that applied to PodTemplateSpec
	// +kubebuilder:validation:XValidation:rule="!has(oldSelf) || oldSelf.all(old, self.exists(new, old.time == new.time && old == new))", message="existing entries cannot be modified or removed in template overrides"
	// +kubebuilder:validation:XValidation:rule="!has(oldSelf) || size(self) >= size(oldSelf)", message="pod template override entries cannot be deleted"
	// +listType=map
	// +listMapKey=time
	JobTemplateOverrides []JobTemplateOverride `json:"job,omitempty"`
	// podTemplateOverrides defines overrides that applied to PodTemplateSpec
	// +kubebuilder:validation:XValidation:rule="!has(oldSelf) || oldSelf.all(old, self.exists(new, old.time == new.time && old == new))", message="existing entries cannot be modified or removed in template overrides"
	// +kubebuilder:validation:XValidation:rule="!has(oldSelf) || size(self) >= size(oldSelf)", message="pod template override entries cannot be deleted"

[andreyvelich]

I’ve been thinking that we could start using JobTemplateOverride instead of the dedicated Labels and Annotations fields we currently expose in the TrainJob.spec API.

The idea would be:

• If targetJob is omitted, the override is applied to the JobSet
• If targetJob is set, the override is applied to the specific Job

One concern is that once we introduce JobTemplateSpecOverride, it could potentially contain fields relevant to both Job and JobSet, which may introduce ambiguity. I’m not entirely sure what the better way to handle that would be, though I also don’t see a clearly better alternative at the moment.

@tenzen-y @kaisoz @mimowo @astefanutti @kannon92 , I’d really appreciate your thoughts on this approach.

[andreyvelich]

Time will be set server-side by Trainer admission mutating webhook when TrainJob is created/updated.

[andreyvelich]

Do you prefer pod and job or podTemplateOverrides and jobTemplateOverrides?

[andreyvelich]

cc @kubeflow/kubeflow-trainer-team @kubeflow/kubeflow-sdk-team @akshaychitneni