Package org.apache.flink.runtime.executiongraph

Class DefaultExecutionGraph

java.lang.Object

org.apache.flink.runtime.executiongraph.DefaultExecutionGraph

All Implemented Interfaces:

AccessExecutionGraph, ExecutionGraph, InternalExecutionGraphAccessor, JobStatusProvider

public class DefaultExecutionGraph
extends Object
implements ExecutionGraph, InternalExecutionGraphAccessor

Default implementation of the ExecutionGraph.

Constructor Summary

Constructors

Constructor

Description

DefaultExecutionGraph(JobInformation jobInformation, ScheduledExecutorService futureExecutor, Executor ioExecutor, Duration rpcTimeout, int executionHistorySizeLimit, ClassLoader userClassLoader, BlobWriter blobWriter, PartitionGroupReleaseStrategy.Factory partitionGroupReleaseStrategyFactory, ShuffleMaster<?> shuffleMaster, JobMasterPartitionTracker partitionTracker, ExecutionDeploymentListener executionDeploymentListener, ExecutionStateUpdateListener executionStateUpdateListener, long initializationTimestamp, VertexAttemptNumberStore initialAttemptCounts, VertexParallelismStore vertexParallelismStore, boolean isDynamic, ExecutionJobVertex.Factory executionJobVertexFactory, List<org.apache.flink.core.execution.JobStatusHook> jobStatusHooks, MarkPartitionFinishedStrategy markPartitionFinishedStrategy, TaskDeploymentDescriptorFactory taskDeploymentDescriptorFactory, List<org.apache.flink.core.execution.JobStatusChangedListener> jobStatusChangedListeners, ExecutionPlanSchedulingContext executionPlanSchedulingContext)

Method Summary

Modifier and Type	Method	Description
void	addNewJobVertices(List<JobVertex> topologicallySortedNewlyJobVertices, JobManagerJobMetricGroup jobManagerJobMetricGroup, VertexParallelismStore newVerticesParallelismStore)	Adds new job vertices to the execution graph based on the provided list of topologically sorted job vertices.
Map<String,org.apache.flink.util.OptionalFailure<org.apache.flink.api.common.accumulators.Accumulator<?,?>>>	aggregateUserAccumulators()	Merges all accumulator results from the tasks previously executed in the Executions.
void	attachJobGraph(List<JobVertex> verticesToAttach, JobManagerJobMetricGroup jobManagerJobMetricGroup)
void	cancel()
ResultPartitionID	createResultPartitionId(IntermediateResultPartitionID resultPartitionId)
void	deleteBlobs(List<PermanentBlobKey> blobKeys)
void	deregisterExecution(Execution exec)
void	enableCheckpointing(CheckpointCoordinatorConfiguration chkConfig, List<MasterTriggerRestoreHook<?>> masterHooks, CheckpointIDCounter checkpointIDCounter, CompletedCheckpointStore checkpointStore, StateBackend checkpointStateBackend, CheckpointStorage checkpointStorage, CheckpointStatsTracker statsTracker, CheckpointsCleaner checkpointsCleaner, String changelogStorageName)
void	failGlobal(Throwable t)	Fails the execution graph globally.
void	failJob(Throwable cause, long timestamp)
Optional<AccessExecution>	findExecution(ExecutionAttemptID attemptId)
Optional<String>	findVertexWithAttempt(ExecutionAttemptID attemptId)
StringifiedAccumulatorResult[]	getAccumulatorResultsStringified()	Returns the a stringified version of the user-defined accumulators.
Map<String,org.apache.flink.util.SerializedValue<org.apache.flink.util.OptionalFailure<Object>>>	getAccumulatorsSerialized()	Gets a serialized accumulator map.
Iterable<ExecutionVertex>	getAllExecutionVertices()	Returns an iterable containing all execution vertices for this execution graph.
Map<IntermediateDataSetID,IntermediateResult>	getAllIntermediateResults()
Map<JobVertexID,ExecutionJobVertex>	getAllVertices()	Returns a map containing all job vertices for this execution graph.
org.apache.flink.api.common.ArchivedExecutionConfig	getArchivedExecutionConfig()	Returns the serializable ArchivedExecutionConfig.
final BlobWriter	getBlobWriter()
Optional<String>	getChangelogStorageName()	Returns the changelog storage name for this ExecutionGraph.
CheckpointCoordinator	getCheckpointCoordinator()
CheckpointCoordinatorConfiguration	getCheckpointCoordinatorConfiguration()	Returns the CheckpointCoordinatorConfiguration or null if checkpointing is disabled.
CheckpointStatsSnapshot	getCheckpointStatsSnapshot()	Returns a snapshot of the checkpoint statistics or null if checkpointing is disabled.
Optional<String>	getCheckpointStorageName()	Returns the checkpoint storage name for this ExecutionGraph.
List<ShuffleDescriptor>	getClusterPartitionShuffleDescriptors(IntermediateDataSetID intermediateDataSetID)	Get the shuffle descriptors of the cluster partitions ordered by partition number.
EdgeManager	getEdgeManager()
ExecutionDeploymentListener	getExecutionDeploymentListener()
ExecutionGraphID	getExecutionGraphID()
ExecutionVertex	getExecutionVertexOrThrow(ExecutionVertexID id)
Throwable	getFailureCause()
ErrorInfo	getFailureInfo()	Returns the exception that caused the job to fail.
Executor	getFutureExecutor()	Returns the ExecutionContext associated with this ExecutionGraph.
org.apache.flink.configuration.Configuration	getJobConfiguration()
org.apache.flink.api.common.JobID	getJobID()	Returns the JobID for this execution graph.
org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor	getJobMasterMainThreadExecutor()
String	getJobName()	Returns the job name for the execution graph.
JobType	getJobType()	Returns the JobType for this execution graph.
ExecutionJobVertex	getJobVertex(JobVertexID id)	Returns the job vertex for the given JobVertexID.
JobVertexInputInfo	getJobVertexInputInfo(JobVertexID jobVertexId, IntermediateDataSetID resultId)	Get the input info of a certain input of a certain job vertex.
String	getJsonPlan()	Returns the job plan as a JSON string.
KvStateLocationRegistry	getKvStateLocationRegistry()
MarkPartitionFinishedStrategy	getMarkPartitionFinishedStrategy()
long	getNumberOfRestarts()	Gets the number of restarts, including full restarts and fine grained restarts.
int	getNumFinishedVertices()
PartitionGroupReleaseStrategy	getPartitionGroupReleaseStrategy()
JobMasterPartitionTracker	getPartitionTracker()
int	getPendingOperatorCount()	Retrieves the count of pending operators waiting to be transferred to job vertices in the adaptive execution of batch jobs.
Map<ExecutionAttemptID,Execution>	getRegisteredExecutions()
ResultPartitionAvailabilityChecker	getResultPartitionAvailabilityChecker()
IntermediateResultPartition	getResultPartitionOrThrow(IntermediateResultPartitionID id)	Gets the intermediate result partition by the given partition ID, or throw an exception if the partition is not found.
SchedulingTopology	getSchedulingTopology()
ShuffleMaster<?>	getShuffleMaster()
org.apache.flink.api.common.JobStatus	getState()	Returns the current JobStatus for this execution graph.
Optional<String>	getStateBackendName()	Returns the state backend name for this ExecutionGraph.
long	getStatusTimestamp(org.apache.flink.api.common.JobStatus status)	Returns the timestamp for the given JobStatus.
String	getStreamGraphJson()	Returns the stream graph as a JSON string.
TaskDeploymentDescriptorFactory	getTaskDeploymentDescriptorFactory()
CompletableFuture<org.apache.flink.api.common.JobStatus>	getTerminationFuture()	Returns the termination future of this ExecutionGraph.
ClassLoader	getUserClassLoader()
Iterable<ExecutionJobVertex>	getVerticesTopologically()	Returns an iterable containing all job vertices for this execution graph in the order they were created.
void	incrementRestarts()
void	initFailureCause(Throwable t, long timestamp)
void	initializeJobVertex(ExecutionJobVertex ejv, long createTimestamp, Map<IntermediateDataSetID,JobVertexInputInfo> jobVertexInputInfos)	Initialize the given execution job vertex, mainly includes creating execution vertices according to the parallelism, and connecting to the predecessors.
org.apache.flink.util.TernaryBoolean	isChangelogStateBackendEnabled()	Returns whether the state changelog is enabled for this ExecutionGraph.
boolean	isDynamic()
boolean	isStoppable()	Returns whether the job for this execution graph is stoppable.
void	jobVertexFinished()	Called whenever a job vertex reaches state FINISHED (completed successfully).
void	jobVertexUnFinished()
void	notifyExecutionChange(Execution execution, ExecutionState previousState, ExecutionState newExecutionState)
void	notifyNewlyInitializedJobVertices(List<ExecutionJobVertex> vertices)	Notify that some job vertices have been newly initialized, execution graph will try to update scheduling topology.
void	notifySchedulerNgAboutInternalTaskFailure(ExecutionAttemptID attemptId, Throwable t, boolean cancelTask, boolean releasePartitions)
void	registerExecution(Execution exec)
void	registerJobStatusListener(JobStatusListener listener)
void	setInternalTaskFailuresListener(InternalFailuresListener internalTaskFailuresListener)
void	setJsonPlan(String jsonPlan)
void	start(org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor jobMasterMainThreadExecutor)
void	suspend(Throwable suspensionCause)	Suspends the current ExecutionGraph.
boolean	transitionState(org.apache.flink.api.common.JobStatus current, org.apache.flink.api.common.JobStatus newState)
void	transitionToRunning()
void	updateAccumulators(AccumulatorSnapshot accumulatorSnapshot)	Updates the accumulators during the runtime of a job.
boolean	updateState(TaskExecutionStateTransition state)	Updates the state of one of the ExecutionVertex's Execution attempts.
org.apache.flink.api.common.JobStatus	waitUntilTerminal()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.executiongraph.ExecutionGraph

initializeJobVertex

Constructor Details

DefaultExecutionGraph

public DefaultExecutionGraph(JobInformation jobInformation,
 ScheduledExecutorService futureExecutor,
 Executor ioExecutor,
 Duration rpcTimeout,
 int executionHistorySizeLimit,
 ClassLoader userClassLoader,
 BlobWriter blobWriter,
 PartitionGroupReleaseStrategy.Factory partitionGroupReleaseStrategyFactory,
 ShuffleMaster<?> shuffleMaster,
 JobMasterPartitionTracker partitionTracker,
 ExecutionDeploymentListener executionDeploymentListener,
 ExecutionStateUpdateListener executionStateUpdateListener,
 long initializationTimestamp,
 VertexAttemptNumberStore initialAttemptCounts,
 VertexParallelismStore vertexParallelismStore,
 boolean isDynamic,
 ExecutionJobVertex.Factory executionJobVertexFactory,
 List<org.apache.flink.core.execution.JobStatusHook> jobStatusHooks,
 MarkPartitionFinishedStrategy markPartitionFinishedStrategy,
 TaskDeploymentDescriptorFactory taskDeploymentDescriptorFactory,
 List<org.apache.flink.core.execution.JobStatusChangedListener> jobStatusChangedListeners,
 ExecutionPlanSchedulingContext executionPlanSchedulingContext)

Method Details

start

public void start(@Nonnull
 org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor jobMasterMainThreadExecutor)

Specified by:

start in interface ExecutionGraph

getSchedulingTopology

public SchedulingTopology getSchedulingTopology()

Specified by:

getSchedulingTopology in interface ExecutionGraph

getJobMasterMainThreadExecutor

@Nonnull
public org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor getJobMasterMainThreadExecutor()

Specified by:

getJobMasterMainThreadExecutor in interface ExecutionGraph

Specified by:

getJobMasterMainThreadExecutor in interface InternalExecutionGraphAccessor

isChangelogStateBackendEnabled

public org.apache.flink.util.TernaryBoolean isChangelogStateBackendEnabled()

Description copied from interface: AccessExecutionGraph

Returns whether the state changelog is enabled for this ExecutionGraph.

Specified by:

isChangelogStateBackendEnabled in interface AccessExecutionGraph

Returns:

true, if state changelog enabled, false otherwise.

getStateBackendName

public Optional<String> getStateBackendName()

Description copied from interface: AccessExecutionGraph

Returns the state backend name for this ExecutionGraph.

Specified by:

getStateBackendName in interface AccessExecutionGraph

Returns:

The state backend name, or an empty Optional in the case of batch jobs

getCheckpointStorageName

public Optional<String> getCheckpointStorageName()

Description copied from interface: AccessExecutionGraph

Returns the checkpoint storage name for this ExecutionGraph.

Specified by:

getCheckpointStorageName in interface AccessExecutionGraph

Returns:

The checkpoint storage name, or an empty Optional in the case of batch jobs

getChangelogStorageName

public Optional<String> getChangelogStorageName()

Description copied from interface: AccessExecutionGraph

Returns the changelog storage name for this ExecutionGraph.

Specified by:

getChangelogStorageName in interface AccessExecutionGraph

Returns:

The changelog storage name, or an empty Optional in the case of batch jobs

getPendingOperatorCount

public int getPendingOperatorCount()

Description copied from interface: AccessExecutionGraph

Retrieves the count of pending operators waiting to be transferred to job vertices in the adaptive execution of batch jobs. This value will be zero if the job is submitted with a JobGraph or if it's a streaming job.

Specified by:

getPendingOperatorCount in interface AccessExecutionGraph

Returns:

the number of pending operators.

enableCheckpointing

public void enableCheckpointing(CheckpointCoordinatorConfiguration chkConfig,
 List<MasterTriggerRestoreHook<?>> masterHooks,
 CheckpointIDCounter checkpointIDCounter,
 CompletedCheckpointStore checkpointStore,
 StateBackend checkpointStateBackend,
 CheckpointStorage checkpointStorage,
 CheckpointStatsTracker statsTracker,
 CheckpointsCleaner checkpointsCleaner,
 String changelogStorageName)

Specified by:

enableCheckpointing in interface ExecutionGraph

getCheckpointCoordinator

@Nullable
public CheckpointCoordinator getCheckpointCoordinator()

Specified by:

getCheckpointCoordinator in interface ExecutionGraph

getKvStateLocationRegistry

public KvStateLocationRegistry getKvStateLocationRegistry()

Specified by:

getKvStateLocationRegistry in interface ExecutionGraph

getCheckpointCoordinatorConfiguration

public CheckpointCoordinatorConfiguration getCheckpointCoordinatorConfiguration()

Description copied from interface: AccessExecutionGraph

Returns the CheckpointCoordinatorConfiguration or null if checkpointing is disabled.

Specified by:

getCheckpointCoordinatorConfiguration in interface AccessExecutionGraph

Returns:

JobCheckpointingConfiguration for this execution graph

getCheckpointStatsSnapshot

public CheckpointStatsSnapshot getCheckpointStatsSnapshot()

Description copied from interface: AccessExecutionGraph

Returns a snapshot of the checkpoint statistics or null if checkpointing is disabled.

Specified by:

getCheckpointStatsSnapshot in interface AccessExecutionGraph

Returns:

Snapshot of the checkpoint statistics for this execution graph

setJsonPlan

public void setJsonPlan(String jsonPlan)

Specified by:

setJsonPlan in interface ExecutionGraph

getStreamGraphJson

public String getStreamGraphJson()

Description copied from interface: AccessExecutionGraph

Returns the stream graph as a JSON string.

Specified by:

getStreamGraphJson in interface AccessExecutionGraph

Returns:

stream graph as a JSON string, or null if the job is submitted with a JobGraph or if it's a streaming job.

getJsonPlan

public String getJsonPlan()

Description copied from interface: AccessExecutionGraph

Returns the job plan as a JSON string.

Specified by:

getJsonPlan in interface AccessExecutionGraph

Returns:

job plan as a JSON string

getJobID

public org.apache.flink.api.common.JobID getJobID()

Description copied from interface: AccessExecutionGraph

Returns the JobID for this execution graph.

Specified by:

getJobID in interface AccessExecutionGraph

Specified by:

getJobID in interface InternalExecutionGraphAccessor

Returns:

job ID for this execution graph

getJobName

public String getJobName()

Description copied from interface: AccessExecutionGraph

Returns the job name for the execution graph.

Specified by:

getJobName in interface AccessExecutionGraph

Returns:

job name for this execution graph

isStoppable

public boolean isStoppable()

Description copied from interface: AccessExecutionGraph

Returns whether the job for this execution graph is stoppable.

Specified by:

isStoppable in interface AccessExecutionGraph

Returns:

true, if all sources tasks are stoppable, false otherwise

getJobConfiguration

public org.apache.flink.configuration.Configuration getJobConfiguration()

Specified by:

getJobConfiguration in interface ExecutionGraph

getUserClassLoader

public ClassLoader getUserClassLoader()

Specified by:

getUserClassLoader in interface InternalExecutionGraphAccessor

getState

public org.apache.flink.api.common.JobStatus getState()

Description copied from interface: AccessExecutionGraph

Returns the current JobStatus for this execution graph.

Specified by:

getState in interface AccessExecutionGraph

Specified by:

getState in interface JobStatusProvider

Returns:

job status for this execution graph

getJobType

public JobType getJobType()

Description copied from interface: AccessExecutionGraph

Returns the JobType for this execution graph.

Specified by:

getJobType in interface AccessExecutionGraph

Returns:

job type for this execution graph. It may be null when an exception occurs.

getFailureCause

public Throwable getFailureCause()

Specified by:

getFailureCause in interface ExecutionGraph

getFailureInfo

public ErrorInfo getFailureInfo()

Description copied from interface: AccessExecutionGraph

Returns the exception that caused the job to fail. This is the first root exception that was not recoverable and triggered job failure.

Specified by:

getFailureInfo in interface AccessExecutionGraph

Returns:

failure causing exception, or null

getNumberOfRestarts

public long getNumberOfRestarts()

Description copied from interface: ExecutionGraph

Gets the number of restarts, including full restarts and fine grained restarts. If a recovery is currently pending, this recovery is included in the count.

Specified by:

getNumberOfRestarts in interface ExecutionGraph

Returns:

The number of restarts so far

getNumFinishedVertices

public int getNumFinishedVertices()

Specified by:

getNumFinishedVertices in interface ExecutionGraph

getJobVertex

public ExecutionJobVertex getJobVertex(JobVertexID id)

Description copied from interface: AccessExecutionGraph

Returns the job vertex for the given JobVertexID.

Specified by:

getJobVertex in interface AccessExecutionGraph

Specified by:

getJobVertex in interface ExecutionGraph

Specified by:

getJobVertex in interface InternalExecutionGraphAccessor

Parameters:

id - id of job vertex to be returned

Returns:

job vertex for the given id, or null

getAllVertices

public Map<JobVertexID,ExecutionJobVertex> getAllVertices()

Description copied from interface: AccessExecutionGraph

Returns a map containing all job vertices for this execution graph.

Specified by:

getAllVertices in interface AccessExecutionGraph

Specified by:

getAllVertices in interface ExecutionGraph

Returns:

map containing all job vertices for this execution graph

getVerticesTopologically

public Iterable<ExecutionJobVertex> getVerticesTopologically()

Description copied from interface: AccessExecutionGraph

Returns an iterable containing all job vertices for this execution graph in the order they were created.

Specified by:

getVerticesTopologically in interface AccessExecutionGraph

Specified by:

getVerticesTopologically in interface ExecutionGraph

Returns:

iterable containing all job vertices for this execution graph in the order they were created

getAllIntermediateResults

public Map<IntermediateDataSetID,IntermediateResult> getAllIntermediateResults()

Specified by:

getAllIntermediateResults in interface ExecutionGraph

getAllExecutionVertices

public Iterable<ExecutionVertex> getAllExecutionVertices()

Description copied from interface: AccessExecutionGraph

Returns an iterable containing all execution vertices for this execution graph.

Specified by:

getAllExecutionVertices in interface AccessExecutionGraph

Specified by:

getAllExecutionVertices in interface ExecutionGraph

Returns:

iterable containing all execution vertices for this execution graph

getEdgeManager

public EdgeManager getEdgeManager()

Specified by:

getEdgeManager in interface InternalExecutionGraphAccessor

getExecutionVertexOrThrow

public ExecutionVertex getExecutionVertexOrThrow(ExecutionVertexID id)

Specified by:

getExecutionVertexOrThrow in interface InternalExecutionGraphAccessor

getResultPartitionOrThrow

public IntermediateResultPartition getResultPartitionOrThrow(IntermediateResultPartitionID id)

Description copied from interface: ExecutionGraph

Gets the intermediate result partition by the given partition ID, or throw an exception if the partition is not found.

Specified by:

getResultPartitionOrThrow in interface ExecutionGraph

Specified by:

getResultPartitionOrThrow in interface InternalExecutionGraphAccessor

Parameters:

id - of the intermediate result partition

Returns:

intermediate result partition

getStatusTimestamp

public long getStatusTimestamp(org.apache.flink.api.common.JobStatus status)

Description copied from interface: AccessExecutionGraph

Returns the timestamp for the given JobStatus.

Specified by:

getStatusTimestamp in interface AccessExecutionGraph

Specified by:

getStatusTimestamp in interface JobStatusProvider

Parameters:

status - status for which the timestamp should be returned

Returns:

timestamp for the given job status

getBlobWriter

public final BlobWriter getBlobWriter()

Specified by:

getBlobWriter in interface InternalExecutionGraphAccessor

getFutureExecutor

public Executor getFutureExecutor()

Description copied from interface: InternalExecutionGraphAccessor

Returns the ExecutionContext associated with this ExecutionGraph.

Specified by:

getFutureExecutor in interface InternalExecutionGraphAccessor

Returns:

ExecutionContext associated with this ExecutionGraph

aggregateUserAccumulators

public Map<String,org.apache.flink.util.OptionalFailure<org.apache.flink.api.common.accumulators.Accumulator<?,?>>> aggregateUserAccumulators()

Description copied from interface: ExecutionGraph

Merges all accumulator results from the tasks previously executed in the Executions.

Specified by:

aggregateUserAccumulators in interface ExecutionGraph

Returns:

The accumulator map

getAccumulatorsSerialized

public Map<String,org.apache.flink.util.SerializedValue<org.apache.flink.util.OptionalFailure<Object>>> getAccumulatorsSerialized()

Gets a serialized accumulator map.

Specified by:

getAccumulatorsSerialized in interface AccessExecutionGraph

Returns:

The accumulator map with serialized accumulator values.

getAccumulatorResultsStringified

public StringifiedAccumulatorResult[] getAccumulatorResultsStringified()

Returns the a stringified version of the user-defined accumulators.

Specified by:

getAccumulatorResultsStringified in interface AccessExecutionGraph

Returns:

an Array containing the StringifiedAccumulatorResult objects

setInternalTaskFailuresListener

public void setInternalTaskFailuresListener(InternalFailuresListener internalTaskFailuresListener)

Specified by:

setInternalTaskFailuresListener in interface ExecutionGraph

notifyNewlyInitializedJobVertices

public void notifyNewlyInitializedJobVertices(List<ExecutionJobVertex> vertices)

Description copied from interface: ExecutionGraph

Notify that some job vertices have been newly initialized, execution graph will try to update scheduling topology.

Specified by:

notifyNewlyInitializedJobVertices in interface ExecutionGraph

Parameters:

vertices - The execution job vertices that are newly initialized.

attachJobGraph

public void attachJobGraph(List<JobVertex> verticesToAttach,
 JobManagerJobMetricGroup jobManagerJobMetricGroup)
                    throws JobException

Specified by:

attachJobGraph in interface ExecutionGraph

Throws:

JobException

addNewJobVertices

public void addNewJobVertices(List<JobVertex> topologicallySortedNewlyJobVertices,
 JobManagerJobMetricGroup jobManagerJobMetricGroup,
 VertexParallelismStore newVerticesParallelismStore)
                       throws JobException

Description copied from interface: ExecutionGraph

Adds new job vertices to the execution graph based on the provided list of topologically sorted job vertices.

Specified by:

addNewJobVertices in interface ExecutionGraph

Parameters:

topologicallySortedNewlyJobVertices - a list of job vertices that are to be added, defined in topological order.

jobManagerJobMetricGroup - the metric group associated with the job manager for monitoring and metrics collection.

newVerticesParallelismStore - a store that maintains parallelism information for the newly added job vertices.

Throws:

JobException

initializeJobVertex

public void initializeJobVertex(ExecutionJobVertex ejv,
 long createTimestamp,
 Map<IntermediateDataSetID,JobVertexInputInfo> jobVertexInputInfos)
                         throws JobException

Description copied from interface: ExecutionGraph

Initialize the given execution job vertex, mainly includes creating execution vertices according to the parallelism, and connecting to the predecessors.

Specified by:

initializeJobVertex in interface ExecutionGraph

Parameters:

ejv - The execution job vertex that needs to be initialized.

createTimestamp - The timestamp for creating execution vertices, used to initialize the first Execution with.

jobVertexInputInfos - The input infos of this job vertex.

Throws:

JobException

transitionToRunning

public void transitionToRunning()

Specified by:

transitionToRunning in interface ExecutionGraph

cancel

public void cancel()

Specified by:

cancel in interface ExecutionGraph

suspend

public void suspend(Throwable suspensionCause)

Description copied from interface: ExecutionGraph

Suspends the current ExecutionGraph.

The JobStatus will be directly set to JobStatus.SUSPENDED iff the current state is not a terminal state. All ExecutionJobVertices will be canceled and the onTerminalState() is executed.

The JobStatus.SUSPENDED state is a local terminal state which stops the execution of the job but does not remove the job from the HA job store so that it can be recovered by another JobManager.

Specified by:

suspend in interface ExecutionGraph

Parameters:

suspensionCause - Cause of the suspension

failGlobal

public void failGlobal(Throwable t)

Description copied from interface: InternalExecutionGraphAccessor

Fails the execution graph globally.

This global failure is meant to be triggered in cases where the consistency of the execution graph' state cannot be guaranteed any more (for example when catching unexpected exceptions that indicate a bug or an unexpected call race), and where a full restart is the safe way to get consistency back.

Specified by:

failGlobal in interface InternalExecutionGraphAccessor

Parameters:

t - The exception that caused the failure.

getArchivedExecutionConfig

public org.apache.flink.api.common.ArchivedExecutionConfig getArchivedExecutionConfig()

Returns the serializable ArchivedExecutionConfig.

Specified by:

getArchivedExecutionConfig in interface AccessExecutionGraph

Returns:

ArchivedExecutionConfig which may be null in case of errors

getTerminationFuture

public CompletableFuture<org.apache.flink.api.common.JobStatus> getTerminationFuture()

Description copied from interface: ExecutionGraph

Returns the termination future of this ExecutionGraph. The termination future is completed with the terminal JobStatus once the ExecutionGraph reaches this terminal state and all Execution have been terminated.

Specified by:

getTerminationFuture in interface ExecutionGraph

Returns:

Termination future of this ExecutionGraph.

waitUntilTerminal

@VisibleForTesting
public org.apache.flink.api.common.JobStatus waitUntilTerminal()
                                                        throws InterruptedException

Specified by:

waitUntilTerminal in interface ExecutionGraph

Throws:

InterruptedException

transitionState

public boolean transitionState(org.apache.flink.api.common.JobStatus current,
 org.apache.flink.api.common.JobStatus newState)

Specified by:

transitionState in interface ExecutionGraph

incrementRestarts

public void incrementRestarts()

Specified by:

incrementRestarts in interface ExecutionGraph

initFailureCause

public void initFailureCause(Throwable t,
 long timestamp)

Specified by:

initFailureCause in interface ExecutionGraph

jobVertexFinished

public void jobVertexFinished()

Called whenever a job vertex reaches state FINISHED (completed successfully). Once all job vertices are in the FINISHED state, the program is successfully done.

Specified by:

jobVertexFinished in interface InternalExecutionGraphAccessor

jobVertexUnFinished

public void jobVertexUnFinished()

Specified by:

jobVertexUnFinished in interface InternalExecutionGraphAccessor

failJob

public void failJob(Throwable cause,
 long timestamp)

Specified by:

failJob in interface ExecutionGraph

updateState

public boolean updateState(TaskExecutionStateTransition state)

Description copied from interface: ExecutionGraph

Updates the state of one of the ExecutionVertex's Execution attempts. If the new status if "FINISHED", this also updates the accumulators.

Specified by:

updateState in interface ExecutionGraph

Parameters:

state - The state update.

Returns:

True, if the task update was properly applied, false, if the execution attempt was not found.

createResultPartitionId

@VisibleForTesting
public ResultPartitionID createResultPartitionId(IntermediateResultPartitionID resultPartitionId)

getRegisteredExecutions

public Map<ExecutionAttemptID,Execution> getRegisteredExecutions()

Specified by:

getRegisteredExecutions in interface ExecutionGraph

registerExecution

public void registerExecution(Execution exec)

Specified by:

registerExecution in interface InternalExecutionGraphAccessor

deregisterExecution

public void deregisterExecution(Execution exec)

Specified by:

deregisterExecution in interface InternalExecutionGraphAccessor

updateAccumulators

public void updateAccumulators(AccumulatorSnapshot accumulatorSnapshot)

Description copied from interface: ExecutionGraph

Updates the accumulators during the runtime of a job. Final accumulator results are transferred through the UpdateTaskExecutionState message.

Specified by:

updateAccumulators in interface ExecutionGraph

Parameters:

accumulatorSnapshot - The serialized flink and user-defined accumulators

registerJobStatusListener

public void registerJobStatusListener(JobStatusListener listener)

Specified by:

registerJobStatusListener in interface ExecutionGraph

notifyExecutionChange

public void notifyExecutionChange(Execution execution,
 ExecutionState previousState,
 ExecutionState newExecutionState)

Specified by:

notifyExecutionChange in interface InternalExecutionGraphAccessor

notifySchedulerNgAboutInternalTaskFailure

public void notifySchedulerNgAboutInternalTaskFailure(ExecutionAttemptID attemptId,
 Throwable t,
 boolean cancelTask,
 boolean releasePartitions)

Specified by:

notifySchedulerNgAboutInternalTaskFailure in interface InternalExecutionGraphAccessor

deleteBlobs

public void deleteBlobs(List<PermanentBlobKey> blobKeys)

Specified by:

deleteBlobs in interface InternalExecutionGraphAccessor

getShuffleMaster

public ShuffleMaster<?> getShuffleMaster()

Specified by:

getShuffleMaster in interface InternalExecutionGraphAccessor

getPartitionTracker

public JobMasterPartitionTracker getPartitionTracker()

Specified by:

getPartitionTracker in interface InternalExecutionGraphAccessor

getResultPartitionAvailabilityChecker

public ResultPartitionAvailabilityChecker getResultPartitionAvailabilityChecker()

Specified by:

getResultPartitionAvailabilityChecker in interface ExecutionGraph

getPartitionGroupReleaseStrategy

public PartitionGroupReleaseStrategy getPartitionGroupReleaseStrategy()

Specified by:

getPartitionGroupReleaseStrategy in interface InternalExecutionGraphAccessor

getExecutionDeploymentListener

public ExecutionDeploymentListener getExecutionDeploymentListener()

Specified by:

getExecutionDeploymentListener in interface InternalExecutionGraphAccessor

isDynamic

public boolean isDynamic()

Specified by:

isDynamic in interface InternalExecutionGraphAccessor

findVertexWithAttempt

public Optional<String> findVertexWithAttempt(ExecutionAttemptID attemptId)

Specified by:

findVertexWithAttempt in interface ExecutionGraph

findExecution

public Optional<AccessExecution> findExecution(ExecutionAttemptID attemptId)

Specified by:

findExecution in interface ExecutionGraph

getExecutionGraphID

public ExecutionGraphID getExecutionGraphID()

Specified by:

getExecutionGraphID in interface InternalExecutionGraphAccessor

getClusterPartitionShuffleDescriptors

public List<ShuffleDescriptor> getClusterPartitionShuffleDescriptors(IntermediateDataSetID intermediateDataSetID)

Description copied from interface: InternalExecutionGraphAccessor

Get the shuffle descriptors of the cluster partitions ordered by partition number.

Specified by:

getClusterPartitionShuffleDescriptors in interface InternalExecutionGraphAccessor

getMarkPartitionFinishedStrategy

public MarkPartitionFinishedStrategy getMarkPartitionFinishedStrategy()

Specified by:

getMarkPartitionFinishedStrategy in interface InternalExecutionGraphAccessor

getJobVertexInputInfo

public JobVertexInputInfo getJobVertexInputInfo(JobVertexID jobVertexId,
 IntermediateDataSetID resultId)

Description copied from interface: InternalExecutionGraphAccessor

Get the input info of a certain input of a certain job vertex.

Specified by:

getJobVertexInputInfo in interface InternalExecutionGraphAccessor

Parameters:

jobVertexId - the job vertex id

resultId - the input(intermediate result) id

Returns:

the input info

getTaskDeploymentDescriptorFactory

public TaskDeploymentDescriptorFactory getTaskDeploymentDescriptorFactory()

Specified by:

getTaskDeploymentDescriptorFactory in interface InternalExecutionGraphAccessor