Package org.apache.flink.runtime.minicluster

Class MiniCluster

java.lang.Object
org.apache.flink.runtime.minicluster.MiniCluster
All Implemented Interfaces:
AutoCloseable, org.apache.flink.util.AutoCloseableAsync
public class MiniCluster extends Object implements org.apache.flink.util.AutoCloseableAsync
MiniCluster to execute Flink jobs locally.

  • Constructor Details

    • MiniCluster

      public MiniCluster(MiniClusterConfiguration miniClusterConfiguration)
      Creates a new Flink mini cluster based on the given configuration.
      Parameters:
      miniClusterConfiguration - The configuration for the mini cluster

    • MiniCluster

      public MiniCluster(MiniClusterConfiguration miniClusterConfiguration, Supplier<org.apache.flink.util.Reference<org.apache.flink.runtime.rpc.RpcSystem>> rpcSystemSupplier)

  • Method Details

    • getRestAddress

      public CompletableFuture<URI> getRestAddress()

    • getClusterInformation

      public ClusterInformation getClusterInformation()

    • getIOExecutor

      protected Executor getIOExecutor()

    • isRunning

      public boolean isRunning()
      Checks if the mini cluster was started and is running.

    • start

      public void start() throws Exception
      Starts the mini cluster, based on the configured properties.
      Throws:
      Exception - This method passes on any exception that occurs during the startup of the mini cluster.

    • createDispatcherResourceManagerComponents

      @VisibleForTesting protected Collection<? extends DispatcherResourceManagerComponent> createDispatcherResourceManagerComponents(org.apache.flink.configuration.Configuration configuration, MiniCluster.RpcServiceFactory rpcServiceFactory, BlobServer blobServer, HeartbeatServices heartbeatServices, DelegationTokenManager delegationTokenManager, MetricRegistry metricRegistry, MetricQueryServiceRetriever metricQueryServiceRetriever, org.apache.flink.runtime.rpc.FatalErrorHandler fatalErrorHandler) throws Exception
      Throws:
      Exception

    • createDispatcherResourceManagerComponentFactory

      protected DispatcherResourceManagerComponentFactory createDispatcherResourceManagerComponentFactory()

    • createHighAvailabilityServices

      @VisibleForTesting protected HighAvailabilityServices createHighAvailabilityServices(org.apache.flink.configuration.Configuration configuration, Executor executor) throws Exception
      Throws:
      Exception

    • getHaLeadershipControl

      public Optional<HaLeadershipControl> getHaLeadershipControl()
      Returns HaLeadershipControl if enabled.

      HaLeadershipControl allows granting and revoking leadership of HA components, e.g. JobManager. The method return Optional.empty() if the control is not enabled in MiniClusterConfiguration.

      Enabling this feature disables HighAvailabilityOptions.HA_MODE option.

    • getHaServices

      protected HighAvailabilityServices getHaServices()

    • closeAsync

      public CompletableFuture<Void> closeAsync()
      Shuts down the mini cluster, failing all currently executing jobs. The mini cluster can be started again by calling the start() method again.

      This method shuts down all started services and components, even if an exception occurs in the process of shutting down some component.

      Specified by:
      closeAsync in interface org.apache.flink.util.AutoCloseableAsync
      Returns:
      Future which is completed once the MiniCluster has been completely shut down

    • closeAsyncWithoutCleaningHighAvailabilityData

      public CompletableFuture<Void> closeAsyncWithoutCleaningHighAvailabilityData()

    • startTaskManager

      public void startTaskManager() throws Exception
      Starts additional TaskManager process.

      When the MiniCluster starts up, it always starts MiniClusterConfiguration.getNumTaskManagers() TaskManagers. All TaskManagers are indexed from 0 to the number of TaskManagers, started so far, minus one. This method starts a TaskManager with the next index which is the number of TaskManagers, started so far. The index always increases with each new started TaskManager. The indices of terminated TaskManagers are not reused after terminateTaskManager(int).

      Throws:
      Exception

    • useLocalCommunication

      @VisibleForTesting protected boolean useLocalCommunication()

    • getConfiguration

      @VisibleForTesting public org.apache.flink.configuration.Configuration getConfiguration()

    • overrideRestoreModeForChangelogStateBackend

      @Internal public void overrideRestoreModeForChangelogStateBackend()

    • terminateTaskManager

      public CompletableFuture<Void> terminateTaskManager(int index)
      Terminates a TaskManager with the given index.

      See startTaskManager() to understand how TaskManagers are indexed. This method terminates a TaskManager with a given index but it does not clear the index. The index stays occupied for the lifetime of the MiniCluster and its TaskManager stays terminated. The index is not reused if more TaskManagers are started with startTaskManager().

      Parameters:
      index - index of the TaskManager to terminate
      Returns:
      CompletableFuture of the given TaskManager termination

    • getArchivedExecutionGraph

      public CompletableFuture<ArchivedExecutionGraph> getArchivedExecutionGraph(org.apache.flink.api.common.JobID jobId)

    • listJobs

      public CompletableFuture<Collection<JobStatusMessage>> listJobs()

    • getJobStatus

      public CompletableFuture<org.apache.flink.api.common.JobStatus> getJobStatus(org.apache.flink.api.common.JobID jobId)

    • cancelJob

      public CompletableFuture<Acknowledge> cancelJob(org.apache.flink.api.common.JobID jobId)

    • triggerSavepoint

      public CompletableFuture<String> triggerSavepoint(org.apache.flink.api.common.JobID jobId, String targetDirectory, boolean cancelJob, org.apache.flink.core.execution.SavepointFormatType formatType)

    • triggerDetachedSavepoint

      public CompletableFuture<String> triggerDetachedSavepoint(org.apache.flink.api.common.JobID jobId, String targetDirectory, boolean cancelJob, org.apache.flink.core.execution.SavepointFormatType formatType)

    • triggerCheckpoint

      public CompletableFuture<String> triggerCheckpoint(org.apache.flink.api.common.JobID jobID)

    • triggerCheckpoint

      public CompletableFuture<Long> triggerCheckpoint(org.apache.flink.api.common.JobID jobID, org.apache.flink.core.execution.CheckpointType checkpointType)

    • stopWithSavepoint

      public CompletableFuture<String> stopWithSavepoint(org.apache.flink.api.common.JobID jobId, String targetDirectory, boolean terminate, org.apache.flink.core.execution.SavepointFormatType formatType)

    • stopWithDetachedSavepoint

      public CompletableFuture<String> stopWithDetachedSavepoint(org.apache.flink.api.common.JobID jobId, String targetDirectory, boolean terminate, org.apache.flink.core.execution.SavepointFormatType formatType)

    • disposeSavepoint

      public CompletableFuture<Acknowledge> disposeSavepoint(String savepointPath)

    • getExecutionGraph

      public CompletableFuture<? extends AccessExecutionGraph> getExecutionGraph(org.apache.flink.api.common.JobID jobId)

    • deliverCoordinationRequestToCoordinator

      public CompletableFuture<CoordinationResponse> deliverCoordinationRequestToCoordinator(org.apache.flink.api.common.JobID jobId, String operatorUid, org.apache.flink.util.SerializedValue<CoordinationRequest> serializedRequest)

    • getResourceOverview

      public CompletableFuture<ResourceOverview> getResourceOverview()

    • runDetached

      public void runDetached(JobGraph job) throws JobExecutionException, InterruptedException
      This method executes a job in detached mode. The method returns immediately after the job has been added to the
      Parameters:
      job - The Flink job to execute
      Throws:
      JobExecutionException - Thrown if anything went amiss during initial job launch, or if the job terminally failed.
      InterruptedException

    • executeJobBlocking

      public org.apache.flink.api.common.JobExecutionResult executeJobBlocking(JobGraph job) throws JobExecutionException, InterruptedException
      This method runs a job in blocking mode. The method returns only after the job completed successfully, or after it failed terminally.
      Parameters:
      job - The Flink job to execute
      Returns:
      The result of the job execution
      Throws:
      JobExecutionException - Thrown if anything went amiss during initial job launch, or if the job terminally failed.
      InterruptedException

    • submitJob

      public CompletableFuture<org.apache.flink.api.common.JobSubmissionResult> submitJob(ExecutionPlan executionPlan)

    • requestJobResult

      public CompletableFuture<JobResult> requestJobResult(org.apache.flink.api.common.JobID jobId)

    • requestClusterOverview

      public CompletableFuture<ClusterOverview> requestClusterOverview()

    • getDispatcherGatewayFuture

      @VisibleForTesting protected CompletableFuture<DispatcherGateway> getDispatcherGatewayFuture()

    • createMetricRegistry

      protected MetricRegistryImpl createMetricRegistry(org.apache.flink.configuration.Configuration config, long maximumMessageSizeInBytes)
      Factory method to create the metric registry for the mini cluster.
      Parameters:
      config - The configuration of the mini cluster
      maximumMessageSizeInBytes - the maximum message size

    • createRemoteRpcService

      protected org.apache.flink.runtime.rpc.RpcService createRemoteRpcService(org.apache.flink.configuration.Configuration configuration, String bindAddress, int bindPort, org.apache.flink.runtime.rpc.RpcSystem rpcSystem) throws Exception
      Factory method to instantiate the remote RPC service.
      Parameters:
      configuration - Flink configuration.
      bindAddress - The address to bind the RPC service to.
      bindPort - The port range to bind the RPC service to.
      rpcSystem -
      Returns:
      The instantiated RPC service
      Throws:
      Exception

    • createRemoteRpcService

      protected org.apache.flink.runtime.rpc.RpcService createRemoteRpcService(org.apache.flink.configuration.Configuration configuration, String externalAddress, String externalPortRange, String bindAddress, org.apache.flink.runtime.rpc.RpcSystem rpcSystem) throws Exception
      Factory method to instantiate the remote RPC service.
      Parameters:
      configuration - Flink configuration.
      externalAddress - The external address to access the RPC service.
      externalPortRange - The external port range to access the RPC service.
      bindAddress - The address to bind the RPC service to.
      rpcSystem -
      Returns:
      The instantiated RPC service
      Throws:
      Exception

    • createLocalRpcService

      protected org.apache.flink.runtime.rpc.RpcService createLocalRpcService(org.apache.flink.configuration.Configuration configuration, org.apache.flink.runtime.rpc.RpcSystem rpcSystem) throws Exception
      Factory method to instantiate the local RPC service.
      Parameters:
      configuration - Flink configuration.
      rpcSystem -
      Returns:
      The instantiated RPC service
      Throws:
      Exception

    • invalidateClusterDataset

      public CompletableFuture<Void> invalidateClusterDataset(org.apache.flink.util.AbstractID clusterDatasetId)

    • listCompletedClusterDatasetIds

      public CompletableFuture<Set<org.apache.flink.util.AbstractID>> listCompletedClusterDatasetIds()

    • reportHeartbeat

      public CompletableFuture<Void> reportHeartbeat(org.apache.flink.api.common.JobID jobId, long expiredTimestamp)