Package org.apache.flink.yarn

Class YarnClusterDescriptor

java.lang.Object
org.apache.flink.yarn.YarnClusterDescriptor
All Implemented Interfaces:
AutoCloseable, org.apache.flink.client.deployment.ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>
public class YarnClusterDescriptor extends Object implements org.apache.flink.client.deployment.ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>
The descriptor with deployment information for deploying a Flink cluster on Yarn.

  • Constructor Summary

    Constructors
    Constructor
    Description
    YarnClusterDescriptor(org.apache.flink.configuration.Configuration flinkConfiguration, org.apache.hadoop.yarn.conf.YarnConfiguration yarnConfiguration, org.apache.hadoop.yarn.client.api.YarnClient yarnClient, YarnClusterInformationRetriever yarnClusterInformationRetriever, boolean sharedYarnClient)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    addShipFiles(List<org.apache.hadoop.fs.Path> shipFiles)
    Adds the given files to the list of files to ship.
    void
    close()
     
    org.apache.flink.client.program.ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId>
    deployApplicationCluster(org.apache.flink.client.deployment.ClusterSpecification clusterSpecification, org.apache.flink.client.deployment.application.ApplicationConfiguration applicationConfiguration)
     
    org.apache.flink.client.program.ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId>
    deploySessionCluster(org.apache.flink.client.deployment.ClusterSpecification clusterSpecification)
     
    String
    getClusterDescription()
     
    org.apache.flink.configuration.Configuration
    getFlinkConfiguration()
     
    String
    getNodeLabel()
     
    org.apache.hadoop.yarn.client.api.YarnClient
    getYarnClient()
     
    protected String
    getYarnSessionClusterEntrypoint()
    The class to start the application master with.
    void
    killCluster(org.apache.hadoop.yarn.api.records.ApplicationId applicationId)
     
    static void
    logDetachedClusterInformation(org.apache.hadoop.yarn.api.records.ApplicationId yarnApplicationId, org.slf4j.Logger logger)
     
    org.apache.flink.client.program.ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId>
    retrieve(org.apache.hadoop.yarn.api.records.ApplicationId applicationId)
     
    void
    setLocalJarPath(org.apache.hadoop.fs.Path localJarPath)
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • YarnClusterDescriptor

      public YarnClusterDescriptor(org.apache.flink.configuration.Configuration flinkConfiguration, org.apache.hadoop.yarn.conf.YarnConfiguration yarnConfiguration, org.apache.hadoop.yarn.client.api.YarnClient yarnClient, YarnClusterInformationRetriever yarnClusterInformationRetriever, boolean sharedYarnClient)

  • Method Details

    • getYarnClient

      public org.apache.hadoop.yarn.client.api.YarnClient getYarnClient()

    • getYarnSessionClusterEntrypoint

      protected String getYarnSessionClusterEntrypoint()
      The class to start the application master with. This class runs the main method in case of session cluster.

    • getFlinkConfiguration

      public org.apache.flink.configuration.Configuration getFlinkConfiguration()

    • setLocalJarPath

      public void setLocalJarPath(org.apache.hadoop.fs.Path localJarPath)

    • addShipFiles

      public void addShipFiles(List<org.apache.hadoop.fs.Path> shipFiles)
      Adds the given files to the list of files to ship.

      Note that any file matching "flink-dist*.jar" will be excluded from the upload by YarnApplicationFileUploader.registerMultipleLocalResources(Collection, String, LocalResourceType) since we upload the Flink uber jar ourselves and do not need to deploy it multiple times.

      Parameters:
      shipFiles - files to ship

    • getNodeLabel

      public String getNodeLabel()

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface org.apache.flink.client.deployment.ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

    • retrieve

      public org.apache.flink.client.program.ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId> retrieve(org.apache.hadoop.yarn.api.records.ApplicationId applicationId) throws org.apache.flink.client.deployment.ClusterRetrieveException
      Specified by:
      retrieve in interface org.apache.flink.client.deployment.ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>
      Throws:
      org.apache.flink.client.deployment.ClusterRetrieveException

    • deploySessionCluster

      public org.apache.flink.client.program.ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId> deploySessionCluster(org.apache.flink.client.deployment.ClusterSpecification clusterSpecification) throws org.apache.flink.client.deployment.ClusterDeploymentException
      Specified by:
      deploySessionCluster in interface org.apache.flink.client.deployment.ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>
      Throws:
      org.apache.flink.client.deployment.ClusterDeploymentException

    • deployApplicationCluster

      public org.apache.flink.client.program.ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId> deployApplicationCluster(org.apache.flink.client.deployment.ClusterSpecification clusterSpecification, org.apache.flink.client.deployment.application.ApplicationConfiguration applicationConfiguration) throws org.apache.flink.client.deployment.ClusterDeploymentException
      Specified by:
      deployApplicationCluster in interface org.apache.flink.client.deployment.ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>
      Throws:
      org.apache.flink.client.deployment.ClusterDeploymentException

    • killCluster

      public void killCluster(org.apache.hadoop.yarn.api.records.ApplicationId applicationId) throws org.apache.flink.util.FlinkException
      Specified by:
      killCluster in interface org.apache.flink.client.deployment.ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>
      Throws:
      org.apache.flink.util.FlinkException

    • getClusterDescription

      public String getClusterDescription()
      Specified by:
      getClusterDescription in interface org.apache.flink.client.deployment.ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

    • logDetachedClusterInformation

      public static void logDetachedClusterInformation(org.apache.hadoop.yarn.api.records.ApplicationId yarnApplicationId, org.slf4j.Logger logger)