Package org.apache.flink.yarn

Class Utils

java.lang.Object
org.apache.flink.yarn.Utils
public final class Utils extends Object
Utility class that provides helper methods to work with Apache Hadoop YARN.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final String
    KRB5_FILE_NAME
    KRB5 file name populated in YARN container for secure IT run.
    static final String
    YARN_SITE_FILE_NAME
    Yarn site xml file name populated in YARN container for secure IT run.

  • Method Summary

    Modifier and Type
    Method
    Description
    static void
    addToEnvironment(Map<String,String> environment, String variable, String value)
    Copied method from org.apache.hadoop.yarn.util.Apps.
    static void
    concatWithSpace(StringBuilder sb, String value)
     
    static void
    deleteApplicationFiles(String applicationFilesDir)
    Deletes the YARN application files, e.g., Flink binaries, libraries, etc., from the remote filesystem.
    static String
    generateJvmOptsString(org.apache.flink.configuration.Configuration conf, List<org.apache.flink.configuration.ConfigOption<String>> jvmOptions, boolean hasKrb5)
     
    static org.apache.hadoop.fs.Path
    getPathFromLocalFile(File localFile)
     
    static org.apache.hadoop.fs.Path
    getPathFromLocalFilePathStr(String localPathStr)
     
    static List<org.apache.hadoop.fs.Path>
    getQualifiedRemoteProvidedLibDirs(org.apache.flink.configuration.Configuration configuration, org.apache.hadoop.yarn.conf.YarnConfiguration yarnConfiguration)
     
    static Optional<org.apache.hadoop.fs.Path>
    getQualifiedRemoteProvidedUsrLib(org.apache.flink.configuration.Configuration configuration, org.apache.hadoop.yarn.conf.YarnConfiguration yarnConfiguration)
     
    static String
    getStartCommand(String template, Map<String,String> startCommandValues)
    Replaces placeholders in the template start command with values from startCommandValues.
    static String
    getTaskManagerShellCommand(org.apache.flink.configuration.Configuration flinkConfig, org.apache.flink.runtime.clusterframework.ContaineredTaskManagerParameters tmParams, String configDirectory, String logDirectory, boolean hasLogback, boolean hasLog4j, boolean hasKrb5, Class<?> mainClass, String mainArgs)
    Generates the shell command to start a task manager.
    static org.apache.hadoop.yarn.conf.YarnConfiguration
    getYarnAndHadoopConfiguration(org.apache.flink.configuration.Configuration flinkConfig)
     
    static org.apache.hadoop.yarn.conf.YarnConfiguration
    getYarnConfiguration(org.apache.flink.configuration.Configuration flinkConfig)
    Add additional config entries from the flink config to the yarn config.
    static boolean
    isUsrLibDirectory(org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path path)
     
    static String
    resolveKeytabPath(String workingDir, String keytabPath)
    Resolve keytab path either as absolute path or relative to working directory.
    static void
    setAclsFor(org.apache.hadoop.yarn.api.records.ContainerLaunchContext amContainer, org.apache.flink.configuration.Configuration flinkConfig)
    Sets the application ACLs for the given ContainerLaunchContext based on the values specified in the given Flink configuration.
    static void
    setupYarnClassPath(org.apache.hadoop.conf.Configuration conf, Map<String,String> appMasterEnv)
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • KRB5_FILE_NAME

      public static final String KRB5_FILE_NAME
      KRB5 file name populated in YARN container for secure IT run.
      See Also:

    • YARN_SITE_FILE_NAME

      public static final String YARN_SITE_FILE_NAME
      Yarn site xml file name populated in YARN container for secure IT run.
      See Also:

  • Method Details

    • setupYarnClassPath

      public static void setupYarnClassPath(org.apache.hadoop.conf.Configuration conf, Map<String,String> appMasterEnv)

    • deleteApplicationFiles

      public static void deleteApplicationFiles(String applicationFilesDir)
      Deletes the YARN application files, e.g., Flink binaries, libraries, etc., from the remote filesystem.
      Parameters:
      applicationFilesDir - The application files directory.

    • addToEnvironment

      public static void addToEnvironment(Map<String,String> environment, String variable, String value)
      Copied method from org.apache.hadoop.yarn.util.Apps. It was broken by YARN-1824 (2.4.0) and fixed for 2.4.1 by https://issues.apache.org/jira/browse/YARN-1931

    • resolveKeytabPath

      public static String resolveKeytabPath(String workingDir, String keytabPath)
      Resolve keytab path either as absolute path or relative to working directory.
      Parameters:
      workingDir - current working directory
      keytabPath - configured keytab path.
      Returns:
      resolved keytab path, or null if not found.

    • getTaskManagerShellCommand

      public static String getTaskManagerShellCommand(org.apache.flink.configuration.Configuration flinkConfig, org.apache.flink.runtime.clusterframework.ContaineredTaskManagerParameters tmParams, String configDirectory, String logDirectory, boolean hasLogback, boolean hasLog4j, boolean hasKrb5, Class<?> mainClass, String mainArgs)
      Generates the shell command to start a task manager.
      Parameters:
      flinkConfig - The Flink configuration.
      tmParams - Parameters for the task manager.
      configDirectory - The configuration directory for the config.yaml
      logDirectory - The log directory.
      hasLogback - Uses logback?
      hasLog4j - Uses log4j?
      mainClass - The main class to start with.
      Returns:
      A String containing the task manager startup command.

    • getStartCommand

      public static String getStartCommand(String template, Map<String,String> startCommandValues)
      Replaces placeholders in the template start command with values from startCommandValues.

      If the default template ConfigConstants.DEFAULT_YARN_CONTAINER_START_COMMAND_TEMPLATE is used, the following keys must be present in the map or the resulting command will still contain placeholders:

      • java = path to the Java executable
      • jvmmem = JVM memory limits and tweaks
      • jvmopts = misc options for the Java VM
      • logging = logging-related configuration settings
      • class = main class to execute
      • args = arguments for the main class
      • redirects = output redirects
      Parameters:
      template - a template start command with placeholders
      startCommandValues - a replacement map placeholder -> value
      Returns:
      the start command with placeholders filled in

    • generateJvmOptsString

      public static String generateJvmOptsString(org.apache.flink.configuration.Configuration conf, List<org.apache.flink.configuration.ConfigOption<String>> jvmOptions, boolean hasKrb5)

    • getQualifiedRemoteProvidedLibDirs

      public static List<org.apache.hadoop.fs.Path> getQualifiedRemoteProvidedLibDirs(org.apache.flink.configuration.Configuration configuration, org.apache.hadoop.yarn.conf.YarnConfiguration yarnConfiguration) throws IOException
      Throws:
      IOException

    • isUsrLibDirectory

      public static boolean isUsrLibDirectory(org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path path) throws IOException
      Throws:
      IOException

    • getQualifiedRemoteProvidedUsrLib

      public static Optional<org.apache.hadoop.fs.Path> getQualifiedRemoteProvidedUsrLib(org.apache.flink.configuration.Configuration configuration, org.apache.hadoop.yarn.conf.YarnConfiguration yarnConfiguration) throws IOException, IllegalArgumentException
      Throws:
      IOException
      IllegalArgumentException

    • getYarnAndHadoopConfiguration

      public static org.apache.hadoop.yarn.conf.YarnConfiguration getYarnAndHadoopConfiguration(org.apache.flink.configuration.Configuration flinkConfig)

    • getYarnConfiguration

      public static org.apache.hadoop.yarn.conf.YarnConfiguration getYarnConfiguration(org.apache.flink.configuration.Configuration flinkConfig)
      Add additional config entries from the flink config to the yarn config.
      Parameters:
      flinkConfig - The Flink configuration object.
      Returns:
      The yarn configuration.

    • setAclsFor

      public static void setAclsFor(org.apache.hadoop.yarn.api.records.ContainerLaunchContext amContainer, org.apache.flink.configuration.Configuration flinkConfig)
      Sets the application ACLs for the given ContainerLaunchContext based on the values specified in the given Flink configuration. Only ApplicationAccessType.VIEW_APP and ApplicationAccessType.MODIFY_APP ACLs are set, and only if they are configured in the Flink configuration. If the viewAcls or modifyAcls string contains the WILDCARD_ACL constant, it will replace the entire string with the WILDCARD_ACL. The resulting map is then set as the application acls for the given container launch context.
      Parameters:
      amContainer - the ContainerLaunchContext to set the ACLs for.
      flinkConfig - the Flink configuration to read the ACL values from.

    • getPathFromLocalFile

      public static org.apache.hadoop.fs.Path getPathFromLocalFile(File localFile)

    • getPathFromLocalFilePathStr

      public static org.apache.hadoop.fs.Path getPathFromLocalFilePathStr(String localPathStr)

    • concatWithSpace

      public static void concatWithSpace(StringBuilder sb, String value)