Package org.apache.flink.fs.gs.utils

Class ConfigUtils

java.lang.Object
org.apache.flink.fs.gs.utils.ConfigUtils
public class ConfigUtils extends Object
Utilities class for configuration of Hadoop and Google Storage.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static interface 
    ConfigUtils.ConfigContext
    Interface that provides context-specific config helper functions, factored out to support unit testing

  • Constructor Summary

    Constructors
    Constructor
    Description
    ConfigUtils()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    static Optional<String>
    getGcsRootUrl(org.apache.hadoop.conf.Configuration hadoopConfig)
     
    static org.apache.hadoop.conf.Configuration
    getHadoopConfiguration(org.apache.flink.configuration.Configuration flinkConfig, ConfigUtils.ConfigContext configContext)
    Loads the Hadoop configuration, by loading from a Hadoop conf dir (if one exists) and then overlaying properties derived from the Flink config.
    static Optional<com.google.auth.oauth2.GoogleCredentials>
    getStorageCredentials(org.apache.hadoop.conf.Configuration hadoopConfig, ConfigUtils.ConfigContext configContext)
    Creates an (optional) GoogleCredentials instance for the given Hadoop config and environment.
    static String
    stringifyHadoopConfig(org.apache.hadoop.conf.Configuration hadoopConfig)
    Helper to serialize a Hadoop config to a string, for logging.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • ConfigUtils

      public ConfigUtils()

  • Method Details

    • getHadoopConfiguration

      public static org.apache.hadoop.conf.Configuration getHadoopConfiguration(org.apache.flink.configuration.Configuration flinkConfig, ConfigUtils.ConfigContext configContext)
      Loads the Hadoop configuration, by loading from a Hadoop conf dir (if one exists) and then overlaying properties derived from the Flink config.
      Parameters:
      flinkConfig - The Flink config
      configContext - The config context.
      Returns:
      The Hadoop config.

    • getStorageCredentials

      public static Optional<com.google.auth.oauth2.GoogleCredentials> getStorageCredentials(org.apache.hadoop.conf.Configuration hadoopConfig, ConfigUtils.ConfigContext configContext)
      Creates an (optional) GoogleCredentials instance for the given Hadoop config and environment.
      Parameters:
      hadoopConfig - The Hadoop config.
      configContext - The config context.
      Returns:
      The optional GoogleCredentials instance.

    • getGcsRootUrl

      public static Optional<String> getGcsRootUrl(org.apache.hadoop.conf.Configuration hadoopConfig)

    • stringifyHadoopConfig

      public static String stringifyHadoopConfig(org.apache.hadoop.conf.Configuration hadoopConfig) throws RuntimeException
      Helper to serialize a Hadoop config to a string, for logging.
      Parameters:
      hadoopConfig - The Hadoop config.
      Returns:
      A string with the Hadoop properties.
      Throws:
      RuntimeException - On underlying IO failure