Package org.apache.flink.runtime.util

Class ZooKeeperUtils

java.lang.Object
org.apache.flink.runtime.util.ZooKeeperUtils
public class ZooKeeperUtils extends Object
Class containing helper functions to interact with ZooKeeper.

  • Field Details

    • HA_STORAGE_SUBMITTED_EXECUTION_PLAN_PREFIX

      public static final String HA_STORAGE_SUBMITTED_EXECUTION_PLAN_PREFIX
      The prefix of the submitted job graph file.
      See Also:

    • HA_STORAGE_COMPLETED_CHECKPOINT

      public static final String HA_STORAGE_COMPLETED_CHECKPOINT
      The prefix of the completed checkpoint file.
      See Also:

    • RESOURCE_MANAGER_NODE

      public static final String RESOURCE_MANAGER_NODE
      The prefix of the resource manager node.
      See Also:

  • Method Details

    • getLeaderPathForJob

      public static String getLeaderPathForJob(org.apache.flink.api.common.JobID jobId)

    • getJobsPath

      public static String getJobsPath()

    • getCheckpointIdCounterPath

      public static String getCheckpointIdCounterPath()

    • getLeaderPath

      public static String getLeaderPath()

    • getDispatcherNode

      public static String getDispatcherNode()

    • getResourceManagerNode

      public static String getResourceManagerNode()

    • getRestServerNode

      public static String getRestServerNode()

    • getLeaderLatchPath

      public static String getLeaderLatchPath()

    • getLeaderPath

      public static String getLeaderPath(String suffix)

    • generateConnectionInformationPath

      public static String generateConnectionInformationPath(String path)

    • isConnectionInfoPath

      public static boolean isConnectionInfoPath(String path)

    • generateLeaderLatchPath

      public static String generateLeaderLatchPath(String path)

    • startCuratorFramework

      public static CuratorFrameworkWithUnhandledErrorListener startCuratorFramework(org.apache.flink.configuration.Configuration configuration, org.apache.flink.runtime.rpc.FatalErrorHandler fatalErrorHandler)
      Starts a CuratorFramework instance and connects it to the given ZooKeeper quorum.
      Parameters:
      configuration - Configuration object containing the configuration values
      fatalErrorHandler - FatalErrorHandler fatalErrorHandler to handle unexpected errors of CuratorFramework
      Returns:
      CuratorFrameworkWithUnhandledErrorListener instance

    • startCuratorFramework

      @VisibleForTesting public static CuratorFrameworkWithUnhandledErrorListener startCuratorFramework(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFrameworkFactory.Builder builder, org.apache.flink.runtime.rpc.FatalErrorHandler fatalErrorHandler)
      Starts a CuratorFramework instance and connects it to the given ZooKeeper quorum from a builder.
      Parameters:
      builder - CuratorFrameworkFactory.Builder A builder for curatorFramework.
      fatalErrorHandler - FatalErrorHandler fatalErrorHandler to handle unexpected errors of CuratorFramework
      Returns:
      CuratorFrameworkWithUnhandledErrorListener instance

    • isZooKeeperRecoveryMode

      public static boolean isZooKeeperRecoveryMode(org.apache.flink.configuration.Configuration flinkConf)
      Returns whether HighAvailabilityMode.ZOOKEEPER is configured.

    • getZooKeeperEnsemble

      public static String getZooKeeperEnsemble(org.apache.flink.configuration.Configuration flinkConf) throws org.apache.flink.configuration.IllegalConfigurationException
      Returns the configured ZooKeeper quorum (and removes whitespace, because ZooKeeper does not tolerate it).
      Throws:
      org.apache.flink.configuration.IllegalConfigurationException

    • createLeaderRetrievalService

      public static DefaultLeaderRetrievalService createLeaderRetrievalService(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client)
      Creates a DefaultLeaderRetrievalService instance with ZooKeeperLeaderRetrievalDriver.
      Parameters:
      client - The CuratorFramework ZooKeeper client to use
      Returns:
      DefaultLeaderRetrievalService instance.

    • createLeaderRetrievalService

      public static DefaultLeaderRetrievalService createLeaderRetrievalService(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client, String path, org.apache.flink.configuration.Configuration configuration)
      Creates a DefaultLeaderRetrievalService instance with ZooKeeperLeaderRetrievalDriver.
      Parameters:
      client - The CuratorFramework ZooKeeper client to use
      path - The path for the leader retrieval
      configuration - configuration for further config options
      Returns:
      DefaultLeaderRetrievalService instance.

    • createLeaderRetrievalDriverFactory

      public static ZooKeeperLeaderRetrievalDriverFactory createLeaderRetrievalDriverFactory(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client)
      Creates a LeaderRetrievalDriverFactory implemented by ZooKeeper.
      Parameters:
      client - The CuratorFramework ZooKeeper client to use
      Returns:
      LeaderRetrievalDriverFactory instance.

    • createLeaderRetrievalDriverFactory

      public static ZooKeeperLeaderRetrievalDriverFactory createLeaderRetrievalDriverFactory(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client, String path)
      Creates a LeaderRetrievalDriverFactory implemented by ZooKeeper.
      Parameters:
      client - The CuratorFramework ZooKeeper client to use
      path - The parent path that shall be used by the client.
      Returns:
      LeaderRetrievalDriverFactory instance.

    • createLeaderRetrievalDriverFactory

      public static ZooKeeperLeaderRetrievalDriverFactory createLeaderRetrievalDriverFactory(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client, String path, org.apache.flink.configuration.Configuration configuration)
      Creates a LeaderRetrievalDriverFactory implemented by ZooKeeper.
      Parameters:
      client - The CuratorFramework ZooKeeper client to use
      path - The path for the leader zNode
      configuration - configuration for further config options
      Returns:
      LeaderRetrievalDriverFactory instance.

    • writeLeaderInformationToZooKeeper

      public static void writeLeaderInformationToZooKeeper(LeaderInformation leaderInformation, org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework curatorFramework, BooleanSupplier hasLeadershipCheck, String connectionInformationPath) throws Exception
      Throws:
      Exception

    • readLeaderInformation

      public static LeaderInformation readLeaderInformation(byte[] data) throws IOException, ClassNotFoundException
      Throws:
      IOException
      ClassNotFoundException

    • createExecutionPlans

      public static ExecutionPlanStore createExecutionPlans(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client, org.apache.flink.configuration.Configuration configuration) throws Exception
      Creates a DefaultExecutionPlanStore instance with ZooKeeperStateHandleStore, ZooKeeperExecutionPlanStoreWatcher and ZooKeeperExecutionPlanStoreUtil.
      Parameters:
      client - The CuratorFramework ZooKeeper client to use
      configuration - Configuration object
      Returns:
      DefaultExecutionPlanStore instance
      Throws:
      Exception - if the submitted execution plan store cannot be created

    • createCompletedCheckpoints

      public static CompletedCheckpointStore createCompletedCheckpoints(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client, org.apache.flink.configuration.Configuration configuration, int maxNumberOfCheckpointsToRetain, SharedStateRegistryFactory sharedStateRegistryFactory, Executor ioExecutor, Executor executor, org.apache.flink.core.execution.RecoveryClaimMode recoveryClaimMode) throws Exception
      Creates a DefaultCompletedCheckpointStore instance with ZooKeeperStateHandleStore.
      Parameters:
      client - The CuratorFramework ZooKeeper client to use
      configuration - Configuration object
      maxNumberOfCheckpointsToRetain - The maximum number of checkpoints to retain
      executor - to run ZooKeeper callbacks
      recoveryClaimMode - the mode in which the job is being restored
      Returns:
      DefaultCompletedCheckpointStore instance
      Throws:
      Exception - if the completed checkpoint store cannot be created

    • getPathForJob

      public static String getPathForJob(org.apache.flink.api.common.JobID jobId)
      Returns the JobID as a String (with leading slash).

    • createZooKeeperStateHandleStore

      public static <T extends Serializable> ZooKeeperStateHandleStore<T> createZooKeeperStateHandleStore(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client, String path, RetrievableStateStorageHelper<T> stateStorage) throws Exception
      Creates an instance of ZooKeeperStateHandleStore.
      Type Parameters:
      T - Type of state
      Parameters:
      client - ZK client
      path - Path to use for the client namespace
      stateStorage - RetrievableStateStorageHelper that persist the actual state and whose returned state handle is then written to ZooKeeper
      Returns:
      ZooKeeperStateHandleStore instance
      Throws:
      Exception - ZK errors

    • createCheckpointIDCounter

      public static ZooKeeperCheckpointIDCounter createCheckpointIDCounter(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client)
      Creates a ZooKeeperCheckpointIDCounter instance.
      Parameters:
      client - The CuratorFramework ZooKeeper client to use
      Returns:
      ZooKeeperCheckpointIDCounter instance

    • createFileSystemStateStorage

      public static <T extends Serializable> FileSystemStateStorageHelper<T> createFileSystemStateStorage(org.apache.flink.configuration.Configuration configuration, String prefix) throws IOException
      Creates a FileSystemStateStorageHelper instance.
      Type Parameters:
      T - Type of the state objects
      Parameters:
      configuration - Configuration object
      prefix - Prefix for the created files
      Returns:
      FileSystemStateStorageHelper instance
      Throws:
      IOException - if file system state storage cannot be created

    • generateZookeeperPath

      public static String generateZookeeperPath(String... paths)
      Creates a ZooKeeper path of the form "/a/b/.../z".

    • splitZooKeeperPath

      public static String[] splitZooKeeperPath(String path)
      Splits the given ZooKeeper path into its parts.
      Parameters:
      path - path to split
      Returns:
      splited path

    • trimStartingSlash

      public static String trimStartingSlash(String path)

    • useNamespaceAndEnsurePath

      public static org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework useNamespaceAndEnsurePath(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client, String path) throws Exception
      Returns a facade of the client that uses the specified namespace, and ensures that all nodes in the path exist.
      Parameters:
      client - ZK client
      path - the new namespace
      Returns:
      ZK Client that uses the new namespace
      Throws:
      Exception - ZK errors

    • createTreeCache

      public static org.apache.flink.shaded.curator5.org.apache.curator.framework.recipes.cache.TreeCache createTreeCache(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client, String pathToNode, org.apache.flink.util.function.RunnableWithException nodeChangeCallback)
      Creates a TreeCache that only observes a specific node.
      Parameters:
      client - ZK client
      pathToNode - full path of the node to observe
      nodeChangeCallback - callback to run if the node has changed
      Returns:
      tree cache

    • createTreeCache

      public static org.apache.flink.shaded.curator5.org.apache.curator.framework.recipes.cache.TreeCache createTreeCache(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client, String pathToNode, org.apache.flink.shaded.curator5.org.apache.curator.framework.recipes.cache.TreeCacheSelector selector)

    • deleteZNode

      public static void deleteZNode(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework curatorFramework, String path) throws Exception
      Throws:
      Exception