Package org.apache.flink.connector.testframe.container

Class FlinkContainers

java.lang.Object
org.apache.flink.connector.testframe.container.FlinkContainers
All Implemented Interfaces:
org.junit.jupiter.api.extension.AfterAllCallback, org.junit.jupiter.api.extension.BeforeAllCallback, org.junit.jupiter.api.extension.Extension
public class FlinkContainers extends Object implements org.junit.jupiter.api.extension.BeforeAllCallback, org.junit.jupiter.api.extension.AfterAllCallback
A Flink cluster running JM and TMs on containers.

This containerized Flink cluster is based on Testcontainers, which simulates a truly distributed environment for E2E tests. This class can also be used as an Extension of JUnit 5 so that the lifecycle of the cluster can be easily managed by JUnit Jupiter engine.

Example usage


 public class E2ETest {
     // Create a Flink cluster using default configurations.
     // Remember to declare it as "static" as required by
     // JUnit 5.
     @RegisterExtension
     static FlinkContainers flink =
          FlinkContainers.builder().build();

 ---

     // To work together with other containers
     static TestcontainersSettings testcontainersSettings =
          TestcontainersSettings.builder()
                 .dependsOn(kafkaContainer)
                 .build());

     @RegisterExtension
     static FlinkContainers flink =
             FlinkContainers.builder()
              .withTestcontainersSettings(testcontainersSettings)
              .build();

 ---
     // Customize a Flink cluster
     static FlinkContainersSettings flinkContainersSettings =
          FlinkContainersSettings.builder()
             .numTaskManagers(3)
             .numSlotsPerTaskManager(6)
             .baseImage(
               "ghcr.io/apache/flink-docker:1.16-snapshot-scala_2.12-java11-debian")
             .enableZookeeperHA()
             .build();

     static TestcontainersSettings testcontainersSettings =
          TestcontainersSettings.builder()
                 .setLogger(
                      LoggerFactory.getLogger(E2ETest.class))
                 .build());

     @RegisterExtension
     static FlinkContainers flink =
         FlinkContainers.builder()
           .withFlinkContainersSettings(flinkContainersSettings)
           .withTestcontainersSettings(testcontainersSettings)
           .build();
     ...
 }

See FlinkContainersSettings and TestcontainersSettings for available options.

Prerequisites

Docker environment is required in the running machine since this class is based on Testcontainers.

Make sure you have an already-built flink-dist either under the current project, or specify the path manually in builder.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static final class 
    FlinkContainers.Builder
    The FlinkContainers builder.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final Duration
    DEFAULT_TIMEOUT
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    afterAll(org.junit.jupiter.api.extension.ExtensionContext context)
     
    void
    beforeAll(org.junit.jupiter.api.extension.ExtensionContext context)
     
    static FlinkContainers.Builder
    builder()
    Creates a builder for FlinkContainers.
    org.testcontainers.containers.GenericContainer<?>
    getJobManager()
    Gets JobManager container.
    String
    getJobManagerHost()
    Gets JobManager's hostname on the host machine.
    int
    getJobManagerPort()
    Gets JobManager's port on the host machine.
    org.apache.flink.client.program.rest.RestClusterClient<org.apache.flink.client.deployment.StandaloneClusterId>
    getRestClusterClient()
    Gets REST client connected to JobManager.
    List<org.testcontainers.containers.GenericContainer<?>>
    getTaskManagers()
    Gets TaskManager containers.
    boolean
    isStarted()
    Gets the running state of the cluster.
    void
    restartJobManager(org.apache.flink.util.function.RunnableWithException afterFailAction)
    Restarts JobManager container.
    void
    restartTaskManager(org.apache.flink.util.function.RunnableWithException afterFailAction)
    Restarts all TaskManager containers.
    void
    start()
    Starts all containers.
    void
    stop()
    Stops all containers.
    org.apache.flink.api.common.JobID
    submitJob(org.apache.flink.test.util.JobSubmission job)
    Submits the given job to the cluster.
    void
    submitSQLJob(org.apache.flink.test.util.SQLJobSubmission job)
    Submits an SQL job to the running cluster.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • DEFAULT_TIMEOUT

      public static final Duration DEFAULT_TIMEOUT

  • Method Details

    • builder

      public static FlinkContainers.Builder builder()
      Creates a builder for FlinkContainers.

    • start

      public void start() throws Exception
      Starts all containers.
      Throws:
      Exception

    • stop

      public void stop()
      Stops all containers.

    • isStarted

      public boolean isStarted()
      Gets the running state of the cluster.

    • getJobManager

      public org.testcontainers.containers.GenericContainer<?> getJobManager()
      Gets JobManager container.

    • getTaskManagers

      public List<org.testcontainers.containers.GenericContainer<?>> getTaskManagers()
      Gets TaskManager containers.

    • getJobManagerHost

      public String getJobManagerHost()
      Gets JobManager's hostname on the host machine.

    • getJobManagerPort

      public int getJobManagerPort()
      Gets JobManager's port on the host machine.

    • getRestClusterClient

      @Nullable public org.apache.flink.client.program.rest.RestClusterClient<org.apache.flink.client.deployment.StandaloneClusterId> getRestClusterClient()
      Gets REST client connected to JobManager.

    • restartJobManager

      public void restartJobManager(org.apache.flink.util.function.RunnableWithException afterFailAction) throws Exception
      Restarts JobManager container.

      Note that the REST port will be changed because the new JM container will be mapped to another random port. Please make sure to get the REST cluster client again after this method is invoked.

      Throws:
      Exception

    • restartTaskManager

      public void restartTaskManager(org.apache.flink.util.function.RunnableWithException afterFailAction) throws Exception
      Restarts all TaskManager containers.
      Throws:
      Exception

    • submitSQLJob

      public void submitSQLJob(org.apache.flink.test.util.SQLJobSubmission job) throws IOException, InterruptedException
      Submits an SQL job to the running cluster.

      NOTE: You should not use '\t'.

      Throws:
      IOException
      InterruptedException

    • submitJob

      public org.apache.flink.api.common.JobID submitJob(org.apache.flink.test.util.JobSubmission job) throws IOException, InterruptedException
      Submits the given job to the cluster.
      Parameters:
      job - job to submit
      Throws:
      IOException
      InterruptedException

    • beforeAll

      public void beforeAll(org.junit.jupiter.api.extension.ExtensionContext context) throws Exception
      Specified by:
      beforeAll in interface org.junit.jupiter.api.extension.BeforeAllCallback
      Throws:
      Exception

    • afterAll

      public void afterAll(org.junit.jupiter.api.extension.ExtensionContext context)
      Specified by:
      afterAll in interface org.junit.jupiter.api.extension.AfterAllCallback