Package org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies

Class DefaultRollingPolicy<IN,BucketID>

java.lang.Object
org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.DefaultRollingPolicy<IN,BucketID>
All Implemented Interfaces:
Serializable, RollingPolicy<IN,BucketID>
@PublicEvolving public final class DefaultRollingPolicy<IN,BucketID> extends Object implements RollingPolicy<IN,BucketID>
The default implementation of the RollingPolicy.

This policy rolls a part file if:

  1. there is no open part file,
  2. the current file has reached the maximum bucket size (by default 128MB),
  3. the current file is older than the roll over interval (by default 60 sec), or
  4. the current file has not been written to for more than the allowed inactivityTime (by default 60 sec).
See Also:

  • Method Details

    • shouldRollOnCheckpoint

      public boolean shouldRollOnCheckpoint(PartFileInfo<BucketID> partFileState) throws IOException
      Description copied from interface: RollingPolicy
      Determines if the in-progress part file for a bucket should roll on every checkpoint.
      Specified by:
      shouldRollOnCheckpoint in interface RollingPolicy<IN,BucketID>
      Parameters:
      partFileState - the state of the currently open part file of the bucket.
      Returns:
      True if the part file should roll, {@link false} otherwise.
      Throws:
      IOException

    • shouldRollOnEvent

      public boolean shouldRollOnEvent(PartFileInfo<BucketID> partFileState, IN element) throws IOException
      Description copied from interface: RollingPolicy
      Determines if the in-progress part file for a bucket should roll based on its current state, e.g. its size.
      Specified by:
      shouldRollOnEvent in interface RollingPolicy<IN,BucketID>
      Parameters:
      partFileState - the state of the currently open part file of the bucket.
      element - the element being processed.
      Returns:
      True if the part file should roll, {@link false} otherwise.
      Throws:
      IOException

    • shouldRollOnProcessingTime

      public boolean shouldRollOnProcessingTime(PartFileInfo<BucketID> partFileState, long currentTime)
      Description copied from interface: RollingPolicy
      Determines if the in-progress part file for a bucket should roll based on a time condition.
      Specified by:
      shouldRollOnProcessingTime in interface RollingPolicy<IN,BucketID>
      Parameters:
      partFileState - the state of the currently open part file of the bucket.
      currentTime - the current processing time.
      Returns:
      True if the part file should roll, {@link false} otherwise.

    • getMaxPartSize

      public long getMaxPartSize()
      Returns the maximum part file size before rolling.
      Returns:
      Max size in bytes

    • getRolloverInterval

      public long getRolloverInterval()
      Returns the maximum time duration a part file can stay open before rolling.
      Returns:
      Time duration in milliseconds

    • getInactivityInterval

      public long getInactivityInterval()
      Returns time duration of allowed inactivity after which a part file will have to roll.
      Returns:
      Time duration in milliseconds

    • builder

      public static DefaultRollingPolicy.PolicyBuilder builder()
      Creates a new DefaultRollingPolicy.PolicyBuilder that is used to configure and build an instance of DefaultRollingPolicy.

    • create

      @Deprecated public static DefaultRollingPolicy.PolicyBuilder create()
      Deprecated.
      This method is Deprecated, use builder() instead.