Package org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners

Class DateTimeBucketAssigner<IN>

java.lang.Object
org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.DateTimeBucketAssigner<IN>
All Implemented Interfaces:
Serializable, BucketAssigner<IN,String>
@PublicEvolving public class DateTimeBucketAssigner<IN> extends Object implements BucketAssigner<IN,String>
A BucketAssigner that assigns to buckets based on current system time.

The DateTimeBucketer will create directories of the following form: /{basePath}/{dateTimePath}/. The basePath is the path that was specified as a base path when creating the new bucket. The dateTimePath is determined based on the current system time and the user provided format string.

DateTimeFormatter is used to derive a date string from the current system time and the date format string. The default format string is "yyyy-MM-dd--HH" so the rolling files will have a granularity of hours.

Example: 


 BucketAssigner bucketAssigner = new DateTimeBucketAssigner("yyyy-MM-dd--HH");

This will create for example the following bucket path: /base/1976-12-31-14/

See Also:

  • Constructor Details

    • DateTimeBucketAssigner

      public DateTimeBucketAssigner()
      Creates a new DateTimeBucketAssigner with format string "yyyy-MM-dd--HH".

    • DateTimeBucketAssigner

      public DateTimeBucketAssigner(String formatString)
      Creates a new DateTimeBucketAssigner with the given date/time format string.
      Parameters:
      formatString - The format string that will be given to SimpleDateFormat to determine the bucket id.

    • DateTimeBucketAssigner

      public DateTimeBucketAssigner(ZoneId zoneId)
      Creates a new DateTimeBucketAssigner with format string "yyyy-MM-dd--HH" using the given timezone.
      Parameters:
      zoneId - The timezone used to format DateTimeFormatter for bucket id.

    • DateTimeBucketAssigner

      public DateTimeBucketAssigner(String formatString, ZoneId zoneId)
      Creates a new DateTimeBucketAssigner with the given date/time format string using the given timezone.
      Parameters:
      formatString - The format string that will be given to DateTimeFormatter to determine the bucket path.
      zoneId - The timezone used to format DateTimeFormatter for bucket id.

  • Method Details

    • getBucketId

      public String getBucketId(IN element, BucketAssigner.Context context)
      Description copied from interface: BucketAssigner
      Returns the identifier of the bucket the provided element should be put into.
      Specified by:
      getBucketId in interface BucketAssigner<IN,String>
      Parameters:
      element - The current element being processed.
      context - The context used by the current bucket assigner.
      Returns:
      A string representing the identifier of the bucket the element should be put into. The actual path to the bucket will result from the concatenation of the returned string and the base path provided during the initialization of the file sink.

    • getSerializer

      public org.apache.flink.core.io.SimpleVersionedSerializer<String> getSerializer()
      Specified by:
      getSerializer in interface BucketAssigner<IN,String>
      Returns:
      A SimpleVersionedSerializer capable of serializing/deserializing the elements of type BucketID. That is the type of the objects returned by the BucketAssigner.getBucketId(Object, BucketAssigner.Context).

    • toString

      public String toString()
      Overrides:
      toString in class Object