class JsonFilters extends StructFilters
The class provides API for applying pushed down source filters to rows with
a struct schema parsed from JSON records. The class should be used in this way:
1. Before processing of the next row, JacksonParser (parser for short) resets the internal
state of JsonFilters by calling the reset() method.
2. The parser reads JSON fields one-by-one in streaming fashion. It converts an incoming
field value to the desired type from the schema. After that, it sets the value to an instance
of InternalRow at the position according to the schema. Order of parsed JSON fields can
be different from the order in the schema.
3. Per every JSON field of the top-level JSON object, the parser calls skipRow by passing
an InternalRow in which some of fields can be already set, and the position of the JSON
field according to the schema.
3.1 skipRow finds a group of predicates that refers to this JSON field.
3.2 Per each predicate from the group, skipRow decrements its reference counter.
3.2.1 If predicate reference counter becomes 0, it means that all predicate attributes have
been already set in the internal row, and the predicate can be applied to it. skipRow
invokes the predicate for the row.
3.3 skipRow applies predicates until one of them returns false. In that case, the method
returns true to the parser.
3.4 If all predicates with zero reference counter return true, the final result of
the method is false which tells the parser to not skip the row.
4. If the parser gets true from JsonFilters.skipRow, it must not call the method anymore
for this internal row, and should go the step 1.
Besides of StructFilters assumptions, JsonFilters assumes that:
skipRow()can be called for any valid index of the struct fields, and in any order.- After
skipRow()returnstrue, the internal state ofJsonFilterscan be inconsistent, so,skipRow()must not be called for the current row anymore withoutreset().
- Alphabetic
- By Inheritance
- JsonFilters
- StructFilters
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
-
new
JsonFilters(pushedFilters: Seq[Filter], schema: StructType)
- pushedFilters
The pushed down source filters. The filters should refer to the fields of the provided schema.
- schema
The required schema of records from datasource files.
Type Members
-
case class
JsonPredicate(predicate: BasePredicate, totalRefs: Int) extends Product with Serializable
Stateful JSON predicate that keeps track of its dependent references in the current row via
refCount.Stateful JSON predicate that keeps track of its dependent references in the current row via
refCount.- predicate
The predicate compiled from pushed down source filters.
- totalRefs
The total amount of all filters references which the predicate compiled from.
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native() @HotSpotIntrinsicCandidate()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
val
filters: Array[Filter]
- Attributes
- protected
- Definition Classes
- StructFilters
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
def
reset(): Unit
Reset states of all predicates by re-initializing reference counters.
Reset states of all predicates by re-initializing reference counters.
- Definition Classes
- JsonFilters → StructFilters
-
def
skipRow(row: InternalRow, index: Int): Boolean
Applies predicates (compiled filters) associated with the row field value at the position
indexonly if other predicates dependencies are already set in the given row.Applies predicates (compiled filters) associated with the row field value at the position
indexonly if other predicates dependencies are already set in the given row.Note: If the function returns
true,refCountof some predicates can be not decremented.- row
The row with fully or partially set values.
- index
The index of already set value.
- returns
trueif at least one of applicable predicates (all dependent row values are set) returnfalse. It returnsfalseif all predicates returntrue.
- Definition Classes
- JsonFilters → StructFilters
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toPredicate(filters: Seq[Filter]): BasePredicate
Compiles source filters to a predicate.
Compiles source filters to a predicate.
- Definition Classes
- StructFilters
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
Deprecated Value Members
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] ) @Deprecated
- Deprecated