Packages

class StreamingQueryListenerBus extends SparkListener with ListenerBus[StreamingQueryListener, Event]

A bus to forward events to StreamingQueryListeners. This one will send received StreamingQueryListener.Events to the Spark listener bus. It also registers itself with Spark listener bus, so that it can receive StreamingQueryListener.Events and dispatch them to StreamingQueryListeners.

Note 1: Each bus and its registered listeners are associated with a single SparkSession and StreamingQueryManager. So this bus will dispatch events to registered listeners for only those queries that were started in the associated SparkSession.

Note 2: To rebuild Structured Streaming UI in SHS, this bus will be registered into org.apache.spark.scheduler.ReplayListenerBus. We check sparkListenerBus defined or not to determine how to process StreamingQueryListener.Event. If false, it means this bus is used to replay all streaming query event from eventLog.

Linear Supertypes
ListenerBus[StreamingQueryListener, Event], Logging, SparkListener, SparkListenerInterface, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. StreamingQueryListenerBus
  2. ListenerBus
  3. Logging
  4. SparkListener
  5. SparkListenerInterface
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new StreamingQueryListenerBus(sparkListenerBus: Option[LiveListenerBus])

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def addListener(listener: StreamingQueryListener): Unit
    Definition Classes
    ListenerBus
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  7. def doPostEvent(listener: StreamingQueryListener, event: Event): Unit

    Dispatch events to registered StreamingQueryListeners.

    Dispatch events to registered StreamingQueryListeners. Only the events associated queries started in the same SparkSession as this ListenerBus will be dispatched to the listeners.

    Attributes
    protected
    Definition Classes
    StreamingQueryListenerBus → ListenerBus
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  10. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  11. def getTimer(listener: StreamingQueryListener): Option[Timer]
    Attributes
    protected
    Definition Classes
    ListenerBus
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  13. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  14. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  15. def isIgnorableException(e: Throwable): Boolean
    Attributes
    protected
    Definition Classes
    ListenerBus
  16. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  17. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  18. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  19. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  20. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  21. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  22. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  23. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  24. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  25. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  26. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  27. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  28. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  30. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  31. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  32. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  33. def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  34. def onApplicationStart(applicationStart: SparkListenerApplicationStart): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  35. def onBlockManagerAdded(blockManagerAdded: SparkListenerBlockManagerAdded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  36. def onBlockManagerRemoved(blockManagerRemoved: SparkListenerBlockManagerRemoved): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  37. def onBlockUpdated(blockUpdated: SparkListenerBlockUpdated): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  38. def onEnvironmentUpdate(environmentUpdate: SparkListenerEnvironmentUpdate): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  39. def onExecutorAdded(executorAdded: SparkListenerExecutorAdded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  40. def onExecutorBlacklisted(executorBlacklisted: SparkListenerExecutorBlacklisted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  41. def onExecutorBlacklistedForStage(executorBlacklistedForStage: SparkListenerExecutorBlacklistedForStage): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  42. def onExecutorExcluded(executorExcluded: SparkListenerExecutorExcluded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  43. def onExecutorExcludedForStage(executorExcludedForStage: SparkListenerExecutorExcludedForStage): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  44. def onExecutorMetricsUpdate(executorMetricsUpdate: SparkListenerExecutorMetricsUpdate): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  45. def onExecutorRemoved(executorRemoved: SparkListenerExecutorRemoved): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  46. def onExecutorUnblacklisted(executorUnblacklisted: SparkListenerExecutorUnblacklisted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  47. def onExecutorUnexcluded(executorUnexcluded: SparkListenerExecutorUnexcluded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  48. def onJobEnd(jobEnd: SparkListenerJobEnd): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  49. def onJobStart(jobStart: SparkListenerJobStart): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  50. def onNodeBlacklisted(nodeBlacklisted: SparkListenerNodeBlacklisted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  51. def onNodeBlacklistedForStage(nodeBlacklistedForStage: SparkListenerNodeBlacklistedForStage): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  52. def onNodeExcluded(nodeExcluded: SparkListenerNodeExcluded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  53. def onNodeExcludedForStage(nodeExcludedForStage: SparkListenerNodeExcludedForStage): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  54. def onNodeUnblacklisted(nodeUnblacklisted: SparkListenerNodeUnblacklisted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  55. def onNodeUnexcluded(nodeUnexcluded: SparkListenerNodeUnexcluded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  56. def onOtherEvent(event: SparkListenerEvent): Unit
    Definition Classes
    StreamingQueryListenerBus → SparkListener → SparkListenerInterface
  57. def onResourceProfileAdded(event: SparkListenerResourceProfileAdded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  58. def onSpeculativeTaskSubmitted(speculativeTask: SparkListenerSpeculativeTaskSubmitted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  59. def onStageCompleted(stageCompleted: SparkListenerStageCompleted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  60. def onStageExecutorMetrics(executorMetrics: SparkListenerStageExecutorMetrics): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  61. def onStageSubmitted(stageSubmitted: SparkListenerStageSubmitted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  62. def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  63. def onTaskGettingResult(taskGettingResult: SparkListenerTaskGettingResult): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  64. def onTaskStart(taskStart: SparkListenerTaskStart): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  65. def onUnpersistRDD(unpersistRDD: SparkListenerUnpersistRDD): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  66. def onUnschedulableTaskSetAdded(unschedulableTaskSetAdded: SparkListenerUnschedulableTaskSetAdded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  67. def onUnschedulableTaskSetRemoved(unschedulableTaskSetRemoved: SparkListenerUnschedulableTaskSetRemoved): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  68. def post(event: Event): Unit

    Post a StreamingQueryListener event to the added StreamingQueryListeners.

    Post a StreamingQueryListener event to the added StreamingQueryListeners. Note that only the QueryStarted event is posted to the listener synchronously. Other events are dispatched to Spark listener bus. This method is guaranteed to be called by queries in the same SparkSession as this listener.

  69. def postToAll(event: Event): Unit

    Override the parent postToAll to remove the query id from activeQueryRunIds after all the listeners process QueryTerminatedEvent.

    Override the parent postToAll to remove the query id from activeQueryRunIds after all the listeners process QueryTerminatedEvent. (SPARK-19594)

    Definition Classes
    StreamingQueryListenerBus → ListenerBus
  70. final def removeAllListeners(): Unit
    Definition Classes
    ListenerBus
  71. final def removeListener(listener: StreamingQueryListener): Unit
    Definition Classes
    ListenerBus
  72. def removeListenerOnError(listener: StreamingQueryListener): Unit
    Definition Classes
    ListenerBus
  73. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  74. def toString(): String
    Definition Classes
    AnyRef → Any
  75. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  76. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  77. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from ListenerBus[StreamingQueryListener, Event]

Inherited from Logging

Inherited from SparkListener

Inherited from SparkListenerInterface

Inherited from AnyRef

Inherited from Any

Ungrouped