Index

B C D E F I M N O P R S
All Classes and Interfaces|All Packages|Constant Field Values

B

builder() - Static method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

builder() - Static method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunInformation

builder() - Static method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunInformation.MaintenanceStats

builder() - Static method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunSpec

buildMaintenanceRunSpec() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceService

Generates a maintenance service run-specification containing realms in states ACTIVE and INACTIVE as "to retain" and realms in state PURGING as "to purge".

C

check() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

countFromLastRunMultiplier() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

Maintenance service sizes the bloom-filters used to hold the identified references and objects according to the expression lastRun.numberOfIdentified * countFromLastRunMultiplier.

createdAtGraceTime() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

Objects and references that have been created after a maintenance run has started are never purged.

D

DEFAULT_COUNT_FROM_LAST_RUN_MULTIPLIER - Static variable in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

DEFAULT_CREATED_AT_GRACE_TIME - Static variable in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

DEFAULT_CREATED_AT_GRACE_TIME_STRING - Static variable in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

DEFAULT_DELETE_BATCH_SIZE - Static variable in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

DEFAULT_EXPECTED_OBJ_COUNT - Static variable in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

DEFAULT_EXPECTED_REFERENCE_COUNT - Static variable in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

DEFAULT_INITIALIZED_FPP - Static variable in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

DEFAULT_MAX_ACCEPTABLE_FPP - Static variable in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

DEFAULT_RETAINED_RUNS - Static variable in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

deleteBatchSize() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

Size of the delete-batches when purging objects.

detailedInformation() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunInformation

Human-readable detailed information, possibly including technical error information.

Discussion Not all databases offer support to perform "prefix key" deletions, which are, for example, necessary to purge a whole realm. Some databases do support "deleting a huge number of rows". Some have another API for prefix-key deletions, for example, Google's BigTable dropRowRange on the table-admin-client. Relational databases may require different configurations with respect to isolation level to run those maintenance operations in a "better" way. Some databases do not support such "prefix-key deletions" at all, for example, Apache Cassandra or RocksDb or Amazon's DynamoDb. Backend implementations therefore expose whether it can leverage "prefix-key deletions" when one or more realms are to be purged. If a Backend does not support "prefix-key deletions", the whole repository has to be scanned. Purging unreferenced data The other maintenance operations like purging a catalog or unreferenced objects or references a two-step approach that works even for large multi-tenant setups: Memoize the current timestamp, subtract some amount to account for expected wall-clock drifts. Identify all objects and references that must be retained, memoize those in a probabilistic data structure (bloom filter). See below. Scan the whole database to identify the objects and references that were not identified as being referenced in the previous step. Delete the unreferenced objects and references if, and only if, their createdAtMicros() timestamp is less than the timestamp memoized in the first step. Identifying objects and references Implementations of @ApplicationScoped PerRealmRetainedIdentifier are called to identify the references and objects that have to be retained for a realm. Implementations of @ApplicationScoped ObjTypeRetainedIdentifier are called for each identified object of the requested object type. Realm status The maintenance service implementation will check the current status of the realm to retain and to purge, that the status is valid for being retained (valid: ACTIVE and INACTIVE) and being purged (valid: PURGING). Realms that have been asked to be purged and for which no data has been encountered will be state-transitioned to PURGED. System realm The system realm is maintained like every other realm. Future export use cases (TBD/TBC) These can be useful in a hosted and multi-tenant SaaS environment, when an export of the data for a particular realm is requested. Export live/referenced objects, filtered by realm. A possible implementation would hook into the implementation of PerRealmRetainedIdentifier via a delegate over Persistence. The actual approach and implementation is therefore out of the scope of the maintenance service. Low-level export, filtered by realm. This one is different from the one above, as it would export references and all object-parts, in contrast to fully materialized objects. A possible implementation would hook into the scanning-part of the maintenance service implementation. - Search tag in package org.apache.polaris.persistence.nosql.maintenance.api

Section

E

expectedObjCount() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

Provides the expected number of objects in all realms to retain, defaults to 100000L, must be at least 100000.

expectedReferenceCount() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

Provides the expected number of references in all realms to retain, defaults to 100L, must be at least 100.

F

filterInitializedFpp() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

False-positive-probability (FPP) used to initialize the bloom-filters for identified references and objects.

finished() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunInformation

Future export use cases (TBD/TBC) These can be useful in a hosted and multi-tenant SaaS environment, when an export of the data for a particular realm is requested. Export live/referenced objects, filtered by realm. A possible implementation would hook into the implementation of PerRealmRetainedIdentifier via a delegate over Persistence. The actual approach and implementation is therefore out of the scope of the maintenance service. Low-level export, filtered by realm. This one is different from the one above, as it would export references and all object-parts, in contrast to fully materialized objects. A possible implementation would hook into the scanning-part of the maintenance service implementation. - Search tag in package org.apache.polaris.persistence.nosql.maintenance.api

Section

I

identifiedObjs() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunInformation

Number of invocations of RetainedCollector.retainObject(ObjRef).

identifiedReferences() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunInformation

Number of invocations of RetainedCollector.retainReference(String).

Identifying objects and references Implementations of @ApplicationScoped PerRealmRetainedIdentifier are called to identify the references and objects that have to be retained for a realm. Implementations of @ApplicationScoped ObjTypeRetainedIdentifier are called for each identified object of the requested object type. Realm status The maintenance service implementation will check the current status of the realm to retain and to purge, that the status is valid for being retained (valid: ACTIVE and INACTIVE) and being purged (valid: PURGING). Realms that have been asked to be purged and for which no data has been encountered will be state-transitioned to PURGED. System realm The system realm is maintained like every other realm. Future export use cases (TBD/TBC) These can be useful in a hosted and multi-tenant SaaS environment, when an export of the data for a particular realm is requested. Export live/referenced objects, filtered by realm. A possible implementation would hook into the implementation of PerRealmRetainedIdentifier via a delegate over Persistence. The actual approach and implementation is therefore out of the scope of the maintenance service. Low-level export, filtered by realm. This one is different from the one above, as it would export references and all object-parts, in contrast to fully materialized objects. A possible implementation would hook into the scanning-part of the maintenance service implementation. - Search tag in package org.apache.polaris.persistence.nosql.maintenance.api

Section

includeSystemRealm() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunSpec

Whether to run maintenance on the system realm, defaults to true.

M

MaintenanceConfig - Interface in org.apache.polaris.persistence.nosql.maintenance.api

Maintenance service configuration.

MaintenanceRunInformation - Interface in org.apache.polaris.persistence.nosql.maintenance.api

MaintenanceRunInformation.MaintenanceStats - Interface in org.apache.polaris.persistence.nosql.maintenance.api

maintenanceRunLog() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceService

Retrieve information about recent maintenance runs.

MaintenanceRunSpec - Interface in org.apache.polaris.persistence.nosql.maintenance.api

Configures a maintenance run.

MaintenanceService - Interface in org.apache.polaris.persistence.nosql.maintenance.api

maxAcceptableFilterFpp() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

Expected maximum false-positive-probability (FPP) used to check the bloom-filters for identified references and objects.

N

newer() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunInformation.MaintenanceStats

Number of items that were written after the calculated grace time.

O

objectScanRateLimitPerSecond() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

Optionally limit the number of objects scanned per second.

objStats() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunInformation

org.apache.polaris.persistence.nosql.maintenance.api - package org.apache.polaris.persistence.nosql.maintenance.api

Maintenance operations include a bunch of tasks that are regularly executed against a backend database.

P

performMaintenance(MaintenanceRunSpec) - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceService

Perform maintenance.

perRealmPerObjTypeStats() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunInformation

perRealmReferenceStats() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunInformation

purged() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunInformation.MaintenanceStats

Number of scanned items that have been purged.

purgedRealms() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunInformation

Number of realms that were directly purges, if the backend supports this.

Purging unreferenced data The other maintenance operations like purging a catalog or unreferenced objects or references a two-step approach that works even for large multi-tenant setups: Memoize the current timestamp, subtract some amount to account for expected wall-clock drifts. Identify all objects and references that must be retained, memoize those in a probabilistic data structure (bloom filter). See below. Scan the whole database to identify the objects and references that were not identified as being referenced in the previous step. Delete the unreferenced objects and references if, and only if, their createdAtMicros() timestamp is less than the timestamp memoized in the first step. Identifying objects and references Implementations of @ApplicationScoped PerRealmRetainedIdentifier are called to identify the references and objects that have to be retained for a realm. Implementations of @ApplicationScoped ObjTypeRetainedIdentifier are called for each identified object of the requested object type. Realm status The maintenance service implementation will check the current status of the realm to retain and to purge, that the status is valid for being retained (valid: ACTIVE and INACTIVE) and being purged (valid: PURGING). Realms that have been asked to be purged and for which no data has been encountered will be state-transitioned to PURGED. System realm The system realm is maintained like every other realm. Future export use cases (TBD/TBC) These can be useful in a hosted and multi-tenant SaaS environment, when an export of the data for a particular realm is requested. Export live/referenced objects, filtered by realm. A possible implementation would hook into the implementation of PerRealmRetainedIdentifier via a delegate over Persistence. The actual approach and implementation is therefore out of the scope of the maintenance service. Low-level export, filtered by realm. This one is different from the one above, as it would export references and all object-parts, in contrast to fully materialized objects. A possible implementation would hook into the scanning-part of the maintenance service implementation. - Search tag in package org.apache.polaris.persistence.nosql.maintenance.api

Section

R

Realm status The maintenance service implementation will check the current status of the realm to retain and to purge, that the status is valid for being retained (valid: ACTIVE and INACTIVE) and being purged (valid: PURGING). Realms that have been asked to be purged and for which no data has been encountered will be state-transitioned to PURGED. System realm The system realm is maintained like every other realm. Future export use cases (TBD/TBC) These can be useful in a hosted and multi-tenant SaaS environment, when an export of the data for a particular realm is requested. Export live/referenced objects, filtered by realm. A possible implementation would hook into the implementation of PerRealmRetainedIdentifier via a delegate over Persistence. The actual approach and implementation is therefore out of the scope of the maintenance service. Low-level export, filtered by realm. This one is different from the one above, as it would export references and all object-parts, in contrast to fully materialized objects. A possible implementation would hook into the scanning-part of the maintenance service implementation. - Search tag in package org.apache.polaris.persistence.nosql.maintenance.api

Section

realmsToProcess() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunSpec

realmsToPurge() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunSpec

referenceScanRateLimitPerSecond() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

Optionally limit the number of references scanned per second.

referenceStats() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunInformation

retained() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunInformation.MaintenanceStats

Number of scanned items that were retained, because those were indicated to be retained by a realm identifier or obj-type identifier.

retainedRuns() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceConfig

Number of retained maintenance run objects, must be at least 2.

S

scanned() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunInformation.MaintenanceStats

Number of scanned items.

started() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunInformation

statusMessage() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunInformation

Human-readable status message.

success() - Method in interface org.apache.polaris.persistence.nosql.maintenance.api.MaintenanceRunInformation

System realm The system realm is maintained like every other realm. Future export use cases (TBD/TBC) These can be useful in a hosted and multi-tenant SaaS environment, when an export of the data for a particular realm is requested. Export live/referenced objects, filtered by realm. A possible implementation would hook into the implementation of PerRealmRetainedIdentifier via a delegate over Persistence. The actual approach and implementation is therefore out of the scope of the maintenance service. Low-level export, filtered by realm. This one is different from the one above, as it would export references and all object-parts, in contrast to fully materialized objects. A possible implementation would hook into the scanning-part of the maintenance service implementation. - Search tag in package org.apache.polaris.persistence.nosql.maintenance.api

Section

B C D E F I M N O P R S
All Classes and Interfaces|All Packages|Constant Field Values