Package org.apache.polaris.service.catalog.iceberg

Class LocalIcebergCatalog

java.lang.Object
org.apache.iceberg.BaseMetastoreCatalog
org.apache.iceberg.view.BaseMetastoreViewCatalog
org.apache.polaris.service.catalog.iceberg.LocalIcebergCatalog
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.iceberg.catalog.Catalog, org.apache.iceberg.catalog.SupportsNamespaces, org.apache.iceberg.catalog.ViewCatalog, SupportsNotifications
public class LocalIcebergCatalog extends org.apache.iceberg.view.BaseMetastoreViewCatalog implements org.apache.iceberg.catalog.SupportsNamespaces, SupportsNotifications, Closeable
Defines the relationship between PolarisEntities and Iceberg's business logic.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    class 
    LocalIcebergCatalog.BasePolarisTableOperations
    An implementation of TableOperations that integrates with LocalIcebergCatalog.

    Nested classes/interfaces inherited from class org.apache.iceberg.view.BaseMetastoreViewCatalog

    org.apache.iceberg.view.BaseMetastoreViewCatalog.BaseMetastoreViewCatalogTableBuilder, org.apache.iceberg.view.BaseMetastoreViewCatalog.BaseViewBuilder

    Nested classes/interfaces inherited from class org.apache.iceberg.BaseMetastoreCatalog

    org.apache.iceberg.BaseMetastoreCatalog.BaseMetastoreCatalogTableBuilder

    Nested classes/interfaces inherited from interface org.apache.iceberg.catalog.Catalog

    org.apache.iceberg.catalog.Catalog.TableBuilder

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final Predicate<Exception>
    SHOULD_RETRY_REFRESH_PREDICATE
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    LocalIcebergCatalog(org.apache.polaris.core.PolarisDiagnostics diagnostics, org.apache.polaris.core.persistence.resolver.ResolverFactory resolverFactory, org.apache.polaris.core.persistence.PolarisMetaStoreManager metaStoreManager, org.apache.polaris.core.context.CallContext callContext, org.apache.polaris.core.persistence.resolver.PolarisResolutionManifestCatalogView resolvedEntityView, org.apache.polaris.core.auth.PolarisPrincipal principal, TaskExecutor taskExecutor, StorageAccessConfigProvider storageAccessConfigProvider, FileIOFactory fileIOFactory, PolarisEventDispatcher polarisEventDispatcher, PolarisEventMetadataFactory eventMetadataFactory)
     
    LocalIcebergCatalog(org.apache.polaris.core.PolarisDiagnostics diagnostics, org.apache.polaris.core.persistence.resolver.ResolverFactory resolverFactory, org.apache.polaris.core.persistence.PolarisMetaStoreManager metaStoreManager, org.apache.polaris.core.context.CallContext callContext, org.apache.polaris.core.persistence.resolver.PolarisResolutionManifestCatalogView resolvedEntityView, org.apache.polaris.core.auth.PolarisPrincipal principal, TaskExecutor taskExecutor, StorageAccessConfigProvider storageAccessConfigProvider, FileIOFactory fileIOFactory, PolarisEventDispatcher polarisEventDispatcher, PolarisEventMetadataFactory eventMetadataFactory, IdempotencyRequestContext idempotencyRequestContext)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.iceberg.catalog.Catalog.TableBuilder
    buildTable(org.apache.iceberg.catalog.TableIdentifier identifier, org.apache.iceberg.Schema schema)
     
    org.apache.iceberg.view.ViewBuilder
    buildView(org.apache.iceberg.catalog.TableIdentifier identifier)
     
    void
    close()
     
    void
    createNamespace(org.apache.iceberg.catalog.Namespace namespace)
     
    void
    createNamespace(org.apache.iceberg.catalog.Namespace namespace, Map<String,String> metadata)
     
    protected String
    defaultWarehouseLocation(org.apache.iceberg.catalog.TableIdentifier tableIdentifier)
     
    boolean
    dropNamespace(org.apache.iceberg.catalog.Namespace namespace)
     
    boolean
    dropTable(org.apache.iceberg.catalog.TableIdentifier tableIdentifier, boolean purge)
     
    boolean
    dropView(org.apache.iceberg.catalog.TableIdentifier identifier)
     
    void
    initialize(String name, Map<String,String> properties)
     
    List<org.apache.iceberg.catalog.Namespace>
    listNamespaces()
     
    List<org.apache.iceberg.catalog.Namespace>
    listNamespaces(org.apache.iceberg.catalog.Namespace namespace)
     
    org.apache.polaris.core.persistence.pagination.Page<org.apache.iceberg.catalog.Namespace>
    listNamespaces(org.apache.iceberg.catalog.Namespace namespace, org.apache.polaris.core.persistence.pagination.PageToken pageToken)
     
    List<org.apache.iceberg.catalog.TableIdentifier>
    listTables(org.apache.iceberg.catalog.Namespace namespace)
     
    org.apache.polaris.core.persistence.pagination.Page<org.apache.iceberg.catalog.TableIdentifier>
    listTables(org.apache.iceberg.catalog.Namespace namespace, org.apache.polaris.core.persistence.pagination.PageToken pageToken)
     
    List<org.apache.iceberg.catalog.TableIdentifier>
    listViews(org.apache.iceberg.catalog.Namespace namespace)
     
    org.apache.polaris.core.persistence.pagination.Page<org.apache.iceberg.catalog.TableIdentifier>
    listViews(org.apache.iceberg.catalog.Namespace namespace, org.apache.polaris.core.persistence.pagination.PageToken pageToken)
     
    Map<String,String>
    loadNamespaceMetadata(org.apache.iceberg.catalog.Namespace namespace)
     
    org.apache.iceberg.view.View
    loadView(org.apache.iceberg.catalog.TableIdentifier identifier)
    Override to fix a bug in BaseMetastoreViewCatalog.loadView(org.apache.iceberg.catalog.TableIdentifier) where newViewOps(org.apache.iceberg.catalog.TableIdentifier) is called twice, causing redundant metadata fetches.
    String
    name()
     
    boolean
    namespaceExists(org.apache.iceberg.catalog.Namespace namespace)
     
    protected org.apache.iceberg.TableOperations
    newTableOps(org.apache.iceberg.catalog.TableIdentifier tableIdentifier)
     
    org.apache.iceberg.TableOperations
    newTableOps(org.apache.iceberg.catalog.TableIdentifier tableIdentifier, boolean makeMetadataCurrentOnCommit)
     
    protected org.apache.iceberg.view.ViewOperations
    newViewOps(org.apache.iceberg.catalog.TableIdentifier identifier)
     
    protected Map<String,String>
    properties()
     
    org.apache.iceberg.Table
    registerTable(org.apache.iceberg.catalog.TableIdentifier identifier, String metadataFileLocation)
     
    org.apache.iceberg.Table
    registerTable(org.apache.iceberg.catalog.TableIdentifier identifier, String metadataFileLocation, boolean overwrite)
    Register a table with optional overwrite semantics.
    org.apache.iceberg.view.View
    registerView(org.apache.iceberg.catalog.TableIdentifier identifier, String metadataFileLocation)
     
    boolean
    removeProperties(org.apache.iceberg.catalog.Namespace namespace, Set<String> properties)
     
    void
    renameTable(org.apache.iceberg.catalog.TableIdentifier from, org.apache.iceberg.catalog.TableIdentifier to)
     
    void
    renameView(org.apache.iceberg.catalog.TableIdentifier from, org.apache.iceberg.catalog.TableIdentifier to)
     
    boolean
    sendNotification(org.apache.iceberg.catalog.TableIdentifier identifier, org.apache.polaris.service.types.NotificationRequest notificationRequest)
     
    void
    setCatalogFileIo(org.apache.iceberg.io.FileIO fileIO)
     
    void
    setMetaStoreManager(org.apache.polaris.core.persistence.PolarisMetaStoreManager newMetaStoreManager)
     
    void
    setMetaStoreManager(org.apache.polaris.core.persistence.PolarisMetaStoreManager newMetaStoreManager, Consumer<org.apache.polaris.service.catalog.iceberg.MetadataFileCleanup> deleteLogic)
    Sets the meta store manager and optionally a custom logic for deleting removed metadata files.
    boolean
    setProperties(org.apache.iceberg.catalog.Namespace namespace, Map<String,String> properties)
     
    String
    transformTableLikeLocation(org.apache.iceberg.catalog.TableIdentifier tableIdentifier, String location)
    Based on configuration settings, for callsites that need to handle potentially setting a new base location for a TableLike entity, produces the transformed location if applicable, or else the unaltered specified location.

    Methods inherited from class org.apache.iceberg.BaseMetastoreCatalog

    fullTableName, isValidIdentifier, isValidMetadataIdentifier, loadTable, metricsReporter, toString

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

    Methods inherited from interface org.apache.iceberg.catalog.Catalog

    createTable, createTable, createTable, createTable, dropTable, invalidateTable, newCreateTableTransaction, newCreateTableTransaction, newCreateTableTransaction, newCreateTableTransaction, newReplaceTableTransaction, newReplaceTableTransaction, newReplaceTableTransaction, newReplaceTableTransaction, tableExists

    Methods inherited from interface org.apache.iceberg.catalog.ViewCatalog

    invalidateView, viewExists

  • Field Details

    • SHOULD_RETRY_REFRESH_PREDICATE

      public static final Predicate<Exception> SHOULD_RETRY_REFRESH_PREDICATE

  • Constructor Details

    • LocalIcebergCatalog

      public LocalIcebergCatalog(org.apache.polaris.core.PolarisDiagnostics diagnostics, org.apache.polaris.core.persistence.resolver.ResolverFactory resolverFactory, org.apache.polaris.core.persistence.PolarisMetaStoreManager metaStoreManager, org.apache.polaris.core.context.CallContext callContext, org.apache.polaris.core.persistence.resolver.PolarisResolutionManifestCatalogView resolvedEntityView, org.apache.polaris.core.auth.PolarisPrincipal principal, TaskExecutor taskExecutor, StorageAccessConfigProvider storageAccessConfigProvider, FileIOFactory fileIOFactory, PolarisEventDispatcher polarisEventDispatcher, PolarisEventMetadataFactory eventMetadataFactory)
      Parameters:
      callContext - the current CallContext
      resolvedEntityView - accessor to resolved entity paths that have been pre-vetted to ensure this catalog instance only interacts with authorized resolved paths.
      taskExecutor - Executor we use to register cleanup task handlers

    • LocalIcebergCatalog

      public LocalIcebergCatalog(org.apache.polaris.core.PolarisDiagnostics diagnostics, org.apache.polaris.core.persistence.resolver.ResolverFactory resolverFactory, org.apache.polaris.core.persistence.PolarisMetaStoreManager metaStoreManager, org.apache.polaris.core.context.CallContext callContext, org.apache.polaris.core.persistence.resolver.PolarisResolutionManifestCatalogView resolvedEntityView, org.apache.polaris.core.auth.PolarisPrincipal principal, TaskExecutor taskExecutor, StorageAccessConfigProvider storageAccessConfigProvider, FileIOFactory fileIOFactory, PolarisEventDispatcher polarisEventDispatcher, PolarisEventMetadataFactory eventMetadataFactory, IdempotencyRequestContext idempotencyRequestContext)

  • Method Details

    • name

      public String name()
      Specified by:
      name in interface org.apache.iceberg.catalog.Catalog
      Specified by:
      name in interface org.apache.iceberg.catalog.ViewCatalog
      Overrides:
      name in class org.apache.iceberg.view.BaseMetastoreViewCatalog

    • setCatalogFileIo

      public void setCatalogFileIo(org.apache.iceberg.io.FileIO fileIO)

    • initialize

      public void initialize(String name, Map<String,String> properties)
      Specified by:
      initialize in interface org.apache.iceberg.catalog.Catalog
      Specified by:
      initialize in interface org.apache.iceberg.catalog.ViewCatalog
      Overrides:
      initialize in class org.apache.iceberg.view.BaseMetastoreViewCatalog

    • setMetaStoreManager

      public void setMetaStoreManager(org.apache.polaris.core.persistence.PolarisMetaStoreManager newMetaStoreManager)

    • setMetaStoreManager

      public void setMetaStoreManager(org.apache.polaris.core.persistence.PolarisMetaStoreManager newMetaStoreManager, Consumer<org.apache.polaris.service.catalog.iceberg.MetadataFileCleanup> deleteLogic)
      Sets the meta store manager and optionally a custom logic for deleting removed metadata files. This allows the caller (e.g. during commitTransaction) to provide a collector instead of immediate deletion, avoiding premature deletes that could lead to corruption on rollback.

    • properties

      protected Map<String,String> properties()
      Overrides:
      properties in class org.apache.iceberg.BaseMetastoreCatalog

    • registerTable

      public org.apache.iceberg.Table registerTable(org.apache.iceberg.catalog.TableIdentifier identifier, String metadataFileLocation)
      Specified by:
      registerTable in interface org.apache.iceberg.catalog.Catalog
      Overrides:
      registerTable in class org.apache.iceberg.BaseMetastoreCatalog

    • registerTable

      public org.apache.iceberg.Table registerTable(org.apache.iceberg.catalog.TableIdentifier identifier, String metadataFileLocation, boolean overwrite)
      Register a table with optional overwrite semantics.

      When overwrite is false (the default) this behaves like a normal register and will fail if the table already exists. When overwrite is true and the named table already exists, this method updates the table's stored metadata-location to point at the provided metadata file. The overwrite path performs additional validation to ensure the supplied metadata file and its location are consistent with the table's resolved storage configuration.

      Specified by:
      registerTable in interface org.apache.iceberg.catalog.Catalog
      Parameters:
      identifier - the table identifier
      metadataFileLocation - the metadata file location
      overwrite - if true, update existing table metadata; if false, throw exception if table exists
      Returns:
      the registered table

    • buildTable

      public org.apache.iceberg.catalog.Catalog.TableBuilder buildTable(org.apache.iceberg.catalog.TableIdentifier identifier, org.apache.iceberg.Schema schema)
      Specified by:
      buildTable in interface org.apache.iceberg.catalog.Catalog
      Overrides:
      buildTable in class org.apache.iceberg.view.BaseMetastoreViewCatalog

    • buildView

      public org.apache.iceberg.view.ViewBuilder buildView(org.apache.iceberg.catalog.TableIdentifier identifier)
      Specified by:
      buildView in interface org.apache.iceberg.catalog.ViewCatalog
      Overrides:
      buildView in class org.apache.iceberg.view.BaseMetastoreViewCatalog

    • newTableOps

      public org.apache.iceberg.TableOperations newTableOps(org.apache.iceberg.catalog.TableIdentifier tableIdentifier, boolean makeMetadataCurrentOnCommit)

    • newTableOps

      protected org.apache.iceberg.TableOperations newTableOps(org.apache.iceberg.catalog.TableIdentifier tableIdentifier)
      Specified by:
      newTableOps in class org.apache.iceberg.BaseMetastoreCatalog

    • defaultWarehouseLocation

      protected String defaultWarehouseLocation(org.apache.iceberg.catalog.TableIdentifier tableIdentifier)
      Specified by:
      defaultWarehouseLocation in class org.apache.iceberg.BaseMetastoreCatalog

    • dropTable

      public boolean dropTable(org.apache.iceberg.catalog.TableIdentifier tableIdentifier, boolean purge)
      Specified by:
      dropTable in interface org.apache.iceberg.catalog.Catalog

    • listTables

      public List<org.apache.iceberg.catalog.TableIdentifier> listTables(org.apache.iceberg.catalog.Namespace namespace)
      Specified by:
      listTables in interface org.apache.iceberg.catalog.Catalog

    • listTables

      public org.apache.polaris.core.persistence.pagination.Page<org.apache.iceberg.catalog.TableIdentifier> listTables(org.apache.iceberg.catalog.Namespace namespace, org.apache.polaris.core.persistence.pagination.PageToken pageToken)

    • renameTable

      public void renameTable(org.apache.iceberg.catalog.TableIdentifier from, org.apache.iceberg.catalog.TableIdentifier to)
      Specified by:
      renameTable in interface org.apache.iceberg.catalog.Catalog

    • createNamespace

      public void createNamespace(org.apache.iceberg.catalog.Namespace namespace)
      Specified by:
      createNamespace in interface org.apache.iceberg.catalog.SupportsNamespaces

    • createNamespace

      public void createNamespace(org.apache.iceberg.catalog.Namespace namespace, Map<String,String> metadata)
      Specified by:
      createNamespace in interface org.apache.iceberg.catalog.SupportsNamespaces

    • namespaceExists

      public boolean namespaceExists(org.apache.iceberg.catalog.Namespace namespace)
      Specified by:
      namespaceExists in interface org.apache.iceberg.catalog.SupportsNamespaces

    • dropNamespace

      public boolean dropNamespace(org.apache.iceberg.catalog.Namespace namespace) throws org.apache.iceberg.exceptions.NamespaceNotEmptyException
      Specified by:
      dropNamespace in interface org.apache.iceberg.catalog.SupportsNamespaces
      Throws:
      org.apache.iceberg.exceptions.NamespaceNotEmptyException

    • setProperties

      public boolean setProperties(org.apache.iceberg.catalog.Namespace namespace, Map<String,String> properties) throws org.apache.iceberg.exceptions.NoSuchNamespaceException
      Specified by:
      setProperties in interface org.apache.iceberg.catalog.SupportsNamespaces
      Throws:
      org.apache.iceberg.exceptions.NoSuchNamespaceException

    • removeProperties

      public boolean removeProperties(org.apache.iceberg.catalog.Namespace namespace, Set<String> properties) throws org.apache.iceberg.exceptions.NoSuchNamespaceException
      Specified by:
      removeProperties in interface org.apache.iceberg.catalog.SupportsNamespaces
      Throws:
      org.apache.iceberg.exceptions.NoSuchNamespaceException

    • loadNamespaceMetadata

      public Map<String,String> loadNamespaceMetadata(org.apache.iceberg.catalog.Namespace namespace) throws org.apache.iceberg.exceptions.NoSuchNamespaceException
      Specified by:
      loadNamespaceMetadata in interface org.apache.iceberg.catalog.SupportsNamespaces
      Throws:
      org.apache.iceberg.exceptions.NoSuchNamespaceException

    • listNamespaces

      public List<org.apache.iceberg.catalog.Namespace> listNamespaces()
      Specified by:
      listNamespaces in interface org.apache.iceberg.catalog.SupportsNamespaces

    • listNamespaces

      public List<org.apache.iceberg.catalog.Namespace> listNamespaces(org.apache.iceberg.catalog.Namespace namespace) throws org.apache.iceberg.exceptions.NoSuchNamespaceException
      Specified by:
      listNamespaces in interface org.apache.iceberg.catalog.SupportsNamespaces
      Throws:
      org.apache.iceberg.exceptions.NoSuchNamespaceException

    • listNamespaces

      public org.apache.polaris.core.persistence.pagination.Page<org.apache.iceberg.catalog.Namespace> listNamespaces(org.apache.iceberg.catalog.Namespace namespace, org.apache.polaris.core.persistence.pagination.PageToken pageToken) throws org.apache.iceberg.exceptions.NoSuchNamespaceException
      Throws:
      org.apache.iceberg.exceptions.NoSuchNamespaceException

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Overrides:
      close in class org.apache.iceberg.BaseMetastoreCatalog
      Throws:
      IOException

    • listViews

      public List<org.apache.iceberg.catalog.TableIdentifier> listViews(org.apache.iceberg.catalog.Namespace namespace)
      Specified by:
      listViews in interface org.apache.iceberg.catalog.ViewCatalog

    • listViews

      public org.apache.polaris.core.persistence.pagination.Page<org.apache.iceberg.catalog.TableIdentifier> listViews(org.apache.iceberg.catalog.Namespace namespace, org.apache.polaris.core.persistence.pagination.PageToken pageToken)

    • newViewOps

      protected org.apache.iceberg.view.ViewOperations newViewOps(org.apache.iceberg.catalog.TableIdentifier identifier)
      Specified by:
      newViewOps in class org.apache.iceberg.view.BaseMetastoreViewCatalog

    • loadView

      public org.apache.iceberg.view.View loadView(org.apache.iceberg.catalog.TableIdentifier identifier)
      Override to fix a bug in BaseMetastoreViewCatalog.loadView(org.apache.iceberg.catalog.TableIdentifier) where newViewOps(org.apache.iceberg.catalog.TableIdentifier) is called twice, causing redundant metadata fetches. This implementation reuses the same ViewOperations instance.
      Specified by:
      loadView in interface org.apache.iceberg.catalog.ViewCatalog
      Overrides:
      loadView in class org.apache.iceberg.view.BaseMetastoreViewCatalog

    • registerView

      public org.apache.iceberg.view.View registerView(org.apache.iceberg.catalog.TableIdentifier identifier, String metadataFileLocation)
      Specified by:
      registerView in interface org.apache.iceberg.catalog.ViewCatalog
      Overrides:
      registerView in class org.apache.iceberg.view.BaseMetastoreViewCatalog

    • dropView

      public boolean dropView(org.apache.iceberg.catalog.TableIdentifier identifier)
      Specified by:
      dropView in interface org.apache.iceberg.catalog.ViewCatalog

    • renameView

      public void renameView(org.apache.iceberg.catalog.TableIdentifier from, org.apache.iceberg.catalog.TableIdentifier to)
      Specified by:
      renameView in interface org.apache.iceberg.catalog.ViewCatalog

    • sendNotification

      public boolean sendNotification(org.apache.iceberg.catalog.TableIdentifier identifier, org.apache.polaris.service.types.NotificationRequest notificationRequest)
      Specified by:
      sendNotification in interface SupportsNotifications

    • transformTableLikeLocation

      public String transformTableLikeLocation(org.apache.iceberg.catalog.TableIdentifier tableIdentifier, String location)
      Based on configuration settings, for callsites that need to handle potentially setting a new base location for a TableLike entity, produces the transformed location if applicable, or else the unaltered specified location.