Package org.apache.polaris.spark.utils

Class PolarisCatalogUtils

java.lang.Object

org.apache.polaris.spark.utils.PolarisCatalogUtils

Direct Known Subclasses:

PolarisCatalogSpark4Utils

public abstract class PolarisCatalogUtils
extends Object

Base class with helpers used by the Polaris Spark client.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	TABLE_PATH_KEY
static final String	TABLE_PROVIDER_KEY

Constructor Summary

Constructors

Constructor	Description
PolarisCatalogUtils()

Method Summary

Modifier and Type	Method	Description
static org.apache.iceberg.rest.auth.OAuth2Util.AuthSession	getAuthSession(org.apache.iceberg.spark.SparkCatalog sparkCatalog)	Get the catalogAuth field inside the RESTSessionCatalog used by Iceberg Spark Catalog use reflection.
static boolean	isTableWithSparkManagedLocation(Map<String,String> properties)	For tables whose location is managed by Spark Session Catalog, there will be no location or path in the properties.
abstract org.apache.spark.sql.connector.catalog.Table	loadV1SparkTable(GenericTable genericTable, org.apache.spark.sql.connector.catalog.Identifier identifier, String catalogName)	Return a Spark V1Table for formats that do not use DataSourceV2.
static org.apache.spark.sql.connector.catalog.Table	loadV2SparkTable(GenericTable genericTable)	Load spark table using DataSourceV2.
protected static Map<String,String>	normalizeTablePropertiesForLoadSparkTable(GenericTable genericTable)	Normalize table properties for loading Spark tables by ensuring the TABLE_PATH_KEY is properly set.
static boolean	useDelta(String provider)	Check whether the table provider is delta.
static boolean	useHudi(String provider)
static boolean	useIceberg(String provider)	Check whether the table provider is iceberg.
static boolean	usePaimon(String provider)	Check whether the table provider is paimon.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

TABLE_PROVIDER_KEY

public static final String TABLE_PROVIDER_KEY

See Also:

• Constant Field Values

TABLE_PATH_KEY

public static final String TABLE_PATH_KEY

See Also:

• Constant Field Values

Constructor Details

PolarisCatalogUtils

public PolarisCatalogUtils()

Method Details

useIceberg

public static boolean useIceberg(String provider)

Check whether the table provider is iceberg.

useDelta

public static boolean useDelta(String provider)

Check whether the table provider is delta.

useHudi

public static boolean useHudi(String provider)

usePaimon

public static boolean usePaimon(String provider)

Check whether the table provider is paimon.

isTableWithSparkManagedLocation

public static boolean isTableWithSparkManagedLocation(Map<String,String> properties)

For tables whose location is managed by Spark Session Catalog, there will be no location or path in the properties.

normalizeTablePropertiesForLoadSparkTable

protected static Map<String,String> normalizeTablePropertiesForLoadSparkTable(GenericTable genericTable)

Normalize table properties for loading Spark tables by ensuring the TABLE_PATH_KEY is properly set. DataSourceV2 requires the path property on table loading.

loadV2SparkTable

public static org.apache.spark.sql.connector.catalog.Table loadV2SparkTable(GenericTable genericTable)

Load spark table using DataSourceV2.

Returns:

V2Table if DataSourceV2 is available for the table format. For delta table, it returns DeltaTableV2.

loadV1SparkTable

public abstract org.apache.spark.sql.connector.catalog.Table loadV1SparkTable(GenericTable genericTable,
 org.apache.spark.sql.connector.catalog.Identifier identifier,
 String catalogName)

Return a Spark V1Table for formats that do not use DataSourceV2. Currently, this is being used for Hudi tables

getAuthSession

public static org.apache.iceberg.rest.auth.OAuth2Util.AuthSession getAuthSession(org.apache.iceberg.spark.SparkCatalog sparkCatalog)

Get the catalogAuth field inside the RESTSessionCatalog used by Iceberg Spark Catalog use reflection. TODO: Deprecate this function once the iceberg client is updated to 1.9.0 to use AuthManager and the capability of injecting an AuthManger is available. Related iceberg PR: https://github.com/apache/iceberg/pull/12655