Monday, November 21, 2022

Partitioned column data not displaying in External Table over Delta Lake in Serverless SQL Pool

Issue: You are trying to create an external table over a delta lake folder path in your Azure Data Lake. The said delta lake table is paritioned over a few columns. For example, see below where the delta lake table is paritioned over fields, 'TheYear' and 'TheMonth'. 

Delta Lake

The path was then created as an external table in Azure Synapse, and queried upon, where you can see all columns from the delta lake except for the paritioned columns, please see screenshot below,



Cause: This seems to be because this is not a feature, Serverless SQL pool has started to support(reference).

Solution: However there is an alternate way of achieveing the same using a SQL view with OPENROWSET functionality. Then use this view to query the table further.

sample syntax as below, 

    CREATE VIEW vName AS 
    SELECT TOP 10 *
    FROM OPENROWSET(
            BULK 'Gold/Database/Table1/',
            DATA_SOURCE = 'DeltaLakeStorage',
            FORMAT = 'delta'
        )
        WITH (  TheYear VARCHAR(10),
        TheMonth VARCHAR(10),
        type VARCHAR(100),
        count BIGINT
            ) 



No comments:

Post a Comment