Cleanup of Retention Window in Partioned Tables – sp_tblCleanupRetentionWindow

In my GitHub collection of SQL Scripts is now available a new stored procedure sp_tblCleanupRetentionWindow under the TablesManagement/Partitioning folder. Help for the function is available in the repository Wiki.

Scenario Overview

You have huge tables in the MS SQL Database and to be able to efectively manage the amount of data, you have partitioned the table.

As the amount of the data grows over the time, you start thinking about regular maintenance of such huge table and drop old not needed data.

You need to clean the oldest partitions and MERGE the partitions in the partitoin fuction to keep the number of partitions reasonable (especially if new partitions are created as new data are arriving).

To cleanup partitions prior SQL Server 2016 you have to perform partition switching to cleanup the partitions and then you could MERGE the partitions. On SQL Server 2016 and above the situation is simplified as you can use the TUNRCATE TABLE WITH PARTITIONS. But again to merge partition, you must put exact partition boundary values etc.

To automate regular maintenance you need to write sripts to handle all the needed tasks.

sp_tblCleanupRetentionWindow

As mentioned in the scenario above, the sp_tblCleanupRetentionWindow is the procedure which takes care about all the operations needed to cleanup the table prior specified retention window.

Parameters

ParameterNameData TypeDefaultDescription
@pfNamenvarchar(128)NULLName of the partition function. All associated partition schemes and tables will be cleaned according specified @retentionWindow
@retentionWindowsql_variantNULL
  • Specifies retention window
  • @retentionWindow must be of the dame exact data type as the partition fuction
  • All partitions prior the partition containging @retentionWindow value wil be cleared.If partitions contain data, partitions are first truncated.
  • All partitions prior partition containing the @retentionWindow will be merged into the first (leftmost) partition.
  • If @retentionWindow is part of the first (leftmost) partition, no cleanup happens.
@infoOnlybit1When 1 prints only information about affected tables and partitions and does not perform any cleanup. When 0 does the actual cleanup

How It Operates

Procedure takes @pfName parameter, which identifies partition function used by partitioned table(s). Then based on the @retentionWindow parameter it idetnfifies partition containing the @retentionWindow value.

The @retentionWindow parameter identifies the lowest value which must be kept in the table. All records prior that one can be cleared.

If the @retentionWindow value is part of the first (leftmost) partition, the procedure ends and does not perform any cleanup as there is nothing to cleanup. It cannot clean the partition containing the @retentionWindow value.

If the @retentionWindow is part of second and higher partitions, it scan for all Partitions Shemes using the @pfName partition function.

After identification of the partition schemes it identifies all the tables using the identified partitions shemes.

Once the tables are identified, it loops through all the tables and TRUNCATES all the partitions prior the partition containing the @retentionWindow value.

After the truncation the procedure ALTERS the parition function and merges all the partitions prior the partition containing the @retentionWindow value to the first (leftmost) partition.

It keeps always the first leftmost partition empty.

By the scrip the procedure is deployed into the [master] database and marked as system stored procedure. This allows simple call of the procedure from within all databases on the SQL Server instance and also thanks to that the procedure is running in the context of the database from within it is being called.

Sample

The sample below does cleanup fo all tables associated with the pf_partDate partition function. Cleanup and merge all partitions prior partition containign value of @retentionWindow = '2020-12-15'. pf_partDate is using the date data type.

If we take a closer look on the pf_partDate partition function by using the sp_HelpPartitionFunction we will find, it has multiple partitions and is used by multipel partition schemes and tables.

sp_HelpPartitionFunction 'pf_partDate', 1

Info Only

If we execute the procedure with @infoOnly=1 then only information about affected partitions and tables will be provided.

DECLARE @retentionWindow date = '2020-12-05'

EXEC sp_tblCleanupRetentionWindow
    @pfName             = 'pf_partDate'
    ,@retentionWindow   = @retentionWindow
    ,@infoOnly          = 1

Info Only Output

Below is sample output of the procedure which informs about actions which will be performed if the @infoOnly would be set to 0.

sp_tblCleanupRetentionWindow v0.10 (2021-05-07) (C) 2017-2021 Pavel Pawlowski
=============================================================================
Cleans retention window for all tables associated with partition function

Feedback mail to: pavel.pawlowski@hotmail.cz
Repository:       https://github.com/PavelPawlowski/SQL-Scripts
-----------------------------------------------------------------------------
Cleaning retention window for partition function [pf_partDate]
 
Partitions to Cleanup:
----------------------
[          ]    [x] <  [2020-11-01]
[2020-11-01] <= [x] <  [2020-12-01]
 
Affected Partition Schemes And Tables:
--------------------------------------
[ps_date1]
  - [dbo].[tblDate1]
  - [dbo].[tblDate11]
[ps_date2]
  - [dbo].[tblDate2]

From the output we can see, that there are two partitions which would be affected. First partition containing data prior 2020-10-01 and second partition containing data between the 2020-10-01 and 2020-11-01.

Also it informs, that there are two partitions schemes. Partition scheme [ps_date1] which is used by tables [dbo].[tblDate1] and [dbo].[tblDate11] and partition scheme [ps_date2] which is used by table [dbo].[tblDate2].

Cleanup

If we execute the procedure with @infoOnly=0 then information about affected partitions and tables will be provided as well as actual cleanup will be done.

DECLARE @retentionWindow date = '2020-12-05'

EXEC sp_tblCleanupRetentionWindow
    @pfName             = 'pf_partDate'
    ,@retentionWindow   = @retentionWindow
    ,@infoOnly          = 0

Cleanup output

sp_tblCleanupRetentionWindow v0.10 (2021-05-07) (C) 2017-2021 Pavel Pawlowski
=============================================================================
Cleans retention window for all tables associated with partition function

Feedback mail to: pavel.pawlowski@hotmail.cz
Repository:       https://github.com/PavelPawlowski/SQL-Scripts
-----------------------------------------------------------------------------
Cleaning retention window for partition function [pf_partDate]
 
Partitions to Cleanup:
----------------------
[          ]    [x] <  [2020-11-01]
[2020-11-01] <= [x] <  [2020-12-01]
 
Affected Partition Schemes And Tables:
--------------------------------------
[ps_date1]
  - [dbo].[tblDate1]
  - [dbo].[tblDate11]
[ps_date2]
  - [dbo].[tblDate2]
 
CLEANUP PROCESS
---------------------------------------------------------------
2021-05-18 23:54:11.4600180 +02:00 - Starting CLEANUP Process
2021-05-18 23:54:11.4610182 +02:00 - [dbo].[tblDate2] PARTITION 1 not empty. Starting TRUNCATE
2021-05-18 23:54:11.4620186 +02:00 - [dbo].[tblDate2] PARTITION 1 TRUNCATE completed
2021-05-18 23:54:11.4710215 +02:00 - [dbo].[tblDate11] PARTITION for range [2020-11-01] <= [x] <  [2020-12-01] not empty. Starting TRUNCATE
2021-05-18 23:54:11.4760227 +02:00 - [dbo].[tblDate11] PARTITION for range [2020-11-01] <= [x] <  [2020-12-01] TRUNCATE completed
2021-05-18 23:54:11.4770230 +02:00 - [pf_partDate] range [2020-11-01] <= [x] <  [2020-12-01] : Start MERGE into PARTITION 1
2021-05-18 23:54:11.4790237 +02:00 - [pf_partDate] range [2020-11-01] <= [x] <  [2020-12-01] : MERGE into PARTITION 1 completed. New range of PARTITION 1: [x] <  [2020-12-01]
2021-05-18 23:54:11.4790237 +02:00 - CLEANUP Process COMPLETED

Compared to the @infoOnly=1 output we see additional section “CLEANUP PROCESS’, which informs about the actual cleanup steps performed.

Procedure identified, that partition 1 of the [dbo].[tblDate2] was not empty and therefore performed TRUNCATE of the partition 1. Then it identified that the table [dbo].[tblDate11] has data in partition covering range between dates 2020-11-01 and 2020-12-01, therefore performed also truncate of that partitions. As there is no information about any truncate of the table [dbo].[tblDate1] it means the table had both affected partitions empty. The same is related to the other afected partitions of the [dbo].[tblDate2] and [dbo].[tblDate11].

After the partition cleanup it started the multi-step partition merge process. It merged each affected partition into the left most partition and informs about the new range of the leftmost partition 1.

If we again utilize the sp_HelpPartitionFunction we can see, that the partitions were properly cleared.

Azure support

The procedure supports also execution in Azure SQL Databases as well as Instances or Azure Synapse Analytics.

When deploying to Azure SQL Database or Synapse Analytics, comment-out the unsuported USE statement or ignore the eventual error as stated in the header of the procedure source file.

/* *****************************************************************************************
                                      AZURE SQL DB Notice

   Comment-out the unsupported USE [master] when running in Azure SQL DB/Synapse Analytics
   or ignore error caused by unsupported USE statement
******************************************************************************************** */

USE [master]
GO

SQL Server Version Limitation

Because the procedure internally is using TRUNCATE TABLE WITH PARTITION , procedure is limited only to the SQL Server 2016 and above or Azure SQL Database, Azure Managed Instance and Azure Synapse Analytics.

Summary

The sp_tblCleanupRetentionWindow heavily simplifies the management of huge partitioned table and moves the complete maintenance of retention window into a single call of a stored procedure. Hopefully you will find the procedure usefull.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s