Configuring the Cluster Sensitivity Profiler

In addition to the generic configuration, there are additional parameters for the Cluster Sensitivity Profiler that can be optionally edited.

You need the DataCatalogCspRuleManager role, to create, to deploy new Custom Sensitivity Profiler rules, to create new regex expressions, and to run validations on newly created rules.

Go to Profilers and select your data lake.
Go to Profilers > Configs.
Select Cluster Sensitivity Profiler.
The Detail page is displayed which contains the following sections:
Use the toggle button to enable or disable the profiler.
Select a schedule to run the profiler. This is implemented as a quartz cron expression.

note
Quartz CRON jobs (in VM-based environments) use the UTC timezone instead of the local timezone of the user.

For more information, see Understanding the Cron Expression generator.
Select Last Run Check and set a period if needed.

note

The Last Run Check enables profilers to avoid profiling the same asset on each scheduled run.

If you have scheduled a cron job, for example set to start in about an hour, and have enabled the Last Run Check configuration for two days, this setup ensures that the job scheduler filters out any asset which was already profiled in the last two days.

If the Last Run Check configuration is disabled, assets will be picked up for profiling as per the job cron schedule, honoring the asset filter rules.
Set the sample settings for VM-based environments:
1. Select the Sample Data Size.
  1. From the drop down, select the type of sample data size.
  2. Enter the value based on the previously selected type.
Continue with the resource settings.
1. In Advanced Options, set the following:
  - Number of Executors - Enter the number of executors to launch for running this profiler.
  - Executor Cores - Enter the number of cores to be used for each executor.
  - Executor Memory - Enter the amount of memory in GB to be used per executor process.
  - Driver Cores - Enter the number of cores to be used for the driver process.
  - Driver Memory - Enter the memory to be used for the driver processes.
  note
  For more information, see Configuring SPARK on YARN Applications and Tuning Resource Allocation.
Click Save to apply the configuration changes to the selected profiler.

Add Asset Filter Rules as needed to customize the selection and deselection of assets which the profiler profiles.

note

Profiler configurations apply to both scheduled and on-demand profiler jobs.
Asset filtering rules apply to assets, such as tables, and not to complete databases.
Multiple asset filtering rules are evaluated together as if connected by the OR operator.
In VM based environments, Deny lists are prioritized over Allow lists.

For example adding a regular expression for a database to the Deny list and adding a regular expression for a table within the first database to the Allow list will result in both entities filtered out. On the other hand, you can include all entities except one from a database by adding the database to the Allow list. Then, add the particular entity from the database to the Deny List.

Set your Deny List and Allow-list.

The profiler will skip profiling assets that meet any criteria in the Deny List and will include assets that meet any criteria in the Allow List.

Select the Deny-list or Allow List tab.
Click Add New to define new rules.

Select the key from the drop-down list and the relevant operator. You can select from the following:


Key	Operator
Database name	equals starts with ends with
Name (of asset)	equals contains starts with ends with
Owner (of asset)	equals contains starts with ends with
Creation date¹	greater than less than

Enter the value corresponding to the key. For example, you can enter a string as mentioned in the previous example.
Click Add Rule. Once a rule is added (enabled by default), you can toggle the state of the new rule to enable it or disable it as needed.

¹ By Creation Date, Greater than 7 days means an asset older than seven days. Less than 7 days means an asset younger than seven days.