Cloudera Docs

Prerequisites for the provisioning credential

To allow Cloudera to create resources on your Azure account, you must create app-based or certificate based credentials. The credential allows Cloudera to access and provision a set of resources in your Azure account.

Cloudera uses an app-based credential or certificate based credential to authenticate your Azure account and obtain authorization to create resources on your behalf.

The app-based credential requires that you to manually configure the service principal created within your Azure Active Directory. The app-based method requires Owner role to be able to create a service principal, which must be given Contributor role or its equivalent.

Certificate based authentication uses asymmetric authentication based on private-keys and public-keys, and does not rely on sharing secrets. When using the certificate based credential, Cloudera generates a private key and a certificate (public key) pair, and shares the certificate with you. After the certificate is shared, you must register it with Azure.

To meet Azure prerequisites for Cloudera:
  1. Review the provided policies.
  2. Obtain the subscription and tenant ID.
  3. Create an app registration on Azure or generate and register Azure certificate credential

Azure permissions

Your Azure administrator must create custom roles in the Azure subscription.

  1. The administrator must create a custom roles containing the following sets of permissions sufficient for registering an environment and creating Cloudera Data Hub clusters and Cloudera Operational Database:
    1. The administrator must create a custom role containing one of the following sets of permissions on the scope of the resource group used for Cloudera:
      1. Option 1: Use the Role definition 1: Allows Cloudera to access and use only a single existing resource group and create service endpoints if you would like to use service endpoints.
      2. Option 2: Use the Role definition 2: Allows Cloudera to access and use only a single existing resource group and create private endpoints if you would like to use private endpoints.
    2. In order to use the Azure Marketplace images that Cloudera publishes, the administrator also needs to grant the service principal the additional Azure permissions on the scope of your Azure subscription. The Role definition for Azure Marketplace images policy includes these permissions.
  2. Additionally, if you would like to provision other Cloudera services (Cloudera Data Engineering, Cloudera DataFlow, Cloudera Data Warehouse, or Cloudera AI), you should assign the built-in Contributor Azure role either at the resource group level (if you are providing your own resource group) or at the Azure subscription level (if Cloudera is creating resource groups).
    • If you need Cloudera Data Warehouse only and Cloudera is creating resource groups, you can use the minimal policy documented in Cloudera Data Warehouse documentation.
    • If you need Cloudera DataFlow only and Cloudera is creating resource groups, you can use the minimal policy documented in Cloudera DataFlow documentation.
    • If you need Cloudera AI only and Cloudera is creating resource groups, you can use the minimal policy documented in Cloudera AI documentation.

Role definition 1: Allows Cloudera to access and use only a single existing resource group and create service endpoints

The following role definition allows Cloudera to create resources only within the specified resource group:
{
    "Name": "Cloudera Management Console Azure Operator For Single Resource Group",
    "IsCustom": true,
    "Description": "Can use Cloudera Management Console managed clusters and resources updated for single resource group.",
    "Actions": [
      "Microsoft.Storage/storageAccounts/read",
      "Microsoft.Storage/storageAccounts/write",
      "Microsoft.Storage/storageAccounts/blobServices/write",
      "Microsoft.Storage/storageAccounts/blobServices/containers/delete",
      "Microsoft.Storage/storageAccounts/blobServices/containers/read",
      "Microsoft.Storage/storageAccounts/blobServices/containers/write",
      "Microsoft.Storage/storageAccounts/fileServices/write",
      "Microsoft.Storage/storageAccounts/listkeys/action",
      "Microsoft.Storage/storageAccounts/regeneratekey/action",
      "Microsoft.Storage/storageAccounts/delete",
      "Microsoft.Storage/locations/deleteVirtualNetworkOrSubnets/action",
      "Microsoft.Network/virtualNetworks/read",
      "Microsoft.Network/virtualNetworks/write",
      "Microsoft.Network/virtualNetworks/delete",
      "Microsoft.Network/virtualNetworks/subnets/read",
      "Microsoft.Network/virtualNetworks/subnets/write",
      "Microsoft.Network/virtualNetworks/subnets/delete",
      "Microsoft.Network/virtualNetworks/subnets/join/action",
      "Microsoft.Network/publicIPAddresses/read",
      "Microsoft.Network/publicIPAddresses/write",
      "Microsoft.Network/publicIPAddresses/delete",
      "Microsoft.Network/publicIPAddresses/join/action",
      "Microsoft.Network/networkInterfaces/read",
      "Microsoft.Network/networkInterfaces/write",
      "Microsoft.Network/networkInterfaces/delete",
      "Microsoft.Network/networkInterfaces/join/action",
      "Microsoft.Network/networkInterfaces/ipconfigurations/read",
      "Microsoft.Network/networkSecurityGroups/read",
      "Microsoft.Network/networkSecurityGroups/write",
      "Microsoft.Network/networkSecurityGroups/delete",
      "Microsoft.Network/networkSecurityGroups/join/action",
      "Microsoft.Network/virtualNetworks/subnets/joinViaServiceEndpoint/action",
      "Microsoft.Network/loadBalancers/delete",
      "Microsoft.Network/loadBalancers/read",
      "Microsoft.Network/loadBalancers/write",
      "Microsoft.Network/loadBalancers/backendAddressPools/join/action",
      "Microsoft.Compute/availabilitySets/read",
      "Microsoft.Compute/availabilitySets/write",
      "Microsoft.Compute/availabilitySets/delete",
      "Microsoft.Compute/disks/read",
      "Microsoft.Compute/disks/write",
      "Microsoft.Compute/disks/delete",
      "Microsoft.Compute/images/read",
      "Microsoft.Compute/images/write",
      "Microsoft.Compute/images/delete",
      "Microsoft.Compute/virtualMachines/read",
      "Microsoft.Compute/virtualMachines/write",
      "Microsoft.Compute/virtualMachines/delete",
      "Microsoft.Compute/virtualMachines/start/action",
      "Microsoft.Compute/virtualMachines/restart/action",
      "Microsoft.Compute/virtualMachines/deallocate/action",
      "Microsoft.Compute/virtualMachines/powerOff/action",
      "Microsoft.Compute/virtualMachines/vmSizes/read",
      "Microsoft.Authorization/roleAssignments/read",
      "Microsoft.Resources/subscriptions/resourceGroups/read",
      "Microsoft.Resources/deployments/read",
      "Microsoft.Resources/deployments/write",
      "Microsoft.Resources/deployments/delete",
      "Microsoft.Resources/deployments/operations/read",
      "Microsoft.Resources/deployments/operationstatuses/read",
      "Microsoft.Resources/deployments/exportTemplate/action",
      "Microsoft.Resources/subscriptions/read",
      "Microsoft.ManagedIdentity/userAssignedIdentities/read",
      "Microsoft.ManagedIdentity/userAssignedIdentities/assign/action",
      "Microsoft.DBforPostgreSQL/servers/read",
      "Microsoft.DBforPostgreSQL/servers/write",
      "Microsoft.DBforPostgreSQL/servers/delete",
      "Microsoft.DBforPostgreSQL/servers/virtualNetworkRules/write",
      "Microsoft.DBforPostgreSQL/flexibleServers/read",
      "Microsoft.DBforPostgreSQL/flexibleServers/write",
      "Microsoft.DBforPostgreSQL/flexibleServers/delete",
      "Microsoft.DBforPostgreSQL/flexibleServers/start/action",
      "Microsoft.DBforPostgreSQL/flexibleServers/stop/action",
      "Microsoft.DBforPostgreSQL/flexibleServers/firewallRules/write",
      "Microsoft.DBforPostgreSQL/flexibleServers/configurations/write", 
      "Microsoft.Resources/deployments/cancel/action",
      "Microsoft.Resources/deployments/whatIf/action"
    ],
    "NotActions": [],
    "DataActions": [
      "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read",
      "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write",
      "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete",
      "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action"
    ],
    "NotDataActions": [],
    "AssignableScopes": [
      "/subscriptions/{SUBSCRIPTION-ID}/resourceGroups/{RESOURCE-GROUP-NAME}"
    ]
  }

When creating the role definition, make sure to:

  • Replace the {SUBSCRIPTION-ID} with your actual subscription ID.

  • Replace the {RESOURCE-GROUP-NAME} with the ID of your existing resource group

Role definition 2: Allows Cloudera to access and use only a single existing resource group and create private endpoints

The following role definition allows Cloudera to create resources only within the specified resource group:

{
    "Name": "Cloudera Management Console Azure Operator for Single Resource Group",
    "IsCustom": true,
    "Description": "Can use Cloudera Management Console managed clusters and resources, updated for use with single resource group for all resources.",
    "Actions": [
      "Microsoft.Storage/storageAccounts/read",
      "Microsoft.Storage/storageAccounts/write",
      "Microsoft.Storage/storageAccounts/blobServices/write",
      "Microsoft.Storage/storageAccounts/blobServices/containers/delete",
      "Microsoft.Storage/storageAccounts/blobServices/containers/read",
      "Microsoft.Storage/storageAccounts/blobServices/containers/write",
      "Microsoft.Storage/storageAccounts/fileServices/write",
      "Microsoft.Storage/storageAccounts/listkeys/action",
      "Microsoft.Storage/storageAccounts/regeneratekey/action",
      "Microsoft.Storage/storageAccounts/delete",
      "Microsoft.Storage/locations/deleteVirtualNetworkOrSubnets/action",
      "Microsoft.Network/virtualNetworks/read",
      "Microsoft.Network/virtualNetworks/write",
      "Microsoft.Network/virtualNetworks/delete",
      "Microsoft.Network/virtualNetworks/subnets/read",
      "Microsoft.Network/virtualNetworks/subnets/write",
      "Microsoft.Network/virtualNetworks/subnets/delete",
      "Microsoft.Network/virtualNetworks/subnets/join/action",
      "Microsoft.Network/publicIPAddresses/read",
      "Microsoft.Network/publicIPAddresses/write",
      "Microsoft.Network/publicIPAddresses/delete",
      "Microsoft.Network/publicIPAddresses/join/action",
      "Microsoft.Network/networkInterfaces/read",
      "Microsoft.Network/networkInterfaces/write",
      "Microsoft.Network/networkInterfaces/delete",
      "Microsoft.Network/networkInterfaces/join/action",
      "Microsoft.Network/networkInterfaces/ipconfigurations/read",
      "Microsoft.Network/networkSecurityGroups/read",
      "Microsoft.Network/networkSecurityGroups/write",
      "Microsoft.Network/networkSecurityGroups/delete",
      "Microsoft.Network/networkSecurityGroups/join/action",
      "Microsoft.Compute/availabilitySets/read",
      "Microsoft.Compute/availabilitySets/write",
      "Microsoft.Compute/availabilitySets/delete",
      "Microsoft.Compute/disks/read",
      "Microsoft.Compute/disks/write",
      "Microsoft.Compute/disks/delete",
      "Microsoft.Compute/images/read",
      "Microsoft.Compute/images/write",
      "Microsoft.Compute/images/delete",
      "Microsoft.Compute/virtualMachines/read",
      "Microsoft.Compute/virtualMachines/write",
      "Microsoft.Compute/virtualMachines/delete",
      "Microsoft.Compute/virtualMachines/powerOff/action",
      "Microsoft.Compute/virtualMachines/start/action",
      "Microsoft.Compute/virtualMachines/restart/action",
      "Microsoft.Compute/virtualMachines/deallocate/action",
      "Microsoft.Compute/virtualMachines/vmSizes/read",
      "Microsoft.Authorization/roleAssignments/read",
      "Microsoft.Resources/subscriptions/resourceGroups/read",
      "Microsoft.Resources/deployments/read",
      "Microsoft.Resources/deployments/write",
      "Microsoft.Resources/deployments/delete",
      "Microsoft.Resources/deployments/operations/read",
      "Microsoft.Resources/deployments/operationstatuses/read",
      "Microsoft.Resources/deployments/exportTemplate/action",
      "Microsoft.Resources/subscriptions/read",
      "Microsoft.ManagedIdentity/userAssignedIdentities/read",
      "Microsoft.ManagedIdentity/userAssignedIdentities/assign/action",
      "Microsoft.DBforPostgreSQL/servers/read",
      "Microsoft.DBforPostgreSQL/servers/write",
      "Microsoft.DBforPostgreSQL/servers/delete",
      "Microsoft.DBforPostgreSQL/flexibleServers/read",
      "Microsoft.DBforPostgreSQL/flexibleServers/write",
      "Microsoft.DBforPostgreSQL/flexibleServers/delete",
      "Microsoft.DBforPostgreSQL/flexibleServers/start/action",
      "Microsoft.DBforPostgreSQL/flexibleServers/stop/action",
      "Microsoft.DBforPostgreSQL/flexibleServers/firewallRules/write",
      "Microsoft.DBforPostgreSQL/flexibleServers/configurations/write",
      "Microsoft.DBforPostgreSQL/flexibleServers/privateEndpointConnectionsApproval/action", 
      "Microsoft.Network/privateDnsZones/read",
      "Microsoft.Network/privateEndpoints/read",
      "Microsoft.Network/privateEndpoints/write",
      "Microsoft.Network/privateEndpoints/delete",
      "Microsoft.Network/privateEndpoints/privateDnsZoneGroups/read",
      "Microsoft.Network/privateEndpoints/privateDnsZoneGroups/write",
      "Microsoft.DBforPostgreSQL/servers/privateEndpointConnectionsApproval/action",
      "Microsoft.Network/privateDnsZones/A/read",
      "Microsoft.Network/privateDnsZones/A/write",
      "Microsoft.Network/privateDnsZones/A/delete",
      "Microsoft.Network/privateDnsZones/join/action",
      "Microsoft.Network/privateDnsZones/write",
      "Microsoft.Network/privateDnsZones/delete",
      "Microsoft.Network/privateDnsZones/virtualNetworkLinks/read", 
      "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write",
      "Microsoft.Network/privateDnsZones/virtualNetworkLinks/delete",        
      "Microsoft.Network/virtualNetworks/join/action",
      "Microsoft.Network/loadBalancers/delete",
      "Microsoft.Network/loadBalancers/read",
      "Microsoft.Network/loadBalancers/write",
      "Microsoft.Network/loadBalancers/backendAddressPools/join/action",
      "Microsoft.Resources/deployments/cancel/action",
      "Microsoft.Resources/deployments/whatIf/action"
    ],
    "NotActions": [],
    "DataActions": [
      "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read",
      "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write",
      "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete",
      "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action"
    ],
    "NotDataActions": [],
    "AssignableScopes": [
      "/subscriptions/{SUBSCRIPTION-ID}/resourceGroups/{RESOURCE-GROUP-NAME}"
    ]
  }

When creating the role definition, make sure to:

  • Replace the {SUBSCRIPTION-ID} with your actual subscription ID.

  • Replace the {RESOURCE-GROUP-NAME} with the name of your existing resource group

Role definition for Azure Marketplace images

In order to use the Azure Marketplace images, you need to grant the service principal additional Azure permissions on the scope of your Azure subscription. The following policy includes these permissions:

{
    "properties": {
        "roleName": "Cloudera Management Console Azure Operator for Azure Marketplace",
        "description": "Can use Azure Marketplace images read and accept image terms if the corresponding setting is enabled in Management Console -> Global Settings -> Azure Settings-> Terms & Conditions auto acceptance. Scope must be subscription level.",
        "assignableScopes": [
            "/subscriptions/{SUBSCRIPTION-ID}"
        ],
        "permissions": [
            {
                "actions": [
                    "Microsoft.MarketplaceOrdering/offertypes/publishers/offers/plans/agreements/read",
                    "Microsoft.MarketplaceOrdering/offertypes/publishers/offers/plans/agreements/write"                    
                ],
                "notActions": [],
                "dataActions": [],
                "notDataActions": []
            }
        ]
    }
}

When creating the role definition, make sure to:

  • Replace the {SUBSCRIPTION-ID} with your actual subscription ID.

Obtain subscription and tenant ID

Obtain subscription and tenant ID. You need them in order to create a provisioning credential for Azure.

These steps should be performed by someone who has the Owner built-in Azure role and the Application Developer role in Azure Active Directory.
  • You can obtain both the Subscription ID and Tenant ID from Azure CLI by using the following Azure CLI command:
    az account list|jq '.[]|{"subscriptionId": .id, "tenantId": .tenantId, "state": .state}'
  • You can obtain your Azure Subscription ID from your Azure Portal > Subscriptions:
  • You can obtain your Azure Tenant ID from your Azure Portal > Azure Active Directory > Properties:

Create an app registration and assign a role to it

Create an app registration and assign a role to it. You need it in order to create a provisioning credential for Azure.

  1. On Azure Portal, navigate to the Azure Active Directory > App Registrations and click on + New Registration:

  2. Register a new application as follows and then click Register:

  3. Once your app registration is created, you will be redirected to the app registration's overview page. Copy and save the Application ID before closing this page. You will need to provide it to Cloudera later:

  4. Next, navigate to Certificates & secrets and generate a new secret by clicking + New client secret, providing a description and expiration time, and clicking Add:

  5. Copy and save the Client secret value. You will need to provide it to Cloudera later.

  6. Next, you need to assign a role to your application. To do that, browse to Subscriptions, click on your subscription, and choose Access control (IAM).

  7. Click Add > Add role assignment and then assign the Contributor role or the custom role to your newly created application by:

    • Under Role, selecting Contributor or the custom role.
    • Typing your app name under Select and then selecting it:

  8. Once done, click Save.

What to do next

Once you have this setup ready, you can Create a provisioning credential for Azure in Cloudera.

Generate and register Azure certificate credentials

You need to generate and register the certificate received by Cloudera to use certificate based authentication with Azure.

  1. Create a new Azure credential and obtain the generated certificate using the cdpcli environments create-azure-credential as shown in the following example: 
    $ cdp environments create-azure-credential --credential-name cert-example --app-based authenticationType=CERTIFICATE
{
    "credential": {
        "credentialName": "cert-example",
        "crn": "crn:cdp:environments:us-west-1:3d338b81-4268-48f6-b754-c92eee05b4b1:credential:811733c2-15a3-4228-a6e9-e2bdfc68c06a",
        "cloudPlatform": "AZURE",
        "description": "",
        "azureCredentialProperties": {
            "authenticationType": "CERTIFICATE",
            "certificate": {
                "status": "KEY_GENERATED",
                "expiration": "1732711849000",
                "expirationDate": "2024-11-27T12:50:49+00:00",
                "sha512": "<sha512sum string>",
                "base64": "<base64 encoded X509 certificate>"
            }
        }
    }
}

    The generated certificate is returned as a base64 string in the output.

  2. Decode the generated certificate, and save it in a file.
  3. Create new app credentials in Azure using the saved certificate credentials.
    You can create the credential either in the Azure portal or you can use Azure CLI as shown in the following example:
    $ az ad sp create-for-rbac  --name cert-creds-example --role Contributor  --scopes /subscriptions/1b6e7d18-9a12-4800-9742-eb5451184004 --cert @certificate.pem
Certificate expires 2024-11-27 12:50:49+00:00. Adjusting key credential end date to match.
Creating 'Contributor' role assignment under scope '/subscriptions/1b6e7d18-9a12-4800-9742-eb5451184004'
The output includes credentials that you must protect. Be sure that you do not include these credentials in your code or check the credentials into your source control. For more information, see https://aka.ms/azadsp-cli
{
  "appId": "f92436f3-0e57-4eb5-bf46-18804198d726",
  "displayName": "cert-creds-example",
  "password": null,
  "tenant": "d96dc590-2ee6-42ba-b48f-1aef42cf2bc3"
}
```
  4. Update the certificate credentials with the appId, subscriptionId and tenantId to activate it as shown in the following example: 
    $ cdp environments update-azure-credential --credential-name cert-example --subscription-id 1b6e7d18-9a12-4800-9742-eb5451184004 --tenant-id d96dc590-2ee6-42ba-b48f-1aef42cf2bc3 --app-based authenticationType=CERTIFICATE,applicationId=f92436f3-0e57-4eb5-bf46-18804198d726
{
    "credential": {
        "credentialName": "cert-example",
        "crn": "crn:cdp:environments:us-west-1:3d338b81-4268-48f6-b754-c92eee05b4b1:credential:811733c2-15a3-4228-a6e9-e2bdfc68c06a",
        "cloudPlatform": "AZURE",
        "description": "",
        "azureCredentialProperties": {
            "subscriptionId": "1b6e7d18-9a12-4800-9742-eb5451184004",
            "tenantId": "d96dc590-2ee6-42ba-b48f-1aef42cf2bc3",
            "appId": "f92436f3-0e57-4eb5-bf46-18804198d726",
            "authenticationType": "CERTIFICATE",
            "certificate": {
                "status": "ACTIVE",
                "expiration": "1732711849000",
                "expirationDate": "2024-11-27T12:50:49+00:00",
                "sha512": "<sha512sum string>",
                "base64": "<base64 encoded X509 certificate>"
            }
        }
    }
}

    The certificate status is now ACTIVE. The credential can be used to create Cloudera environments, Data Lakes and Data Services.