Adding and Editing Metadata

Required Role: Metadata Administrator (or Full Administrator)

Adding and Editing Metadata Using the Cloudera Navigator Console

1. Run a search in the Navigator UI.
2. Click an entity link returned in the search. The Details tab displays.
3. To the left of the Details tab, click Actions > Edit Metadata.... The Edit Metadata dialog box drops down.
4. Add metadata fields:
   • In the Name field, type a new display name.
   • In the Description field, type a description.
   • Managed Metadata
     1. Click the and select a property.
     2. Click the value field after the : to display type-specific selection controls such as integer spinners and date selection controls. Either type the value or use the controls to select a value.
     Click the plus icon (+) to add another managed property key-value pair or another value for a given key.
   • Custom Metadata: In the Tags field, type a tag and press Enter or Tab to create new tag entries.
   • Key-Value Pairs
     1. Click to add a key-value pair.
     2. Type a key and a value. You can specify special characters (for example, ".", " ") in the name, but it makes searching for the entity more difficult because some characters collide with special characters in the search syntax.
5. Click Save. The new metadata appears in the Managed Metadata or Custom Metadata pane.

Editing MapReduce Custom Metadata

You can associate custom metadata with arbitrary configuration parameters to MapReduce jobs and job executions. The configuration parameters to be extracted by Navigator can be specified statically or dynamically.

Editing HDFS Custom Metadata Using Metadata Files

You can add tags and properties to HDFS entities using metadata files. With metadata files, you can assign metadata to entities in bulk and create metadata before it is extracted. A metadata file is a JSON file with the following structure:

{
  "name" : "aName",
  "description" : "a description",
  "properties" : {
    "prop1" : "value1", "prop2" : "value2"
  },
  "tags" : [ "tag1" ]
}

Editing HDFS and Hive Metadata Using the Navigator Metadata API

You can use the Cloudera Navigator APIs to modify the custom metadata of HDFS or Hive entities, whether the entities have been extracted or not. If an entity has been extracted when the API is called, the metadata is applied immediately. If the entity has not been extracted, you can preregister metadata, which is then applied once the entity is extracted. Metadata is saved regardless of whether or not a matching entity is extracted, and Navigator does not perform any cleanup of unused metadata.

curl http://fqdn-n.example.com:port/api/APIversion/entities/?query=-internalType:* -u username:password -X GET 

Custom metadata provided through the API overwrites existing metadata. For example, if you call the API with an empty name and description, empty array for tags, and empty dictionary for properties, the call removes this metadata. If you omit the tags or properties fields, the existing values remain unchanged.

Modifying custom metadata using HDFS metadata files and the metadata API at the same time is not supported. You must use one or the other, because the two methods work differently. Metadata specified in files is merged with existing metadata, whereas the API overwrites metadata. Also, the updates provided by metadata files wait in a queue before being merged, but API changes are committed immediately. Some inconsistency can occur if a metadata file is merged when the API is in use.

• PUT

  curl http://fqdn-n.example.com:port/api/APIversion/entities/identity -u username:password -X PUT -H\
  "Content-Type: application/json" -d '{properties}'

  where identity is an entity ID and properties are:
  • name - Name metadata.
  • description - Description metadata.
  • tags - Tag metadata.
  • properties - Custom metadata properties. The format is {key: value}.
  • customProperties - Managed metadata properties. The format is {namespace: {key: value}}. If a property is assigned a value that does not conform to type constraints, an error is returned.
  All existing naming rules apply, and if any value is invalid, the entire request is denied.
• POST

  curl http://fqdn-n.example.com:port/api/APIversion/entities/ -u username:password -X POST -H\
  "Content-Type: application/json" -d '{properties}'

  where properties are:
  • sourceId (required) - An existing source ID. After the first extraction, you can retrieve source IDs using the call:

    curl http://fqdn-n.example.com:port/api/APIversion/entities/?query=type:SOURCE -u username:password -X GET

    For example:

    [ ...
    {  {
    "identity": "61cfefd303d4284b7f5014b701f2c76d",
    "originalName": "source.listing",
    "originalDescription": null,
    "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2",
    "firstClassParentId": null,
    "parentPath": "/user/hdfs/.cm/distcp/2016-02-03_487",
    ...
    "properties": {
    "__cloudera_internal__hueLink":
    "http://fqdn-2.example.com:8888/filebrowser/#/user/hdfs/.cm/distcp/2016-02-03_487/source.listing"
     },
    "technicalProperties": null,
    "fileSystemPath": "/user/hdfs/.cm/distcp/2016-02-03_487/source.listing",
    "type": "FILE",
    "size": 92682,
    "created": "2016-02-03T21:12:16.587Z",
    "lastModified": "2016-02-03T21:12:16.587Z",
    "lastAccessed": "2016-02-03T21:12:16.587Z",
    "permissions": "rw-r--r--",
    "owner": "hdfs",
    "group": "supergroup",
    "blockSize": 134217728,
    "mimeType": "application/octet-stream",
    "replication": 3,
    "userEntity": false,
    "deleted": false,
    "sourceType": "HDFS",
    "metaClassName": "fselement",
    "packageName": "nav",
    "internalType": "fselement"
    }, ...

    If you have multiple services of a given type, you must specify the source ID that contains the entity you expect it to match.
  • parentPath - The path of the parent entity, defined as:
    • HDFS file or directory - fileSystemPath of the parent directory. (Do not provide this field if the entity affected is the root directory.) Example parentPath for /user/admin/input_dir: /user/admin. If you add metadata to a directory, the metadata does not propagate to any files or folders in that directory.
    • Hive database - If you are updating database metadata, do not specify this field.
    • Hive table or view - The name of database containing the table or view. Example for a table in the default database: default.
    • Hive column - database name/table name/view name. Example for a column in the sample_07 table: default/sample_07.
  • originalName (required) - The name as defined by the source system.
    • HDFS file or directory- Name of file or directory (ROOT if the entity is the root directory). Example originalName for /user/admin/input_dir: input_dir.
    • Hive database, table, view, or column - The name of the database, table, view, or column.
      • Example for default database: default
      • Example for sample_07 table: sample_07
  • name - Name metadata.
  • description - Description metadata.
  • tags - Tag metadata.
  • properties - Custom metadata properties. The format is {key: value}.
  All existing naming rules apply, and if any value is invalid, the entire request is denied.

Accessing and Editing Metadata with the Cloudera Navigator SDK

To facilitate editing metadata with the Cloudera Navigator APIs, Cloudera provides a Cloudera Navigator SDK at GitHub. Cloudera Navigator SDK is a client library that provides functionality for extracting and enriching metadata with custom models, entities, and relationships. See GitHub cloudera/navigator-sdk for details.