Data Catalog
What is a Data Catalog?
Section titled “What is a Data Catalog?”A data catalog functions as a centralized repository for an organization’s data assets, serves as a marketplace for data products. It provides tools for the comprehensive searching, discovery, and assessment of data quality and suitability.
Data catalog addresses the data discovery challenge within data mesh strategy. Similar to a library’s card catalog, it offers extensive data context, thereby enabling both technical and non-technical users to access information autonomously, without requiring intervention from IT personnel.
Key Functions of a Data Catalog:
- Data Discovery: Enables quick searching and location of products based on attributes and descriptions.
- Metadata Management: Provides information about data source, format, structure, lineage and quality.
- Data Understanding: Provides context, definitions, and descriptions of data assets, enabling users to understand their meaning and relevance.
- Support Data Governance: Help in managing data access, ensuring compliance with regulations, and enforcing data policies.
- Collaboration: Facilitates access and sharing among data producers and consumers through a common platform for requests and communication.
Microsoft Purview as a Data Cataloging Tool
Section titled “Microsoft Purview as a Data Cataloging Tool”Trimble uses Microsoft Purview as its main enterprise data cataloging tool. This centralized platform helps with data discovery, classification, and management across various data environments, supporting both on-premises, multi-cloud, and SaaS data with AI-powered search. It also aids data governance through features like data discovery, lineage tracing, metadata management, and data quality, allowing users to explore the catalog, define business glossary terms, and manage data access.
Key Features of Microsoft Purview for Data Cataloging:
- Unified Data Catalog: A centralized, searchable repository encompassing an organization’s data assets, thereby facilitating user access to pertinent information across diverse data sources.
- AI-Powered Search: Leverages artificial intelligence and natural language processing to enable efficient data discovery based on business concepts, glossary terms, classifications, and other relevant metadata.
- Data Lineage: Providesdata consumers with the ability to visually trace the flow and transformation of data from its origin to its utilization within analytics systems, - enhancing transparency and comprehension. Data Curation: Offers functionalities for data stewards and subject matter experts to manage business glossaries, automate the tagging of data assets with glossary terms, and uphold data quality standards.
- Metadata Management: Automatically extracts and inventories metadata from scanned data sources, thereby generating a comprehensive overview of data assets and their characteristics.
- Data Quality and Health Monitoring: Incorporates capabilities for assessing and monitoring the quality and integrity of data assets, which is crucial for ensuring reliable data for informed decision-making.
- Cross-Platform Compatibility: Supports data governance across a variety of environments, including on-premises systems, various cloud platforms, and software-as-a-service (SaaS) applications.
- Governance and Access Control: Facilitates responsible data utilization by providing structured workflows for requesting and obtaining data access, ensuring that only authorized users can access sensitive information.
What is a Data Product?
Section titled “What is a Data Product?”A data product is a logical grouping of curated, reusable, and consumable data assets designed to solve a specific business problem or deliver insights. It is a self-contained and offers metadata, documentation, and access controls for easy consumption, with each having measurable value and an owner responsible for its lifecycle.Examples include data tables, pipelines, dashboards, reports, and ML models.
Data Products are:
- Purpose-built: Designed to address a specific business problem.
- Reusable: Applicable across multiple contexts and applications, promoting efficiency and consistency.
- Self-Contained: Comprises components such as metadata, documentation, and access controls to facilitate understanding and utilization.
- Ready for consumption: Curated, readily accessible, and usable.
What is a Data Asset?
Section titled “What is a Data Asset?”A data asset in Microsoft Purview represents a distinct unit of information within a data environment. It includes metadata that describes the data, rather than the data itself.Purview extracts this metadata from various data sources and stores it into the Microsoft Purview Data Map, a centralized and frequently updated repository. This data map curates information concerning assets throughout hybrid, on-premises, and multi-cloud environments.
A centralized metadata catalog in Purview offers substantial value for data discovery and governance. It allows users to quickly find specific data assets and access all relevant metadata in one location, fostering a deeper understanding of data utility. This streamlined approach bypasses informal information channels, thereby accelerating the realization of data’s business value.
Key Components of Microsoft Purview
Section titled “Key Components of Microsoft Purview”Microsoft Purview consists of two main components that collaborate to offer a comprehensive overview and management of an organization’s data.
-
Microsoft Purview Data Map: Data Map is a tool for capturing metadata about data assets from various sources. Data Map provides the foundation for data discovery and data governance. It automatically scans and classifies data across the organization. Primarily used by the Data Product Owners and Data Curators.
-
Microsoft Purview Unified Catalog: It is a centralized, searchable catalog of all data assets and their metadata. It’s built on the Data Map and provides an easy-to-use interface for data consumers to find, understand, and use data.
Governance Domains
Section titled “Governance Domains”Governance domains in Microsoft Purview Unified Catalog provide context for data assets, simplifying management, governance, and discovery of data and business concepts.
At Trimble, these Governance domains reflect the company’s organizational hierarchy, defining ownership and rules for data access, utilization, and sharing. Each domain’s scope outlines the owner’s responsibilities and authority, covering specific data products, business concepts, and compliance requirements.
Microsoft Purview Training Material
Section titled “Microsoft Purview Training Material”For complete onboarding instructions, please consult the Data Catalog How-to Guides. This section offers specific training materials for both Microsoft Purview Consumers and Data Producers. please refer to the relevant How-to Guide.