Three’s Company Too: Metadata, Data and Text Analysis

Metadata used to be a secret shared between system programmers and the data. Metadata described the data in terms of cardinality, data types such as strings vs integers, and primary or foreign key relationships. Details that were never intended or even useful to the end business user or even another program or database.

As connectivity became the rule rather than the exception for computers, the importance and usefulness of sharing data necessitated the ability to share how that data was defined. XML and later JSON were the languages that enabled data interchange by establishing a common data model by establishing a standard description of the data being shared.

Inevitably, the information that could and needed to be expressed by metadata increased in complexity. Metadata started including descriptions and source information. Separate documents called schemas let you describe the structure and restrictions of the data being described. Separating the definitions of the metadata from the data had the benefit of simplifying validation and introducing flexibility. Beyond the ability to ensure there was an enterprise wide data model, it was also possible to reuse data but with different metadata and schema.

Metadata, the Lingua Franca of the Internet

As metadata grew in expressivity and ubiquity to facilitate data interchange, so did the general connectivity of computers. More and more content such as documents, videos, sound was natively digital. As the amount of digital content increased, so did the difficulty in finding exactly what you were looking for.

Structural metadata offered no help, but text analysis did. Initially this was limited to keyword search, which was only as useful if the person searching could correctly guess the right terms to match exactly what was being searched for. Once text analysis was coupled with taxonomies it was possible to have a better understanding of what the content was about and most importantly what other concepts and content were related to.

Content about ‘NASDAQ’, ‘FTSE’ and China’s ‘SSE’ would be understood to have a relationship to a broader concept of ‘stock market’. Taxonomies eventually became ontologies. The relationships between terms, also called entities or concepts, in a taxonomy are limited to narrower and broader. The most famous taxonomy is the one in biology: domain, kingdom, phylum, class, order, family, genus, and species.

On the other hand, ontologies are able to capture a multiplicity of relationships. Rather than a top down tree structure branching out, ontologies resemble a graph. This level of complexity better captures how business users understand their content and data. It is more suited to the variety, complexity, and dynamic nature of the world our data is attempting to describe. Ontologies are able to capture the relationships between business processes and entities that reflect the real world. The consequence is that information becomes understood by both machine and human users.

Text Analysis Makes Metadata Feasible. Semantics Makes It Powerful.

It is hard to overstate the value of metadata and schemas, but they are only part of the puzzle. When dealing with the volumes of information, content, and data that an average organization contends with on a daily basis, manual processes for adding metadata are impossible. Text analysis is a ‘catch all’ term to describe any process or algorithm that takes human language text as an input and outputs additional metadata about that text. This metadata could include classification, sentiment, keyword extraction, etc.

To ensure the metadata extracted is of high-quality, Ontotext uses knowledge graphs to increase the performance of its text analysis services. The knowledge graph holds the ontology, the description of the concepts and their relationships, and the existing instances of those concepts and relationships.

For example, the ontology would describe that every company is registered in a country. The data would state that the USA is a country and Facebook is a company that is registered in the USA. This machine understandable but humanly sensible way of representing the world helps the text analysis deal with the ambiguity of language and mitigates errors such as confusing homonyms and homographs. Both are notoriously tricky for text analysis.

The added advantage of this approach is that the metadata that is associated with the content is semantic. There is a link between the concept extracted and the ontology, which means that concept and now the text too have context, which enables more accurate search and further analytics. The text analysis is also able to identify gaps in the knowledge graph such as concepts and relationships previously unidentified. This new knowledge is added to the graph further enhancing the text analysis to produce more and better metadata.

The Value of Text Analysis and Semantic Metadata

With text analysis producing valuable (and semantic) metadata, companies derive a number of benefits.

Better search

Without metadata, the chances of finding anything you are looking for are near nil. Finding what you need through file structures on our personal computers is bad enough, never mind an enterprise-wide ICT. Faster and easier knowledge discovery has obvious cost benefits and reduces duplication of effort.

More useful organization of information

With good metadata and especially good semantic metadata, you are able to organize and present information more closely to how a human user understands it. If the user role is also understood in metadata, It is possible as well to dynamically modify information for the specific role’s needs and access privileges.

Better understanding of information

Metadata makes for richer analytics. The time it takes to Extract, Transform and Load (ETL) is significantly reduced when the metadata and its schema are there. That means more accurate data and more timely decision-making based on that data.

Data Governance

Metadata around provenance and permissions simplify data governance issues and provide the full picture of data and its lifecycle.

Universal integration

Metadata has immediate SEO benefits. Just as metadata helps you find what you are looking for so does it help others find you. The standards for semantic metadata are well-developed by the W3C, enabling seamless integration of  internal company’s data and metadata with valuable third party sources.

Machine understandable and human sensible

Enabling machines to do the heavy-lifting to process the huge amounts of information and data that passes through enables business users to more easily get to the information that need to make the right decisions in a timely manner.

Only with good metadata and text analysis is this all possible.

Want to learn more about how text analysis and semantic metadata work and can transform enterprise knowledge management?

White Paper: Text Analytics for Enterprise Use Use the power of text analytics for your enterprise