What Are Knowledge Graph Embeddings?

Imagine you have a big network of information. It’s like a giant spider web where each point (or “node”) represents a person, organization, location, or thing. The lines (or “edges”) connecting these points show the relationships between them. For example, you might have a node representing “Bulgaria” and another node for “Greece“. There’s also a line connecting them because “Bulgaria”  has a “border with” “Greece”. 

This whole interconnected web is called a knowledge graph. 

But here’s the intriguing part: does this knowledge graph capture all possible relationships? One can notice that there is missing information about whether Bulgaria is in Europe. So how can we identify the missing links? 

Embeddings

This is where knowledge graph embedding models, innovative tools designed to predict relationships, come to the rescue. These models have the potential to elucidate the unseen connections, enriching our understanding of the world woven through these interconnected nodes. 

Embeddings are a way to convert the complex web of information into a simpler form that computers can understand better. Specifically, embeddings are mathematical representations (usually vectors) of the nodes (entities) and relationships in the knowledge graph. Think of it as creating a list of numbers for each node and relationship that keeps track of the most important bits of information about them. Knowledge graph embeddings are designed to capture the semantic meaning and structure of the entities and relationships in the graph so that they can be easily manipulated and understood by machine learning algorithms.

Main characteristics of knowledge graph embedding models

Main characteristics of knowledge graph embedding models

Here are some of the key characteristics of these models:

Efficient Computation: Operations on embeddings in a Euclidean space, such as vector addition or finding the distance/similarity between vectors, are computationally efficient. This efficiency is beneficial for tasks like entity resolution, relationship prediction, and graph completion.

Dimensionality Reduction: knowledge graphs can be highly complex and large, consisting of numerous nodes (entities) and edges (relationships). Embeddings help reduce this complexity by mapping high-dimensional graph structures into a lower-dimensional vector space while preserving important information.

Semantic Similarity: By embedding entities and relationships into a continuous space, similar entities and relationships can be positioned closer to each other in the vector space. This makes it easier to perform tasks like clustering, classification, and link prediction.

Knowledge graph embedding methods

Several models and techniques have been developed for creating knowledge graph embeddings. Some popular ones include:

• TransE (Translating Embeddings (Bordes et al. 2013): Projects both entities and relationships into the same space such that the relationship is represented as a translation operation between the entities.
• DistMult (Yang et al. 2014): Uses a bilinear formulation for relationships, with separate embeddings for each entity and a diagonal matrix for each relationship.
• ComplEx (Trouillon et al. 2016): Extends DistMult by using complex-valued embeddings, allowing it to better capture asymmetric relationships.
• Node2Vec and DeepWalk (Grover and Leskovec 2016): Extend the concept of word embeddings (like Word2Vec) to graphs by performing random walks on the graph to create sequences of nodes, which are then embedded similarly to words in sentences.
• Graph Convolutional Networks (R-GCNs) (Schlichtkrull et al. 2018): Use neural network architectures adapted for graph data, allowing for more flexible and powerful embeddings that can incorporate additional node features

Knowledge graph embedding frameworks provide implementations of various models: 

• LibKGE – PyTorch-based library for KGE models training and evaluation with a special focus on reproducible research.
• PyKeen (Python Knowledge Graph Embeddings) – a library that provides implementations of various KGE models
• AmpliGraph– an open-source Python library for KGE models, primarily used for link prediction.
• OpenKE (Open-source Knowledge Embedding) toolkit for knowledge representation learning
• DGL-KE (Deep Graph Library for Knowledge Embedding) – built on top of the Deep Graph Library (DGL) and is optimized for training large-scale knowledge graph embeddings efficiently using multiple GPUs and machines

How does it work?

TransE model

To illustrate link prediction using the TransE model on the given knowledge graph, we need first to establish entity and relation embeddings. TransE represents entities and relations as vectors in a continuous vector space. 

For a triplet (head, relation, tail), TransE aims to enforce:

head + relation ≈ tail

where head, relation, and tail are the embeddings of the head entity, relation, and tail entity, respectively. The optimization objective is usually to minimize the difference between (head + relation) and tail.

Let’s consider a simplified scenario with low-dimensional embeddings for illustrative purposes. Assume all embeddings are in 2-dimensional space.

For the example with countries in the knowledge graph presented above, we can accept that the following vectors are created for nodes (countries and continents) and relationships (borderWith and containedInPlace).

Calculate Expected Outcomes for Known Triplets:

 1. (Romania, containedInPlace, Europe):

Romania + containedInPlace ≈ Europe

(0.5, 0.2) + (0.1, 0.1) = (0.6, 0.3)

This closely approximates Europe = (0.4, 0.3).

2. (Serbia, containedInPlace, Europe):

Serbia + containedInPlace ≈ Europe

(0.7, 0.1) + (0.1, 0.1) = (0.8, 0.2)

3. (Romania, borderWith, Serbia):

Romania + borderWith ≈ Serbia

(0.5, 0.2) + (0.2, 0.0) = (0.7, 0.2)

4. (Greece, containedInPlace, Europe):

Greece + containedInPlace ≈ Europe

(0.3, 0.7) + (0.1, 0.1) = (0.4, 0.8)

5. (Bulgaria, borderWith, Romania):

Bulgaria + borderWith ≈ Romania

(0.6, 0.5) + (0.2, 0.0) = (0.8, 0.5)

6.(Bulgaria, borderWith, Greece):

Bulgaria + borderWith ≈ Greece

(0.6, 0.5) + (0.2, 0.0) = (0.8, 0.5)

Link Prediction:

For link prediction, we can predict unknown links by evaluating which entities, when added with a relationship vector, yield results close to a known entity embedding.

Predicting (Bulgaria, containedInPlace, ?): 

Using containedInPlace:

Bulgaria + containedInPlace = (0.6, 0.5) + (0.1, 0.1) = (0.7, 0.6)

Compare (0.7, 0.6) to all location vectors and find which entity is closest: likely candidates include possibly “Europe” based on proximity in this vector space.

Predicting (?, borderWith, Greece):

? + borderWith ≈ Greece

For ? + (0.2, 0.0) ≈ (0.3, 0.7), reverse the operation:

? = Greece − borderWith = (0.3, 0.7) − (0.2, 0.0) = (0.1, 0.7)

Again, compare this result to other entities; closer vectors might indicate good candidates.

It’s important to note that these embeddings are illustrative. In practice, they would be learned from data through optimization algorithms and would likely be in a higher-dimensional space to accurately reflect complex relationships.

Why are knowledge graph embeddings important?

 Knowledge graph embeddings aim to achieve several goals:

1. Simplifying complex data:
   • Knowledge graphs are often incredibly complex, with millions of entities and relations. By converting this complexity into knowledge graph embeddings, we can make the information easier to work with, store, and process.
2. Machine learning Improvement:
   • Machine learning algorithms work better with numerical data. When we turn entities and relations into embeddings, we can feed this data into machine learning models to perform tasks like link prediction, entity classification, knowledge graph completion, or similarity search.
3. Search and recommendations enhancement:
   • In a search engine, knowledge graph embeddings can help improve search results by better understanding the relationships between different pieces of information. Similarly, they can power recommendation systems by suggesting new friends on social media or new movies to watch based on your interests.
4. Accessing hidden patterns:
   • With embeddings, we can uncover hidden patterns or relationships in the data that might not be immediately obvious. This can be valuable for research, business insights, and more.

To sum up, knowledge graph embeddings take a complex knowledge graph of information and turn it into a more manageable form that makes it easier for computers to analyze, understand, and derive valuable insights from it.