Is “ReLiK” the Future of NLP? (2024)

Let’s say you’re reading about your favorite sports team in the news and you come across the name “Jordan.” If you like basketball, you might think of Michael Jordan right away. What if, though, the article was about someone else? What if it was about soccer player Jordan Henderson? Or when you look for the word “apple,” you might think it means a phone, but it actually means a fruit. This kind of misunderstanding is a common problem for machines that try to understand and process language used by people. Tasks like Entity Linking (EL) and Relation Extraction (RE) help them figure out who or what we’re talking about.

Entity Linking (EL) is similar to determining whether the name “Jordan” in a text refers to Michael Jordan, a basketball legend, or Jordan, a Middle Eastern country. On the other hand, Relation Extraction (RE) is like figuring out that “Michael Jordan” and “Chicago Bulls” are linked by the relationship “played for.” These tasks help machines organize unstructured text into meaningful data for search engines and knowledge graphs.

Here comes ReLiK (Retrieve and LinK), a state-of-the-art system created by some highly intelligent experts at the Sapienza NLP Group in Rome. ReLiK stands for “Retrieve and LinK,” and it’s meant to handle both EL and RE more quickly and correctly than current methods. What makes ReLiK special is that it achieves cutting-edge results on a “academic budget” without expensive hardware.

ReLiK: The Reader-Retriever Paradigm

This system is made up of two parts: the Retriever and the Reader. The Retriever is like a very smart librarian. When you give it a text, it quickly finds all possible “candidates” — potential entities or relationships that match the text. The Reader, on the other hand, is like a careful researcher who looks through all of these possibilities at once to see which ones really match the text. The following flowchart illustrates the steps involved in ReLiK’s Retriever-Reader Paradigm, showing how the system processes input text to output linked entities and extracted relations:

Retriever: The Smart Librarian

The Retriever is based on a bi-encoder architecture that is similar to Dense Passage Retrieval (DPR). How it works:

Dense Representations: The Retriever takes a piece of text and turns it into a dense vector, which is a mathematical representation of what the text means. It does the same for potential candidate entities or relations.

Similarity Matching: Next, it figures out how similar these dense vectors are to each other. This helps it find the best candidates from a huge list (think of a huge library of possibilities).

Multi-label Noise Contrastive Estimation (NCE) for Efficiency: A method called multi-label NCE is used to train the Retriever. This lets the Retriever learn quickly by comparing the right candidates with a lot of wrong ones during each training round.

Reader: The Dedicated Researcher

The Reader takes over after the Retriever is done with its job. It does the following:

Single Forward Pass: Older methods would process each candidate separately that was slow and used a lot of resources. But ReLiK’s Reader does all of the processing in one pass. Similar to comparing many puzzle pieces at once to the picture instead of one by one.

Transformers for Contextualization: To look at both the text and the candidates at the same time, the Reader uses a Transformer encoder specifically a DeBERTa-v3 in two sizes (base and large). Transformers are great at tasks like EL and RE because they can understand what each word means in the context of the rest of the sentence.

Span Identification: The reader also figures out which parts of the text (called “spans”) are related to things or people. To make sure accurate identification it uses special tokens to mark the beginning and end of these spans.

Entity Linking (EL)

The researchers used the AIDA-CoNLL dataset, which is a standard test for EL tasks to see how well ReLiK does in Entity Linking. It was also put to the test on MSNBC and KORE 50 datasets to see how well it works with different kinds of text.

How ReLiK Outperforms the Competition

No Dependence on a Mention-Entities Dictionary: Many traditional systems rely on a “mention-entities dictionary,” which is like a pre-built cheat sheet that tells the system all possible meanings of a word or phrase. This is not required by ReLiK, which makes it more adaptable especially when dealing with new or uncommon things it hasn’t seen before.

High Performance: ReLiK performed well in both in-domain and out-of-domain tests, despite the data being very different from the training data. That means ReLiK can handle different types of texts that is very important because it means it can be used in many real-life situations.

Faster and more efficient: The base version of ReLiK (ReLiKB) was faster and needed fewer parameters (the building blocks of a machine learning model) than many of its rivals, even those that used bi-encoder architectures. The bigger version (ReLiKL) broke records for accuracy while still being very fast, especially on difficult datasets like KORE 50 that have a lot of tricky disambiguations.

Relation Extraction and Closed Information Extraction (cIE)

But ReLiK is more than just linking things together; it’s also about knowing how those things connect to each other. The researchers put ReLiK to the test on two well-known sets of data for Relation Extraction:

NYT (New York Times): With 24 different types of relationships, this dataset provided a robust test of ReLiK’s abilities.

CONLL04: A smaller set of data with only 5 types of relationships that can be used to test how well ReLiK works with less data.

The team used the REBEL dataset for Closed Information Extraction (cIE), which is a mix of EL and RE. There are thousands of relationship types from Wikidata and entity labels from Wikipedia in this huge dataset.

ReLiK vs. Other Systems

Seq2Seq Models: Sequence-to-sequence (seq2seq) models build relationships one step at a time in traditional systems. It’s the same as cooking each ingredient separately before putting them all together. Not ReLiK, though. It does everything at once, like a chef who cooks everything in one pot to save time and energy.

Better Speed and Flexibility: ReLiK was not just competitive in RE and cIE tasks; it was a clear winner. For instance, on the NYT dataset, ReLiK did better than other models in terms of both speed and accuracy. Plus, it was up to 35 times faster than some of the best systems on the REBEL dataset. This level of efficiency is very important, especially when working with big datasets or when there aren’t many computers available.

What’s next for ReLiK

Researchers think ReLiK could be even more useful than it already is. What makes them happy is this:

Testing in More Areas: ReLiK has already done well on a number of benchmarks, but it’s always interesting to see how it does with different kinds of text, such as social media posts, legal documents, and even scientific papers. This might help make models that are more reliable and work well in all situations.

Dynamic Entity Linking: One problem that ReLiK could solve next is how to deal with “emerging entities,” which are new or uncommon things that become important all of a sudden. Think about how a new celebrity or planet that was just found would need to be linked correctly even though they weren’t in the training data at first.

Automatic Verbalization: The team also wants to teach ReLiK how to write its own descriptions for entities and relationships. This would make it even more useful and less reliant on data that already exists. Imagine if ReLiK did more than just connect “Jordan” to Michael Jordan. It would also write a short bio about his basketball career on the spot!

My personal Perspective on ReLiK

ReLiK is a big step forward in both EL and RE thanks to its unique Retriever-Reader architecture. ReLiK excels at completing these tasks simultaneously with high efficiency and accuracy, even on a “academic budget.” This balance between high performance and resource efficiency is very important especially now that more and more people want AI solutions that can be scaled up and used by everyone.

In terms of technology, ReLiK’s ability to process data in a single forward pass is a game-changer. In the past, models might have needed more than one iteration. This not only speeds up processing times but also lowers the amount of work that needs to be done on the machine. This makes it easier for smaller research teams or organizations that don’t have the money of tech giants to do advanced NLP tasks.

From my point of view, the development of ReLiK shows a growing trend in the NLP community: the desire to make models that are not only strong, but also useful and easy to use by many people. ReLiK does very well in tests that aren’t in its own domain, which suggests that it can generalize well. This could lead to more robust and flexible AI systems in the future.

But this efficiency also brings up some interesting questions about how AI will grow in the future. In particular, there is a fine line between making models work better and making them more flexible. While ReLiK’s architecture works very well for EL and RE tasks, it remains to be seen how easily it can be changed or expanded to work with other NLP tasks or how it might connect with new AI ideas like multimodal learning, which involves understanding data that comes in many forms, such as text, images, and audio.

Orlando, R., Huguet-Cabot, P. L., Barba, E., & Navigli, R. (2024). ReLiK: Retrieve and LinK, Fast and Accurate Entity Linking and Relation Extraction on an Academic Budget. arXiv preprint arXiv:2408.00103. (Link)

Barba, E., Orlando, R., Cabot, P. L. H., & Navigli, R. ReLiK: Retrieve, Read and LinK: Fast and Accurate Entity Linking and Relation Extraction on an Academic Budget (Link).