"Cell \u001b[0;32mIn[357], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m scores \u001b[38;5;241m=\u001b[39m evaluation_scores(dev_gold_mentions, \u001b[38;5;28mset\u001b[39m(\u001b[43mmost_probable_method\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdf_dev_pred\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdf_kb\u001b[49m\u001b[43m)\u001b[49m))\n\u001b[1;32m 2\u001b[0m print_evaluation_scores(scores)\n\u001b[1;32m 4\u001b[0m \u001b[38;5;28;01massert\u001b[39;00m scores[\u001b[38;5;241m0\u001b[39m] \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m.64\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mPrecision should be above 64\u001b[39m\u001b[38;5;124m%\u001b[39m\u001b[38;5;124m\"\u001b[39m\n",
"Cell \u001b[0;32mIn[356], line 14\u001b[0m, in \u001b[0;36mmost_probable_method\u001b[0;34m(df, df_kb)\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"An entity linker that resolves each mention to the most probably entity in a knowledge base.\u001b[39;00m\n\u001b[1;32m 3\u001b[0m \n\u001b[1;32m 4\u001b[0m \u001b[38;5;124;03mArguments:\u001b[39;00m\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 11\u001b[0m \u001b[38;5;124;03m position and the predicted entity label of each span.\u001b[39;00m\n\u001b[1;32m 12\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m 13\u001b[0m \u001b[38;5;66;03m# YOUR CODE HERE\u001b[39;00m\n\u001b[0;32m---> 14\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mNotImplementedError\u001b[39;00m()\n",
➡️ Before you start, make sure that you are familiar with the **[study guide](https://liu-nlp.ai/text-mining/logistics/)**, in particular the rules around **cheating and plagiarism** (found in the course memo).
➡️ If you use code from external sources (e.g. StackOverflow, ChatGPT, ...) as part of your solutions, don't forget to add a reference to these source(s) (for example as a comment above your code).
➡️ Make sure you fill in all cells that say **`YOUR CODE HERE`** or **YOUR ANSWER HERE**. You normally shouldn't need to modify any of the other cells.
</div>
%% Cell type:markdown id: tags:
# L3: Information Extraction
%% Cell type:markdown id: tags:
Information extraction (IE) is the task of identifying named entities and semantic relations between these entities in text data. In this lab we will focus on two sub-tasks in IE, **named entity recognition** (identifying mentions of entities) and **entity linking** (matching these mentions to entities in a knowledge base).
%% Cell type:code id: tags:
``` python
# Define some helper functions that are used in this notebook
from IPython.display import display, HTML
def success():
display(HTML('<div class="alert alert-success"><strong>Checks have passed!</strong></div>'))
```
%% Cell type:markdown id: tags:
## Dataset
%% Cell type:markdown id: tags:
The main data set for this lab is a collection of news wire articles in which mentions of **named entities** have been annotated with **page names** from the [English Wikipedia](https://en.wikipedia.org/wiki/). The next code cell loads the training and the development parts of the data into Pandas data frames.
%% Cell type:code id: tags:
``` python
import bz2
import csv
import pandas as pd
import numpy as np
with bz2.open('ner-train.tsv.bz2', mode='rt', encoding='utf-8') as source:
Note that for implementing this function, it doesn’t matter what exactly `gold` and `pred` will contain, except that they will be Python `set` objects.
Let's also define a convenience function that prints the scores nicely:
result = evaluation_scores(example_gold, example_pred)
print_evaluation_scores(result)
# Check if the scores appear correct
assert np.isclose(result, (.5, 1. / 3, .4)).all(), "Should be close to the expected values"
success()
```
%% Output
Precision: 0.500, Recall: 0.333, F1: 0.400
%% Cell type:markdown id: tags:
## Problem 2: Named entity recognition
%% Cell type:markdown id: tags:
One of the first tasks that an information extraction system has to solve is to locate and classify (mentions of) named entities, such as persons and organizations, a task usually known as **named entity recognition (NER)**. For this lab, we will consider a slightly simplified version of NER, by only looking at the _spans_ of tokens containing an entity mention, without the actual entity label.
The English language models in spaCy feature a full-fledged [named entity recognizer](https://spacy.io/usage/linguistic-features#named-entities) that identifies a variety of entities, and can be updated with new entity types by the user. We therefore start by loading spaCy. _However,_ the data that we will be using has already been tokenized (following the conventions of the [Penn Treebank](ftp://ftp.cis.upenn.edu/pub/treebank/public_html/tokenization.html)), so we need to prevent spaCy from using its own tokenizer on top of this. We therefore override spaCy’s tokenizer with the default one that simply splits on whitespace:
%% Cell type:code id: tags:
``` python
import spacy
from spacy.tokenizer import Tokenizer
nlp = spacy.load('en_core_web_md') # Let’s use the "medium" (md) model this time
nlp.tokenizer = Tokenizer(nlp.vocab) # ...but override the tokenizer
```
%% Cell type:markdown id: tags:
Your task in this problem is to **evaluate the performance of spaCy’s NER component** when predicting entity spans in the **development data**.
This can be done in the following three steps:
1. Write a function `gold_spans()` that takes a DataFrame and returns a set of triples of the form `(sentence_id, start_position, end_position)`, one for each entity mention _in the dataset_.
2. Write a function `pred_spans()` that takes a DataFrame, runs spaCy’s NER on each sentence, and returns a set of triples (in the same form as above), one for each entity mention _predicted by spaCy_.
3. Evaluate the results using your function from Problem 1.
We ask you to implement `gold_spans()` and `pred_spans()` as _generator functions_ that “yield” a single triple at a time, and provide stubs of such functions below that you can use as a starting point. (If you're not familiar with the `yield` keyword in Python, check out [this brief explanation](https://www.nbshare.io/notebook/851988260/Python-Yield/).)
%% Cell type:code id: tags:
``` python
def gold_spans(df):
"""Yield the gold-standard mention spans in a data frame.
Arguments:
df: A data frame.
Yields:
The gold-standard mention spans in the specified data frame as
triples consisting of the sentence id, start position, and end
position of each span.
"""
# Hint: The Pandas method .itertuples() is useful for iterating over rows in a DataFrame
for row in df.itertuples():
yield row[1], row[3], row[4]
```
%% Cell type:code id: tags:solution
``` python
def pred_spans(df):
"""Run and evaluate spaCy's NER.
Arguments:
df: A data frame.
Yields:
The predicted mention spans in the specified data frame as
triples consisting of the sentence id, start position, and end
position of each span.
"""
ner = nlp.get_pipe("ner")
for row in df.itertuples():
sentence = row[2]
doc = nlp(sentence)
for ent in doc.ents:
yield row[1], ent.start, ent.end
```
%% Cell type:markdown id: tags:
#### 🤞 Putting it all together
The following cell shows how you can put it all together and produce the evaluation report, provided you have implemented the functions as generator functions. You should get a precision above 50%, with a recall above 70%, and an F1-score above 60%.
assert scores[0] > .50, "Precision should be above 50%."
assert scores[1] > .70, "Recall should be above 70%."
success()
```
%% Output
%% Cell type:markdown id: tags:
## Problem 3: Error analysis
%% Cell type:markdown id: tags:
As you can see in Problem 2, the span accuracy of the named entity recognizer is far from perfect. In particular, only slightly more than half of the predicted spans are correct according to the gold standard. Your next task is to analyse this result in more detail.
Below is a function that uses spaCy’s span visualizer to visualize sentences containing _at least one mistake_ (i.e., either a false positive, a false negative, or both):
%% Cell type:code id: tags:
``` python
from collections import defaultdict
from spacy import displacy
from spacy.tokens import Span
def error_report(df, spans_gold, spans_pred):
"""Run and evaluate spaCy's NER.
Arguments:
df: A data frame.
spans_gold: The set of gold-standard entity spans from the data frame.
spans_pred: The set of predicted entity spans from the data frame.
Yields:
The predicted mention spans in the specified data frame as
triples consisting of the sentence id, start position, and end
position of each span.
"""
gold_by_sid = defaultdict(set)
for (sentence_id, span_s, span_e) in spans_gold:
gold_by_sid[sentence_id].add((span_s, span_e))
pred_by_sid = defaultdict(set)
for (sentence_id, span_s, span_e) in spans_pred:
pred_by_sid[sentence_id].add((span_s, span_e))
for row in df.drop_duplicates('sentence_id').itertuples():
if gold_by_sid[row.sentence_id] == pred_by_sid[row.sentence_id]:
continue
doc = nlp(row.sentence)
doc.spans["sc"] = [
Span(doc, span_s, span_e, "GOLD") for (span_s, span_e) in gold_by_sid[row.sentence_id]
] + [
Span(doc, span_s, span_e, "PRED") for (span_s, span_e) in pred_by_sid[row.sentence_id]
]
yield doc
```
%% Cell type:markdown id: tags:
Let’s use a small sample of the training data to inspect this way. The following cell renders sentences containing mistakes that the automated prediction makes based on the _first 500 rows_ of the training data (you may have to click on “Show more outputs” at the bottom to see all of them):
%% Cell type:code id: tags:solution
``` python
df_inspect = df_train[:500]
spans_inspect_pred = set(pred_spans(df_inspect))
for doc in error_report(df_inspect, set(gold_spans(df_inspect)), spans_inspect_pred):
Can you see any patterns in the mistakes from the sample above? **Write a short text** that summarizes your observations!
%% Cell type:markdown id: tags:
We can see four types of mistakes that our model consistently makes. It labels points in time such as "tomorrow or Monday", and it labels numbers such as "16.4 percent" or "two". It also labels "the" in front of words such as "the European...". It also misses non english words.
%% Cell type:markdown id: tags:
### Task 3.2
Based on your insights from the error analysis, you should be able to improve the automated prediction that you implemented in Problem 2. While the best way to do this would be to [update spaCy’s NER model](https://spacy.io/usage/linguistic-features#updating) using domain-specific training data, for this lab it suffices to **write code to post-process the output** produced by spaCy. To filter out specific labels it is useful to know the named entity label scheme, which can be found in the [model's documentation](https://spacy.io/models/en#en_core_web_sm).
%% Cell type:code id: tags:solution
``` python
def pred_spans_improved(df):
"""Run and evaluate spaCy's NER, with post-processing to improve the results.
Arguments:
df: A data frame.
Yields:
The predicted mention spans in the specified data frame as
triples consisting of the sentence id, start position, and end
assert scores_improved[-1] > .8, "F1-score should be above 0.8"
success()
```
%% Output
%% Cell type:markdown id: tags:
### Task 3.3
Before moving on, we ask you to **store the outputs of the improved named entity recognizer in a new data frame**. This new frame should have the same layout as the original data frame for the _development data_ that you loaded above, but should contain the *predicted* start and end positions for each token span, rather than the gold positions. As the `label` of each span, you can use the special value `--NME--` for now.
%% Cell type:code id: tags:
``` python
def df_with_pred_spans(df):
"""Make a new DataFrame with *predicted* NER spans.
Arguments:
df: A data frame.
Returns:
A *new* data frame with the same layout as `df`, but containing
the predicted start and end positions for each token span.
Run the following cell to run your function and display the first few lines of the new data frame:
%% Cell type:code id: tags:solution
``` python
df_dev_pred = df_with_pred_spans(df_dev)
display(df_dev_pred.head())
```
%% Output
%% Cell type:markdown id: tags:
## Problem 4: Entity linking
%% Cell type:markdown id: tags:
Now that we have a method for predicting mention spans, we turn to the task of **entity linking**, which amounts to predicting the knowledge base entity that is referenced by a given mention. In our case, for each span, we want to predict the Wikipedia page that this mention references.
%% Cell type:markdown id: tags:
### Task 4.1
Start by **extending the generator function** that you implemented in Problem 2 to **labelled spans**.
%% Cell type:code id: tags:
``` python
def gold_mentions(df):
"""Yield the gold-standard mentions in a data frame.
Args:
df: A data frame.
Yields:
The gold-standard mention spans in the specified data frame as
quadruples consisting of the sentence id, start position, end
position and entity label of each span.
"""
for row in df.itertuples():
yield row[1], row[3], row[4], row[5]
```
%% Cell type:markdown id: tags:
#### 🤞 Test your code
To test your code, you can run the following cell, which checks if one of the expected tuples is included in the results:
%% Cell type:code id: tags:solution
``` python
dev_gold_mentions = set(gold_mentions(df_dev))
assert ('1094-020', 0, 1, 'Seattle_Mariners') in dev_gold_mentions, "An expected tuple is not included in the results"
success()
```
%% Output
%% Cell type:markdown id: tags:
### Task 4.2
A naive baseline for entity linking on our data set is to link each mention span to the Wikipedia page name that we get when we join the tokens in the span by underscores, as is standard in Wikipedia page names. Suppose, for example, that a span contains the two tokens
Jimi Hendrix
The baseline Wikipedia page name for this span would be
Jimi_Hendrix
**Implement this naive baseline and evaluate its performance!**
%% Cell type:markdown id: tags:
**_Important:_** Here and in the remainder of this lab, you should base your experiments on the _predicted spans_ that you computed in Problem 3.
%% Cell type:code id: tags:solution
``` python
def baseline(df):
"""A naive baseline for entity linking that "predicts" Wikipedia
page names from the tokens in the mention span.
Arguments:
df: A data frame.
Yields:
The predicted mention spans in the specified data frame as
quadruples consisting of the sentence id, start position, end
position and the predicted entity label of each span.
Again, we can turn to the evaluation measures that we implemented in Problem 1. The expected precision should be around 29%, with an F1-score around 28%.
assert scores[0] > .28, "Precision should be above 28%"
assert scores[-1] > .27, "F1-score should be above 27%"
success()
```
%% Output
Precision: 0.301, Recall: 0.274, F1: 0.287
%% Cell type:markdown id: tags:
## Problem 5: Extending the training data using the knowledge base
%% Cell type:markdown id: tags:
State-of-the-art approaches to entity linking exploit information in knowledge bases. In our case, where Wikipedia is the knowledge base, one particularly useful type of information are links to other Wikipedia pages. In particular, we can interpret the anchor texts (the highlighted texts that you click on) as mentions of the entities (pages) that they link to. This allows us to harvest long lists of mention–entity pairings.
The following cell loads a data frame summarizing anchor texts and page references harvested from the first paragraphs of the English Wikipedia. The data frame also contains all entity mentions in the training data (but not the development or the test data).
To understand what information is available in this data, the following cell shows the entry for the anchor text `Sweden`.
%% Cell type:code id: tags:
``` python
df_kb.loc[df_kb.mention == 'Sweden']
```
%% Output
mention entity prob
17436 Sweden Sweden 0.985768
17437 Sweden Sweden_national_football_team 0.014173
17438 Sweden Sweden_men's_national_ice_hockey_team 0.000059
%% Cell type:markdown id: tags:
As you can see, each row of the data frame contains a pair $(m, e)$ of a mention $m$ and an entity $e$, as well as the conditional probability $P(e|m)$ for mention $m$ referring to entity $e$. These probabilities were estimated based on the frequencies of mention–entity pairs in the knowledge base. The example shows that the anchor text ‘Sweden’ is most often used to refer to the entity [Sweden](http://en.wikipedia.org/wiki/Sweden), but in a few cases also to refer to Sweden’s national football and ice hockey teams. Note that references are sorted in decreasing order of probability, so that the most probable pairing come first.
%% Cell type:markdown id: tags:
**Implement an entity linking method** that resolves each mention to the most probable entity in the data frame. If the mention is not included in the data frame, you can predict the generic label `--NME--`.
%% Cell type:code id: tags:solution
``` python
def most_probable_method(df, df_kb):
"""An entity linker that resolves each mention to the most probably entity in a knowledge base.
Arguments:
df: A data frame containing the mention spans.
df_kb: A data frame containing the knowledge base.
Yields:
The predicted mention spans in the specified data frame as
quadruples consisting of the sentence id, start position, end
position and the predicted entity label of each span.
"""
# YOUR CODE HERE
raise NotImplementedError()
```
%% Cell type:markdown id: tags:
### 🤞 Test your code
We run the same evaluation as before. The expected precision should now be around 65%, with an F1-score just around 59%.
4 assert scores[0] > .64, "Precision should be above 64%"
Cell In[356], line 14, in most_probable_method(df, df_kb)
2 """An entity linker that resolves each mention to the most probably entity in a knowledge base.
3
4 Arguments:
(...)
11 position and the predicted entity label of each span.
12 """
13 # YOUR CODE HERE
---> 14 raise NotImplementedError()
NotImplementedError:
%% Cell type:markdown id: tags:
## Problem 6: Context-sensitive disambiguation
%% Cell type:markdown id: tags:
Consider the entity mention ‘Lincoln’. The most probable entity for this mention turns out to be [Lincoln, Nebraska](http://en.wikipedia.org/Lincoln,_Nebraska); but in pages about American history, we would be better off to predict [Abraham Lincoln](http://en.wikipedia.org/Abraham_Lincoln). This suggests that we should try to disambiguate between different entity references based on the textual context on the page from which the mention was taken. Your task in this last problem is to implement this idea.
Set up a dictionary that contains, for each mention $m$ that can refer to more than one entity $e$, a separate Naive Bayes classifier that is trained to predict the correct entity $e$, given the textual context of the mention. As the prior probabilities of the classifier, choose the probabilities $P(e|m)$ that you used in Problem 5. To let you estimate the context-specific probabilities, we have compiled a data set with mention contexts:
This data frame contains, for each ambiguous mention $m$ and each knowledge base entity $e$ to which this mention can refer, up to 100 randomly selected contexts in which $m$ is used to refer to $e$. For this data, a **context** is defined as the 5 tokens to the left and the 5 tokens to the right of the mention. Here are a few examples:
%% Cell type:code id: tags:
``` python
df_contexts.head()
```
%% Cell type:markdown id: tags:
Note that, in each context, the position of the mention is indicated by the `@` symbol.
From this data frame, it is easy to select the data that you need to train the classifiers – the contexts and corresponding entities for all mentions. To illustrate this, the following cell shows how to select all contexts that belong to the mention ‘Lincoln’:
Implement the context-sensitive disambiguation method and evaluate its performance. Do this in two parts, first implementing a function that builds the classifiers _(refer to the text above for a detailed description)_, then implementing a prediction function that uses these classifiers to perform the entity prediction.
Here are some more **hints** that may help you along the way:
1. The prior probabilities for a Naive Bayes classifier can be specified using the `class_prior` option. You will have to provide the probabilities in the same order as the alphabetically sorted class (entity) names.
2. Not all mentions in the knowledge base are ambiguous, and therefore not all mentions have context data. If a mention has only one possible entity, pick that one. If a mention has no entity at all, predict the `--NME--` label.
%% Cell type:code id: tags:solution
``` python
def build_entity_classifiers(df_kb, df_contexts):
"""Build Naive Bayes classifiers for entity prediction.
Arguments:
df_kb: A data frame with the knowledge base.
df_contexts: A data frame with contexts for each mention.
Returns:
A dictionary where the keys are mentions and the values are Naive Bayes
classifiers trained to predict the correct entity, given the textual
context of the mention (as described in detail above).
Finally, the cell below evaluates the results as before. You should expect to see a small (around 1 unit) increase in each of precision, recall, and F1.
<strong>After you have solved the lab,</strong> write a <em>brief</em> reflection (max. one A4 page) on the question(s) below. Remember:
<ul>
<li>You are encouraged to discuss this part with your lab partner, but you should each write up your reflection <strong>individually</strong>.</li>
<li><strong>Do not put your answers in the notebook</strong>; upload them in the separate submission opportunity for the reflections on Lisam.</li>
</ul>
</div>
%% Cell type:markdown id: tags:
1. In Problem 3, you performed an error analysis and implemented some post-processing to improve the model’s evaluation scores. How could you improve the model’s performance further, and what kind of resources (such as data, compute, etc.) would you need for that? Discuss this based on two or three concrete examples from the error analysis.
2. How does the “context” data from Problem 6 help to disambiguate between different entities? Can you think of other types of “context” that you could use for disambiguation? Illustrate this with a specific example.
➡️ Before you submit, **make sure the notebook can be run from start to finish** without errors. For this, _restart the kernel_ and _run all cells_ from top to bottom. In Jupyter Notebook version 7 or higher, you can do this via "Run$\rightarrow$Restart Kernel and Run All Cells..." in the menu (or the "⏩" button in the toolbar).
