Metric on Jiang Yi(姜祎)'s Homepage

ROUGE (Recall-Oriented Understudy)

Thu, 09 May 2024 17:35:20 +0800

ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. It includes several automatic evaluation methods that measure the similarity between summaries.

Preliminaries

ROUGE-N: N-gram co-occurrence statistics

Definitions

Given any string $y=y_1y_2\cdots y_K$, where $y_i,i\in{1,\dots,K}$ are characters and an integer $n\geq1$, we define the set of $n$-gram to be

$$ G_n(y) = \{ y_1\cdots y_n, y_2\cdots y_{n+1}, \dots, y_{K-n+1}\cdots y_K\} $$

NOTE that this is a set with unique elements, for example, $G_2(abab)=\{ab, ba\}$.

Given any two strings $s$ and $y$, we define substring count $C(s,y)$ to tbe the number of appearances of $s$ as a substring of $y$. For example, $C(ab, abcbab)=2$ since $ab$ appear in $abcbab$ twice in position $1$ and $5$.

Base Version

ROUGE-N is an n-gram recall between a candidate summary $\hat{y}$ and a set of reference summaries $S=\{y_1,\dots,y_n\}$. ROUGE-N is defined as follows:

$$ \text{ROUGE-N}(\hat{y}, S) = \frac{\sum_{i=1}^n\sum_{s\in G_N(y_i)}C(s,\hat{y})}{\sum_{i=1}^n\sum_{s\in G_N(y_i)}C(s, y_i)} $$

Features of ROUGE-N:

the denominator increases as we add more references, since there might exists multiple good summaries.
A candidate summary that contains words shared by more references is favored by the ROUGE-N measure.

Multiple references

When there are multiple references for a candidate summary, it is suggests to use the following formula:

$$ \text{ROUGE-N}(\hat{y}, S) = \arg\max_{i=1,\dots,n}\text{ROUGE-N}(\hat{y}, \{y_i\}) $$

the above formula is favored for the following reasons:

There is no single “best” reference summary, the multiple reference formula allows ROUGE-N to take into account of all the possible reference summaries and provide a more accurate measure of the quality of the generated summary.
The multiple reference formula is more robust. If a reference summary contains a typo or a grammatical error, this can affect the ROUGE-N score.
The multiple reference formula can provide a more comprehensive evaluation of the generated summary, since it can allow ROUGE-N to evaluate the generated summary against a wider range pf possible reference summaries.

ROUGE-L

A sequence $Z=[z_1,\dots,z_m]$ is a subsequence of another sequence $X=[x_1,\dots,x_n]$ if there exists a strict increasing sequence $[i_1,\dots,i_k]$ of indices of $X$ such that for all $j=1,\dots,k$, we have $x_{i_j}=z_j$.

Given two sequences $X$ and $Y$, the longest common subsequences (LCS) of $X$ and $Y$ is a common subsequences with maximum length.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


def longest_common_subsequence(x: str, y: str) -> int:
 """compute the length of LCS of x and y

 Args:
 x (str): a string of length m
 y (str): a string of length n

 Return:
 int: the length of LCS of x and y
 """
 m, n = len(x), len(y)
 dp = [[0 for _ in range(m + 1)] for _ in range(n + 1)]

 for i in range(1, m + 1):
 for j in range(1, n + 1):
 if x[i] == y[j]:
 dp[i][j] = dp[i - 1][j - 1] + 1
 else:
 dp[i][j] = max(dp[i - 1][j], dp[i][j - 1])

 return dp[m][n]

Sentence-level LCS

The intuition of sentence-level LCS is that the longer the LCS of two summary sentences is, the more similar the two summaries are.

Given two summaries $X$ of length $m$ and $Y$ of length $n$, assuming $X$ is a reference summary sentence and $Y$ is a candidate summary sentence, the LCS-based recall, precision and F-measure are defined as follows:

$$ R_{LCS} = \frac{LCS(X, Y)}{m}, P_{LCS} = \frac{LCS(X, Y)}{n}, R_{LCS} = \frac{(1+\beta^2)R_{LCS}P_{LCS}}{R_{LCS}+\beta^2P_{LCS}} $$

the above formula is called ROUGE-L. $\beta$ is a hyperparameter.

Features of ROUGE-L are listed as follows:

It doesn’t require consecutive matches but in-sequence matches, which is more reasonable than n-grams.
It automatically includes longest in-sequence common n-grams, there fore no predefined n-gram length is necessary.
It’s value is less tan or equal to the minimum of unigram F-measure of $X$ and $Y$.
The disadvantage of ROUGE-L is that it only counts the main in-sequences words, therefore, other alternative LCSes and shorter sequences are not reflected in the final score.

Summary-level LCS

We can apply sentence-level LCS-based F-measure score to summary level. Given a reference summary of $u$ sentences $\{r_1,\dots,r_u\}$ containing a total of $m$ words and a candidate summary of $v$ sentences $\{c_1,\dots,c_v\}$ containing a total of $n$ words, the summary-level LCS-based recall, precision and F-measure are defined as follows:

$$ R_{LCS} = \frac{\sum_{i=1}^u\max_{j}LCS(r_i, c_j)}{m}, P_{LCS} = \frac{\sum_{i=1}^v \max_{j}LCS(r_i, c_j)}{n}, R_{LCS} = \frac{(1+\beta^2)R_{LCS}P_{LCS}}{R_{LCS}+\beta^2P_{LCS}} $$

ROUGE-W

The basic LCS has a problem that it doesn’t differentiate LCSes of different spatial relations within their embedding sequences.

To improve the basic LSC method, we can simply remember the length of consecutive matches encountered so fat to a regular two dimensional dynamic program table computing LCS. we call this weighted LCS (WLCS) and use $k$ to indicate the length of the current consecutive matches ending at words $x_i$ and $y_j$.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34


def weighted_longest_common_subsequence(x: str, y: str, f: Callable):
 """use dynamic programming to compute WLCS with a weighted function f

 Args:
 x (str): a candidate summary containing m words
 y (str): a reference summary containing n words
 f (Callable): a function satisfies f(x+y) > f(x) + f(y)

 Return:
 the WLCS score

 Reference:
 https://aclanthology.org/W04-1013
 """
 x, y = x.split(), y.split()
 m, n = len(x), len(y)
 c = [[0 for _ in range(m + 1)] for _ in range(n + 1)]
 w = [[0 for _ in range(m + 1)] for _ in range(n + 1)]
 for i in range(1, m + 1):
 for j in range(1, n + 1):
 if x[i] == y[j]:
 # the length of consecutive matches at (i - 1, j - 1)
 k = w[i - 1][j - 1]
 c[i][j] = c[i - 1][j - 1] + f(k + 1) - f(k)
 # remember the length of consecutive matches at (i - 1, j - 1)
 w[i][j] = k + 1
 else:
 if c[i - 1][j] > c[i][j - 1]:
 c[i][j] = c[i - 1][j]
 w[i][j] = 0 # no match at (i, j)
 elif c[i][j] = c[i][j - 1]:
 w[i][j] = 0 # no match at (i, j)

 return c[m][n]

where c[i][j] stores the WLCS score ending at word x[i] of x and y[i] of y. w stores the length of consecutive matches at c[i][j]. f is a function of consecutive matches at c[i][j].

Recall, precision, F-score based on WLCS can be computed as follows:

$$ R_{WLCS} = f^{-1}\left(\frac{WLCS(X, Y)}{f(m)}\right), P_{WLCS} = f^{-1}\left(\frac{WLCS(X, Y)}{f(n)}\right), R_{LCS} = \frac{(1+\beta^2)R_{WLCS}P_{WLCS}}{R_{WLCS}+\beta^2P_{WLCS}} $$

where $f$ is the inverse function of $f$. We call the WLCS-based F-measure as ROUGE-W. Usually, a function $f$ that has a close form inverse is preferred.

ROUGE-S

Skip-bigram is any pair of words in their sentence order, allowing for arbitrary gaps. Skip-bigram co-occurrence statistics measure the overlap of skip-bigrams between a candidate translation and a set of reference translations. A sentence with $n$ words will have $\binom{n}{2}=n(n-1)/2$ skip-bigrams.

Recall, precision, F-score based on skip-bigram can be computed as follows:

$$ R_{\mathrm{SKIP2}} = \frac{\mathrm{SKIP2}(X, Y)}{m}, P_{\mathrm{SKIP2}} = \frac{\mathrm{SKIP2}(X, Y)}{n}, R_{\mathrm{SKIP2}} = \frac{(1+\beta^2)R_{\mathrm{SKIP2}}P_{\mathrm{SKIP2}}}{R_{\mathrm{SKIP2}}+\beta^2P_{\mathrm{SKIP2}}} $$

where $\mathrm{SKIP2}(X, Y)$ is the number of skip-bigram matches between $X$ and $Y$. The F-score is called ROUGE-S.

ROUGE-SU

One problem of ROUGE-S is that it doesn’t given any credit to a candidate sentence if the sentence doesn’t have any word pair co-occurring with its references.

To fix this problem, we extend ROUGE-S with the addition of unigram as counting unit. The extended version is called ROUGE-SU. We can also obtain ROUGE-SU from ROUGE-S by adding a begin-of-sentence marker at the beginning of candidate and reference sentences.

Reference

BLEU (Bilingual Evaluation Understudy)

Thu, 25 Apr 2024 22:46:53 +0800

BLEU (Bilingual Evaluation Understudy) is a widely used metric that evaluates the quality of the translated text with respect to the reference translations.

Introduction

The formula of BLEU is defined as follows:

$$ \mathrm{BLEU}_ {w_n}(\hat{S}, S) = \mathrm{BP} \cdot \exp\left(\sum_{n=1}^Nw_n\log p_n(\hat{S}, S) \right) $$

where

$\mathrm{BP}$ represents the Brevity Penalty to penalize the translations that are shorter than the reference translations.
$p_n$ represents the modified $n$-gram precision score
$w_n$ represents the weight for $p_n$, it satisfies $0\leq w_n\leq1$ and $\sum_{n=1}^Nw_n=1$.

Now we interprets three parts in detail.

Interpretation

Definitions

Given any string $y=y_1y_2\cdots y_K$, where $y_i,i\in{1,\dots,K}$ are characters and an integer $n\geq1$, we define the set of $n$-gram to be

$$ G_n(y) = \{ y_1\cdots y_n, y_2\cdots y_{n+1}, \dots, y_{K-n+1}\cdots y_K\} $$

NOTE that this is a set with unique elements, for example, $G_2(abab)=\{ab, ba\}$.

Modified precision score

We start from the one candidate translation $\hat{y}$ and one reference translation $y$. The modified $n$-gram is defined as

$$ p_n(\hat{y}, y) = \frac{\sum_{s\in G_n(\hat{y})}\min (C(s,\hat{y}), C(s,y))}{\sum_{s\in G_n(\hat{y})}C(s,\hat{y})} $$

The quantity measures how many $n$-grams of the reference translation $y$ appears in the candidate translation $\hat{y}$. In case that $\hat{y}$ is too short, we take a minimum between $C(s,\hat{y})$ and $ C(s,y)$. Then we normalize to make $p_n(\hat{y}, y)$ comparable among multiple translation pairs.

Now, suppose we have a candidate translation corpus, $\hat{S}=\{\hat{y}^1,\dots,\hat{y}^M\}$, and for each candidate translation $\hat{y}^i$, we have a reference translation corpus (there are multiple translations can represent the same meaning) $S_i=\{y^{i,1},\dots,y^{i,N_i}\}$. We define $S=\{S_1,\dots,S_M\}$, then our modified $n$-gram precision is defined as

$$ p_n(\hat{S}, S) = \frac{\sum_{i=1}^M\sum_{s\in G_n(\hat{y})}\min (C(s,\hat{y}), \max_{y\in S_i}C(s,y))}{\sum_{i=1}^M\sum_{s\in G_n(\hat{y})}C(s,\hat{y})} $$

note that we have replaced $\min (C(s,\hat{y}), C(s,y))$ with $\min (C(s,\hat{y}), \max_{y\in S_i}C(s,y))$ since there are multiple reference translation, we use the most similar one. So this is to say: “There are multiple answer, go to choose the best one and compute the score.”

BP (Brevity Penalty)

Candidate translations longer than their references are already penalized by the modified $n$-gram precision measure. Now, to penalize those translations shorter than the reference translations, we need add an penalty term. This is when brevity penalty comes out:

$$ \mathrm{BP}=\begin{cases}1&\text{ if }c > r\\ \exp(1-\frac{r}{c})&\text{ if }c \leq r \end{cases} $$

where

$c$ is the number of words or tokens of the candidate corpus. That is,

$$ c = \sum_{i=1}^M\mathrm{length}(\hat{y}^i) $$

$r$ is the number of words or tokens of the effective reference corpus length, where the effective reference is defined as the reference translation whose length is as close to the corresponding candidate translation as possible. That is

$$ r = \sum_{i=1}^M\mathrm{length}(y^{i,j}),\text{ where } y^{i,j}=\arg\min_{y\in S_i}|\mathrm{length}(\hat{y}^i)-\mathrm{length}(y)| $$

with this penalty term, we wish the model to output the translations with the same length as the reference translations.

Weight

The weight measures the importance of different precision score, in the original paper, the uniform weights are adopted, that is

$$ w_i = \frac{1}{N}, \ \text{ for } i=1,\dots,N $$

Final definition

The final definition of the BLEU is given by

$$ \mathrm{BLEU}_ {w}(\hat{S}, S) = \mathrm{BP} \cdot \exp\left(\sum_{n=1}^\infty w_n\log p_n(\hat{S}, S) \right) $$

usually, the upper-bound of the above summation can be reduced to $\max_{i=1,\dots,M}\mathrm{length}(\hat{y}^i)$.

Analysis

Disadvantages of BLEU:

BLEU compares overlap in tokens from the predictions and references, instead of comparing meaning. This can lead to discrepancies between BLEU scores and human ratings.
BLEU scores are not comparable across different datasets, nor are they comparable across different languages.
BLEU scores can vary greatly depending on which parameters are used to generate the scores, especially when different tokenization and normalization techniques are used.
BLEU ignores synonym or similar expression, which causes refuses of reasonable translation.
BLEU is affected by common words.

Python Implementation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54


import math
from typing import Set, List


def compute_n_gram_set(s: str, n: int) -> Set[str]:
 return set(s[i : i + n] for i in range(len(s) - n))


def modified_n_gram_precision(S_hat: List[str], S: List[List[str]], n: int) -> float:
 if len(S_hat) < n:
 return 0.0
 numerator: int = 0
 denominator: int = 0

 for index, y_hat in enumerate(S_hat):
 n_gram_set = compute_n_gram_set(y_hat, n)
 if not n_gram_set:
 continue
 # print(n_gram_set)
 for n_gram in n_gram_set:
 candidate_substr_count = y_hat.count(n_gram)
 best_ref_substr_count, best_ref_len = max(
 [(y.count(n_gram), len(y)) for y in S[index]], key=lambda x: x[0]
 )
 numerator += min(candidate_substr_count, best_ref_substr_count)
 denominator += candidate_substr_count

 return numerator / denominator


def brevity_penalty(S_hat: List[str], S: List[List[str]]) -> float:
 r: int = 0
 c: int = 0
 for index, y_hat in enumerate(S_hat):
 c += len(y_hat)
 best_match_ref = min(S[index], key=lambda x: abs(len(x) - len(y_hat)))
 r += len(best_match_ref)
 return 1.0 if c > r else math.exp(1 - r / c)


def compute_bleu_score(S_hat: List[str], S: List[List[str]]) -> float:
 assert len(S_hat) == len(S)

 # take N as a sufficiently large integer
 # N = max(len(y_hat) for y_hat in S_hat)
 N = sum(len(y_hat) for y_hat in S_hat)

 bp = brevity_penalty(S_hat, S)
 precisions = [modified_n_gram_precision(S_hat, S, n) for n in range(1, N + 1)]

 # bleu_score = bp * exp(p_n)
 return bp * math.exp(
 sum(math.log(precision) for precision in precisions if precision != 0)
 )