1 Mutaxe

Non Pairwise Comparison Definition Essay


Decision-making, as a way to discover the preference of ranking, has been used in various fields. However, owing to the uncertainty in group decision-making, how to rank alternatives by incomplete pairwise comparisons has become an open issue. In this paper, an improved method is proposed for ranking of alternatives by incomplete pairwise comparisons using Dempster-Shafer evidence theory and information entropy. Firstly, taking the probability assignment of the chosen preference into consideration, the comparison of alternatives to each group is addressed. Experiments verified that the information entropy of the data itself can determine the different weight of each group's choices objectively. Numerical examples in group decision-making environments are used to test the effectiveness of the proposed method. Moreover, the divergence of ranking mechanism is analyzed briefly in conclusion section.

1. Introduction

The increasing trend toward decision-making in various fields requires computational methods for discovering the preferences of ranking. In fact, methods for finding and predicting preferences in a reasonable way are among the very hot topics in recent science study, such as information systems [1], control systems [2], social choices [3, 4], and so on.

The term “pairwise comparisons” generally refers to any process of comparing entities in pairs to judge which of each entity is preferred or has a greater amount of quantitative property. Prominent psychometrician Thurstone first introduced a scientific approach to use pairwise comparisons for measurement in 1927, which he referred to as the law of comparative judgment [5]. Thurstone demonstrated that the method can be used to order items along a dimension such as preference or importance using an interval-type scale. The Bradley-Terry-Luce (BTL) model was applied to pairwise comparison data to scale preferences [6, 7]. The BTL model was identical to Thurstone's model if the simple logistic function was used. Thurstone used the normal distribution in applications of the model, which the method of pairwise comparisons was used as an approach to measuring perceived intensity of physical stimuli, attitudes, preferences, choices, and values. He also studied implications of the theory he developed for opinion polls and political voting [8]. If an individual or organization expresses a preference between two mutually distinct alternatives, this preference can be expressed as a pairwise comparison. If pairwise comparisons are in fact transitive, then pairwise comparisons for a list alternatives (A1, A2, A3,…, An−1, An) can take the form

A1 ⪰ A2 ⪰ A3 ⪰  ⋯  ⪰ An−1 ⪰ An


and it means that the alternative Ai is preferred to Aj, in which i < j. Or the alternatives can be expressed as

A1 ≻ A2 ≻ A3 ≻  ⋯  ≻ An−1 ≻ An


and it means that the alternative Ai is strictly preferred to Aj, if i < j.

Although pairwise comparison is a well-known technique in decision-making, in some cases we have to be faced with the problem of incomplete judgement to preference. For instance, if the number of the alternatives n is large, the experts may not give the full comparison one by one. In order to overcome such problem, a decision support system (DSS) based on fuzzy information axiom (FIA) is developed in [9]. However, calculation procedure of information axiom is not only incommodious but also difficult for decision makers and it is hard to deal with the incomplete pairwise problems. For the purpose of reducing the complexity of calculation and preference eliciting process, [10] points out that some comparison between alternative can be skipped and a method is proposed to derive the priorities of n alternatives from an incomplete n × n pairwise comparison matrices in [11]. Furthermore, [12] introduces a fuzzy multiexpert multicriteria decision-making method in possibility measure to handle the difficulty of conflict aggregation process. Moreover, the well-known methods of eigenvector or geometric mean are used to ranking pairwise comparison also introduced in [11, 13]. But all of the above references only focus on the complexity of calculation with the complete comparison. However, some methods have been proposed to solve the incomplete comparison problem. Shiraishi et al. proposed a heuristic method which is based on a property of a coefficient of the characteristic polynomial of pairwise comparison matrices [14]. But the solving is mainly depending on the polynomial which has infinitely many solutions, and it is difficult to get the best candidate. In [15], a least squares type method is proposed to directly calculate the priority vector as the solution of a constrained optimization problem instead of calculating the missing entries of pairwise comparison matrices. In [16], a new centroid-index ranking method of fuzzy number in decision-making was proposed by Yong and Qi. He also proposed a method to concern with the ranking of decision alternatives based on preference judgments made on decision alternatives over a number of criteria to Multiple-criteria decision-making (MCDM) problem in [17]. And in fuzzy group decision-making, an optimal consensus method, in which the limit of each expert's compromise was under consideration in the process of reaching group consensus, was proposed by Liu et al. in [18]. Similarly, in order to calculate the priority vector of incomplete preference, a fuzzy preference relation by a goal programming approach was proposed by Xu in [19], and then he developed a method for incomplete fuzzy preference [20]. Subsequently, the eigenvector method and least square method are proposed with incomplete fuzzy preference relation in [21, 22]. The fuzzy decision-making can not only reduce the complexity of calculation, but also be used in incomplete pairwise comparison, but it is important that the little variety of preference will change the result mainly. Moreover, a method for learning valued preference structures using a natural extension of so-called pairwise classification was proposed by Hullermeier and Furnkranz, which may have a potential application in fuzzy classification [23, 24]. A weighted voting procedure is used in the Hullermeier's proposed method, but the problem is that the voting procedure is always lost in the dead cycle.

In this paper, a discount rate is derived from the information entropy, which determines the certainty or uncertainty of the preference. Then an improved method to calculate the probability assignment of preference using an index introduced in [25] with the discount rate is proposed. Comparing to the current method, the incomplete comparison results of the proposed method are completely dependendent on the information itself instead of the human factor. The paper is organized as follows. Some fundamental and quantifying principles of Dempster-Shafer evidence theory are given in Section 2. Then, the ranking procedures by comparing with single group of experts and with independent groups are introduced in Section 3. In this section, the Dempster-Shafer evidence theory model and imprecise Dirichlet model, which give a basic ranking, are concerned. Next, the proposed enhanced method on the case when comparisons are supplied by independent groups of experts is presented in Section 4. In this section, a weight derived from the entropy of BPA's probability is used in Dempster-Shafer evidence theory and improves the existing ranking method mentioned in the previous section. Finally, the conclusions are given in Section 5.

2. Preliminaries

2.1. Dempster-Shafer Evidence Theory

Dempster-Shafer evidence theory can be divided into probability distribution function, plausibility function, and Dempster evidence combination rule [26, 27]. It is rather flexible for many applied problems.

Definition 1 . —

Assume frame of discernment is θ; then function m : 2θ → [0,1] satisfies m(ϕ) = 0, and ∑Aθm(A) = 1 is called the basic probability distribution of frame of discernment θ.

For ∀Aθ, m(A) is the basic probability of A. The meaning of m(A) is that if AΩ and AΩ, thus m(A) is the accurate trust degree of A; if A = Ω, thus m(A) means that the trust degree of A can not be allocated accurately.

Definition 2 . —

As for ∀Aθ, the defined function Bel : m : 2θ → [0,1] by Bel(A) = ∑BAm(B) is called the belief function of θ.

Definition 3 . —

As for ∀Aθ, pl is called the plausibility function of Bel in .

The relation between belief function and plausibility function is that Bel(A) and pl(A) are, respectively, referred to as the lower limit and the upper limit function of pl(A) ≥ Bel(A).

Even if they are the same evidences, the probability assignment might be different when they came from different sources. Then the orthogonal method is used to combine these functions by dempster-shafer evidence theory.

Assume m1, m2,…, mn are the basic probability assignment functions of 2Ω, and their orthogonal m = m1m2 ⊕ ⋯⊕mn are


in which k−1 = 1 − ∑Ai=A1≤inmi(Ai).

Several of the algorithms basic to Dempster-Shafer's evidence theory are as follows.

  • (1)

    It is known that if we assume frame of discernment of some field is Ω = S1, S2,…, Sn and propositions A, B,… are the subsets of Ω, the inference rule shall be among which E, H are the logic groupings of the proposition, CF is the certainty factor, which is measured by ci and called credibility. For any proposition A, the certainty factor CF of credibility A shall satisfy
    • (a)

      ci ≥ 0, 1 ≤ in,

    • (b)

      1≤inci ≤ 1.

  • (2)

    Evidence description: assume m is the defined basic probability assignment function of  2Ω, then it shall meet the following conditions during calculation:

    • (a)

      m({Si}) ≥ 0, SiΩ,

    • (b)

      1≤in(m({Si}) ≥ 0) ≤ 1,

    • (c)

      m(Ω) = 1 − ∑1≤inm({Si}),

    • (d)

      m(A) = 0,  AΩ,  and |A| > 1 or |A| = 0  among which |A| means the factor numbers of proposition A.

  • (3)

    Inaccurate inference model:

    • (a)

      suppose A is one part proposition of regular condition. Under the condition of evidence E, the matching degree of proposition A and evidence E is


    • (b)

      the definition of part proposition A in regular condition is

      CER = MD(AE) · f(A).


2.2. Quantifying the Uncertainty in the Dempster-Shafer Evidence Theory

The uncertain factor considered in evidence theory includes both the uncertainty associated with randomness and the uncertainty associated with granularity. The measure of the granular uncertainty associated with a subset is the specificity measure introduced by [28].

Definition 4 . —

Let B be a nonempty subset of X. A measure of specificity of B, Sp(B), is defined, using Card to denote the cardinality of a set, as


It essentially measures the degree to which B has exactly one element. The larger the specificity, the lesser the uncertainty. In [29], the measure to the case of a probability assignment function m was extended.

Definition 5 . —

Assume m has focal element Bj, j = 1 to q. Then


can be denoted as the expected specificity of the focal elements.

Noting that Sp(m)∈[0,1] and the larger the Sp(m), the less the uncertainty will be. It is clear that the specificity is smallest when m is the vacuous belief function, m(X) = 1. In this case, Sp(m) = 0.

Klir [30] and Ayyub [31] introduced a related measure which he called nonspecificity.

Definition 6 . —

If m is a belief function with focal element Bj, j = 1 to q, then its nonspecificity is defined as


where |Bj| = Card(Bj).

It is clear that this takes its largest value ln⁡(n), where n is the cardinality of X for the vacuous belief function. It takes its smallest value when m is Bayesian where |Bj | = 1; in this case ln⁡(|Bj|) = 0 and N[Sp(m)] = 0.

Yager [32] made this definition of nonspecificity cointensive with the preceding definition of specificity by normalization and negation; hence


The standard measure of uncertainty associated with a probability distribution is the Shannon entropy.

Definition 7 . —

If X = {x1,…, xn} and if P is a probability distribution on X such that pi is the probability of xi, then the Shannon entropy of P is


Any extension of this to belief function must be such that it reduced to the Shannon entropy when the belief structure is Bayesian. Yager [29] suggested an extension of the Shannon entropy to belief structures using the measure of dissonance.

Definition 8 . —

If m has focal element Bj, then the extension of the Shannon entropy to belief structures is


For a given set of weights, it can be noted that the smaller the focal elements, the larger the entropy. In particular, the smallest a nonempty set can be is one element. And as we knew, the larger the entropy, the larger the uncertainty. For a given set of weights, the smallest entropy for a Bayesian belief structure shall be one element, because one element always means very certainty information of the preference and the entropy is 0, and the largest entropy occurred when all elements have equal probability. From this, we can conclude that, for any uncertainty and certainty information, the fewer the element the smaller the entropy, and the sparser the probability of element the smaller the entropy. Figure 1 shows the relationship between the assignment of element and entropy, in which the more balance the assignment of element, the higher the entropy.

Figure 1

The relationship between the assignment of element and entropy.

3. Ranking Procedure by Incomplete Pairwise Comparison

Here we suppose there is a set of alternative Λ = {A1, A2,…, An} with n elements. Then we can get 2n − 1 subset with n elements as


The pairwise comparison means that an expert has chosen some subset of alternatives from Ω and compared them pairwisely. If the paired comparisons have been done for all the subset in Ω, we call it complete pairwise comparison, otherwise, we call it incomplete pairwise comparison. For complete pairwise comparison, we always get the pairwise comparison matrix that looks like Table 1 in which we give two alternatives as an example. It is supposed that experts only compare subset of alternatives without providing preference value or weights of preference. If an expert chooses one comparison preference, then the value 1 is added to the corresponding cell in the pairwise comparison matrix. For example, c12 is the number of experts who choose the comparison preference {A1}⪰{A2}. There are many references [13, 23, 33] that have studied the complete pairwise comparison matrix. But in most occasions which have a lot of alternatives, the experts can not give the preference one by one. They only choose limited preferences of all the alternatives. Thus, we get the incomplete pairwise comparison matrix that looks like Table 2 with three alternatives, which contain some uncertainty and (or) conflicted information. In complete pairwise comparison matrix, we can calculate the probability of each subset {Ai} with cij, but, in incomplete pairwise comparison, we can not get the assignment of the probability of all the subset. For instant, if an expert chooses the preference {A1}⪰{A2, A3}, we do not know how the probability is distributed among the preferences {A1}⪰{A2} and {A1}⪰{A3} without any other supplement information. In such situation, we can apply the framework of Dempster-Shafer evidence theory to the considered sets of preferences. Now we suppose there are three alternatives which need to prefer each other. In order to simplify the format, we denote all the subset of the three alternatives as in Table 3, and denote Bij as the preference BiBj. For example, B16 means the preference {A1}⪰{A2, A3}. Then we define its BPA for every pairwise comparison in the extended pairwise comparison matrix as follows:


Table 1

The complete pairwise comparison matrix.

Table 2

The incomplete pairwise comparison matrix.

Table 3

The subset of three alternatives and their short notations.

3.1. The Ranking Method with One Group of Preference

We assume the experts choose the following preferences from Utkin's works [25]:

  •   five experts: {A1, A3}⪰{A2} = B52,
  •   two experts: {A3}⪰{A1, A2, A3} = B37,
  •   three experts: {A1}⪰{A3} = B13.

Using the Dempster-Shafer evidence theory we described in previous section, we can get the belief function of the preferences {Ai}⪰Λ which means {Ai} is the best choice of all the subset (or alternative):

  •   Bel({A1}⪰Λ) = m(B13) = 0.3,
  •   Bel({A2}⪰Λ) = 0,
  •   Bel({A3}⪰Λ) = m(B37) = 0.2

and the plausibility functions of the preference:

  •   pl({A1}⪰Λ) = m(B52) + m(B13) = 0.8,
  •   pl({A2}⪰Λ) = 0,
  •   pl({A3}⪰Λ) = m(B52) + m(B37) = 0.7.

From the belief function and the plausibility function, it can be concluded that the best ranking of the alternatives is A1A3A2.

3.2. The Ranking Method with Two Independent Groups of Preference

In fact, in order to obtain more consensus result of the preference, we always choose more than one independent group experts to give their preference of alternatives. Thus we can use the well-established method for combining the independent information with the Dempster-Shafer evidence rule of combination.

Here we suppose there are two groups of experts without loss of generality, and denote the preference obtained from the first and second groups of experts by upper indices (1) and (2), respectively. The combined rule refers to the contents introduced in preliminaries. Now we assume that the first group (with five experts) provides the following preferences:

  •   two experts (c11 = 2): {A1}⪰{A2, A3} = B16(1),
  •   three experts (c12 = 3): {A1, A2}⪰{A3} = B43(1).

And the second group (with ten experts) provides the following judgements:

  •   five experts (c21 = 5): {A1, A3}⪰{A2} = B52(2),
  •   two experts (c22 = 2): {A3}⪰{A1, A2} = B34(2),
  •   three experts (c23 = 3): {A1}⪰{A3} = B13(2).

Then we get the BPA's of all preferences:

  • m1(B16(1)) = 0.4,  m1(B43(1)) = 0.6,
  • m2(B52(2)) = 0.5,  m2(B34(2)) = 0.2,  m2(B13(2)) = 0.3,

and the preference intersections for Dempster-Shafer combination rule are in Table 4.

Table 4

The preference intersections for Dempster-Shafer combination rule.

According to the table, we calculate the weight of conflict first:


Then the probability of the assignment for the nonzero combined BPA's preference can be calculated as follows:


After that, we can get the belief and plausibility functions of alternatives A1, A2, A3 or preferences {Ai}⪰Λ, i = 1,2, 3 as follows:

  •   Bel({A1}⪰Λ) = m12(B12(12)) + m12(B13(12)) = 1,
  •   pl({A1}⪰Λ) = 1.

The belief and plausibility functions of {A2}⪰Λ and {A3}⪰Λ are 0. So we can only get the result of preference from that the “best” alternative is A1 and can not get any information of preference about A2 and A3. From [25], the author thought the main reason of the situation in previous example was the small number of expert judgements, and the other reason was the used assumption that the sources of evidence were absolutely reliable. So he improved the ranking method with the imprecise Dirichlet model.

3.3. The Ranking Method with Imprecise Dirichlet Model

As we described at the final part of the last subsection, the main difficulty of the proposed ranking method is the possible small number of experts. In order to overcome this difficulty, an imprecise Dirichlet model (IDM) introduced by Walley [34] was applied to extend the belief and plausibility functions such that a lack of sufficient statistical data could be taken into account [35, 36]. With the method, we can get the extended belief and plausibility functions as follows:


Here the hyperparameter s determines how quickly upper and lower probability of events converge as statistical data accumulate, and it should be taken to be 1 or 2; N is the number of expert judgements.

However, the main advantage of the IDM is that it produces the cautious inference. In particular, if N = 0, then Bels(A) = 0 and pls(A) = 1. In the case N, it can be stated for any s: Bels(A) = Bel(A), pls(A) = pl(A). If we denote μ = N/(N + s), then there holds m*(Ai) = μ · m(Ai). One can see from the last expression for m*(Ai) that μ is the discount rate characterizing the reliability of a source of evidence and it depends on the number of estimates N. Because the total probability assignment ∑m(Ai) = 1, we assign the left probability to m*(B77); that means that the experts do not know which alternative is better than the others and is indicated by the preference {A1, A2, A3}⪰{A1, A2, A3}. By using the discount rate μ with s = 1, we can get μ1 = 5/6≃0.83 for the first group and μ2 = 10/11≃0.91. Hence we can rewrite the preference intersection for Dempster-Shafer combination rule in Table 5.

Table 5

The preference intersections for modified Dempster-Shafer combination rule.

And the modified probability assignment and conflict weight are


Now the combined BPA in Table 5 are



So the belief and plausibility functions of {Ai}⪰Λ are


In the same way,


It can be seen from the above results that the “best” ranking is A1A3A2.

4. Improved Method and Numerical Analysis

The main advantage of IDM method is that it allows us to deal with comparisons of arbitrary groups of alternatives. It gives the possibility to use the framework of Dempster-Shafer evidence theory and to compute the belief and plausibility functions of alternatives or ranking and provides a way to make cautious decisions when the number of expert estimates is rather small. However, the method depends excessively on the number of experts rather than data itself. If we change the number of experts, it may lead to a different result. For instant, we assume the number of first group of experts is 50, and 20 experts choose the preference {A1}⪰{A2, A3} = B16(1), and 30 experts choose the preference {A1, A2}⪰{A3} = B43(1). Then we get the same probability assignment for the first group of experts. The only change is the discount rate from 0.83 to μ1 = 50/51 = 0.98. Here we recompute the probability assignment and conflict weight as follows:


Now we recompute the combined BPA in Table 5



So the belief and plausibility functions of {Ai}⪰Λ are


In the same way,


Thus we get that the “best” ranking is A1A3A2 by pessimistic decision-making and A1A2A3

When you're choosing between many different options, how do you decide on the best way forward?

This is especially challenging if your choices are quite different from one another, if decision criteria are subjective, or if you don't have objective data to use for your decision.

Paired Comparison Analysis helps you to work out the relative importance of a number of different options – the classical case of "comparing apples with oranges."

In this article and video, we'll explore how you can use Paired Comparison Analysis to make decisions.

Get your most important options to stand out from the crowd.

About the Tool

Paired Comparison Analysis (also known as Pairwise Comparison) helps you work out the importance of a number of options relative to one another.

This makes it easy to choose the most important problem to solve, or to pick the solution that will be most effective. It also helps you set priorities where there are conflicting demands on your resources.

The tool is particularly useful when you don't have objective data to use to make your decision. It's also an ideal tool to use to compare different, subjective options, for example, where you need to decide the relative importance of qualifications, skills, experience, and teamworking ability when hiring people for a new role.

Decisions like these are often much harder to make than, for example, comparing three similar IT systems, where Decision Matrix Analysis or some form of financial analysis can help you decide.

How to Use the Tool

To use the technique, download our free worksheet, and then follow these six steps:

  1. Make a list of all of the options that you want to compare. Assign each option a letter (A, B, C, D, and so on) and note this down.
  2. Mark your options as both the row and column headings on the worksheet. This is so that you can compare options with one-another.


On the table, the cells where you will compare an option with itself are blocked out. The cells on the table where you would be duplicating a comparison are also blocked out. This ensures that you make each comparison only once.

  1. Within each of the blank cells, compare the option in the row with the option in the column. Decide which of the two options is most important, and write down the letter of the most important option in the cell. 
  2. Score the difference in importance between the options, running from zero (no difference/same importance) to, say, three (major difference/one much more important than the other.)
  3. Finally, consolidate the results by adding up the values for each of the options. You may want to convert these values into a percentage of the total score.
  4. Use your common sense, and manually adjust the results if necessary.


For example, a philanthropist is choosing between several different nonprofit organizations that are asking for funding. To maximize impact, she only wants to contribute to a few of these, and she has the following options:

  • An overseas development project.
  • A local educational project.
  • A bequest for her university.
  • Disaster relief.

First, she draws up the Paired Comparison Analysis table in Figure 1.

Figure 1 – Example Paired Comparison Analysis Table (not filled in):

 A: Overseas DevelopmentB: Local EducationalC: UniversityD: Disaster Relief
A: Overseas Development    
B: Local Educational    
C: University    
D: Disaster

Then she compares options, writes down the letter of the most important option, and scores their difference in importance to her. Figure 2 illustrates this step of the process.

Figure 2 – Example Paired Comparison Analysis Table (filled in):

 A: Overseas DevelopmentB: Local EducationalC: UniversityD: Disaster Relief
A: Overseas Development A, 2C, 1A, 1
B: Local Educational  C, 1B, 1
C: University
  C, 2
D: Disaster

Finally, she adds up the A, B, C, and D values and converts each into a percentage of the total. These calculations yield the following totals:

  • A = 3 (37.5 percent).
  • B = 1 (12.5 percent).
  • C = 4 (50 percent).
  • D = 0.

Here, she decides to make a bequest to her university (C) and to allocate some funding to overseas development (A).

Key Points

Paired Comparison Analysis is useful for weighing up the relative importance of different options. It's particularly helpful where priorities aren't clear, where the options are completely different, where evaluation criteria are subjective, or where they're competing in importance.

The tool provides a framework for comparing each option against all others, and helps to show the difference in importance between factors.

Download Worksheet

Get the Free Newsletter

Learn essential career skills every week, and get your bonus Time Management: "Seven Quick Wins" Checklist FREE when you subscribe.

Leave a Comment


Your email address will not be published. Required fields are marked *