Truthicize - Informations

What is the aim of Truthicize ?

Truthicize aims at finding the truth amongst several people answers to multiple question polls (quiz).
The idea is to identify what are the plausible most reliable sources and to take into account their opinion more than less reliables sources. It is aimed to be used:

As a vulgarisation platform to illustrate the power of democratic choices to find the truth and to illustrate the use of dedicated methods to do so
As a tool for e-democracy, for taking multiple decisions that have the most chances to be the good ones
As a tool for collect real world benchmarks

How to use it (Create a quiz) ?

Log in (or create an account if it is your first quiz)
Define your quiz (see the documentation here)
Start the quiz
Invite people to answer

How to use it (Answer a quiz) ?

Just find the quiz you want to answer (by looking at the open quiz or by using the short url that was given to you) and start answering.

How it works ?

Does it work ?

To have convincing results you have to face a population of people that have an averagely good expertise. If the global expertise is very low, there are no chances to nd the truth.
If the global expertise is very high, our method will nd the truth, but just as well as majority voting on each question (since if everyone has small chances to give a wrong answer, then mosts people will give the true answer of each question, that is then easily identified).
Our method outperforms majority voting when we are on the remaining situations.

Data collection - Privacy ?

One of the aims of Truthicize is to collect data for performing experimental evaluation of methods developed by academic researchers. So to create a quiz or to answer it, you have to agree on that data collection.
The corresponding benchmarks will be made available for all academic researchers.
We will keep only track of the structure of the questions, answers and claims from participants, i.e. the collected datas will not be the sentences of the questions and of the answers, and we also do not keep track of the pseudo of the participants.
So there is strictly no privacy issue, since we will not know what the questions and answers where about, and we will not know anything about the participants identity.
See here for an example of what are the resulting benchmark associated to a quiz.

How it works ?

Truthicize uses a method called «Sources & Facts» in order to evaluate the reliability of answers to questions and of sources of information (people who answer the quiz). The idea behind «Sources & Facts» is to identify what are the most plausible reliable sources and to take into account their opinion more than the less reliables sources.
Thus Truthicize can provide better performances than majority vote on each question for finding the truth.
More specifically, we suppose that we initially have no information about the reliability of the sources and we define an iterative procedure to determine their reliability.
At the beginning, we assign the same reliability to all the sources, then we compare the answers to the different questions. In order to find the true information and reward thesources, we rely on the idea of Condorcet’s Jury Theorem (which is also the theoretical basis on the experiments on «Wisdom of Crowds»), which states that it is more likely that the majority of the individuals will choose the correct solution.

More precisely, at each iteration, the sources give strength to the facts they claim about the different objects. With the sum of the obtained strengths, we got the confidence of each fact.
We want to reward the sources that provide pieces of information that is confirmed by others, and then that are more likely true. To reward the sources, the objects take part to a vote where they rank their related facts from most reliable to least reliable ones. We use scoring-based voting rules to assign a score to each rank of facts.
Then the new reliability of each source is computed by combining all these scores. But, we wish to give the reliability of the source, i.e. an estimation of the probability of this source to find the true facts. So, we have to normalize the reliability of the sources to ensure that this reliability is between 0 and 1. There are at least two ways to normalize the reliability. The first one (normalization A) favors sources that provide the most of correct answers. The reliability of the source is divided by the total number of objects in the graph. The second (normalization C) favors sources that are more careful and do not fail often. The reliability of the source is divided by the number of objects on which it claims a fact.
Then a new iteration begins with the updated reliability of each source. The algorithm stops when the process converges, i.e. when the similarity between the reliability of the sources of the previous and the current iteration is small enough, or when 30 iterations has been performed.

Corresponding Research Papers

2023 Quentin Elsaesser, Patricia Everaere, Sébastien Konieczny, Voting-based Methods for Evaluating Sources and Facts Reliability in International Conference on Tools with Artificial Intelligence (ICTAI 2023). pages 178-185. 2023. [pdf]

Other Related Papers

2022 Joseph Singleton and Richard Booth, Towards an axiomatic approach to truth discovery. in Journal of Autonomous Agents and Multi-Agent Systems 36, 2 (2022), 1-49.

2010 Je Pasternack and Dan Roth., Knowing what to believe (when you al- ready know something). in Proceedings of the 23rd International Conference on Computational Linguistics (COLING 2010). 877–885.

2008 Xiaoxin Yin, Jiawei Han, and Philip S. Yu., Truth Discovery with Multiple Con icting Information Providers on the Web. in IEEE Transactions on Knowledge and Data Engineering 20, 6 (2008), 796–808.

1999 Jon M Kleinberg., Authoritative sources in a hyperlinked environment. in Journal of the ACM (JACM) 46, 5 (1999), 604–632.

1785 Marquis de Condorcet., Essai sur l’application de l’analyse la probabilit des d cisions rendues la pluralit des voix. in Imprimerie royale Paris.

Informations provided by the users/participants of a quiz

Participants
Agathe:
Brazil -> Brasilia
Australia -> Camberra
Romane:
Brazil -> Rio
Australia -> Camberra
Marie:
Brazil -> Brasilia
Australia -> Sydney
Alienor:
Brazil -> Brasilia
Australia -> Sydney

Informations that are kept for benchmarks

fact(Q1,F1)
fact(Q1,F2)
fact(Q2,F3)
fact(Q2,F4)

claim(U1,F2)
claim(U1,F4)
claim(U2,F1)
claim(U2,F4)

claim(U3,F2)
claim(U3,F3)
claim(U4,F2)
claim(U4,F3)

The benchmarks that are collected from answer do not keep any information on the identity of the participants, on the text of the questions and answers, assuring a full privacy of the polls (quiz). The only information that is kept is the structure of the graphs of answers, where participants, questions and answers are replaced by variables (U1,U2,Q1,Q2,F1,F2,…)