Originally Published - September 23, 2020

Upshot One is a question & answer protocol built on Ethereum. On it, randomly selected agents are incentivized to answer questions correctly via peer prediction. Series of questions and answers can be forked to reflect varying subjective preferences.

A question & answer protocol is simply a service where questions are met with answers. Upshot One is a question & answer protocol that makes it prohibitively expensive for questions to be matched with incorrect answers. In this article, we describe how Upshot One accomplishes this. At a high level, Upshot One operates in five steps:

1. Asking: Questions are asked

2. Answering: Agents place stakes and answer questions

3. Selection: Agents are randomly selected according to their stake to answer questions

4. Scoring: Answers of selected agents are scored and questions are resolved (i.e. matched with an answer)

5. Forking: Series of questions and answers can be forked in the event of a successful attack or to reflect differing preferences

Anyone can ask a question on Upshot One. To do this, question-askers must provide some information and a monetary reward. In particular, they must…

1. list the possible answers to their questions (e.g. “yes” and “no” for a “yes-or-no” question)

2. state how much confidence agents should have in their answers (Confidence is measured as the total amount of stake placed by agents in their hopes of answering questions. If the total stake does not outweigh this confidence criterion, a question will not be resolved.)

3. define filter criteria that limits whose answers can be considered (e.g. “people that hold X governance token” or “agents on this whitelist of beta testers”)

4. pay a fee that is awarded to successful agents, paid in any currency (this incentivizes agents to participate in Upshot One)

Agents (who could be anyone) stake money to answer questions. The more money they stake, the higher likelihood that their answers will be selected to resolve questions. Agents may lose this stake if they fail to answer enough questions. The exact set of questions that are petitioned for resolution is randomly formed, so an agent’s only recourse to save their stake is to answer as many questions as possible. Thus, the size of an agent’s stake reflects the number of questions they’ve answered. As we’ll see, an agent’s stake reflects the quality of their answers as well.

As mentioned, the more an agent stakes, the more likely their answers will be used to resolve questions. Three agents are sampled randomly without replacement, which means that an agent can be selected multiple times. Random sampling ensures that the selection of any particular agent is never guaranteed (so long as there is more than one agent), which contributes to Upshot One’s security. Though it is possible (albeit expensive as a large stake would need to be placed) to dominate a selection (and thus possibly manipulate resolutions), the “threat” of being chosen alongside a non-colluding agent implies that not every agent needs to be queried for answers to find a correct answer — just a representative set of three agents is queried — and this contributes to Upshot One’s efficiency. Sampling without replacement ensures that creating many (say N) fake accounts that each stake some small amount X is never more profitable than staking X*N from a single account.

Random sampling requires a random seed to ensure that samples are not predictable (i.e. are truly random). Upshot One secures random seeds in a two-step procedure. First, seeds are selected in advance, before a selection. In particular, the blockhash of a future Ethereum block is chosen, which is very difficult to predict well-ahead of its creation. Second, after the chosen blockhash is actually calculated, it is used as input to Chainlink’s verifiable random function (VRF) service, which outputs cryptographically secure random numbers. It is this output that is used as a seed for Upshot One’s random agent selection.

Selected agents’ answers are then fed to Upshot One’s peer prediction mechanism of choice. Peer prediction is a subfield of mechanism design that attempts to elicit honest information without any means of verification (i.e. without there necessarily being a ground truth). Upshot One employs the Determinant-based Mutual Information Mechanism (DMI-Mechanism) to score agents and obtain answers to questions.

DMI-Mechanism works like this. The answers of at least three agents are aggregated into columns and are indexed by their respective questions. These columns are paired with one another and then divided in half. Each “pairing of half-columns” is transformed into a matrix that lists the number of overlapping answers between agents. For a “yes-or-no” question, a 2x2 matrix would be formed and entry (1,0) represents the number of times the first paired agent said “yes” whenever the second paired agent said no. The matrix’s determinant is then calculated and multiplied by that of the other matrix. This product (and thus an agent’s score) is likely nonnegative because if two agents’ answers are negatively correlated, then both matrices are likely to have a nonpositive determinant, so their product is likely to be nonnegative. This product is then calculated for every agent pairing, and the sum of these pairings that include agent X is agent X’s score. This procedure, along with a technical description of DMI-Mechanism and its properties, is given in the paper that introduces DMI-Mechanism.

Why do we take the determinant of some matrix? The determinant of these matrices detects answering strategies, or any systematic way of answering questions apart from answering honestly. For example, if an ambivalent agent always says “yes” despite the honest answers to some questions being a mix of “yes” and “no,” then, whenever an ambivalent agent is paired with an honest agent, their score will be lower. Both the product of diagonal elements (“a” and “d” in the image below) and the product of off-diagonal elements (“b” and “c”) will be positive, thus lowering the determinant. Hence, the determinant of the matrix, and thus the ambivalent agent’s score, will be less than had they answered honestly.

DMI-Mechanism ensures that answering questions honestly is strictly the most profitable strategy for agents, and it requires only three agents for this property to hold (hence why we just select three agents). In fact, DMI-Mechanism is the first peer prediction mechanism to admit this property for a constant number of agents. DMI-Mechanism examines the correlation of answers between different agents and scores agents based on how well their answers correlate to others. In other words, agents’ answers determine how agents themselves are scored, which in turn determines how much they’re paid. Hence, an agent’s stake is a reflection of not only how many questions they’ve answered, but also the quality of their answers — Why would they stake a wealth of money if they know that they have low-quality answers and will thus receive poor returns?

After answers are scored, questions are resolved to the answers of the highest-scoring agent.

Though it is expensive to manipulate a question’s resolution, it is possible. Furthermore, a question may be inherently subjective in nature (e.g. “Do you like this outfit?”), and one may wish to subscribe to a feed of resolutions that reflects their unique preferences (e.g. If you said “yes,” you like this outfit, but the question resolved to “no,” then maybe you should use a different series of resolutions to curate your fashion). To this end, Upshot One employs subjectivocracy (“subjective governance”), which means that anyone can fork (i.e. split away from) a series of resolutions. Anyone that reads Upshot One can then choose to focus on the series that best suits their preferences. How they choose a series is entirely up to them (one may choose the series of resolutions that best fits their preferences, that is growing fastest, that is most cited, etc.).

Upshot One is a question & answer protocol on the Ethereum blockchain that associates questions to honest answers. We’ve covered the five steps that comprise Upshot One: Asking, Answering, Selection, Scoring, and Forking. With this post and future content, we hope to make Upshot One accessible to everyone so anyone can join us on this journey of ensuring that questions are answered honestly.

To stay up date, subscribe to our Medium, and follow us on Twitter. For further insight into how Upshot One works, check out upshot.io, read our whitepaper, or reach out directly to us!

Acknowledgements to my co-founder Kenny Peluso for collaboration on this post.

Originally Published - September 23, 2020

Upshot One is a question & answer protocol built on Ethereum. On it, randomly selected agents are incentivized to answer questions correctly via peer prediction. Series of questions and answers can be forked to reflect varying subjective preferences.

A question & answer protocol is simply a service where questions are met with answers. Upshot One is a question & answer protocol that makes it prohibitively expensive for questions to be matched with incorrect answers. In this article, we describe how Upshot One accomplishes this. At a high level, Upshot One operates in five steps:

1. Asking: Questions are asked

2. Answering: Agents place stakes and answer questions

3. Selection: Agents are randomly selected according to their stake to answer questions

4. Scoring: Answers of selected agents are scored and questions are resolved (i.e. matched with an answer)

5. Forking: Series of questions and answers can be forked in the event of a successful attack or to reflect differing preferences

Anyone can ask a question on Upshot One. To do this, question-askers must provide some information and a monetary reward. In particular, they must…

1. list the possible answers to their questions (e.g. “yes” and “no” for a “yes-or-no” question)

2. state how much confidence agents should have in their answers (Confidence is measured as the total amount of stake placed by agents in their hopes of answering questions. If the total stake does not outweigh this confidence criterion, a question will not be resolved.)

3. define filter criteria that limits whose answers can be considered (e.g. “people that hold X governance token” or “agents on this whitelist of beta testers”)

4. pay a fee that is awarded to successful agents, paid in any currency (this incentivizes agents to participate in Upshot One)

Agents (who could be anyone) stake money to answer questions. The more money they stake, the higher likelihood that their answers will be selected to resolve questions. Agents may lose this stake if they fail to answer enough questions. The exact set of questions that are petitioned for resolution is randomly formed, so an agent’s only recourse to save their stake is to answer as many questions as possible. Thus, the size of an agent’s stake reflects the number of questions they’ve answered. As we’ll see, an agent’s stake reflects the quality of their answers as well.

As mentioned, the more an agent stakes, the more likely their answers will be used to resolve questions. Three agents are sampled randomly without replacement, which means that an agent can be selected multiple times. Random sampling ensures that the selection of any particular agent is never guaranteed (so long as there is more than one agent), which contributes to Upshot One’s security. Though it is possible (albeit expensive as a large stake would need to be placed) to dominate a selection (and thus possibly manipulate resolutions), the “threat” of being chosen alongside a non-colluding agent implies that not every agent needs to be queried for answers to find a correct answer — just a representative set of three agents is queried — and this contributes to Upshot One’s efficiency. Sampling without replacement ensures that creating many (say N) fake accounts that each stake some small amount X is never more profitable than staking X*N from a single account.

Random sampling requires a random seed to ensure that samples are not predictable (i.e. are truly random). Upshot One secures random seeds in a two-step procedure. First, seeds are selected in advance, before a selection. In particular, the blockhash of a future Ethereum block is chosen, which is very difficult to predict well-ahead of its creation. Second, after the chosen blockhash is actually calculated, it is used as input to Chainlink’s verifiable random function (VRF) service, which outputs cryptographically secure random numbers. It is this output that is used as a seed for Upshot One’s random agent selection.

Selected agents’ answers are then fed to Upshot One’s peer prediction mechanism of choice. Peer prediction is a subfield of mechanism design that attempts to elicit honest information without any means of verification (i.e. without there necessarily being a ground truth). Upshot One employs the Determinant-based Mutual Information Mechanism (DMI-Mechanism) to score agents and obtain answers to questions.

DMI-Mechanism works like this. The answers of at least three agents are aggregated into columns and are indexed by their respective questions. These columns are paired with one another and then divided in half. Each “pairing of half-columns” is transformed into a matrix that lists the number of overlapping answers between agents. For a “yes-or-no” question, a 2x2 matrix would be formed and entry (1,0) represents the number of times the first paired agent said “yes” whenever the second paired agent said no. The matrix’s determinant is then calculated and multiplied by that of the other matrix. This product (and thus an agent’s score) is likely nonnegative because if two agents’ answers are negatively correlated, then both matrices are likely to have a nonpositive determinant, so their product is likely to be nonnegative. This product is then calculated for every agent pairing, and the sum of these pairings that include agent X is agent X’s score. This procedure, along with a technical description of DMI-Mechanism and its properties, is given in the paper that introduces DMI-Mechanism.

Why do we take the determinant of some matrix? The determinant of these matrices detects answering strategies, or any systematic way of answering questions apart from answering honestly. For example, if an ambivalent agent always says “yes” despite the honest answers to some questions being a mix of “yes” and “no,” then, whenever an ambivalent agent is paired with an honest agent, their score will be lower. Both the product of diagonal elements (“a” and “d” in the image below) and the product of off-diagonal elements (“b” and “c”) will be positive, thus lowering the determinant. Hence, the determinant of the matrix, and thus the ambivalent agent’s score, will be less than had they answered honestly.

DMI-Mechanism ensures that answering questions honestly is strictly the most profitable strategy for agents, and it requires only three agents for this property to hold (hence why we just select three agents). In fact, DMI-Mechanism is the first peer prediction mechanism to admit this property for a constant number of agents. DMI-Mechanism examines the correlation of answers between different agents and scores agents based on how well their answers correlate to others. In other words, agents’ answers determine how agents themselves are scored, which in turn determines how much they’re paid. Hence, an agent’s stake is a reflection of not only how many questions they’ve answered, but also the quality of their answers — Why would they stake a wealth of money if they know that they have low-quality answers and will thus receive poor returns?

After answers are scored, questions are resolved to the answers of the highest-scoring agent.

Though it is expensive to manipulate a question’s resolution, it is possible. Furthermore, a question may be inherently subjective in nature (e.g. “Do you like this outfit?”), and one may wish to subscribe to a feed of resolutions that reflects their unique preferences (e.g. If you said “yes,” you like this outfit, but the question resolved to “no,” then maybe you should use a different series of resolutions to curate your fashion). To this end, Upshot One employs subjectivocracy (“subjective governance”), which means that anyone can fork (i.e. split away from) a series of resolutions. Anyone that reads Upshot One can then choose to focus on the series that best suits their preferences. How they choose a series is entirely up to them (one may choose the series of resolutions that best fits their preferences, that is growing fastest, that is most cited, etc.).

Upshot One is a question & answer protocol on the Ethereum blockchain that associates questions to honest answers. We’ve covered the five steps that comprise Upshot One: Asking, Answering, Selection, Scoring, and Forking. With this post and future content, we hope to make Upshot One accessible to everyone so anyone can join us on this journey of ensuring that questions are answered honestly.

To stay up date, subscribe to our Medium, and follow us on Twitter. For further insight into how Upshot One works, check out upshot.io, read our whitepaper, or reach out directly to us!

Acknowledgements to my co-founder Kenny Peluso for collaboration on this post.