Subscribe to MetricsDAO
MetricsDAOBounty Review Process Overview
Bounties form a big part of MetricsDAO’s mission to deliver high-quality analytics solutions. As our partners rely on accurate, timely, and informative analytics, it is important that MetricsDAO evaluates bounty submissions according to a specific set of criteria. These are Accuracy & Completeness, Insights, Visualizations, and Definitions. Submissions that get decent scores in these criteria are qualified to receive bounty payouts. Those that get a perfect score are then entitled to the gran...
MetricsDAOGoldfinch Deep Dive
SummaryGoldfinch is a decentralized credit platform that enables loans without requiring on-chain collateral. This novel approach increases access to capital and expands the crypto lending market to users that otherwise would not have sufficient on-chain assets to qualify for a loan. Goldfinch’s “trust through consensus” mechanism allows lenders to directly evaluate the creditworthiness of potential borrowers, supporting but removing the need for on-chain collateral. Ultimately, this creates ...
MetricsDAOIntroducing: MetricsDAO
Uniting the best analytical minds in the space to build the future of crypto analytics. Join MetricsDAO on DiscordThe DAO MissionThe purpose of MetricsDAO is to empower analytical minds and enable blockchains and protocols to succeed. It's critical for blockchains and all ecosystem participants to have the data, insights, visualizations and metrics that enable an active, educated and informed community. MetricsDAO will provide the framework for insights to be delivered on-demand and as-n...
<100 subscribers
MetricsDAOBounty Review Process Overview
Bounties form a big part of MetricsDAO’s mission to deliver high-quality analytics solutions. As our partners rely on accurate, timely, and informative analytics, it is important that MetricsDAO evaluates bounty submissions according to a specific set of criteria. These are Accuracy & Completeness, Insights, Visualizations, and Definitions. Submissions that get decent scores in these criteria are qualified to receive bounty payouts. Those that get a perfect score are then entitled to the gran...
MetricsDAOGoldfinch Deep Dive
SummaryGoldfinch is a decentralized credit platform that enables loans without requiring on-chain collateral. This novel approach increases access to capital and expands the crypto lending market to users that otherwise would not have sufficient on-chain assets to qualify for a loan. Goldfinch’s “trust through consensus” mechanism allows lenders to directly evaluate the creditworthiness of potential borrowers, supporting but removing the need for on-chain collateral. Ultimately, this creates ...
MetricsDAOIntroducing: MetricsDAO
Uniting the best analytical minds in the space to build the future of crypto analytics. Join MetricsDAO on DiscordThe DAO MissionThe purpose of MetricsDAO is to empower analytical minds and enable blockchains and protocols to succeed. It's critical for blockchains and all ecosystem participants to have the data, insights, visualizations and metrics that enable an active, educated and informed community. MetricsDAO will provide the framework for insights to be delivered on-demand and as-n...
Subscribe to MetricsDAO
<100 subscribers
Share Dialog
Share Dialog
Goal: Qualify submissions in terms of quality and veracity.
MetricsDAO provides a 6 step process for Organized, On-Demand Analytics Delivery, or as we like to call it, OODAD.
QA for Solutions is the 5th step, which we experimented with for the first time with OlympusDAO. Here’s how we did it, what we learned and how we plan to improve!
Timeline: We gave ourselves a week to review and grade the 11 OlympusDAO submissions based on a predefined rubric.
Reviewers: After putting the word out on Discord, we found 2 community members to review the 11 Olympus DAO submissions. Reviewers had to know basic SQL and have experience with crypto analytics. Our third reviewer was a member of the core team.
Incentive: Reviewers will earn a reviewer-specific POAP.
Peer-Review Process: The Core Team member was in charge of reviewing the other reviewers’ work.
QA for OlympusDAO solutions went very smoothly. The reviewers broadly agreed with each other, and it was easy to identify winning submissions. We were able to use average scores to validate submissions, and we finished grading in a week.
Quality of submissions varied widely. The worst quality submissions were directly tied to users doing the bare minimum. The best quality ones were done by experienced users. Experience matters but it is not the only factor determining quality – better incentives should be put in place.
Incentivizing better quality submissions
Payment:
We want to experiment with awarding higher quality submissions with a higher payout.
We will also be paying more for harder questions.
Setting Expectations:
The level of difficulty for each Challenge will be stated from the get-go. This should help analysts know which challenge they can tackle effectively.
The rubric should be made very public and shared more broadly ahead of time.
Payouts should also be announced ahead of time.
Knowledge-Sharing & Mentorship:
As more bounty rounds are completed we will have existing grand prize submissions to share out as examples to follow.
The #support channel will be the place for users to ask questions about the rubric and get help easily. We want to promote education and knowledge sharing as much as possible!
Our Mentor-Circle is planning to host more workshops where community members can learn from each other.
Dispute & resolution mechanism:
We want to implement a dispute & resolution mechanism in case one reviewer disagrees with everyone else. A few experienced graders should carry somewhat more weight.
Attracting more reviewers:
As the number of submissions increases, we will need more reviewers. Being a Reviewer is an excellent way for talented analysts with SQL and crypto skills to get involved at a low time commitment (2-3 hours a week, maximum).
For upcoming programs, like Convex, we plan to offer a Reviewer POAP to track reviewer contributions. Additionally - from experience, Reviewers have indicated that seeing the submissions of others is an excellent way to improve their own work by getting exposure to a range of different techniques.
If you have any interest in being a Reviewer for upcoming programs, please contact Snowslinger.ust!
We’d love to hear your thoughts and feedback so please let us know in Discord!
MetricsDAO is uniting the best analytical minds in the space to build the future of crypto analytics.
Goal: Qualify submissions in terms of quality and veracity.
MetricsDAO provides a 6 step process for Organized, On-Demand Analytics Delivery, or as we like to call it, OODAD.
QA for Solutions is the 5th step, which we experimented with for the first time with OlympusDAO. Here’s how we did it, what we learned and how we plan to improve!
Timeline: We gave ourselves a week to review and grade the 11 OlympusDAO submissions based on a predefined rubric.
Reviewers: After putting the word out on Discord, we found 2 community members to review the 11 Olympus DAO submissions. Reviewers had to know basic SQL and have experience with crypto analytics. Our third reviewer was a member of the core team.
Incentive: Reviewers will earn a reviewer-specific POAP.
Peer-Review Process: The Core Team member was in charge of reviewing the other reviewers’ work.
QA for OlympusDAO solutions went very smoothly. The reviewers broadly agreed with each other, and it was easy to identify winning submissions. We were able to use average scores to validate submissions, and we finished grading in a week.
Quality of submissions varied widely. The worst quality submissions were directly tied to users doing the bare minimum. The best quality ones were done by experienced users. Experience matters but it is not the only factor determining quality – better incentives should be put in place.
Incentivizing better quality submissions
Payment:
We want to experiment with awarding higher quality submissions with a higher payout.
We will also be paying more for harder questions.
Setting Expectations:
The level of difficulty for each Challenge will be stated from the get-go. This should help analysts know which challenge they can tackle effectively.
The rubric should be made very public and shared more broadly ahead of time.
Payouts should also be announced ahead of time.
Knowledge-Sharing & Mentorship:
As more bounty rounds are completed we will have existing grand prize submissions to share out as examples to follow.
The #support channel will be the place for users to ask questions about the rubric and get help easily. We want to promote education and knowledge sharing as much as possible!
Our Mentor-Circle is planning to host more workshops where community members can learn from each other.
Dispute & resolution mechanism:
We want to implement a dispute & resolution mechanism in case one reviewer disagrees with everyone else. A few experienced graders should carry somewhat more weight.
Attracting more reviewers:
As the number of submissions increases, we will need more reviewers. Being a Reviewer is an excellent way for talented analysts with SQL and crypto skills to get involved at a low time commitment (2-3 hours a week, maximum).
For upcoming programs, like Convex, we plan to offer a Reviewer POAP to track reviewer contributions. Additionally - from experience, Reviewers have indicated that seeing the submissions of others is an excellent way to improve their own work by getting exposure to a range of different techniques.
If you have any interest in being a Reviewer for upcoming programs, please contact Snowslinger.ust!
We’d love to hear your thoughts and feedback so please let us know in Discord!
MetricsDAO is uniting the best analytical minds in the space to build the future of crypto analytics.
No comments yet