OSO_Giveth_Pipeline_By_DAOstar.md

Giveth Data Pipeline Documentation

This document outlines the Giveth data ingestion pipeline integrated into the DAGSTER-based ETL system. It describes what data is pulled, and where it lands in the data lake.

---

Overview

The pipeline fetches:

• QF Rounds Metadata
• Project Metadata for each QF Round ( Only projects eligible for Giveth's rounds are pulled )

Each record is flattened and normalized using Polars DataFrame operations. The output is materialized into the DuckDB data lake with asset names:

• giveth__qf_rounds
• giveth__projects_by_round

---

QF Rounds Field Extraction

Field	Extracted
id	✅
name	✅
title	✅
description	✅
slug	✅
isActive	✅
allocatedFund	✅
allocatedFundUSD	✅
allocatedFundUSDPreferred	✅
allocatedTokenSymbol	✅
allocatedTokenChainId	✅
maximumReward	✅
minimumPassportScore	✅
minMBDScore	✅
minimumValidUsdValue	✅
eligibleNetworks	✅
beginDate	✅
endDate	✅
qfStrategy	✅
bannerBgImage	✅
sponsorsImgs	✅
isDataAnalysisDone	✅
clusterMatchingSyncAt	✅

---

Project Field Extraction

Field	Extracted
id	✅
title	✅
slug	✅
slugHistory	✅
description	✅
descriptionSummary	✅
traceCampaignId	✅
givingBlocksId	✅
changeId	✅
website	✅
youtube	✅
creationDate	✅
updatedAt	✅
latestUpdateCreationDate	✅
organization	✅
coOrdinates	✅
image	✅
impactLocation	✅
categories	✅
qfRounds	✅
balance	✅
stripeAccountId	✅
walletAddress	✅
verified	✅
verificationStatus	✅
isImported	✅
giveBacks	✅
donations	✅
qualityScore	✅
contacts	✅
reactions	✅
addresses	✅
socialMedia	✅
anchorContracts	✅
status	✅
adminUser	✅
adminUserId	✅
statusHistory	✅
projectVerificationForm	✅
featuredUpdate	✅
projectPower	❌
projectFuturePower	❌
projectInstantPower	❌
verificationFormStatus	✅
socialProfiles	✅
projectEstimatedMatchingView	✅
totalDonations	✅
totalTraceDonations	✅
totalReactions	✅
totalProjectUpdates	✅
sumDonationValueUsdForActiveQfRound	✅
countUniqueDonorsForActiveQfRound	✅
countUniqueDonors	✅
listed	✅
isGivbackEligible	✅
reviewStatus	✅
projectUrl	✅
prevStatusId	✅
projectUpdate	❌
projectUpdates	❌
adminJsBaseUrl	✅
reaction	✅
campaigns	✅
estimatedMatching	✅

---

Data Lake Assets

Asset Name	Description
giveth__qf_rounds	Contains one row per QF Round with all relevant metadata fields from the Giveth API
giveth__projects_by_round	Contains all projects that participated in QF rounds, along with round-level association

---

Future Work

• Map these fields to DAOIP-5 standard
• Set up error handling for unexpected nested or null fields