Civil Transparency Experiments in Python

CivilTransparency.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Civil Transparency Experiments in Python\n",
    "\n",
    "Exploring and auditing the https://civil.co world and ecosystem for higher-quality, more sustainable journalism.\n",
    "\n",
    "As discussed at https://community.civil.co/t/transparency-reproducible-vote-tallies-etc-digging-in-to-the-blockchain/147\n",
    "\n",
    "So far, this is just the beginning of getting on board in Python with Ethereum, web3, Civil, PLCR etc.\n",
    "so we can e.g. some day audit the vote counts announced to the community."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparation\n",
    "\n",
    "Do this, ideally in a virtual environment, before starting up your Jupyter notebook or python3 environment:\n",
    "\n",
    "Learn about Ethereum in Python:\n",
    "[Ethereum Smart Contracts in Python: a comprehensive(ish) guide](https://hackernoon.com/ethereum-smart-contracts-in-python-a-comprehensive-ish-guide-771b03990988)\n",
    "\n",
    "`pip install web3 ens jupyter`\n",
    "\n",
    "Install a local eth server, or get keys to connect to the eth network via https://infura.io or whatever.\n",
    "\n",
    "`export WEB3_INFURA_API_KEY=xxxx`\n",
    "\n",
    "`export WEB3_PROVIDER_URI=wss://mainnet.infura.io/ws/v3/xxxx`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import urllib.request\n",
    "from web3 import Web3\n",
    "from web3.auto import w3\n",
    "from ens import ENS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8306033"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w3.eth.blockNumber"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "ns = ENS.fromWeb3(w3)                                                                                                        \n",
    "cvltcr = ns.address('civiltcr.eth')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0xBd5a95A66dd4E78Bcb597198Df222c4EdDC14DA7'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cvltcr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cvltcr == Web3.toChecksumAddress(cvltcr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0x00c26eC65d2D84667212b820bfd25C60b59c60f5'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ns.owner('civiltcr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "ctcr = json.loads(urllib.request.urlopen('https://raw.githubusercontent.com/joincivil/Civil/master/packages/artifacts/v1/CivilTCR.json').read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "cabi = ctcr['abi']\n",
    "cvl = w3.eth.contract(address=cvltcr, abi=cabi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "cvl = w3.eth.contract(address=cvltcr, abi=cabi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "cplcr = json.loads(urllib.request.urlopen('https://raw.githubusercontent.com/joincivil/Civil/master/packages/artifacts/v1/CivilPLCRVoting.json').read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "cplcrabi = cplcr['abi']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "cplcrx = w3.eth.contract(address=cvltcr, abi=cplcrabi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'CivilTCR'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cvl.functions.name().call() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['name', 'civilVoting', 'government', 'parameterizer', 'token', 'voting']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[c['name'] for c in cabi if 'name' in c and c['inputs'] == []]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0xc625FC42ab6d07746b953aE98b4Ec22622E1B9A9'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cvl.functions.government().call()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0x55656b8a58dF94C1e8B5142F8da973301452eA65'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cvl.functions.civilVoting().call()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['name',\n",
       " 'civilVoting',\n",
       " 'government',\n",
       " 'isWhitelisted',\n",
       " 'deposit',\n",
       " 'claimRewards',\n",
       " 'appWasMade',\n",
       " 'challengeRequestAppealExpiries',\n",
       " 'listings',\n",
       " 'challengeExists',\n",
       " 'challenges',\n",
       " 'tokenClaims',\n",
       " 'appeals',\n",
       " 'exit',\n",
       " 'canBeWhitelisted',\n",
       " 'parameterizer',\n",
       " 'withdraw',\n",
       " 'updateStatuses',\n",
       " 'token',\n",
       " 'voting',\n",
       " '_AppealRequested',\n",
       " '_AppealGranted',\n",
       " '_FailedChallengeOverturned',\n",
       " '_SuccessfulChallengeOverturned',\n",
       " '_GrantedAppealChallenged',\n",
       " '_GrantedAppealOverturned',\n",
       " '_GrantedAppealConfirmed',\n",
       " '_GovernmentTransfered',\n",
       " '_Application',\n",
       " '_Challenge',\n",
       " '_Deposit',\n",
       " '_Withdrawal',\n",
       " '_ApplicationWhitelisted',\n",
       " '_ApplicationRemoved',\n",
       " '_ListingRemoved',\n",
       " '_ListingWithdrawn',\n",
       " '_TouchAndRemoved',\n",
       " '_ChallengeFailed',\n",
       " '_ChallengeSucceeded',\n",
       " '_RewardClaimed',\n",
       " 'apply',\n",
       " 'requestAppeal',\n",
       " 'grantAppeal',\n",
       " 'transferGovernment',\n",
       " 'updateStatus',\n",
       " 'challenge',\n",
       " 'challengeGrantedAppeal',\n",
       " 'claimReward',\n",
       " 'determineReward',\n",
       " 'voterReward',\n",
       " 'challengeCanBeResolved',\n",
       " 'appealCanBeResolved',\n",
       " 'appealChallengeCanBeResolved']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[c['name'] for c in cabi if 'name' in c]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['INITIAL_POLL_NONCE', 'telemetry', 'pollNonce', 'token']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[c['name'] for c in cplcrabi if 'name' in c and c['inputs'] == []]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['getTotalNumberOfTokensForWinningOption',\n",
       " 'INITIAL_POLL_NONCE',\n",
       " 'getInsertPointForNumTokens',\n",
       " 'startPoll',\n",
       " 'voteTokenBalance',\n",
       " 'commitVotes',\n",
       " 'telemetry',\n",
       " 'getLastNode',\n",
       " 'revealPeriodActive',\n",
       " 'isPassed',\n",
       " 'pollMap',\n",
       " 'getLockedTokens',\n",
       " 'commitVote',\n",
       " 'didCommit',\n",
       " 'revealVotes',\n",
       " 'validPosition',\n",
       " 'pollExists',\n",
       " 'pollNonce',\n",
       " 'rescueTokens',\n",
       " 'attrUUID',\n",
       " 'commitPeriodActive',\n",
       " 'didReveal',\n",
       " 'revealVote',\n",
       " 'getNumPassingTokens',\n",
       " 'rescueTokensInMultiplePolls',\n",
       " 'getNumTokens',\n",
       " 'getCommitHash',\n",
       " 'isExpired',\n",
       " 'withdrawVotingRights',\n",
       " 'pollEnded',\n",
       " 'token',\n",
       " '_VoteCommitted',\n",
       " '_VoteRevealed',\n",
       " '_PollCreated',\n",
       " '_VotingRightsGranted',\n",
       " '_VotingRightsWithdrawn',\n",
       " '_TokensRescued',\n",
       " 'requestVotingRights',\n",
       " 'getNumLosingTokens',\n",
       " 'getTotalNumberOfTokensForLosingOption']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[c['name'] for c in cplcrabi if 'name' in c]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## All challenges to date\n",
    "Crude: just look at all of them by sequential challengeID, starting at 0 (invalid it seems)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, '0x0000000000000000000000000000000000000000', False, 0, 0]\n",
      "[373276072225334878979, '0x17918Cff28025059e15cEc69910Bc0662e94696B', True, 5000000000000000000000, 102277490000000000000000]\n",
      "[1403136403455656613885, '0x5E3E872aDb9266adCc47A32758b9329b235963E8', True, 7500000000000000000000, 126313990000000000000000]\n",
      "[177271168219167682318, '0xF1C5e806d642F7E2f73a679Ba430A5a61D58318B', True, 5000000000000000000000, 26152000000000000000000]\n",
      "[15064416691696182456, '0xF1C5e806d642F7E2f73a679Ba430A5a61D58318B', True, 5000000000000000000000, 2025480000000000000000]\n",
      "[1945613519490114942929, '0x14B9c0B60665D221c8Aa53CfF5C25e0EF6F7dA23', True, 5000000000000000000000, 17927490000000000000000]\n",
      "[69300185130732208524, '0xF1C5e806d642F7E2f73a679Ba430A5a61D58318B', True, 5000000000000000000000, 2624510000000000000000]\n",
      "[2500000000000000000000, '0x14B9c0B60665D221c8Aa53CfF5C25e0EF6F7dA23', False, 5000000000000000000000, 0]\n",
      "[0, '0x0000000000000000000000000000000000000000', False, 0, 0]\n",
      "[0, '0x0000000000000000000000000000000000000000', False, 0, 0]\n"
     ]
    }
   ],
   "source": [
    "for challengeID in range(10):\n",
    "    print(cvl.functions.challenges(challengeID).call())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, what does that data mean? ....."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Look up a specific challenged newsroom\n",
    "\n",
    "`mmviii [2008]`: https://registry.civil.co/listing/0xfcd4bd67481b8ffb9682988ef73dd72cf64dc0ae\n",
    "\n",
    "Blockchain Trailblazers: \n",
    "https://registry.civil.co/listing/0x49db84c3dbea7240293c7cd943827d59a8dc766b\n",
    "\n",
    "Active challenge as of 2019-08-07:\n",
    "\n",
    "blockleadersio: https://registry.civil.co/listing/0x2cd80996528073ca45da96ad96311e02ce9bc1c8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def challengeInfo(newsroomHex):\n",
    "    newsroomAddr = Web3.toInt(hexstr=newsroomHex)\n",
    "    listingAddr = Web3.toChecksumAddress(newsroomAddr)\n",
    "    listing = cvl.functions.listings(listingAddr).call()\n",
    "    # listing = cvl.functions.listings(listingAddr)\n",
    "    print(\"Listing data: %s\" % listing)\n",
    "    challengeID = Web3.toInt(listing[4])\n",
    "    if challengeID == 0:\n",
    "        print(\"No active challenge for this newsroom\")\n",
    "    else:\n",
    "        challengeData = cvl.functions.challenges(challengeID).call()\n",
    "        print(\"ChallengeInfo for challenge %d:\\n %s\" % (challengeID, challengeData))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "mmviiiHex = '0xfcd4bd67481b8ffb9682988ef73dd72cf64dc0ae'\n",
    "BlockchainTrailblazersHex = '0x49db84c3dbea7240293c7cd943827d59a8dc766b'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "blockleadersioHex = '0x2cd80996528073ca45da96ad96311e02ce9bc1c8'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Listing data: [0, False, '0x0000000000000000000000000000000000000000', 0, 0]\n",
      "No active challenge for this newsroom\n"
     ]
    }
   ],
   "source": [
    "challengeInfo(mmviiiHex)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Listing data: [1564000518, False, '0x1170Bb018bC6C1C6716d404DcDFdaA7E35037483', 0, 7]\n",
      "ChallengeInfo for challenge 7:\n",
      " [2500000000000000000000, '0x14B9c0B60665D221c8Aa53CfF5C25e0EF6F7dA23', False, 5000000000000000000000, 0]\n"
     ]
    }
   ],
   "source": [
    "challengeInfo(blockleadersioHex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And an approved newsroom:\n",
    "\n",
    "Africa-OnTheRise: https://registry.civil.co/listing/0x6af6e3e92a5c68e155f493f9671e0d4a5a01a002"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now what?\n",
    "\n",
    "Read more tips at  https://community.civil.co/t/transparency-reproducible-vote-tallies-etc-digging-in-to-the-blockchain/147 ..."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "63px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}