APIs data Collection.ipynb

{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "<center>\n    <img src=\"https://gitlab.com/ibm/skills-network/courses/placeholder101/-/raw/master/labs/module%201/images/IDSNlogo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\"  />\n</center>\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "# **Collecting Job Data Using APIs**\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Estimated time needed: **45 to 60** minutes\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "## Objectives\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "After completing this lab, you will be able to:\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "-   Collect job data from GitHub Jobs API\n-   Store the collected data into an excel spreadsheet. \n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "## Warm-Up Exercise\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Before you attempt the actual lab, here is a fully solved warmup exercise that will help you to learn how to access an API.\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Using an API, let us find out who currently are on the International Space Station (ISS).<br> The API at [http://api.open-notify.org/astros.json](http://api.open-notify.org/astros.json?cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ) gives us the information of astronauts currently on ISS in json format.<br>\nYou can read more about this API at [http://open-notify.org/Open-Notify-API/People-In-Space/](http://open-notify.org/Open-Notify-API/People-In-Space?cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ)\n"
        },
        {
            "cell_type": "code",
            "execution_count": 2,
            "metadata": {},
            "outputs": [],
            "source": "import requests # you need this module to make an API call"
        },
        {
            "cell_type": "code",
            "execution_count": 3,
            "metadata": {},
            "outputs": [],
            "source": "api_url = \"http://api.open-notify.org/astros.json\" # this url gives use the astronaut data"
        },
        {
            "cell_type": "code",
            "execution_count": 4,
            "metadata": {},
            "outputs": [],
            "source": "response = requests.get(api_url) # Call the API using the get method and store the\n                                # output of the API call in a variable called response."
        },
        {
            "cell_type": "code",
            "execution_count": 5,
            "metadata": {},
            "outputs": [],
            "source": "if response.ok:             # if all is well() no errors, no network timeouts)\n    data = response.json()  # store the result in json format in a variable called data\n                            # the variable data is of type dictionary."
        },
        {
            "cell_type": "code",
            "execution_count": 6,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "{'message': 'success', 'number': 7, 'people': [{'craft': 'ISS', 'name': 'Sergey Ryzhikov'}, {'craft': 'ISS', 'name': 'Kate Rubins'}, {'craft': 'ISS', 'name': 'Sergey Kud-Sverchkov'}, {'craft': 'ISS', 'name': 'Mike Hopkins'}, {'craft': 'ISS', 'name': 'Victor Glover'}, {'craft': 'ISS', 'name': 'Shannon Walker'}, {'craft': 'ISS', 'name': 'Soichi Noguchi'}]}\n"
                }
            ],
            "source": "print(data)   # print the data just to check the output or for debugging"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Print the number of astronauts currently on ISS.\n"
        },
        {
            "cell_type": "code",
            "execution_count": 7,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "7\n"
                }
            ],
            "source": "print(data.get('number'))"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Print the names of the astronauts currently on ISS.\n"
        },
        {
            "cell_type": "code",
            "execution_count": 8,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "There are 7 astronauts on ISS\nAnd their names are :\nSergey Ryzhikov\nKate Rubins\nSergey Kud-Sverchkov\nMike Hopkins\nVictor Glover\nShannon Walker\nSoichi Noguchi\n"
                }
            ],
            "source": "astronauts = data.get('people')\nprint(\"There are {} astronauts on ISS\".format(len(astronauts)))\nprint(\"And their names are :\")\nfor astronaut in astronauts:\n    print(astronaut.get('name'))\n    "
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Hope the warmup was helpful. Good luck with your next lab!\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "## Lab: Collect Jobs Data using GitHub Jobs API\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Before you start doing this lab, get familier with the GitHub Jobs API.<br>\nThe documentation for the GitHub Jobs API can be found at <https://jobs.github.com/api><br>\n\n<li>Understand what urls to use.<br>\n<li>Understand what parameters have to be passed.<br>\n<li>Understand the format of the output data.</li>\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "### Objective: Determine the number of jobs currently open for various technologies\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Collect the number of job postings for the following languages using the API:\n\n-   C\n-   C#\n-   C++\n-   Java\n-   JavaScript\n-   Python\n-   Scala\n-   Oracle\n-   SQL Server\n-   MySQL Server\n-   PostgreSQL\n-   MongoDB\n"
        },
        {
            "cell_type": "code",
            "execution_count": 9,
            "metadata": {},
            "outputs": [],
            "source": "#Import required libraries\nimport requests"
        },
        {
            "cell_type": "code",
            "execution_count": 10,
            "metadata": {},
            "outputs": [],
            "source": "baseurl = \"https://jobs.github.com/positions.json\""
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Write a function to get the number of jobs for the given technology.<br>\n_Note:_ The API gives a maximum of 50 jobs per page.<br>\nIf you get 50 jobs per page, it means there could be some more job listings available.<br>\nSo if you get 50 jobs per page you should make another API call for next page to check for more jobs.<br>\nIf you get less than 50 jobs per page, you can take it as the final count.<br>\n"
        },
        {
            "cell_type": "code",
            "execution_count": 11,
            "metadata": {},
            "outputs": [],
            "source": "def get_number_of_jobs(technology):\n    number_of_jobs = 0\n    #your code goes here\n    return technology,number_of_jobs"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Call the function for Python and check if it is working.\n"
        },
        {
            "cell_type": "code",
            "execution_count": 12,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "('python', 0)\n"
                }
            ],
            "source": "print(get_number_of_jobs('python'))"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "### Store the results in an excel file\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Call the API for all the given technologies above and write the results in an excel spreadsheet.\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "If you do not know how create excel file using python, double click here for **hints**.\n\n<!--\n\nfrom openpyxl import Workbook        # import Workbook class from module openpyxl\nwb=Workbook()                        # create a workbook object\nws=wb.active                         # use the active worksheet\nws.append(['Country','Continent'])   # add a row with two columns 'Country' and 'Continent'\nws.append(['Eygpt','Africa'])        # add a row with two columns 'Egypt' and 'Africa'\nws.append(['India','Asia'])          # add another row\nws.append(['France','Europe'])       # add another row\nwb.save(\"countries.xlsx\")            # save the workbook into a file called countries.xlsx\n\n\n-->\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Create a python list of all technologies for which you need to find the number of jobs postings.\n"
        },
        {
            "cell_type": "code",
            "execution_count": 13,
            "metadata": {},
            "outputs": [],
            "source": "#your code goes here\nimport numpy as np\nimport pandas as pd\nimport requests"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Import libraries required to create excel spreadsheet\n"
        },
        {
            "cell_type": "code",
            "execution_count": 14,
            "metadata": {},
            "outputs": [
                {
                    "ename": "ModuleNotFoundError",
                    "evalue": "No module named 'openpyxl'",
                    "output_type": "error",
                    "traceback": [
                        "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
                        "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
                        "\u001b[0;32m<ipython-input-14-fcd964cfa1e7>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# your code goes here\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mopenpyxl\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mWorkbook\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
                        "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'openpyxl'"
                    ]
                }
            ],
            "source": "# your code goes here\nfrom openpyxl import Workbook "
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Create a workbook and select the active worksheet\n"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "# your code goes here\nws=wb.active"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Find the number of jobs postings for each of the technology in the above list.\nWrite the technology name and the number of jobs postings into the excel spreadsheet.\n"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "#your code goes here\nws.append(['Technology','Number_of_jobs'])   # add a row with two columns 'Country' and 'Continent'\nws.append(['C','{}']) \nws.append(['C#','{}'])        # add a row with two columns 'Egypt' and 'Africa'\nws.append(['C++','{}'])          # add another row\nws.append(['Java','{}'])       # add another row\nws.append(['JavaScript','{}']) \nws.append(['Python','{}']) \nws.append(['Oracle','{}']) \nws.append(['SQL Server','{}']) \nws.append(['MySQL Server','{}']) \nws.append(['PostgreSQL','{}']) \nws.append(['MongoDB','{}']) "
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Save into an excel spreadsheet named 'github-job-postings.xlsx'.\n"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "#your code goes here\nwb.save(\"countries.xlsx\")\n\n\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "## Authors\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Ramesh Sannareddy\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "### Other Contributors\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Rav Ahuja\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "## Change Log\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "| Date (YYYY-MM-DD) | Version | Changed By        | Change Description                 |\n| ----------------- | ------- | ----------------- | ---------------------------------- |\n| 2020-10-17        | 0.1     | Ramesh Sannareddy | Created initial version of the lab |\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": " Copyright \u00a9 2020 IBM Corporation. This notebook and its source code are released under the terms of the [MIT License](https://cognitiveclass.ai/mit-license?cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ).\n"
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.7",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.7.10"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 4
}