Created
September 12, 2023 15:02
-
-
Save GvdDool/3840487dbe658c42f2d5128000bbed9f to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "nbformat": 4, | |
| "nbformat_minor": 0, | |
| "metadata": { | |
| "colab": { | |
| "provenance": [], | |
| "authorship_tag": "ABX9TyMspErCXPsAG6yE99/E7Sz6", | |
| "include_colab_link": true | |
| }, | |
| "kernelspec": { | |
| "name": "python3", | |
| "display_name": "Python 3" | |
| }, | |
| "language_info": { | |
| "name": "python" | |
| } | |
| }, | |
| "cells": [ | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "view-in-github", | |
| "colab_type": "text" | |
| }, | |
| "source": [ | |
| "<a href=\"https://colab.research.google.com/github/GvdDool/Notebook-test/blob/master/SimplifyFileSharing.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "#Simplify File Sharing\n", | |
| "##Coding Example for working with Google Drive's Shared Folders in Collaboration Projects\n", | |
| "\n", | |
| "Recently, a data-sharing problem came up again, and I thought this would be a good time to design a method to work with shared folders. I am an independent GIScience professional, and I often have to work with different organisations simultaneously, and no organisation works with the data the same way because they all have their own culture and work ethics. Luckily, there are some practices they have in common, and one of them is working with a cloud-based data management system, often Google, but this could also be One-Drive (from Microsoft) of Dropbox.\n", | |
| "\n", | |
| "In this post, I will try to explain how I work, using Python, with Shared Folders in the Google Ecosystem." | |
| ], | |
| "metadata": { | |
| "id": "p8LjwdFEW5uQ" | |
| } | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "" | |
| ], | |
| "metadata": { | |
| "id": "4b58FePiXC9l" | |
| } | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "##Use Case\n", | |
| "File management is very individual when working on a local machine, and (hopefully) standardised, or at least has some standardisation, when working in an organisation. Working with shared folders is an option when you don't have direct access to a production folder, but the organisation can share a specifically designated working folder with you to exchange files. In this example, an organisation has granted access to a folder called DATA on their Google Drive repository, and it's agreed that we can use this folder to exchange files.\n", | |
| "\n", | |
| "##Local file management\n", | |
| "To explain quickly, for people unfamiliar with Google Drive file sharing, the process starts with receiving an email inviting you to contribute to a specific folder; see the invitation (left) below. In the invitation is a button which will open a web browser with the Google Drive interface (right) associated with the Google email of the receiver of the invitation." | |
| ], | |
| "metadata": { | |
| "id": "a2S6mcpRZyPQ" | |
| } | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "" | |
| ], | |
| "metadata": { | |
| "id": "-QroJWjXZ2gM" | |
| } | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "There are a few important pieces of information hidden in the interface, and understanding them early will help with the rest of the process.\n", | |
| "- In the URL (top of the screen), there is a masked ID, this is the ID Google is using to track all the operations on this folder, and this is the ID we will fetch in the Python code later in this post.\n", | |
| "- Then it says: \"Shared with me\" and the name of the shared folder; also, this is important because when we mount the Google Drive to the CoLab notebook, we will see that this category is not available.\n", | |
| "- And finally, we see the files and folders under Data; this means we can access the information we need, and add new files to the folder. Still, there could be a problem with the folder's security settings, so a good test at this stage is to create a small text file, and drag-and-drop in the folder \"ExternalData\", to verify that you have full access.\n", | |
| "\n", | |
| "To make the \"Shared with me\" folder accessible, we need to link this folder to the local/personal drive. We can do this by creating a Shortcut, but this is a manual step, and will be different for everyone. To access a shared with you folder or file in Google Colab, you have to:\n", | |
| "1. Go to Shared with me in Google Drive.\n", | |
| "2. Select the folder or file you want to access.\n", | |
| "3. Right-click on it and choose \"Add shortcut to drive\", and a pop-up window will appear, \"Select MyDrive\", then click on \"Add Shortcut\".\n", | |
| "4. Place the shortcut in a location on your drive that you can find back easily; in the setup I use, the location for Shortcuts is \"__Shared\", ensuring that the folder with the shortcuts is at the top of the folder list under \"MyDrive\", and then a subdirectory for the organisation.\n", | |
| "5. Rename the shortcut to a meaningful name; I use \"DataDevelopement\" in this example. The file location and name conventions are very personal, and it doesn't matter for the program where the files are stored or how they are called, but having some structure can save some headaches later." | |
| ], | |
| "metadata": { | |
| "id": "CocsNHrIaDxm" | |
| } | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "" | |
| ], | |
| "metadata": { | |
| "id": "9UMxX69raD08" | |
| } | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "With the local file system organised, and the personal Google Drive configured, we can try to work with this shared folder in a Python notebook and automate the file sharing in the project.\n", | |
| "\n", | |
| "<br>\n", | |
| "\n", | |
| "\n", | |
| "---\n", | |
| "\n", | |
| "\n", | |
| "#Installation\n", | |
| "This project is based on a Google Colab, or \"Collaboratory\", notebook, and the advantage of using this environment is that it allows you to write and execute Python in your browser, with\n", | |
| "- Zero configuration required\n", | |
| "- Access to GPUs free of charge\n", | |
| "- Easy sharing\n", | |
| "\n", | |
| "These are very important points when working with organisations which have their internal structures, but as an external collaborator, you are not often getting direct access to the code base (and this can have many different reasons, from security concerns to project management constraints); I will share a gist to the notebook at the end of this post.\n", | |
| "Importing modules and packages\n", | |
| "For this example, only the necessary packages are loaded to the notebook's runtime, and we need a few specific libraries to work with the shared drive.\n", | |
| "\n", | |
| "##Google autotorisation\n", | |
| "\n" | |
| ], | |
| "metadata": { | |
| "id": "nyOcqCKkaD4T" | |
| } | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 2, | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/" | |
| }, | |
| "id": "g37f5WuoWwlO", | |
| "outputId": "0528d706-d059-4e4b-c2ed-40fba6f492e3" | |
| }, | |
| "outputs": [ | |
| { | |
| "output_type": "stream", | |
| "name": "stdout", | |
| "text": [ | |
| "Mounted at /content/gdrive\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "from oauth2client.client import GoogleCredentials\n", | |
| "\n", | |
| "from google.colab import auth as google_auth\n", | |
| "google_auth.authenticate_user()\n", | |
| "\n", | |
| "from google.colab import drive\n", | |
| "drive.mount('/content/gdrive')" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "Using oauth2client and the Google Credentials will make working with the files easier. There are alternatives, like downloading a JSON file with the credentials, and there will be situations where working with the JSON file will be preferred over using Google Credentials, but as this is a project with no sensitive data using the oauth2client library is giving enough protection.\n", | |
| "\n", | |
| "##pydrive" | |
| ], | |
| "metadata": { | |
| "id": "xUg-orVjaD7Q" | |
| } | |
| }, | |
| { | |
| "cell_type": "code", | |
| "source": [ | |
| "from pydrive.auth import GoogleAuth\n", | |
| "from pydrive.drive import GoogleDrive\n", | |
| "\n", | |
| "gauth = GoogleAuth()\n", | |
| "gauth.credentials = GoogleCredentials.get_application_default()\n", | |
| "drive = GoogleDrive(gauth)" | |
| ], | |
| "metadata": { | |
| "id": "rnKLhEyObUE0" | |
| }, | |
| "execution_count": 3, | |
| "outputs": [] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "pydrive is a wrapper library of google-api-python-client that simplifies many common Google Drive API tasks, and one of these functionalities is handling get responses when querying the Google Drive File system. Google Drive stores all objects by ID, and the IDs are linked by relational information in the objects. It is possible to access this information by the API (see the next code block), but the wrapper is doing all the heavy lifting for us when we create a GoogleDriveFileList instance with the parameters of Files.list() as dict. Calling GetList() will get all files that match your query as a list of GoogleDriveFile.\n", | |
| "\n", | |
| "##Google API Client" | |
| ], | |
| "metadata": { | |
| "id": "31EQjaUAZ2vj" | |
| } | |
| }, | |
| { | |
| "cell_type": "code", | |
| "source": [ | |
| "# Google API client:\n", | |
| "from googleapiclient.discovery import build\n", | |
| "\n", | |
| "# Initialize the Google Drive API client\n", | |
| "drive_service = build('drive', 'v3')" | |
| ], | |
| "metadata": { | |
| "id": "nvAZ1QyYbcL8" | |
| }, | |
| "execution_count": 4, | |
| "outputs": [] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "The Google API Client is a large library, and it has many functionalities, but for this project, we only need one module: build. The build module constructs a Resource Object for interacting with an API and returns the methods for interacting with the service. The pydrive library will handle the basic functions very well, like creating, updating, and deleting files, but there are a few moments (in this project) when we need more advanced functionality, and having access to the \"service\" allows us to extract information not captured by the pydrive methods.\n", | |
| "\n", | |
| "\n", | |
| "\n", | |
| "> This concludes the configuration of the notebook. In this example, we don't need more libaries than the loaded libaries for file management, and with the libraries loaded, we can have a look at what they are doing.\n", | |
| "\n", | |
| "<br>\n", | |
| "\n", | |
| "---\n", | |
| "\n", | |
| "##File Management in the Notebook\n", | |
| "Up to this point, a few things have happened:\n", | |
| "Google authorisation is set up,\n", | |
| "We created access to the drive (for read/write access), and\n", | |
| "The Pydrive package is available to navigate on the drive\n", | |
| "\n", | |
| "Hopefully, when you are following along and running the code, you will see the image on the right, after having refreshed the panel. You can see the Shortcut in the image as a folder under \"__Shared\", and we don't see the \"Shared with me\" section, but because we have the Shortcut, we don't need to see the \"Shared with me\" files.\n", | |
| "\n" | |
| ], | |
| "metadata": { | |
| "id": "6TEJtDjCbbXo" | |
| } | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "" | |
| ], | |
| "metadata": { | |
| "id": "500FgcAPbbhl" | |
| } | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "Google Drive works differently than the file management in the local Operating Systems, the physical location of the files is not important because the objects are managed by ID in an unstructured DataLake, and we can access the files and folders by the ID.\n", | |
| "Unfortunately, while os.path (in Python) has walk functions to go over the file system, a similar method doesn't exist for Google Drive (or I am not aware of this method). However, we can use the pydrive library, and walk manually through the folders in the directory tree, and luckily, we know where we want to go from the path of the folder. So, we don't need to walk through the whole structure, but we can use the folder names of the data path to go deeper into the folder tree.\n", | |
| "So, we loop over the small list (in this example, three items) to find the ID and use this ID to go to the next level. Note that the fourth level is commented out; we will get to this level in the second part of the file handling section of this notebook.\n", | |
| "\n", | |
| "The loop, in the code block below, starts in the root, and when it finds an item in the list, the loop will use the ID of the object to go to the next level on the list, and if an item is not found, the code will prompt that the folder is not found and will not look for any folder deeper in the structure. The loop concludes with either the ID of the Shortcut folder or prompts that the folder is not found." | |
| ], | |
| "metadata": { | |
| "id": "evQTSniPbboi" | |
| } | |
| }, | |
| { | |
| "cell_type": "code", | |
| "source": [ | |
| "# File handling testing:\n", | |
| "# There are in this example three folder levels:\n", | |
| "# /content/gdrive/MyDrive/__Shared/<your Project>/DataDevelopment\n", | |
| "\n", | |
| "your_Project = \"YourFolderName\"\n", | |
| "\n", | |
| "# Update these to your structure:\n", | |
| "folderList1 = [\"__Shared\", your_Project ,\"DataDevelopment\"] #, \"ExternalData\"]\n", | |
| "\n", | |
| "# Trying to copy the created dummy file:\n", | |
| "boo_foundFolder = False\n", | |
| "fileID = \"root\"\n", | |
| "level = 0\n", | |
| "\n", | |
| "# View all folders and file in your Google Drive\n", | |
| "# First loop over the list:\n", | |
| "print(\"File and Folder structure - check with IDs\")\n", | |
| "for folderName in folderList1:\n", | |
| " print(f\"Checking: {folderName}\")\n", | |
| "\n", | |
| " if boo_foundFolder or fileID == \"root\": #first run\n", | |
| " boo_foundFolder = False\n", | |
| "\n", | |
| " fileList = drive.ListFile({'q': f\"'{fileID}' in parents and trashed=false\"}).GetList()\n", | |
| " for file in fileList:\n", | |
| " # Testing the name:\n", | |
| " if(file['title'] == folderName):\n", | |
| " fileID = file['id']\n", | |
| " boo_foundFolder = True\n", | |
| "\n", | |
| " level += 1\n", | |
| " # end if\n", | |
| " # end for\n", | |
| "\n", | |
| " if boo_foundFolder == False:\n", | |
| " print(f\"folder not found\")\n", | |
| " break\n", | |
| " # end if\n", | |
| " # end if\n", | |
| "# end for\n", | |
| "\n", | |
| "print(f\"Did we find the folder: {boo_foundFolder}\")\n", | |
| "if boo_foundFolder:\n", | |
| " print(fileID)\n", | |
| " ShortCutID = fileID\n", | |
| "else:\n", | |
| " ShortCutID = 0" | |
| ], | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/" | |
| }, | |
| "id": "mH1oQE86gEdN", | |
| "outputId": "08654366-7702-47f6-be65-e79f7c87a9c7" | |
| }, | |
| "execution_count": 6, | |
| "outputs": [ | |
| { | |
| "output_type": "stream", | |
| "name": "stdout", | |
| "text": [ | |
| "File and Folder structure - check with IDs\n", | |
| "Checking: __Shared\n", | |
| "Checking: YourFolderName\n", | |
| "folder not found\n", | |
| "Did we find the folder: False\n" | |
| ] | |
| } | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "At this moment, we have the local file ID for the working folder, but before we can look for files in this location we need to match this local ID with the target ID of the shared folder. To find this information, we have to look deeper into the Google infrastructure, and to do this, we need a helper: the drive_service. We activated the helper while we were loading the project, and we didn't get a warning, which means we have access to the service by using the API, and requesting information by ID.\n", | |
| "The details we need are best collected through a simple function, like the findTargetID function in the next code block. In this function, the <fileID> is the Shortcut ID we found by looping over the names in the folders, and by calling 'drive_service.files().get' and specifying the fields, we get the target ID of the folder (this will be the same ID as in the URL of the web interface of Google Drive (see Figure 1)." | |
| ], | |
| "metadata": { | |
| "id": "43mCbt33gap4" | |
| } | |
| }, | |
| { | |
| "cell_type": "code", | |
| "source": [ | |
| "def findTargetID(fileID, drive_service):\n", | |
| " # The ID of the shared file you want to get ShortcutDetails from\n", | |
| " file_id = fileID\n", | |
| "\n", | |
| " try:\n", | |
| " # Get the file details\n", | |
| " file = drive_service.files().get(fileId=file_id,\n", | |
| " fields=\"id, shortcutDetails\").execute()\n", | |
| "\n", | |
| " # Check if the file is a shortcut\n", | |
| " if 'shortcutDetails' in file:\n", | |
| " shortcut_details = file['shortcutDetails']\n", | |
| " print(\"Shortcut Details:\")\n", | |
| " print(f\"Target ID: {shortcut_details['targetId']}\")\n", | |
| " print(f\"Target MIME Type: {shortcut_details['targetMimeType']}\")\n", | |
| " else:\n", | |
| " print(\"The file is not a shortcut.\")\n", | |
| " # end if\n", | |
| "\n", | |
| " except Exception as e:\n", | |
| " print(f\"An error occurred: {e}\")\n", | |
| "\n", | |
| " return shortcut_details['targetId']\n", | |
| "\n", | |
| "if boo_foundFolder:\n", | |
| " targetID = findTargetID(fileID, drive_service)\n", | |
| " print(targetID)\n", | |
| "else:\n", | |
| " print(\"Folder not found\")" | |
| ], | |
| "metadata": { | |
| "id": "B4rwI1VLgeLi" | |
| }, | |
| "execution_count": 7, | |
| "outputs": [] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "With this target ID, we have access to the actual shared folder on the Google Data Server, and we are not working on the shortcut folder anymore.\n", | |
| "\n", | |
| "\n", | |
| "> To recap, the reason we created the shortcut folder was to be able to see the folder in our mounted list of folders. The category \"Shared with me\" is not mounted, but the shortcuts are. So with this new ID we can look for files.\n", | |
| "\n", | |
| "\n", | |
| "---\n", | |
| "<br>\n", | |
| "\n", | |
| "##Looking for files\n", | |
| "We have now what we need, the Target ID of the folder shared with us at the beginning of the process, and with that ID, all normal file operations are available to us.\n", | |
| "\n", | |
| "We can verify that we have enough permissions on the shared folder by first creating a small text file in the Run Time environment; creating this file also confirms that we have access to the Run Time environment because it will appear in the left panel of the web interface of the CoLab notebook when the file is correctly created.\n", | |
| "\n" | |
| ], | |
| "metadata": { | |
| "id": "hoo1rMFQhIF0" | |
| } | |
| }, | |
| { | |
| "cell_type": "code", | |
| "source": [ | |
| "# Create a test file:\n", | |
| "with open('example.txt', 'w') as f:\n", | |
| " f.write('This is an example file, to test CoLab file sharing')\n", | |
| "# this file is now sitting in the runtime space of the notebook\n", | |
| "# (see left plane, under files)" | |
| ], | |
| "metadata": { | |
| "id": "SeqgPJqohbBz" | |
| }, | |
| "execution_count": null, | |
| "outputs": [] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "Now the idea is to move this file to the \"Shared with me\" folder \"Data\", which we renamed \"DataDevelopment\" in the Shortcut, but the function in the previous section provided the <target ID>, and we can use now this ID to check if the file we just created in the Run Time Environment is available on the shared drive.\n" | |
| ], | |
| "metadata": { | |
| "id": "NYEeNAh2hemX" | |
| } | |
| }, | |
| { | |
| "cell_type": "code", | |
| "source": [ | |
| "if boo_foundFolder:\n", | |
| " print(\"folder found\")\n", | |
| " folderID = targetID\n", | |
| "\n", | |
| " file_on_drive = False\n", | |
| " file_id = 0\n", | |
| "\n", | |
| " # check if the file is on the drive:\n", | |
| " fileList = drive.ListFile({'q': f\"'{folderID}' in parents and trashed=false\"}).GetList()\n", | |
| " for file in fileList:\n", | |
| " if(file['title'] == \"example.txt\"):\n", | |
| " file_on_drive = True\n", | |
| " fileID = file['id']\n", | |
| " # end if\n", | |
| " # end for\n", | |
| "\n", | |
| " if file_on_drive:\n", | |
| " #Overwrites the existing Google drive file.\"\"\"\n", | |
| " file1 = drive.CreateFile({'id': fileID})\n", | |
| " strFileHandling = \"Updated\"\n", | |
| "\n", | |
| " else:\n", | |
| " file1 = drive.CreateFile({\"mimeType\": \"text/csv\",\n", | |
| " \"parents\": [{\"kind\": \"drive#fileLink\",\n", | |
| " \"id\": folderID}]})\n", | |
| " strFileHandling = \"Created\"\n", | |
| " # end if\n", | |
| "\n", | |
| " # creating the binding to the file in the Run-Time environment:\n", | |
| " file1.SetContentFile(\"example.txt\")\n", | |
| "\n", | |
| " # copying the file to the Google Drive:\n", | |
| " file1.Upload()\n", | |
| "\n", | |
| " print(f'{strFileHandling} file %s with mimeType %s' % (file1['title'], file1['mimeType']))\n", | |
| "\n", | |
| "else:\n", | |
| " print(\"folder not found\")\n", | |
| "# end if" | |
| ], | |
| "metadata": { | |
| "id": "iZG_hHinhiuY" | |
| }, | |
| "execution_count": null, | |
| "outputs": [] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "Running the code above will either create a new file in the shared folder or update (overwrite) the file when the file is found.\n", | |
| "Creating a workspace\n", | |
| "\n", | |
| "There is a second reason to use the Shortcut ID to find the Target ID and that is to find items under the shared folder. As mentioned before, Google Drive manages everything by ID, and the Shortcut ID doesn't have any children, so using this ID to find new items will result in an empty list. This can be tested by including the \"ExternalData\" folder name in the first list of folders; the first list will not find this folder. However, restarting the loop with the Target ID will find this folder.\n", | |
| "\n", | |
| "In the code snippet below, a new folder list is created, using the folder names below the \"Shared with me\" folder name. The \"ExternalData\" folder is available (see Figure 1), but \"NewDataFolder\" is not yet created." | |
| ], | |
| "metadata": { | |
| "id": "c6myhyEWhjdE" | |
| } | |
| }, | |
| { | |
| "cell_type": "code", | |
| "source": [ | |
| "# Update these to your structure:\n", | |
| "# ... DataDevelopment/ExternalData/__CoLab_Notebook\n", | |
| "\n", | |
| "# Setting the Working Folder:\n", | |
| "folderList2 = [\"ExternalData\", \"NewDataFolder\"]" | |
| ], | |
| "metadata": { | |
| "id": "AzYCUaFShqHh" | |
| }, | |
| "execution_count": null, | |
| "outputs": [] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "We can use the same loop structure as before, but now instead of starting in the ROOT, we are starting with the Target ID, and the loop will find the \"ExternalData\" folder, but not the new data folder.\n", | |
| "As the working folder doesn't exist yet we can use <drive_service.files> to create this new folder, and with the same method all files that need to be transferred from the Run Time Environment to the \"Shared with me\" folder." | |
| ], | |
| "metadata": { | |
| "id": "NYQfi7_khsbE" | |
| } | |
| }, | |
| { | |
| "cell_type": "code", | |
| "source": [ | |
| "print(\"File and Folder structure - check with TARGET IDs\")\n", | |
| "boo_foundFolder = False\n", | |
| "fileID = targetID\n", | |
| "\n", | |
| "for folderName in folderList2:\n", | |
| " print(f\"Checking: {folderName}\")\n", | |
| "\n", | |
| " if boo_foundFolder or fileID == targetID: #first run\n", | |
| " boo_foundFolder = False\n", | |
| "\n", | |
| " fileList = drive.ListFile({'q': f\"'{fileID}' in parents and trashed=false\"}).GetList()\n", | |
| "\n", | |
| " for file in fileList:\n", | |
| " # Testing the name:\n", | |
| " if(file['title'] == folderName):\n", | |
| " #print('Title: %s, ID: %s' % (file['title'], file['id']))\n", | |
| " fileID = file['id']\n", | |
| " print(f\"{level}: {fileID}\")\n", | |
| "\n", | |
| " boo_foundFolder = True\n", | |
| "\n", | |
| " level += 1\n", | |
| " # end if\n", | |
| " # end for\n", | |
| "\n", | |
| " if boo_foundFolder == False:\n", | |
| " print(f\"folder not found\")\n", | |
| " break\n", | |
| " # end if\n", | |
| " # end if\n", | |
| "# end for\n", | |
| "\n", | |
| "print(f\"Did we find the folder: {boo_foundFolder}\")\n", | |
| "if boo_foundFolder:\n", | |
| " WorkingFolderID = fileID\n", | |
| "else:\n", | |
| " WorkingFolderID = 0" | |
| ], | |
| "metadata": { | |
| "id": "oeuBL3zOi8ga" | |
| }, | |
| "execution_count": null, | |
| "outputs": [] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "source": [ | |
| "def create_folder_in_folder(folder_name,parent_folder_id, drive_service):\n", | |
| "\n", | |
| " file_metadata = {\n", | |
| " 'name' : folder_name,\n", | |
| " 'parents' : [parent_folder_id],\n", | |
| " 'mimeType' : 'application/vnd.google-apps.folder'\n", | |
| " }\n", | |
| "\n", | |
| " file = drive_service.files().create(body=file_metadata, supportsAllDrives=True,\n", | |
| " fields='id').execute()\n", | |
| "\n", | |
| " print ('Folder ID: %s' % file.get('id'))\n", | |
| "\n", | |
| "if WorkingFolderID == 0:\n", | |
| " # fileID is the parent ID from the previous search\n", | |
| " create_folder_in_folder(\"NewDataFolder\", fileID, drive_service)" | |
| ], | |
| "metadata": { | |
| "id": "4L0WiTlJhvlI" | |
| }, | |
| "execution_count": null, | |
| "outputs": [] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "---\n", | |
| "\n", | |
| "\n", | |
| "> Main takeaways: Google Drive File System is ID-driven, and all objects have IDs. The \"Shared with me\" objects are not available in Google Colab, but through a \"Shortcut\", we can access them, and by finding the associated Target ID, we can work directly on the \"Shared with me\" folder, including the objects under the folder initially shared with us.\n", | |
| "\n", | |
| "\n", | |
| "---\n", | |
| "\n", | |
| "#Conclusions\n", | |
| "In this article, we have covered some of the essential aspects of working with shared folders, including:\n", | |
| "- Setting Up Local File Management: We started the process with the reception of an invitation to contribute to a designated Google Drive directory and showed how to structure your local file system to enhance collaborative efficiency.\n", | |
| "- Configuring Google Colab for Collaboration: We discussed the advantages of using Google Colab, a collaborative Python environment, and how to set it up for project collaboration.\n", | |
| "- Importing Necessary Modules and Packages: We provided code examples for importing essential modules and packages, including Google authorization, pydrive for simplifying Google Drive API tasks, and the Google API client for advanced functionality.\n", | |
| "- File Management in the Notebook: You saw how to manage files within the Google Colab environment, including creating and moving files between your local environment and shared folders.\n", | |
| "- Finding Files and Creating Workspaces: We delved into the process of finding files within shared folders using target IDs and creating new folders and workspaces for your projects.\n", | |
| "\n", | |
| "I hope this walkthrough on working with shared folders and files between organizations was helpful and that it gave some insights into how to work with files and folders in a shared folder.\n", | |
| "\n", | |
| "Thank you for reading, and I hope this post has helped you to solve a problem or gave you an idea for the next project.\n", | |
| "\n", | |
| "Google CoLab NoteBook link: https://gist.github.com/GvdDool/ WIP\n", | |
| "\n", | |
| "Disclaimer: The code used in this example is not optimised, but written to illustrate the process (any suggestions on how to improve the code are welcome on the gitHub pages which host this notebook." | |
| ], | |
| "metadata": { | |
| "id": "hcKoRf3yiOal" | |
| } | |
| } | |
| ] | |
| } |
Author
Thanks @MarkFarmiloe, I updated the medium text with some extra information around this step.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In the Medium article the 'print("File and Folder structure - check with TARGET IDs")' code block is missing, which added a bit of confusion :-)