DS4A course repository - tooling files

README.md

DS4A - Workspace

This repo was created to store all my productions and notes throughout the Data Science for All course.

Folders structure

ds4a/
└── week1/
    │   case_1.1_student/
    └── case_20.1_student/
    ...

Utilities

The setup_envs.sh

It was made to fasten up the environments installation. To use it just organize the cases per week folder, and run it as: ./setup_envs.sh weekX. Example: ./setup_envs.sh week1. This will make the script enter each case folder and execute conda env update osx_env.yml.

The open.sh

The open.sh script takes care of doing 3 simple annoying things for you:

• Looks into the osx_env.yml for the correct env name;
• Runs conda activate $CORRECT_ENV_NAME
• And finally: jupyter notebook in the indicated folder.

This one is a little bit more funky. It requires that you once put an alias on your ~/.bashrc or ~/.zshrc as following:

• alias jopen=". ./open.sh"

After doing so, don't forget to either re-open your shell or run a simple source ~/.zshrc or source ~/.bashrc.

The opening of a jupyter notebook with its environment boils down to a much simpler way:

 $ cd ds4a/
 $ jopen week1/case_1.1_student

Important: to use this alias, make sure that you're inside the ds4a repo. root.

functions.sh

#!/bin/sh
parse_yaml() {
   local prefix=$2
   local s='[[:space:]]*' w='[a-zA-Z0-9_]*' fs=$(echo @|tr @ '\034')
   sed -ne "s|^\($s\)\($w\)$s:$s\"\(.*\)\"$s\$|\1$fs\2$fs\3|p" \
        -e "s|^\($s\)\($w\)$s:$s\(.*\)$s\$|\1$fs\2$fs\3|p"  $1 |
   awk -F$fs '{
      indent = length($1)/2;
      vname[indent] = $2;
      for (i in vname) {if (i > indent) {delete vname[i]}}
      if (length($3) > 0) {
         vn=""; for (i=0; i<indent; i++) {vn=(vn)(vname[i])("_")}
         printf("%s%s%s=\"%s\"\n", "'$prefix'",vn, $2, $3);
      }
   }'
}

open.sh

#!/bin/sh

. ./functions.sh

ORIGINAL_PWD=(`pwd`)

cd $1

if [ ! -f osx_env.yml ]; then
  echo "Cant find osx_env.yaml file!"
  exit
fi

# Parser the osx_env file
eval $(parse_yaml osx_env.yml "ds4a_env_")
# Activate the right environment
conda activate $ds4a_env_name
# Open jupyter notebook app
jupyter notebook

cd $ORIGINAL_PWD

setup_envs.sh

#!/bin/sh
cd $1
for D in *; do
	if [ -d "${D}" ]; then
		echo "Now working at"
		echo "${D}"
		echo "..."
		cd ${D}
		conda env update --file osx_env.yml
		echo "All done!"
		echo ""
		cd ..
   	 fi
done