Skip to content

Instantly share code, notes, and snippets.

@coryjog

coryjog/import_wendy_data.ipynb

Created May 17, 2018 20:34
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save coryjog/5fd93c1e4bf1e080014a3d1dc87fd22b to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save coryjog/5fd93c1e4bf1e080014a3d1dc87fd22b to your computer and use it in GitHub Desktop.
Download ZIP
Tutorial to import Windographer .csv file into a notebook
Display the source blob
Display the rendered blob
Raw
import_wendy_data.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load data from Windographer\n",
"Example code to import a Windographer .txt file into a Jupyter notebook.\n",
"\n",
"Pandas has a very powerful, built-in .csv reader. The cells below calls pd.read_csv and stores the data in a variable called mast_data. The following key-word arguments (kwargs) are passed to the [read_csv method](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html):\n",
"\n",
"* filename - name of the text file to read (comma-delimited assumed)\n",
"* index_col - column number to use as the DataFrame index along the left side\n",
"* header - row number to use as the header row which is then translated to the column labels\n",
"* parse_dates - automatically translate something that looks like a date into a date object\n",
"* infer_datetime_format - automatically determine the order of the year, month, day, hour, etc.\n",
"* na_values - flag used to denote invalid data\n",
"* encoding - this forces Python to read those pesky degrees temperature symbols"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Pandas version: 0.22.0\n"
]
}
],
"source": [
"import pandas as pd\n",
"print('Pandas version: {}'.format(pd.__version__))"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>SPD_60_W_AVG [m/s]</th>\n",
" <th>SPD_60_W_SD [m/s]</th>\n",
" <th>SPD_60_W_MAX [m/s]</th>\n",
" <th>SPD_60_E_AVG [m/s]</th>\n",
" <th>SPD_60_E_SD [m/s]</th>\n",
" <th>SPD_60_E_MAX [m/s]</th>\n",
" <th>SPD_40_W_AVG [m/s]</th>\n",
" <th>SPD_40_W_SD [m/s]</th>\n",
" <th>SPD_40_E_AVG [m/s]</th>\n",
" <th>SPD_40_E_SD [m/s]</th>\n",
" <th>SPD_20_E_AVG [m/s]</th>\n",
" <th>SPD_20_E_SD [m/s]</th>\n",
" <th>SPD_20_W_AVG [m/s]</th>\n",
" <th>SPD_20_W_SD [m/s]</th>\n",
" <th>DIR_59_AVG [°]</th>\n",
" <th>DIR_59_SD [°]</th>\n",
" <th>DIR_39_AVG [°]</th>\n",
" <th>DIR_39_SD [°]</th>\n",
" <th>T_1_AVG [°C]</th>\n",
" <th>Unnamed: 20</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Date/Time</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2008-11-23 00:00:00</th>\n",
" <td>12.167</td>\n",
" <td>0.366</td>\n",
" <td>12.960</td>\n",
" <td>12.103</td>\n",
" <td>0.382</td>\n",
" <td>12.960</td>\n",
" <td>10.601</td>\n",
" <td>0.417</td>\n",
" <td>10.776</td>\n",
" <td>0.399</td>\n",
" <td>8.493</td>\n",
" <td>0.590</td>\n",
" <td>8.351</td>\n",
" <td>0.590</td>\n",
" <td>210.0</td>\n",
" <td>1.5</td>\n",
" <td>209.8</td>\n",
" <td>2.0</td>\n",
" <td>9.3</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2008-11-23 00:10:00</th>\n",
" <td>12.475</td>\n",
" <td>0.322</td>\n",
" <td>13.341</td>\n",
" <td>12.393</td>\n",
" <td>0.322</td>\n",
" <td>13.341</td>\n",
" <td>10.889</td>\n",
" <td>0.399</td>\n",
" <td>11.098</td>\n",
" <td>0.399</td>\n",
" <td>8.745</td>\n",
" <td>0.565</td>\n",
" <td>8.604</td>\n",
" <td>0.590</td>\n",
" <td>208.6</td>\n",
" <td>1.7</td>\n",
" <td>208.4</td>\n",
" <td>2.5</td>\n",
" <td>9.7</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2008-11-23 00:20:00</th>\n",
" <td>12.328</td>\n",
" <td>0.259</td>\n",
" <td>12.960</td>\n",
" <td>12.255</td>\n",
" <td>0.295</td>\n",
" <td>12.960</td>\n",
" <td>10.525</td>\n",
" <td>0.475</td>\n",
" <td>10.734</td>\n",
" <td>0.475</td>\n",
" <td>8.644</td>\n",
" <td>0.643</td>\n",
" <td>8.505</td>\n",
" <td>0.616</td>\n",
" <td>209.8</td>\n",
" <td>1.4</td>\n",
" <td>209.4</td>\n",
" <td>2.7</td>\n",
" <td>9.9</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2008-11-23 00:30:00</th>\n",
" <td>12.368</td>\n",
" <td>0.248</td>\n",
" <td>12.960</td>\n",
" <td>12.279</td>\n",
" <td>0.259</td>\n",
" <td>12.960</td>\n",
" <td>10.636</td>\n",
" <td>0.417</td>\n",
" <td>10.840</td>\n",
" <td>0.436</td>\n",
" <td>8.527</td>\n",
" <td>0.565</td>\n",
" <td>8.368</td>\n",
" <td>0.590</td>\n",
" <td>209.4</td>\n",
" <td>1.3</td>\n",
" <td>208.6</td>\n",
" <td>2.8</td>\n",
" <td>9.8</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2008-11-23 00:40:00</th>\n",
" <td>12.111</td>\n",
" <td>0.271</td>\n",
" <td>12.590</td>\n",
" <td>12.047</td>\n",
" <td>0.259</td>\n",
" <td>12.590</td>\n",
" <td>10.341</td>\n",
" <td>0.475</td>\n",
" <td>10.559</td>\n",
" <td>0.455</td>\n",
" <td>8.400</td>\n",
" <td>0.590</td>\n",
" <td>8.260</td>\n",
" <td>0.590</td>\n",
" <td>209.7</td>\n",
" <td>1.3</td>\n",
" <td>208.6</td>\n",
" <td>2.8</td>\n",
" <td>9.5</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" SPD_60_W_AVG [m/s] SPD_60_W_SD [m/s] \\\n",
"Date/Time \n",
"2008-11-23 00:00:00 12.167 0.366 \n",
"2008-11-23 00:10:00 12.475 0.322 \n",
"2008-11-23 00:20:00 12.328 0.259 \n",
"2008-11-23 00:30:00 12.368 0.248 \n",
"2008-11-23 00:40:00 12.111 0.271 \n",
"\n",
" SPD_60_W_MAX [m/s] SPD_60_E_AVG [m/s] \\\n",
"Date/Time \n",
"2008-11-23 00:00:00 12.960 12.103 \n",
"2008-11-23 00:10:00 13.341 12.393 \n",
"2008-11-23 00:20:00 12.960 12.255 \n",
"2008-11-23 00:30:00 12.960 12.279 \n",
"2008-11-23 00:40:00 12.590 12.047 \n",
"\n",
" SPD_60_E_SD [m/s] SPD_60_E_MAX [m/s] \\\n",
"Date/Time \n",
"2008-11-23 00:00:00 0.382 12.960 \n",
"2008-11-23 00:10:00 0.322 13.341 \n",
"2008-11-23 00:20:00 0.295 12.960 \n",
"2008-11-23 00:30:00 0.259 12.960 \n",
"2008-11-23 00:40:00 0.259 12.590 \n",
"\n",
" SPD_40_W_AVG [m/s] SPD_40_W_SD [m/s] \\\n",
"Date/Time \n",
"2008-11-23 00:00:00 10.601 0.417 \n",
"2008-11-23 00:10:00 10.889 0.399 \n",
"2008-11-23 00:20:00 10.525 0.475 \n",
"2008-11-23 00:30:00 10.636 0.417 \n",
"2008-11-23 00:40:00 10.341 0.475 \n",
"\n",
" SPD_40_E_AVG [m/s] SPD_40_E_SD [m/s] \\\n",
"Date/Time \n",
"2008-11-23 00:00:00 10.776 0.399 \n",
"2008-11-23 00:10:00 11.098 0.399 \n",
"2008-11-23 00:20:00 10.734 0.475 \n",
"2008-11-23 00:30:00 10.840 0.436 \n",
"2008-11-23 00:40:00 10.559 0.455 \n",
"\n",
" SPD_20_E_AVG [m/s] SPD_20_E_SD [m/s] \\\n",
"Date/Time \n",
"2008-11-23 00:00:00 8.493 0.590 \n",
"2008-11-23 00:10:00 8.745 0.565 \n",
"2008-11-23 00:20:00 8.644 0.643 \n",
"2008-11-23 00:30:00 8.527 0.565 \n",
"2008-11-23 00:40:00 8.400 0.590 \n",
"\n",
" SPD_20_W_AVG [m/s] SPD_20_W_SD [m/s] DIR_59_AVG [°] \\\n",
"Date/Time \n",
"2008-11-23 00:00:00 8.351 0.590 210.0 \n",
"2008-11-23 00:10:00 8.604 0.590 208.6 \n",
"2008-11-23 00:20:00 8.505 0.616 209.8 \n",
"2008-11-23 00:30:00 8.368 0.590 209.4 \n",
"2008-11-23 00:40:00 8.260 0.590 209.7 \n",
"\n",
" DIR_59_SD [°] DIR_39_AVG [°] DIR_39_SD [°] \\\n",
"Date/Time \n",
"2008-11-23 00:00:00 1.5 209.8 2.0 \n",
"2008-11-23 00:10:00 1.7 208.4 2.5 \n",
"2008-11-23 00:20:00 1.4 209.4 2.7 \n",
"2008-11-23 00:30:00 1.3 208.6 2.8 \n",
"2008-11-23 00:40:00 1.3 208.6 2.8 \n",
"\n",
" T_1_AVG [°C] Unnamed: 20 \n",
"Date/Time \n",
"2008-11-23 00:00:00 9.3 NaN \n",
"2008-11-23 00:10:00 9.7 NaN \n",
"2008-11-23 00:20:00 9.9 NaN \n",
"2008-11-23 00:30:00 9.8 NaN \n",
"2008-11-23 00:40:00 9.5 NaN "
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"filename = 'example_mast_data.txt'\n",
"mast_data = pd.read_csv(filename, \n",
" index_col=0, \n",
" header=8, \n",
" parse_dates=True, \n",
" infer_datetime_format=True,\n",
" na_values=-999,\n",
" encoding='iso-8859-1')\n",
"mast_data.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Assuming you don't have any spaces in your sensor names in Windographer you can trim the column labels using the default space between the sensor name and the units. This gives a much cleaner data table and allows the ability to slice the data by the sensor name. "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>sensor</th>\n",
" <th>SPD_60_W_AVG</th>\n",
" <th>SPD_60_W_SD</th>\n",
" <th>SPD_60_W_MAX</th>\n",
" <th>SPD_60_E_AVG</th>\n",
" <th>SPD_60_E_SD</th>\n",
" <th>SPD_60_E_MAX</th>\n",
" <th>SPD_40_W_AVG</th>\n",
" <th>SPD_40_W_SD</th>\n",
" <th>SPD_40_E_AVG</th>\n",
" <th>SPD_40_E_SD</th>\n",
" <th>SPD_20_E_AVG</th>\n",
" <th>SPD_20_E_SD</th>\n",
" <th>SPD_20_W_AVG</th>\n",
" <th>SPD_20_W_SD</th>\n",
" <th>DIR_59_AVG</th>\n",
" <th>DIR_59_SD</th>\n",
" <th>DIR_39_AVG</th>\n",
" <th>DIR_39_SD</th>\n",
" <th>T_1_AVG</th>\n",
" <th>Unnamed:</th>\n",
" </tr>\n",
" <tr>\n",
" <th>stamp</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2008-11-23 00:00:00</th>\n",
" <td>12.167</td>\n",
" <td>0.366</td>\n",
" <td>12.960</td>\n",
" <td>12.103</td>\n",
" <td>0.382</td>\n",
" <td>12.960</td>\n",
" <td>10.601</td>\n",
" <td>0.417</td>\n",
" <td>10.776</td>\n",
" <td>0.399</td>\n",
" <td>8.493</td>\n",
" <td>0.590</td>\n",
" <td>8.351</td>\n",
" <td>0.590</td>\n",
" <td>210.0</td>\n",
" <td>1.5</td>\n",
" <td>209.8</td>\n",
" <td>2.0</td>\n",
" <td>9.3</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2008-11-23 00:10:00</th>\n",
" <td>12.475</td>\n",
" <td>0.322</td>\n",
" <td>13.341</td>\n",
" <td>12.393</td>\n",
" <td>0.322</td>\n",
" <td>13.341</td>\n",
" <td>10.889</td>\n",
" <td>0.399</td>\n",
" <td>11.098</td>\n",
" <td>0.399</td>\n",
" <td>8.745</td>\n",
" <td>0.565</td>\n",
" <td>8.604</td>\n",
" <td>0.590</td>\n",
" <td>208.6</td>\n",
" <td>1.7</td>\n",
" <td>208.4</td>\n",
" <td>2.5</td>\n",
" <td>9.7</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2008-11-23 00:20:00</th>\n",
" <td>12.328</td>\n",
" <td>0.259</td>\n",
" <td>12.960</td>\n",
" <td>12.255</td>\n",
" <td>0.295</td>\n",
" <td>12.960</td>\n",
" <td>10.525</td>\n",
" <td>0.475</td>\n",
" <td>10.734</td>\n",
" <td>0.475</td>\n",
" <td>8.644</td>\n",
" <td>0.643</td>\n",
" <td>8.505</td>\n",
" <td>0.616</td>\n",
" <td>209.8</td>\n",
" <td>1.4</td>\n",
" <td>209.4</td>\n",
" <td>2.7</td>\n",
" <td>9.9</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2008-11-23 00:30:00</th>\n",
" <td>12.368</td>\n",
" <td>0.248</td>\n",
" <td>12.960</td>\n",
" <td>12.279</td>\n",
" <td>0.259</td>\n",
" <td>12.960</td>\n",
" <td>10.636</td>\n",
" <td>0.417</td>\n",
" <td>10.840</td>\n",
" <td>0.436</td>\n",
" <td>8.527</td>\n",
" <td>0.565</td>\n",
" <td>8.368</td>\n",
" <td>0.590</td>\n",
" <td>209.4</td>\n",
" <td>1.3</td>\n",
" <td>208.6</td>\n",
" <td>2.8</td>\n",
" <td>9.8</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2008-11-23 00:40:00</th>\n",
" <td>12.111</td>\n",
" <td>0.271</td>\n",
" <td>12.590</td>\n",
" <td>12.047</td>\n",
" <td>0.259</td>\n",
" <td>12.590</td>\n",
" <td>10.341</td>\n",
" <td>0.475</td>\n",
" <td>10.559</td>\n",
" <td>0.455</td>\n",
" <td>8.400</td>\n",
" <td>0.590</td>\n",
" <td>8.260</td>\n",
" <td>0.590</td>\n",
" <td>209.7</td>\n",
" <td>1.3</td>\n",
" <td>208.6</td>\n",
" <td>2.8</td>\n",
" <td>9.5</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"sensor SPD_60_W_AVG SPD_60_W_SD SPD_60_W_MAX SPD_60_E_AVG \\\n",
"stamp \n",
"2008-11-23 00:00:00 12.167 0.366 12.960 12.103 \n",
"2008-11-23 00:10:00 12.475 0.322 13.341 12.393 \n",
"2008-11-23 00:20:00 12.328 0.259 12.960 12.255 \n",
"2008-11-23 00:30:00 12.368 0.248 12.960 12.279 \n",
"2008-11-23 00:40:00 12.111 0.271 12.590 12.047 \n",
"\n",
"sensor SPD_60_E_SD SPD_60_E_MAX SPD_40_W_AVG SPD_40_W_SD \\\n",
"stamp \n",
"2008-11-23 00:00:00 0.382 12.960 10.601 0.417 \n",
"2008-11-23 00:10:00 0.322 13.341 10.889 0.399 \n",
"2008-11-23 00:20:00 0.295 12.960 10.525 0.475 \n",
"2008-11-23 00:30:00 0.259 12.960 10.636 0.417 \n",
"2008-11-23 00:40:00 0.259 12.590 10.341 0.475 \n",
"\n",
"sensor SPD_40_E_AVG SPD_40_E_SD SPD_20_E_AVG SPD_20_E_SD \\\n",
"stamp \n",
"2008-11-23 00:00:00 10.776 0.399 8.493 0.590 \n",
"2008-11-23 00:10:00 11.098 0.399 8.745 0.565 \n",
"2008-11-23 00:20:00 10.734 0.475 8.644 0.643 \n",
"2008-11-23 00:30:00 10.840 0.436 8.527 0.565 \n",
"2008-11-23 00:40:00 10.559 0.455 8.400 0.590 \n",
"\n",
"sensor SPD_20_W_AVG SPD_20_W_SD DIR_59_AVG DIR_59_SD \\\n",
"stamp \n",
"2008-11-23 00:00:00 8.351 0.590 210.0 1.5 \n",
"2008-11-23 00:10:00 8.604 0.590 208.6 1.7 \n",
"2008-11-23 00:20:00 8.505 0.616 209.8 1.4 \n",
"2008-11-23 00:30:00 8.368 0.590 209.4 1.3 \n",
"2008-11-23 00:40:00 8.260 0.590 209.7 1.3 \n",
"\n",
"sensor DIR_39_AVG DIR_39_SD T_1_AVG Unnamed: \n",
"stamp \n",
"2008-11-23 00:00:00 209.8 2.0 9.3 NaN \n",
"2008-11-23 00:10:00 208.4 2.5 9.7 NaN \n",
"2008-11-23 00:20:00 209.4 2.7 9.9 NaN \n",
"2008-11-23 00:30:00 208.6 2.8 9.8 NaN \n",
"2008-11-23 00:40:00 208.6 2.8 9.5 NaN "
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mast_data.columns = mast_data.columns.str.split(' ', expand=True).get_level_values(0)\n",
"mast_data.columns.name = 'sensor'\n",
"mast_data.index.name = 'stamp'\n",
"mast_data.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.4"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment