Skip to content

Instantly share code, notes, and snippets.

@programus

programus/python-performance-tips-jp.ipynb

Last active April 13, 2020 14:21
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save programus/fce1bc1f34291d2e455f2d03893cbc32 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save programus/fce1bc1f34291d2e455f2d03893cbc32 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
python-performance-tips-jp.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# PythonのパフォーマンスについてのTIPS"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:14:18.487110Z",
"start_time": "2020-04-13T14:14:18.473767Z"
}
},
"outputs": [],
"source": [
"# just some preparation\n",
"from IPython.core import page\n",
"page.page = print"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 実行時間の測定\n",
"パフォーマンス改善するにまず最初のステップはボトルネックを掴むこと。\n",
"\n",
"人の感覚は信用できないので、感覚的ではなく、実際の実行時間を測定するのは一番重要なことである。\n",
"\n",
"### 1. Pythonコードの実行時間を測定する方法"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:14:23.785805Z",
"start_time": "2020-04-13T14:14:18.518423Z"
},
"_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
"_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 4.22 s, sys: 281 ms, total: 4.5 s\n",
"Wall time: 5.26 s\n"
]
}
],
"source": [
"%%time\n",
"heavy_job = [x ** 3 for x in range(9999999)]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2. Pythonコードの平均実行時間を測定する方法"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:14:24.154768Z",
"start_time": "2020-04-13T14:14:23.790703Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"0.34769509399999965"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# without jupyter notebook\n",
"import timeit\n",
"\n",
"def factorial(n):\n",
" return n * factorial(n - 1) if n > 0 else 1\n",
"\n",
"timeit.timeit('sum(map(factorial, range(500)))', globals = globals(), number = 10)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:14:27.249983Z",
"start_time": "2020-04-13T14:14:24.164705Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"41.2 ms ± 10.6 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
]
}
],
"source": [
"%%timeit -n 10\n",
"\n",
"def factorial(n):\n",
" return n * factorial(n - 1) if n > 0 else 1\n",
"\n",
"sum(map(factorial, range(500)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 3. ステップ毎の実行時間測定"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:14:27.261265Z",
"start_time": "2020-04-13T14:14:27.252388Z"
}
},
"outputs": [],
"source": [
"def relu(x):\n",
" return x if x > 0 else 0\n",
"\n",
"def test():\n",
" result = [relu(x) for x in range(200000)]\n",
" return result"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:14:29.769716Z",
"start_time": "2020-04-13T14:14:27.263935Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" 200006 function calls in 0.644 seconds\n",
"\n",
" Ordered by: standard name\n",
"\n",
" ncalls tottime percall cumtime percall filename:lineno(function)\n",
" 1 0.000 0.000 0.644 0.644 :0(exec)\n",
" 1 0.000 0.000 0.000 0.000 :0(setprofile)\n",
" 200000 0.301 0.000 0.301 0.000 <ipython-input-5-1f90dd81715b>:1(relu)\n",
" 1 0.000 0.000 0.642 0.642 <ipython-input-5-1f90dd81715b>:4(test)\n",
" 1 0.341 0.341 0.642 0.642 <ipython-input-5-1f90dd81715b>:5(<listcomp>)\n",
" 1 0.003 0.003 0.644 0.644 <string>:1(<module>)\n",
" 0 0.000 0.000 profile:0(profiler)\n",
" 1 0.000 0.000 0.644 0.644 profile:0(test())\n",
"\n",
"\n"
]
}
],
"source": [
"# without jupyter notebook\n",
"import profile\n",
"profile.run('test()')"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:14:29.851183Z",
"start_time": "2020-04-13T14:14:29.775567Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" 200005 function calls in 0.068 seconds\n",
"\n",
" Ordered by: internal time\n",
"\n",
" ncalls tottime percall cumtime percall filename:lineno(function)\n",
" 1 0.044 0.044 0.066 0.066 <ipython-input-5-1f90dd81715b>:5(<listcomp>)\n",
" 200000 0.022 0.000 0.022 0.000 <ipython-input-5-1f90dd81715b>:1(relu)\n",
" 1 0.002 0.002 0.068 0.068 <string>:1(<module>)\n",
" 1 0.000 0.000 0.068 0.068 {built-in method builtins.exec}\n",
" 1 0.000 0.000 0.066 0.066 <ipython-input-5-1f90dd81715b>:4(test)\n",
" 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}\n",
" "
]
}
],
"source": [
"%prun test()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4. 行毎の実行時間測定"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:14:29.876325Z",
"start_time": "2020-04-13T14:14:29.857556Z"
}
},
"outputs": [],
"source": [
"# !pip install line_profiler\n",
"%load_ext line_profiler"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:14:30.155742Z",
"start_time": "2020-04-13T14:14:29.884883Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"<line_profiler.line_profiler.LineProfiler at 0x1325b0440>"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Timer unit: 1e-06 s\n",
"\n",
"Total time: 0.076197 s\n",
"File: <ipython-input-5-1f90dd81715b>\n",
"Function: relu at line 1\n",
"\n",
"Line # Hits Time Per Hit % Time Line Contents\n",
"==============================================================\n",
" 1 def relu(x):\n",
" 2 200000 76197.0 0.4 100.0 return x if x > 0 else 0\n",
"\n",
"Total time: 0.238026 s\n",
"File: <ipython-input-5-1f90dd81715b>\n",
"Function: test at line 4\n",
"\n",
"Line # Hits Time Per Hit % Time Line Contents\n",
"==============================================================\n",
" 4 def test():\n",
" 5 1 238025.0 238025.0 100.0 result = [relu(x) for x in range(200000)]\n",
" 6 1 1.0 1.0 0.0 return result\n",
"\n"
]
}
],
"source": [
"# without jupyter notebook\n",
"import line_profiler\n",
"\n",
"lprofile = line_profiler.LineProfiler(test, relu)\n",
"lprofile.run('test()')\n",
"lprofile.print_stats()"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:14:30.402235Z",
"start_time": "2020-04-13T14:14:30.158128Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Timer unit: 1e-06 s\n",
"\n",
"Total time: 0.075953 s\n",
"File: <ipython-input-5-1f90dd81715b>\n",
"Function: relu at line 1\n",
"\n",
"Line # Hits Time Per Hit % Time Line Contents\n",
"==============================================================\n",
" 1 def relu(x):\n",
" 2 200000 75953.0 0.4 100.0 return x if x > 0 else 0\n",
"\n",
"Total time: 0.23546 s\n",
"File: <ipython-input-5-1f90dd81715b>\n",
"Function: test at line 4\n",
"\n",
"Line # Hits Time Per Hit % Time Line Contents\n",
"==============================================================\n",
" 4 def test():\n",
" 5 1 235460.0 235460.0 100.0 result = [relu(x) for x in range(200000)]\n",
" 6 1 0.0 0.0 0.0 return result\n"
]
}
],
"source": [
"%lprun -f test -f relu test()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 検索を早くする\n",
"多量のデータから必要なデータを探し出すのはよくある操作であり、データの量が多いとき時間がかかる操作でもある。\n",
"\n",
"### 5. `list` より `set`\n",
"データの準備"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:14:56.321850Z",
"start_time": "2020-04-13T14:14:30.404680Z"
}
},
"outputs": [],
"source": [
"import random\n",
"data = [i**2 for i in range(10000000)]\n",
"random.shuffle(data)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:14:57.288528Z",
"start_time": "2020-04-13T14:14:56.335654Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 307 ms, sys: 38 ms, total: 345 ms\n",
"Wall time: 876 ms\n"
]
}
],
"source": [
"%%time\n",
"list_data = list(data)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:15:06.832604Z",
"start_time": "2020-04-13T14:14:57.296651Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 3.94 s, sys: 361 ms, total: 4.3 s\n",
"Wall time: 9.48 s\n"
]
}
],
"source": [
"%%time\n",
"set_data = set(data)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`list` からデータを検索する場合"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:15:06.903239Z",
"start_time": "2020-04-13T14:15:06.867691Z"
}
},
"outputs": [],
"source": [
"random.seed(0)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:16:06.592225Z",
"start_time": "2020-04-13T14:15:06.908447Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"852 ms ± 206 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
]
}
],
"source": [
"%%timeit -n 10\n",
"random.choice(data) in list_data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`set` からデータを検索する場合"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:16:06.656258Z",
"start_time": "2020-04-13T14:16:06.611832Z"
}
},
"outputs": [],
"source": [
"random.seed(0)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:16:09.925495Z",
"start_time": "2020-04-13T14:16:06.658616Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"4.55 µs ± 2.6 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
]
}
],
"source": [
"%%timeit -n 10\n",
"random.choice(data) in set_data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 6. `list` より `dict`\n",
"\n",
"データ準備"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:16:14.876826Z",
"start_time": "2020-04-13T14:16:09.929650Z"
}
},
"outputs": [],
"source": [
"integers = [i for i in range(10000000)]\n",
"squares = [i * i for i in integers]\n",
"square_dict = {i: i * i for i in integers}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`list` を利用する場合"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:16:14.932412Z",
"start_time": "2020-04-13T14:16:14.893907Z"
}
},
"outputs": [],
"source": [
"random.seed(0)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:16:21.016209Z",
"start_time": "2020-04-13T14:16:14.945057Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"86.4 ms ± 14.8 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
]
}
],
"source": [
"%%timeit -n 10\n",
"squares[integers.index(random.choice(integers))]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`dict` を利用する場合"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:16:21.029814Z",
"start_time": "2020-04-13T14:16:21.023219Z"
}
},
"outputs": [],
"source": [
"random.seed(0)"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:16:21.075572Z",
"start_time": "2020-04-13T14:16:21.042846Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"3.69 µs ± 977 ns per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
]
}
],
"source": [
"%%timeit -n 10\n",
"square_dict[random.choice(integers)]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 繰り返し処理を早くする\n",
"### 7. `while` vs `for` vs 標準関数"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:16:21.534763Z",
"start_time": "2020-04-13T14:16:21.119381Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 246 ms, sys: 7.94 ms, total: 254 ms\n",
"Wall time: 388 ms\n"
]
},
{
"data": {
"text/plain": [
"499999500000"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"i = 0\n",
"s = 0\n",
"while i < 1000000:\n",
" s += i\n",
" i += 1\n",
"s"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:16:21.732505Z",
"start_time": "2020-04-13T14:16:21.552758Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 133 ms, sys: 3.07 ms, total: 136 ms\n",
"Wall time: 146 ms\n"
]
},
{
"data": {
"text/plain": [
"499999500000"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"s = 0\n",
"for i in range(1000000):\n",
" s += i\n",
"s"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:16:21.766849Z",
"start_time": "2020-04-13T14:16:21.734904Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 23 ms, sys: 323 µs, total: 23.4 ms\n",
"Wall time: 23.7 ms\n"
]
},
{
"data": {
"text/plain": [
"499999500000"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"s = sum(range(1000000))\n",
"s"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 8. 計算結果を再利用する"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:16:21.783369Z",
"start_time": "2020-04-13T14:16:21.769914Z"
}
},
"outputs": [],
"source": [
"some_list = [i * i for i in range(30000)]"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:16:30.726785Z",
"start_time": "2020-04-13T14:16:21.785740Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 7.94 s, sys: 93.3 ms, total: 8.03 s\n",
"Wall time: 8.93 s\n"
]
}
],
"source": [
"%%time\n",
"result = [i / sum(some_list) for i in some_list]"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:16:30.740619Z",
"start_time": "2020-04-13T14:16:30.728980Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 2.91 ms, sys: 974 µs, total: 3.89 ms\n",
"Wall time: 4.11 ms\n"
]
}
],
"source": [
"%%time\n",
"total = sum(some_list)\n",
"result = [i / total for i in some_list]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 関数を早くする\n",
"### 9. 再帰より繰り返し"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:16:33.582872Z",
"start_time": "2020-04-13T14:16:30.744720Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 2.72 s, sys: 25.6 ms, total: 2.75 s\n",
"Wall time: 2.83 s\n"
]
},
{
"data": {
"text/plain": [
"9227465"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"def fib(n):\n",
" return 1 if n in (1, 2) else fib(n - 1) + fib(n - 2)\n",
"fib(35)"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:16:33.597599Z",
"start_time": "2020-04-13T14:16:33.585639Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 36 µs, sys: 0 ns, total: 36 µs\n",
"Wall time: 42 µs\n"
]
},
{
"data": {
"text/plain": [
"9227465"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"def fib(n):\n",
" a, b = 1, 1\n",
" for i in range(n - 2):\n",
" a, b = b, a + b\n",
" return b\n",
"fib(35)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 10. キャッシュを利用する"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:16:36.725211Z",
"start_time": "2020-04-13T14:16:33.601240Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 2.84 s, sys: 30.3 ms, total: 2.87 s\n",
"Wall time: 3.12 s\n"
]
},
{
"data": {
"text/plain": [
"9227465"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"def fib(n):\n",
" return 1 if n in (1, 2) else fib(n - 1) + fib(n - 2)\n",
"fib(35)"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:16:37.848083Z",
"start_time": "2020-04-13T14:16:36.728748Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 143 µs, sys: 1.08 ms, total: 1.23 ms\n",
"Wall time: 1.53 ms\n",
"Compiler : 806 ms\n"
]
},
{
"data": {
"text/plain": [
"9227465"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"import functools\n",
"@functools.lru_cache(maxsize=50)\n",
"def fib(n):\n",
" return 1 if n in (1, 2) else fib(n - 1) + fib(n - 2)\n",
"fib(35)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 11. `numba`\n",
"> numpyと相性が良いが、pandasを利用するときは効果がない"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:16:53.174748Z",
"start_time": "2020-04-13T14:16:37.851164Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"219 ms ± 37.4 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
]
}
],
"source": [
"%%timeit -n 10\n",
"def square(x):\n",
" return x * x\n",
"\n",
"def square_sum(n):\n",
" s = 0\n",
" for i in range(1, n + 1):\n",
" s += square(i)\n",
" return s\n",
"\n",
"square_sum(1000000)"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:10.743690Z",
"start_time": "2020-04-13T14:16:53.181474Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The slowest run took 7.24 times longer than the fastest. This could mean that an intermediate result is being cached.\n",
"160 ms ± 172 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
]
}
],
"source": [
"%%timeit -n 10\n",
"import numba\n",
"\n",
"@numba.jit\n",
"def square(x):\n",
" return x * x\n",
"\n",
"@numba.jit\n",
"def square_sum(n):\n",
" s = 0\n",
" for i in range(1, n + 1):\n",
" s += square(i)\n",
" return s\n",
"\n",
"square_sum(1000000)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 標準関数及びライブラリを利用する\n",
"### 12. `collections.Counter`"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:12.724380Z",
"start_time": "2020-04-13T14:17:10.750540Z"
}
},
"outputs": [],
"source": [
"import collections\n",
"data = [x % 2020 for x in range(10000000)]"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:15.847131Z",
"start_time": "2020-04-13T14:17:12.728670Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 2.94 s, sys: 40.3 ms, total: 2.98 s\n",
"Wall time: 3.11 s\n"
]
},
{
"data": {
"text/plain": [
"4951"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"count_values = {}\n",
"for i in data:\n",
" count = count_values.get(i, 0)\n",
" count_values[i] = count + 1\n",
"count_values[10]"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:17.044684Z",
"start_time": "2020-04-13T14:17:15.850557Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 925 ms, sys: 25.7 ms, total: 950 ms\n",
"Wall time: 1.16 s\n"
]
},
{
"data": {
"text/plain": [
"4951"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"count_values = collections.Counter(data)\n",
"count_values[10]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 13. `collections.ChainMap` で `dict` の合併を早くする"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:19.745882Z",
"start_time": "2020-04-13T14:17:17.068220Z"
}
},
"outputs": [],
"source": [
"dicts = [\n",
" {hex(i * 16 + j): f'v{i}-{j}' for i in range(100000)}\n",
" for j in range(16)\n",
"]"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:20.792386Z",
"start_time": "2020-04-13T14:17:19.778021Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 521 ms, sys: 117 ms, total: 638 ms\n",
"Wall time: 1 s\n"
]
},
{
"data": {
"text/plain": [
"1600000"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"result = {}\n",
"for d in dicts:\n",
" result.update(d)\n",
"len(result)"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:21.244556Z",
"start_time": "2020-04-13T14:17:20.807199Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 274 ms, sys: 55.6 ms, total: 330 ms\n",
"Wall time: 373 ms\n"
]
},
{
"data": {
"text/plain": [
"1600000"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"result = collections.ChainMap(*dicts)\n",
"len(result)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## `numpy` を利用\n",
"### 14. `list` vs `np.array`"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:21.413397Z",
"start_time": "2020-04-13T14:17:21.400902Z"
}
},
"outputs": [],
"source": [
"import numpy as np\n",
"N = 10000000"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:22.582443Z",
"start_time": "2020-04-13T14:17:21.417680Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 626 ms, sys: 69.8 ms, total: 696 ms\n",
"Wall time: 1.16 s\n"
]
}
],
"source": [
"%%time\n",
"a = range(1, N, 3)\n",
"b = range(N, 1, -3)\n",
"result = [3 * aa - 2 * bb for aa, bb in zip(a, b)]"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:22.754833Z",
"start_time": "2020-04-13T14:17:22.587234Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 77.2 ms, sys: 56.9 ms, total: 134 ms\n",
"Wall time: 161 ms\n"
]
}
],
"source": [
"%%time\n",
"a = np.arange(1, N, 3)\n",
"b = np.arange(N, 1, -3)\n",
"result = 3 * a - 2 * b"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 15. `np.ufunc()` vs `math.func()`"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:22.763609Z",
"start_time": "2020-04-13T14:17:22.758585Z"
}
},
"outputs": [],
"source": [
"import math\n",
"N = 10000000"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:23.768837Z",
"start_time": "2020-04-13T14:17:22.766524Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 760 ms, sys: 75.5 ms, total: 835 ms\n",
"Wall time: 995 ms\n"
]
}
],
"source": [
"%%time\n",
"a = range(1, N, 3)\n",
"b = [math.log(x) for x in a]"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:23.908758Z",
"start_time": "2020-04-13T14:17:23.772021Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 79.9 ms, sys: 31 ms, total: 111 ms\n",
"Wall time: 129 ms\n"
]
}
],
"source": [
"%%time\n",
"a = np.arange(1, N, 3)\n",
"b = np.log(a)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 16. `np.where` vs `if`"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:23.948332Z",
"start_time": "2020-04-13T14:17:23.914150Z"
}
},
"outputs": [],
"source": [
"a = np.arange(-1000000, 1000000)"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:24.595519Z",
"start_time": "2020-04-13T14:17:23.951181Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 429 ms, sys: 64 ms, total: 493 ms\n",
"Wall time: 636 ms\n"
]
}
],
"source": [
"%%time\n",
"relu = np.vectorize(lambda x: x if x > 0 else 0)\n",
"result = relu(a)"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:24.617328Z",
"start_time": "2020-04-13T14:17:24.598905Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 7.33 ms, sys: 4 ms, total: 11.3 ms\n",
"Wall time: 13.3 ms\n"
]
}
],
"source": [
"%%time\n",
"relu = lambda x: np.where(x > 0, x, 0)\n",
"result = relu(a)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## pandasを早くする\n",
"### 17. `np.ufunc` vs `pd.apply` vs `pd.applymap`"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:29.820401Z",
"start_time": "2020-04-13T14:17:24.620394Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>a</th>\n",
" <th>b</th>\n",
" <th>c</th>\n",
" <th>d</th>\n",
" <th>e</th>\n",
" <th>f</th>\n",
" <th>g</th>\n",
" <th>h</th>\n",
" <th>i</th>\n",
" <th>j</th>\n",
" <th>...</th>\n",
" <th>q</th>\n",
" <th>r</th>\n",
" <th>s</th>\n",
" <th>t</th>\n",
" <th>u</th>\n",
" <th>v</th>\n",
" <th>w</th>\n",
" <th>x</th>\n",
" <th>y</th>\n",
" <th>z</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>0</td>\n",
" <td>-4</td>\n",
" <td>-7</td>\n",
" <td>4</td>\n",
" <td>-7</td>\n",
" <td>-9</td>\n",
" <td>2</td>\n",
" <td>-2</td>\n",
" <td>5</td>\n",
" <td>-9</td>\n",
" <td>9</td>\n",
" <td>...</td>\n",
" <td>-10</td>\n",
" <td>-10</td>\n",
" <td>3</td>\n",
" <td>-1</td>\n",
" <td>-6</td>\n",
" <td>-2</td>\n",
" <td>4</td>\n",
" <td>7</td>\n",
" <td>-1</td>\n",
" <td>-10</td>\n",
" </tr>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>-1</td>\n",
" <td>10</td>\n",
" <td>-1</td>\n",
" <td>-5</td>\n",
" <td>-8</td>\n",
" <td>2</td>\n",
" <td>8</td>\n",
" <td>7</td>\n",
" <td>-9</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>8</td>\n",
" <td>5</td>\n",
" <td>10</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>-9</td>\n",
" <td>9</td>\n",
" <td>1</td>\n",
" <td>7</td>\n",
" <td>-10</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>-6</td>\n",
" <td>8</td>\n",
" <td>-6</td>\n",
" <td>4</td>\n",
" <td>7</td>\n",
" <td>2</td>\n",
" <td>-5</td>\n",
" <td>-5</td>\n",
" <td>-6</td>\n",
" <td>-8</td>\n",
" <td>...</td>\n",
" <td>-10</td>\n",
" <td>-1</td>\n",
" <td>4</td>\n",
" <td>-3</td>\n",
" <td>3</td>\n",
" <td>5</td>\n",
" <td>-9</td>\n",
" <td>6</td>\n",
" <td>8</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3</td>\n",
" <td>-4</td>\n",
" <td>10</td>\n",
" <td>-2</td>\n",
" <td>-2</td>\n",
" <td>2</td>\n",
" <td>-4</td>\n",
" <td>-10</td>\n",
" <td>3</td>\n",
" <td>-7</td>\n",
" <td>10</td>\n",
" <td>...</td>\n",
" <td>-1</td>\n",
" <td>-7</td>\n",
" <td>-4</td>\n",
" <td>-1</td>\n",
" <td>-8</td>\n",
" <td>7</td>\n",
" <td>-7</td>\n",
" <td>8</td>\n",
" <td>10</td>\n",
" <td>9</td>\n",
" </tr>\n",
" <tr>\n",
" <td>4</td>\n",
" <td>-3</td>\n",
" <td>-9</td>\n",
" <td>8</td>\n",
" <td>-5</td>\n",
" <td>-6</td>\n",
" <td>8</td>\n",
" <td>-2</td>\n",
" <td>-10</td>\n",
" <td>-2</td>\n",
" <td>-5</td>\n",
" <td>...</td>\n",
" <td>-5</td>\n",
" <td>-8</td>\n",
" <td>-6</td>\n",
" <td>6</td>\n",
" <td>8</td>\n",
" <td>3</td>\n",
" <td>-10</td>\n",
" <td>5</td>\n",
" <td>-7</td>\n",
" <td>9</td>\n",
" </tr>\n",
" <tr>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99995</td>\n",
" <td>-2</td>\n",
" <td>1</td>\n",
" <td>-4</td>\n",
" <td>9</td>\n",
" <td>-3</td>\n",
" <td>-1</td>\n",
" <td>7</td>\n",
" <td>3</td>\n",
" <td>-9</td>\n",
" <td>-8</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>-10</td>\n",
" <td>-7</td>\n",
" <td>10</td>\n",
" <td>-2</td>\n",
" <td>4</td>\n",
" <td>-1</td>\n",
" <td>0</td>\n",
" <td>6</td>\n",
" <td>-4</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99996</td>\n",
" <td>-8</td>\n",
" <td>9</td>\n",
" <td>10</td>\n",
" <td>-8</td>\n",
" <td>-6</td>\n",
" <td>8</td>\n",
" <td>-1</td>\n",
" <td>9</td>\n",
" <td>-5</td>\n",
" <td>-3</td>\n",
" <td>...</td>\n",
" <td>7</td>\n",
" <td>10</td>\n",
" <td>2</td>\n",
" <td>-10</td>\n",
" <td>-6</td>\n",
" <td>7</td>\n",
" <td>-7</td>\n",
" <td>-1</td>\n",
" <td>-6</td>\n",
" <td>-5</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99997</td>\n",
" <td>-1</td>\n",
" <td>-10</td>\n",
" <td>7</td>\n",
" <td>-10</td>\n",
" <td>9</td>\n",
" <td>-6</td>\n",
" <td>-3</td>\n",
" <td>0</td>\n",
" <td>-4</td>\n",
" <td>-7</td>\n",
" <td>...</td>\n",
" <td>9</td>\n",
" <td>-6</td>\n",
" <td>7</td>\n",
" <td>-3</td>\n",
" <td>-5</td>\n",
" <td>-9</td>\n",
" <td>4</td>\n",
" <td>-8</td>\n",
" <td>0</td>\n",
" <td>-9</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99998</td>\n",
" <td>-5</td>\n",
" <td>-5</td>\n",
" <td>4</td>\n",
" <td>-3</td>\n",
" <td>-9</td>\n",
" <td>-6</td>\n",
" <td>6</td>\n",
" <td>-3</td>\n",
" <td>-2</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>6</td>\n",
" <td>-3</td>\n",
" <td>-7</td>\n",
" <td>4</td>\n",
" <td>-9</td>\n",
" <td>-7</td>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" <td>-5</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99999</td>\n",
" <td>0</td>\n",
" <td>5</td>\n",
" <td>4</td>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" <td>-2</td>\n",
" <td>-9</td>\n",
" <td>-4</td>\n",
" <td>-4</td>\n",
" <td>10</td>\n",
" <td>...</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" <td>-9</td>\n",
" <td>-2</td>\n",
" <td>-3</td>\n",
" <td>-1</td>\n",
" <td>-7</td>\n",
" <td>2</td>\n",
" <td>6</td>\n",
" <td>10</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>100000 rows × 26 columns</p>\n",
"</div>"
],
"text/plain": [
" a b c d e f g h i j ... q r s t u v w x \\\n",
"0 -4 -7 4 -7 -9 2 -2 5 -9 9 ... -10 -10 3 -1 -6 -2 4 7 \n",
"1 -1 10 -1 -5 -8 2 8 7 -9 0 ... 8 5 10 1 1 -9 9 1 \n",
"2 -6 8 -6 4 7 2 -5 -5 -6 -8 ... -10 -1 4 -3 3 5 -9 6 \n",
"3 -4 10 -2 -2 2 -4 -10 3 -7 10 ... -1 -7 -4 -1 -8 7 -7 8 \n",
"4 -3 -9 8 -5 -6 8 -2 -10 -2 -5 ... -5 -8 -6 6 8 3 -10 5 \n",
"... .. .. .. .. .. .. .. .. .. .. ... .. .. .. .. .. .. .. .. \n",
"99995 -2 1 -4 9 -3 -1 7 3 -9 -8 ... 0 -10 -7 10 -2 4 -1 0 \n",
"99996 -8 9 10 -8 -6 8 -1 9 -5 -3 ... 7 10 2 -10 -6 7 -7 -1 \n",
"99997 -1 -10 7 -10 9 -6 -3 0 -4 -7 ... 9 -6 7 -3 -5 -9 4 -8 \n",
"99998 -5 -5 4 -3 -9 -6 6 -3 -2 0 ... 6 -3 -7 4 -9 -7 3 4 \n",
"99999 0 5 4 4 5 -2 -9 -4 -4 10 ... 2 1 -9 -2 -3 -1 -7 2 \n",
"\n",
" y z \n",
"0 -1 -10 \n",
"1 7 -10 \n",
"2 8 3 \n",
"3 10 9 \n",
"4 -7 9 \n",
"... .. .. \n",
"99995 6 -4 \n",
"99996 -6 -5 \n",
"99997 0 -9 \n",
"99998 -5 2 \n",
"99999 6 10 \n",
"\n",
"[100000 rows x 26 columns]"
]
},
"execution_count": 50,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"df = pd.DataFrame(np.random.randint(-10, 11, size=(100000, 26)), columns=[chr(c) for c in range(ord('a'), ord('z') + 1)])\n",
"df"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:43.785942Z",
"start_time": "2020-04-13T14:17:29.825394Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 10.2 s, sys: 213 ms, total: 10.4 s\n",
"Wall time: 13.8 s\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>a</th>\n",
" <th>b</th>\n",
" <th>c</th>\n",
" <th>d</th>\n",
" <th>e</th>\n",
" <th>f</th>\n",
" <th>g</th>\n",
" <th>h</th>\n",
" <th>i</th>\n",
" <th>j</th>\n",
" <th>...</th>\n",
" <th>q</th>\n",
" <th>r</th>\n",
" <th>s</th>\n",
" <th>t</th>\n",
" <th>u</th>\n",
" <th>v</th>\n",
" <th>w</th>\n",
" <th>x</th>\n",
" <th>y</th>\n",
" <th>z</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>0</td>\n",
" <td>0.103159</td>\n",
" <td>0.096916</td>\n",
" <td>-1.410446</td>\n",
" <td>0.096916</td>\n",
" <td>-1.323249</td>\n",
" <td>0.493151</td>\n",
" <td>-1.325444</td>\n",
" <td>-0.675262</td>\n",
" <td>-1.323249</td>\n",
" <td>-0.499012</td>\n",
" <td>...</td>\n",
" <td>-0.295050</td>\n",
" <td>-0.295050</td>\n",
" <td>-0.848872</td>\n",
" <td>-0.301169</td>\n",
" <td>1.239586</td>\n",
" <td>-1.325444</td>\n",
" <td>-1.410446</td>\n",
" <td>1.410889</td>\n",
" <td>-0.301169</td>\n",
" <td>-0.295050</td>\n",
" </tr>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>-0.301169</td>\n",
" <td>-1.383093</td>\n",
" <td>-0.301169</td>\n",
" <td>1.242586</td>\n",
" <td>-1.134858</td>\n",
" <td>0.493151</td>\n",
" <td>0.843858</td>\n",
" <td>1.410889</td>\n",
" <td>-1.323249</td>\n",
" <td>1.000000</td>\n",
" <td>...</td>\n",
" <td>0.843858</td>\n",
" <td>-0.675262</td>\n",
" <td>-1.383093</td>\n",
" <td>1.381773</td>\n",
" <td>1.381773</td>\n",
" <td>-1.323249</td>\n",
" <td>-0.499012</td>\n",
" <td>1.381773</td>\n",
" <td>1.410889</td>\n",
" <td>-0.295050</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>1.239586</td>\n",
" <td>0.843858</td>\n",
" <td>1.239586</td>\n",
" <td>-1.410446</td>\n",
" <td>1.410889</td>\n",
" <td>0.493151</td>\n",
" <td>1.242586</td>\n",
" <td>1.242586</td>\n",
" <td>1.239586</td>\n",
" <td>-1.134858</td>\n",
" <td>...</td>\n",
" <td>-0.295050</td>\n",
" <td>-0.301169</td>\n",
" <td>-1.410446</td>\n",
" <td>-1.131113</td>\n",
" <td>-0.848872</td>\n",
" <td>-0.675262</td>\n",
" <td>-1.323249</td>\n",
" <td>0.680755</td>\n",
" <td>0.843858</td>\n",
" <td>-0.848872</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3</td>\n",
" <td>0.103159</td>\n",
" <td>-1.383093</td>\n",
" <td>-1.325444</td>\n",
" <td>-1.325444</td>\n",
" <td>0.493151</td>\n",
" <td>0.103159</td>\n",
" <td>-0.295050</td>\n",
" <td>-0.848872</td>\n",
" <td>0.096916</td>\n",
" <td>-1.383093</td>\n",
" <td>...</td>\n",
" <td>-0.301169</td>\n",
" <td>0.096916</td>\n",
" <td>0.103159</td>\n",
" <td>-0.301169</td>\n",
" <td>-1.134858</td>\n",
" <td>1.410889</td>\n",
" <td>0.096916</td>\n",
" <td>0.843858</td>\n",
" <td>-1.383093</td>\n",
" <td>-0.499012</td>\n",
" </tr>\n",
" <tr>\n",
" <td>4</td>\n",
" <td>-1.131113</td>\n",
" <td>-1.323249</td>\n",
" <td>0.843858</td>\n",
" <td>1.242586</td>\n",
" <td>1.239586</td>\n",
" <td>0.843858</td>\n",
" <td>-1.325444</td>\n",
" <td>-0.295050</td>\n",
" <td>-1.325444</td>\n",
" <td>1.242586</td>\n",
" <td>...</td>\n",
" <td>1.242586</td>\n",
" <td>-1.134858</td>\n",
" <td>1.239586</td>\n",
" <td>0.680755</td>\n",
" <td>0.843858</td>\n",
" <td>-0.848872</td>\n",
" <td>-0.295050</td>\n",
" <td>-0.675262</td>\n",
" <td>0.096916</td>\n",
" <td>-0.499012</td>\n",
" </tr>\n",
" <tr>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99995</td>\n",
" <td>-1.325444</td>\n",
" <td>1.381773</td>\n",
" <td>0.103159</td>\n",
" <td>-0.499012</td>\n",
" <td>-1.131113</td>\n",
" <td>-0.301169</td>\n",
" <td>1.410889</td>\n",
" <td>-0.848872</td>\n",
" <td>-1.323249</td>\n",
" <td>-1.134858</td>\n",
" <td>...</td>\n",
" <td>1.000000</td>\n",
" <td>-0.295050</td>\n",
" <td>0.096916</td>\n",
" <td>-1.383093</td>\n",
" <td>-1.325444</td>\n",
" <td>-1.410446</td>\n",
" <td>-0.301169</td>\n",
" <td>1.000000</td>\n",
" <td>0.680755</td>\n",
" <td>0.103159</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99996</td>\n",
" <td>-1.134858</td>\n",
" <td>-0.499012</td>\n",
" <td>-1.383093</td>\n",
" <td>-1.134858</td>\n",
" <td>1.239586</td>\n",
" <td>0.843858</td>\n",
" <td>-0.301169</td>\n",
" <td>-0.499012</td>\n",
" <td>1.242586</td>\n",
" <td>-1.131113</td>\n",
" <td>...</td>\n",
" <td>1.410889</td>\n",
" <td>-1.383093</td>\n",
" <td>0.493151</td>\n",
" <td>-0.295050</td>\n",
" <td>1.239586</td>\n",
" <td>1.410889</td>\n",
" <td>0.096916</td>\n",
" <td>-0.301169</td>\n",
" <td>1.239586</td>\n",
" <td>1.242586</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99997</td>\n",
" <td>-0.301169</td>\n",
" <td>-0.295050</td>\n",
" <td>1.410889</td>\n",
" <td>-0.295050</td>\n",
" <td>-0.499012</td>\n",
" <td>1.239586</td>\n",
" <td>-1.131113</td>\n",
" <td>1.000000</td>\n",
" <td>0.103159</td>\n",
" <td>0.096916</td>\n",
" <td>...</td>\n",
" <td>-0.499012</td>\n",
" <td>1.239586</td>\n",
" <td>1.410889</td>\n",
" <td>-1.131113</td>\n",
" <td>1.242586</td>\n",
" <td>-1.323249</td>\n",
" <td>-1.410446</td>\n",
" <td>-1.134858</td>\n",
" <td>1.000000</td>\n",
" <td>-1.323249</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99998</td>\n",
" <td>1.242586</td>\n",
" <td>1.242586</td>\n",
" <td>-1.410446</td>\n",
" <td>-1.131113</td>\n",
" <td>-1.323249</td>\n",
" <td>1.239586</td>\n",
" <td>0.680755</td>\n",
" <td>-1.131113</td>\n",
" <td>-1.325444</td>\n",
" <td>1.000000</td>\n",
" <td>...</td>\n",
" <td>0.680755</td>\n",
" <td>-1.131113</td>\n",
" <td>0.096916</td>\n",
" <td>-1.410446</td>\n",
" <td>-1.323249</td>\n",
" <td>0.096916</td>\n",
" <td>-0.848872</td>\n",
" <td>-1.410446</td>\n",
" <td>1.242586</td>\n",
" <td>0.493151</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99999</td>\n",
" <td>1.000000</td>\n",
" <td>-0.675262</td>\n",
" <td>-1.410446</td>\n",
" <td>-1.410446</td>\n",
" <td>-0.675262</td>\n",
" <td>-1.325444</td>\n",
" <td>-1.323249</td>\n",
" <td>0.103159</td>\n",
" <td>0.103159</td>\n",
" <td>-1.383093</td>\n",
" <td>...</td>\n",
" <td>0.493151</td>\n",
" <td>1.381773</td>\n",
" <td>-1.323249</td>\n",
" <td>-1.325444</td>\n",
" <td>-1.131113</td>\n",
" <td>-0.301169</td>\n",
" <td>0.096916</td>\n",
" <td>0.493151</td>\n",
" <td>0.680755</td>\n",
" <td>-1.383093</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>100000 rows × 26 columns</p>\n",
"</div>"
],
"text/plain": [
" a b c d e f g \\\n",
"0 0.103159 0.096916 -1.410446 0.096916 -1.323249 0.493151 -1.325444 \n",
"1 -0.301169 -1.383093 -0.301169 1.242586 -1.134858 0.493151 0.843858 \n",
"2 1.239586 0.843858 1.239586 -1.410446 1.410889 0.493151 1.242586 \n",
"3 0.103159 -1.383093 -1.325444 -1.325444 0.493151 0.103159 -0.295050 \n",
"4 -1.131113 -1.323249 0.843858 1.242586 1.239586 0.843858 -1.325444 \n",
"... ... ... ... ... ... ... ... \n",
"99995 -1.325444 1.381773 0.103159 -0.499012 -1.131113 -0.301169 1.410889 \n",
"99996 -1.134858 -0.499012 -1.383093 -1.134858 1.239586 0.843858 -0.301169 \n",
"99997 -0.301169 -0.295050 1.410889 -0.295050 -0.499012 1.239586 -1.131113 \n",
"99998 1.242586 1.242586 -1.410446 -1.131113 -1.323249 1.239586 0.680755 \n",
"99999 1.000000 -0.675262 -1.410446 -1.410446 -0.675262 -1.325444 -1.323249 \n",
"\n",
" h i j ... q r s \\\n",
"0 -0.675262 -1.323249 -0.499012 ... -0.295050 -0.295050 -0.848872 \n",
"1 1.410889 -1.323249 1.000000 ... 0.843858 -0.675262 -1.383093 \n",
"2 1.242586 1.239586 -1.134858 ... -0.295050 -0.301169 -1.410446 \n",
"3 -0.848872 0.096916 -1.383093 ... -0.301169 0.096916 0.103159 \n",
"4 -0.295050 -1.325444 1.242586 ... 1.242586 -1.134858 1.239586 \n",
"... ... ... ... ... ... ... ... \n",
"99995 -0.848872 -1.323249 -1.134858 ... 1.000000 -0.295050 0.096916 \n",
"99996 -0.499012 1.242586 -1.131113 ... 1.410889 -1.383093 0.493151 \n",
"99997 1.000000 0.103159 0.096916 ... -0.499012 1.239586 1.410889 \n",
"99998 -1.131113 -1.325444 1.000000 ... 0.680755 -1.131113 0.096916 \n",
"99999 0.103159 0.103159 -1.383093 ... 0.493151 1.381773 -1.323249 \n",
"\n",
" t u v w x y z \n",
"0 -0.301169 1.239586 -1.325444 -1.410446 1.410889 -0.301169 -0.295050 \n",
"1 1.381773 1.381773 -1.323249 -0.499012 1.381773 1.410889 -0.295050 \n",
"2 -1.131113 -0.848872 -0.675262 -1.323249 0.680755 0.843858 -0.848872 \n",
"3 -0.301169 -1.134858 1.410889 0.096916 0.843858 -1.383093 -0.499012 \n",
"4 0.680755 0.843858 -0.848872 -0.295050 -0.675262 0.096916 -0.499012 \n",
"... ... ... ... ... ... ... ... \n",
"99995 -1.383093 -1.325444 -1.410446 -0.301169 1.000000 0.680755 0.103159 \n",
"99996 -0.295050 1.239586 1.410889 0.096916 -0.301169 1.239586 1.242586 \n",
"99997 -1.131113 1.242586 -1.323249 -1.410446 -1.134858 1.000000 -1.323249 \n",
"99998 -1.410446 -1.323249 0.096916 -0.848872 -1.410446 1.242586 0.493151 \n",
"99999 -1.325444 -1.131113 -0.301169 0.096916 0.493151 0.680755 -1.383093 \n",
"\n",
"[100000 rows x 26 columns]"
]
},
"execution_count": 51,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"df.applymap(lambda x: np.sin(x) + np.cos(x))"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:44.781178Z",
"start_time": "2020-04-13T14:17:43.823672Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 320 ms, sys: 63.6 ms, total: 383 ms\n",
"Wall time: 890 ms\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>a</th>\n",
" <th>b</th>\n",
" <th>c</th>\n",
" <th>d</th>\n",
" <th>e</th>\n",
" <th>f</th>\n",
" <th>g</th>\n",
" <th>h</th>\n",
" <th>i</th>\n",
" <th>j</th>\n",
" <th>...</th>\n",
" <th>q</th>\n",
" <th>r</th>\n",
" <th>s</th>\n",
" <th>t</th>\n",
" <th>u</th>\n",
" <th>v</th>\n",
" <th>w</th>\n",
" <th>x</th>\n",
" <th>y</th>\n",
" <th>z</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>0</td>\n",
" <td>0.103159</td>\n",
" <td>0.096916</td>\n",
" <td>-1.410446</td>\n",
" <td>0.096916</td>\n",
" <td>-1.323249</td>\n",
" <td>0.493151</td>\n",
" <td>-1.325444</td>\n",
" <td>-0.675262</td>\n",
" <td>-1.323249</td>\n",
" <td>-0.499012</td>\n",
" <td>...</td>\n",
" <td>-0.295050</td>\n",
" <td>-0.295050</td>\n",
" <td>-0.848872</td>\n",
" <td>-0.301169</td>\n",
" <td>1.239586</td>\n",
" <td>-1.325444</td>\n",
" <td>-1.410446</td>\n",
" <td>1.410889</td>\n",
" <td>-0.301169</td>\n",
" <td>-0.295050</td>\n",
" </tr>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>-0.301169</td>\n",
" <td>-1.383093</td>\n",
" <td>-0.301169</td>\n",
" <td>1.242586</td>\n",
" <td>-1.134858</td>\n",
" <td>0.493151</td>\n",
" <td>0.843858</td>\n",
" <td>1.410889</td>\n",
" <td>-1.323249</td>\n",
" <td>1.000000</td>\n",
" <td>...</td>\n",
" <td>0.843858</td>\n",
" <td>-0.675262</td>\n",
" <td>-1.383093</td>\n",
" <td>1.381773</td>\n",
" <td>1.381773</td>\n",
" <td>-1.323249</td>\n",
" <td>-0.499012</td>\n",
" <td>1.381773</td>\n",
" <td>1.410889</td>\n",
" <td>-0.295050</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>1.239586</td>\n",
" <td>0.843858</td>\n",
" <td>1.239586</td>\n",
" <td>-1.410446</td>\n",
" <td>1.410889</td>\n",
" <td>0.493151</td>\n",
" <td>1.242586</td>\n",
" <td>1.242586</td>\n",
" <td>1.239586</td>\n",
" <td>-1.134858</td>\n",
" <td>...</td>\n",
" <td>-0.295050</td>\n",
" <td>-0.301169</td>\n",
" <td>-1.410446</td>\n",
" <td>-1.131113</td>\n",
" <td>-0.848872</td>\n",
" <td>-0.675262</td>\n",
" <td>-1.323249</td>\n",
" <td>0.680755</td>\n",
" <td>0.843858</td>\n",
" <td>-0.848872</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3</td>\n",
" <td>0.103159</td>\n",
" <td>-1.383093</td>\n",
" <td>-1.325444</td>\n",
" <td>-1.325444</td>\n",
" <td>0.493151</td>\n",
" <td>0.103159</td>\n",
" <td>-0.295050</td>\n",
" <td>-0.848872</td>\n",
" <td>0.096916</td>\n",
" <td>-1.383093</td>\n",
" <td>...</td>\n",
" <td>-0.301169</td>\n",
" <td>0.096916</td>\n",
" <td>0.103159</td>\n",
" <td>-0.301169</td>\n",
" <td>-1.134858</td>\n",
" <td>1.410889</td>\n",
" <td>0.096916</td>\n",
" <td>0.843858</td>\n",
" <td>-1.383093</td>\n",
" <td>-0.499012</td>\n",
" </tr>\n",
" <tr>\n",
" <td>4</td>\n",
" <td>-1.131113</td>\n",
" <td>-1.323249</td>\n",
" <td>0.843858</td>\n",
" <td>1.242586</td>\n",
" <td>1.239586</td>\n",
" <td>0.843858</td>\n",
" <td>-1.325444</td>\n",
" <td>-0.295050</td>\n",
" <td>-1.325444</td>\n",
" <td>1.242586</td>\n",
" <td>...</td>\n",
" <td>1.242586</td>\n",
" <td>-1.134858</td>\n",
" <td>1.239586</td>\n",
" <td>0.680755</td>\n",
" <td>0.843858</td>\n",
" <td>-0.848872</td>\n",
" <td>-0.295050</td>\n",
" <td>-0.675262</td>\n",
" <td>0.096916</td>\n",
" <td>-0.499012</td>\n",
" </tr>\n",
" <tr>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99995</td>\n",
" <td>-1.325444</td>\n",
" <td>1.381773</td>\n",
" <td>0.103159</td>\n",
" <td>-0.499012</td>\n",
" <td>-1.131113</td>\n",
" <td>-0.301169</td>\n",
" <td>1.410889</td>\n",
" <td>-0.848872</td>\n",
" <td>-1.323249</td>\n",
" <td>-1.134858</td>\n",
" <td>...</td>\n",
" <td>1.000000</td>\n",
" <td>-0.295050</td>\n",
" <td>0.096916</td>\n",
" <td>-1.383093</td>\n",
" <td>-1.325444</td>\n",
" <td>-1.410446</td>\n",
" <td>-0.301169</td>\n",
" <td>1.000000</td>\n",
" <td>0.680755</td>\n",
" <td>0.103159</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99996</td>\n",
" <td>-1.134858</td>\n",
" <td>-0.499012</td>\n",
" <td>-1.383093</td>\n",
" <td>-1.134858</td>\n",
" <td>1.239586</td>\n",
" <td>0.843858</td>\n",
" <td>-0.301169</td>\n",
" <td>-0.499012</td>\n",
" <td>1.242586</td>\n",
" <td>-1.131113</td>\n",
" <td>...</td>\n",
" <td>1.410889</td>\n",
" <td>-1.383093</td>\n",
" <td>0.493151</td>\n",
" <td>-0.295050</td>\n",
" <td>1.239586</td>\n",
" <td>1.410889</td>\n",
" <td>0.096916</td>\n",
" <td>-0.301169</td>\n",
" <td>1.239586</td>\n",
" <td>1.242586</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99997</td>\n",
" <td>-0.301169</td>\n",
" <td>-0.295050</td>\n",
" <td>1.410889</td>\n",
" <td>-0.295050</td>\n",
" <td>-0.499012</td>\n",
" <td>1.239586</td>\n",
" <td>-1.131113</td>\n",
" <td>1.000000</td>\n",
" <td>0.103159</td>\n",
" <td>0.096916</td>\n",
" <td>...</td>\n",
" <td>-0.499012</td>\n",
" <td>1.239586</td>\n",
" <td>1.410889</td>\n",
" <td>-1.131113</td>\n",
" <td>1.242586</td>\n",
" <td>-1.323249</td>\n",
" <td>-1.410446</td>\n",
" <td>-1.134858</td>\n",
" <td>1.000000</td>\n",
" <td>-1.323249</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99998</td>\n",
" <td>1.242586</td>\n",
" <td>1.242586</td>\n",
" <td>-1.410446</td>\n",
" <td>-1.131113</td>\n",
" <td>-1.323249</td>\n",
" <td>1.239586</td>\n",
" <td>0.680755</td>\n",
" <td>-1.131113</td>\n",
" <td>-1.325444</td>\n",
" <td>1.000000</td>\n",
" <td>...</td>\n",
" <td>0.680755</td>\n",
" <td>-1.131113</td>\n",
" <td>0.096916</td>\n",
" <td>-1.410446</td>\n",
" <td>-1.323249</td>\n",
" <td>0.096916</td>\n",
" <td>-0.848872</td>\n",
" <td>-1.410446</td>\n",
" <td>1.242586</td>\n",
" <td>0.493151</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99999</td>\n",
" <td>1.000000</td>\n",
" <td>-0.675262</td>\n",
" <td>-1.410446</td>\n",
" <td>-1.410446</td>\n",
" <td>-0.675262</td>\n",
" <td>-1.325444</td>\n",
" <td>-1.323249</td>\n",
" <td>0.103159</td>\n",
" <td>0.103159</td>\n",
" <td>-1.383093</td>\n",
" <td>...</td>\n",
" <td>0.493151</td>\n",
" <td>1.381773</td>\n",
" <td>-1.323249</td>\n",
" <td>-1.325444</td>\n",
" <td>-1.131113</td>\n",
" <td>-0.301169</td>\n",
" <td>0.096916</td>\n",
" <td>0.493151</td>\n",
" <td>0.680755</td>\n",
" <td>-1.383093</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>100000 rows × 26 columns</p>\n",
"</div>"
],
"text/plain": [
" a b c d e f g \\\n",
"0 0.103159 0.096916 -1.410446 0.096916 -1.323249 0.493151 -1.325444 \n",
"1 -0.301169 -1.383093 -0.301169 1.242586 -1.134858 0.493151 0.843858 \n",
"2 1.239586 0.843858 1.239586 -1.410446 1.410889 0.493151 1.242586 \n",
"3 0.103159 -1.383093 -1.325444 -1.325444 0.493151 0.103159 -0.295050 \n",
"4 -1.131113 -1.323249 0.843858 1.242586 1.239586 0.843858 -1.325444 \n",
"... ... ... ... ... ... ... ... \n",
"99995 -1.325444 1.381773 0.103159 -0.499012 -1.131113 -0.301169 1.410889 \n",
"99996 -1.134858 -0.499012 -1.383093 -1.134858 1.239586 0.843858 -0.301169 \n",
"99997 -0.301169 -0.295050 1.410889 -0.295050 -0.499012 1.239586 -1.131113 \n",
"99998 1.242586 1.242586 -1.410446 -1.131113 -1.323249 1.239586 0.680755 \n",
"99999 1.000000 -0.675262 -1.410446 -1.410446 -0.675262 -1.325444 -1.323249 \n",
"\n",
" h i j ... q r s \\\n",
"0 -0.675262 -1.323249 -0.499012 ... -0.295050 -0.295050 -0.848872 \n",
"1 1.410889 -1.323249 1.000000 ... 0.843858 -0.675262 -1.383093 \n",
"2 1.242586 1.239586 -1.134858 ... -0.295050 -0.301169 -1.410446 \n",
"3 -0.848872 0.096916 -1.383093 ... -0.301169 0.096916 0.103159 \n",
"4 -0.295050 -1.325444 1.242586 ... 1.242586 -1.134858 1.239586 \n",
"... ... ... ... ... ... ... ... \n",
"99995 -0.848872 -1.323249 -1.134858 ... 1.000000 -0.295050 0.096916 \n",
"99996 -0.499012 1.242586 -1.131113 ... 1.410889 -1.383093 0.493151 \n",
"99997 1.000000 0.103159 0.096916 ... -0.499012 1.239586 1.410889 \n",
"99998 -1.131113 -1.325444 1.000000 ... 0.680755 -1.131113 0.096916 \n",
"99999 0.103159 0.103159 -1.383093 ... 0.493151 1.381773 -1.323249 \n",
"\n",
" t u v w x y z \n",
"0 -0.301169 1.239586 -1.325444 -1.410446 1.410889 -0.301169 -0.295050 \n",
"1 1.381773 1.381773 -1.323249 -0.499012 1.381773 1.410889 -0.295050 \n",
"2 -1.131113 -0.848872 -0.675262 -1.323249 0.680755 0.843858 -0.848872 \n",
"3 -0.301169 -1.134858 1.410889 0.096916 0.843858 -1.383093 -0.499012 \n",
"4 0.680755 0.843858 -0.848872 -0.295050 -0.675262 0.096916 -0.499012 \n",
"... ... ... ... ... ... ... ... \n",
"99995 -1.383093 -1.325444 -1.410446 -0.301169 1.000000 0.680755 0.103159 \n",
"99996 -0.295050 1.239586 1.410889 0.096916 -0.301169 1.239586 1.242586 \n",
"99997 -1.131113 1.242586 -1.323249 -1.410446 -1.134858 1.000000 -1.323249 \n",
"99998 -1.410446 -1.323249 0.096916 -0.848872 -1.410446 1.242586 0.493151 \n",
"99999 -1.325444 -1.131113 -0.301169 0.096916 0.493151 0.680755 -1.383093 \n",
"\n",
"[100000 rows x 26 columns]"
]
},
"execution_count": 52,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"df.apply(lambda x: np.sin(x) + np.cos(x))"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:45.051539Z",
"start_time": "2020-04-13T14:17:44.786006Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 136 ms, sys: 37.7 ms, total: 173 ms\n",
"Wall time: 167 ms\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>a</th>\n",
" <th>b</th>\n",
" <th>c</th>\n",
" <th>d</th>\n",
" <th>e</th>\n",
" <th>f</th>\n",
" <th>g</th>\n",
" <th>h</th>\n",
" <th>i</th>\n",
" <th>j</th>\n",
" <th>...</th>\n",
" <th>q</th>\n",
" <th>r</th>\n",
" <th>s</th>\n",
" <th>t</th>\n",
" <th>u</th>\n",
" <th>v</th>\n",
" <th>w</th>\n",
" <th>x</th>\n",
" <th>y</th>\n",
" <th>z</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>0</td>\n",
" <td>0.103159</td>\n",
" <td>0.096916</td>\n",
" <td>-1.410446</td>\n",
" <td>0.096916</td>\n",
" <td>-1.323249</td>\n",
" <td>0.493151</td>\n",
" <td>-1.325444</td>\n",
" <td>-0.675262</td>\n",
" <td>-1.323249</td>\n",
" <td>-0.499012</td>\n",
" <td>...</td>\n",
" <td>-0.295050</td>\n",
" <td>-0.295050</td>\n",
" <td>-0.848872</td>\n",
" <td>-0.301169</td>\n",
" <td>1.239586</td>\n",
" <td>-1.325444</td>\n",
" <td>-1.410446</td>\n",
" <td>1.410889</td>\n",
" <td>-0.301169</td>\n",
" <td>-0.295050</td>\n",
" </tr>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>-0.301169</td>\n",
" <td>-1.383093</td>\n",
" <td>-0.301169</td>\n",
" <td>1.242586</td>\n",
" <td>-1.134858</td>\n",
" <td>0.493151</td>\n",
" <td>0.843858</td>\n",
" <td>1.410889</td>\n",
" <td>-1.323249</td>\n",
" <td>1.000000</td>\n",
" <td>...</td>\n",
" <td>0.843858</td>\n",
" <td>-0.675262</td>\n",
" <td>-1.383093</td>\n",
" <td>1.381773</td>\n",
" <td>1.381773</td>\n",
" <td>-1.323249</td>\n",
" <td>-0.499012</td>\n",
" <td>1.381773</td>\n",
" <td>1.410889</td>\n",
" <td>-0.295050</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>1.239586</td>\n",
" <td>0.843858</td>\n",
" <td>1.239586</td>\n",
" <td>-1.410446</td>\n",
" <td>1.410889</td>\n",
" <td>0.493151</td>\n",
" <td>1.242586</td>\n",
" <td>1.242586</td>\n",
" <td>1.239586</td>\n",
" <td>-1.134858</td>\n",
" <td>...</td>\n",
" <td>-0.295050</td>\n",
" <td>-0.301169</td>\n",
" <td>-1.410446</td>\n",
" <td>-1.131113</td>\n",
" <td>-0.848872</td>\n",
" <td>-0.675262</td>\n",
" <td>-1.323249</td>\n",
" <td>0.680755</td>\n",
" <td>0.843858</td>\n",
" <td>-0.848872</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3</td>\n",
" <td>0.103159</td>\n",
" <td>-1.383093</td>\n",
" <td>-1.325444</td>\n",
" <td>-1.325444</td>\n",
" <td>0.493151</td>\n",
" <td>0.103159</td>\n",
" <td>-0.295050</td>\n",
" <td>-0.848872</td>\n",
" <td>0.096916</td>\n",
" <td>-1.383093</td>\n",
" <td>...</td>\n",
" <td>-0.301169</td>\n",
" <td>0.096916</td>\n",
" <td>0.103159</td>\n",
" <td>-0.301169</td>\n",
" <td>-1.134858</td>\n",
" <td>1.410889</td>\n",
" <td>0.096916</td>\n",
" <td>0.843858</td>\n",
" <td>-1.383093</td>\n",
" <td>-0.499012</td>\n",
" </tr>\n",
" <tr>\n",
" <td>4</td>\n",
" <td>-1.131113</td>\n",
" <td>-1.323249</td>\n",
" <td>0.843858</td>\n",
" <td>1.242586</td>\n",
" <td>1.239586</td>\n",
" <td>0.843858</td>\n",
" <td>-1.325444</td>\n",
" <td>-0.295050</td>\n",
" <td>-1.325444</td>\n",
" <td>1.242586</td>\n",
" <td>...</td>\n",
" <td>1.242586</td>\n",
" <td>-1.134858</td>\n",
" <td>1.239586</td>\n",
" <td>0.680755</td>\n",
" <td>0.843858</td>\n",
" <td>-0.848872</td>\n",
" <td>-0.295050</td>\n",
" <td>-0.675262</td>\n",
" <td>0.096916</td>\n",
" <td>-0.499012</td>\n",
" </tr>\n",
" <tr>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99995</td>\n",
" <td>-1.325444</td>\n",
" <td>1.381773</td>\n",
" <td>0.103159</td>\n",
" <td>-0.499012</td>\n",
" <td>-1.131113</td>\n",
" <td>-0.301169</td>\n",
" <td>1.410889</td>\n",
" <td>-0.848872</td>\n",
" <td>-1.323249</td>\n",
" <td>-1.134858</td>\n",
" <td>...</td>\n",
" <td>1.000000</td>\n",
" <td>-0.295050</td>\n",
" <td>0.096916</td>\n",
" <td>-1.383093</td>\n",
" <td>-1.325444</td>\n",
" <td>-1.410446</td>\n",
" <td>-0.301169</td>\n",
" <td>1.000000</td>\n",
" <td>0.680755</td>\n",
" <td>0.103159</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99996</td>\n",
" <td>-1.134858</td>\n",
" <td>-0.499012</td>\n",
" <td>-1.383093</td>\n",
" <td>-1.134858</td>\n",
" <td>1.239586</td>\n",
" <td>0.843858</td>\n",
" <td>-0.301169</td>\n",
" <td>-0.499012</td>\n",
" <td>1.242586</td>\n",
" <td>-1.131113</td>\n",
" <td>...</td>\n",
" <td>1.410889</td>\n",
" <td>-1.383093</td>\n",
" <td>0.493151</td>\n",
" <td>-0.295050</td>\n",
" <td>1.239586</td>\n",
" <td>1.410889</td>\n",
" <td>0.096916</td>\n",
" <td>-0.301169</td>\n",
" <td>1.239586</td>\n",
" <td>1.242586</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99997</td>\n",
" <td>-0.301169</td>\n",
" <td>-0.295050</td>\n",
" <td>1.410889</td>\n",
" <td>-0.295050</td>\n",
" <td>-0.499012</td>\n",
" <td>1.239586</td>\n",
" <td>-1.131113</td>\n",
" <td>1.000000</td>\n",
" <td>0.103159</td>\n",
" <td>0.096916</td>\n",
" <td>...</td>\n",
" <td>-0.499012</td>\n",
" <td>1.239586</td>\n",
" <td>1.410889</td>\n",
" <td>-1.131113</td>\n",
" <td>1.242586</td>\n",
" <td>-1.323249</td>\n",
" <td>-1.410446</td>\n",
" <td>-1.134858</td>\n",
" <td>1.000000</td>\n",
" <td>-1.323249</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99998</td>\n",
" <td>1.242586</td>\n",
" <td>1.242586</td>\n",
" <td>-1.410446</td>\n",
" <td>-1.131113</td>\n",
" <td>-1.323249</td>\n",
" <td>1.239586</td>\n",
" <td>0.680755</td>\n",
" <td>-1.131113</td>\n",
" <td>-1.325444</td>\n",
" <td>1.000000</td>\n",
" <td>...</td>\n",
" <td>0.680755</td>\n",
" <td>-1.131113</td>\n",
" <td>0.096916</td>\n",
" <td>-1.410446</td>\n",
" <td>-1.323249</td>\n",
" <td>0.096916</td>\n",
" <td>-0.848872</td>\n",
" <td>-1.410446</td>\n",
" <td>1.242586</td>\n",
" <td>0.493151</td>\n",
" </tr>\n",
" <tr>\n",
" <td>99999</td>\n",
" <td>1.000000</td>\n",
" <td>-0.675262</td>\n",
" <td>-1.410446</td>\n",
" <td>-1.410446</td>\n",
" <td>-0.675262</td>\n",
" <td>-1.325444</td>\n",
" <td>-1.323249</td>\n",
" <td>0.103159</td>\n",
" <td>0.103159</td>\n",
" <td>-1.383093</td>\n",
" <td>...</td>\n",
" <td>0.493151</td>\n",
" <td>1.381773</td>\n",
" <td>-1.323249</td>\n",
" <td>-1.325444</td>\n",
" <td>-1.131113</td>\n",
" <td>-0.301169</td>\n",
" <td>0.096916</td>\n",
" <td>0.493151</td>\n",
" <td>0.680755</td>\n",
" <td>-1.383093</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>100000 rows × 26 columns</p>\n",
"</div>"
],
"text/plain": [
" a b c d e f g \\\n",
"0 0.103159 0.096916 -1.410446 0.096916 -1.323249 0.493151 -1.325444 \n",
"1 -0.301169 -1.383093 -0.301169 1.242586 -1.134858 0.493151 0.843858 \n",
"2 1.239586 0.843858 1.239586 -1.410446 1.410889 0.493151 1.242586 \n",
"3 0.103159 -1.383093 -1.325444 -1.325444 0.493151 0.103159 -0.295050 \n",
"4 -1.131113 -1.323249 0.843858 1.242586 1.239586 0.843858 -1.325444 \n",
"... ... ... ... ... ... ... ... \n",
"99995 -1.325444 1.381773 0.103159 -0.499012 -1.131113 -0.301169 1.410889 \n",
"99996 -1.134858 -0.499012 -1.383093 -1.134858 1.239586 0.843858 -0.301169 \n",
"99997 -0.301169 -0.295050 1.410889 -0.295050 -0.499012 1.239586 -1.131113 \n",
"99998 1.242586 1.242586 -1.410446 -1.131113 -1.323249 1.239586 0.680755 \n",
"99999 1.000000 -0.675262 -1.410446 -1.410446 -0.675262 -1.325444 -1.323249 \n",
"\n",
" h i j ... q r s \\\n",
"0 -0.675262 -1.323249 -0.499012 ... -0.295050 -0.295050 -0.848872 \n",
"1 1.410889 -1.323249 1.000000 ... 0.843858 -0.675262 -1.383093 \n",
"2 1.242586 1.239586 -1.134858 ... -0.295050 -0.301169 -1.410446 \n",
"3 -0.848872 0.096916 -1.383093 ... -0.301169 0.096916 0.103159 \n",
"4 -0.295050 -1.325444 1.242586 ... 1.242586 -1.134858 1.239586 \n",
"... ... ... ... ... ... ... ... \n",
"99995 -0.848872 -1.323249 -1.134858 ... 1.000000 -0.295050 0.096916 \n",
"99996 -0.499012 1.242586 -1.131113 ... 1.410889 -1.383093 0.493151 \n",
"99997 1.000000 0.103159 0.096916 ... -0.499012 1.239586 1.410889 \n",
"99998 -1.131113 -1.325444 1.000000 ... 0.680755 -1.131113 0.096916 \n",
"99999 0.103159 0.103159 -1.383093 ... 0.493151 1.381773 -1.323249 \n",
"\n",
" t u v w x y z \n",
"0 -0.301169 1.239586 -1.325444 -1.410446 1.410889 -0.301169 -0.295050 \n",
"1 1.381773 1.381773 -1.323249 -0.499012 1.381773 1.410889 -0.295050 \n",
"2 -1.131113 -0.848872 -0.675262 -1.323249 0.680755 0.843858 -0.848872 \n",
"3 -0.301169 -1.134858 1.410889 0.096916 0.843858 -1.383093 -0.499012 \n",
"4 0.680755 0.843858 -0.848872 -0.295050 -0.675262 0.096916 -0.499012 \n",
"... ... ... ... ... ... ... ... \n",
"99995 -1.383093 -1.325444 -1.410446 -0.301169 1.000000 0.680755 0.103159 \n",
"99996 -0.295050 1.239586 1.410889 0.096916 -0.301169 1.239586 1.242586 \n",
"99997 -1.131113 1.242586 -1.323249 -1.410446 -1.134858 1.000000 -1.323249 \n",
"99998 -1.410446 -1.323249 0.096916 -0.848872 -1.410446 1.242586 0.493151 \n",
"99999 -1.325444 -1.131113 -0.301169 0.096916 0.493151 0.680755 -1.383093 \n",
"\n",
"[100000 rows x 26 columns]"
]
},
"execution_count": 53,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"np.sin(df) + np.cos(df)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 18. サイズが分かる場合、最初のサイズを指定しておく"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:46.157393Z",
"start_time": "2020-04-13T14:17:45.057172Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 921 ms, sys: 28.2 ms, total: 949 ms\n",
"Wall time: 1.09 s\n"
]
},
{
"data": {
"text/plain": [
"(1000, 26)"
]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"df = pd.DataFrame(columns=[chr(c) for c in range(ord('a'), ord('z') + 1)])\n",
"for i in range(1000):\n",
" df.loc[i, :] = range(i, i + 26)\n",
"df.shape"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:47.025882Z",
"start_time": "2020-04-13T14:17:46.170463Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 547 ms, sys: 19.3 ms, total: 566 ms\n",
"Wall time: 836 ms\n"
]
},
{
"data": {
"text/plain": [
"(1000, 26)"
]
},
"execution_count": 55,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"df = pd.DataFrame(np.zeros((1000, 26)), columns=[chr(c) for c in range(ord('a'), ord('z') + 1)])\n",
"for i in range(1000):\n",
" df.loc[i, :] = range(i, i + 26)\n",
"df.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 19. `pandarallel`"
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:17:51.835225Z",
"start_time": "2020-04-13T14:17:47.041652Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: pandarallel in /Users/wangyuan@jp.ibm.com/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages (1.4.2)\r\n",
"Requirement already satisfied: dill in /Users/wangyuan@jp.ibm.com/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages (from pandarallel) (0.3.1.1)\r\n"
]
}
],
"source": [
"!pip install pandarallel\n",
"import pandarallel\n",
"import os\n",
"df = pd.DataFrame(np.random.randint(-10, 11, size=(100000, 26)), columns=[chr(c) for c in range(ord('a'), ord('z') + 1)])"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:18:09.012535Z",
"start_time": "2020-04-13T14:17:51.838322Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 13.4 s, sys: 274 ms, total: 13.7 s\n",
"Wall time: 17.2 s\n"
]
},
{
"data": {
"text/plain": [
"0 13\n",
"1 -62\n",
"2 10\n",
"3 75\n",
"4 -43\n",
" ..\n",
"99995 -16\n",
"99996 40\n",
"99997 12\n",
"99998 -56\n",
"99999 -27\n",
"Length: 100000, dtype: int64"
]
},
"execution_count": 57,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"df.apply(np.sum, axis=1)"
]
},
{
"cell_type": "code",
"execution_count": 58,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:18:19.235336Z",
"start_time": "2020-04-13T14:18:09.018697Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO: Pandarallel will run on 4 workers.\n",
"INFO: Pandarallel will use standard multiprocessing data tranfer (pipe) to transfer data between the main process and workers.\n",
"CPU times: user 58.5 ms, sys: 124 ms, total: 183 ms\n",
"Wall time: 10.2 s\n"
]
},
{
"data": {
"text/plain": [
"0 13\n",
"1 -62\n",
"2 10\n",
"3 75\n",
"4 -43\n",
" ..\n",
"99995 -16\n",
"99996 40\n",
"99997 12\n",
"99998 -56\n",
"99999 -27\n",
"Length: 100000, dtype: int64"
]
},
"execution_count": 58,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"pandarallel.pandarallel.initialize(nb_workers=os.cpu_count())\n",
"df.parallel_apply(np.sum, axis=1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 20. マルチプロセス"
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:18:19.252933Z",
"start_time": "2020-04-13T14:18:19.242260Z"
}
},
"outputs": [],
"source": [
"def busy(n):\n",
" for _ in range(10000000):\n",
" n * n\n",
" return n * n\n",
"\n",
"import multiprocessing"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:18:26.281798Z",
"start_time": "2020-04-13T14:18:19.255377Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 5.51 s, sys: 71.8 ms, total: 5.59 s\n",
"Wall time: 6.99 s\n"
]
},
{
"data": {
"text/plain": [
"[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]"
]
},
"execution_count": 60,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"result = [busy(i) for i in range(10)]\n",
"result"
]
},
{
"cell_type": "code",
"execution_count": 61,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:18:29.559854Z",
"start_time": "2020-04-13T14:18:26.285567Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 11.7 ms, sys: 51.9 ms, total: 63.6 ms\n",
"Wall time: 3.26 s\n"
]
},
{
"data": {
"text/plain": [
"[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]"
]
},
"execution_count": 61,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"with multiprocessing.Pool(os.cpu_count()) as pool:\n",
" result = pool.map(busy, range(10))\n",
"result"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 21. マルチスレッド"
]
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:18:29.578313Z",
"start_time": "2020-04-13T14:18:29.571642Z"
}
},
"outputs": [],
"source": [
"import time\n",
"def io_busy(n):\n",
" time.sleep(5)\n",
" return n * n\n",
"\n",
"import multiprocessing.pool"
]
},
{
"cell_type": "code",
"execution_count": 63,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:18:49.613028Z",
"start_time": "2020-04-13T14:18:29.585748Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 1.82 ms, sys: 2.51 ms, total: 4.33 ms\n",
"Wall time: 20 s\n"
]
},
{
"data": {
"text/plain": [
"[0, 1, 4, 9]"
]
},
"execution_count": 63,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"result = [io_busy(i) for i in range(4)]\n",
"result"
]
},
{
"cell_type": "code",
"execution_count": 64,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:18:54.719191Z",
"start_time": "2020-04-13T14:18:49.617393Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 8.59 ms, sys: 9.5 ms, total: 18.1 ms\n",
"Wall time: 5.09 s\n"
]
},
{
"data": {
"text/plain": [
"[0, 1, 4, 9]"
]
},
"execution_count": 64,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"with multiprocessing.pool.ThreadPool(os.cpu_count()) as pool:\n",
" result = pool.map(io_busy, range(4))\n",
"result"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"PythonのマルチスレッドはCPUたくさん使う処理を並行する場合、パフォーマンスの改善はほとんどない。"
]
},
{
"cell_type": "code",
"execution_count": 65,
"metadata": {
"ExecuteTime": {
"end_time": "2020-04-13T14:19:00.923373Z",
"start_time": "2020-04-13T14:18:54.727288Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 5.11 s, sys: 92.1 ms, total: 5.2 s\n",
"Wall time: 6.18 s\n"
]
},
{
"data": {
"text/plain": [
"[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]"
]
},
"execution_count": 65,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"with multiprocessing.pool.ThreadPool(os.cpu_count()) as pool:\n",
" result = pool.map(busy, range(10))\n",
"result"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.4"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment