Skip to content

Instantly share code, notes, and snippets.

@blacha
Last active November 6, 2021 22:27
Show Gist options
  • Select an option

  • Save blacha/57114537c2263f77232c76ae56f14d8d to your computer and use it in GitHub Desktop.

Select an option

Save blacha/57114537c2263f77232c76ae56f14d8d to your computer and use it in GitHub Desktop.
Docker GDAL COG creation issues
#!/bin/bash
GDAL_CONTAINER=osgeo/gdal:ubuntu-small-latest
TIF_FOLDER=
docker run -v $PWD:$PWD -it $GDAL_CONTAINER gdalbuildvrt \
$PWD/$TIF_FOLDER.vrt \
$PWD/$TIF_FOLDER/*.tif
docker run -v $PWD:$PWD -it $GDAL_CONTAINER gdal_translate \
-of COG \
-co COMPRESS=WEBP \
-co NUM_THREADS=ALL_CPUS \
-co BIGTIFF=YES \
--config BIGTIFF_OVERVIEW YES \
--config GDAL_CACHEMAX 8196 \
$PWD/$TIF_FOLDER.vrt $PWD/$TIF_FOLDER.webp.cog.tif
Samples: 26K of event 'cpu-clock', Event count (approx.): 260850000000
Children Self Command Shared Object Symbol
+ 69.46% 16.13% gdal_translate ld-musl-x86_64.so.1 [.] memset
+ 56.34% 0.00% gdal_translate [kernel.kallsyms] [k] page_fault
+ 56.34% 0.02% gdal_translate [kernel.kallsyms] [k] do_page_fault
+ 56.17% 24.83% gdal_translate [kernel.kallsyms] [k] __do_page_fault
+ 27.35% 1.38% gdal_translate [kernel.kallsyms] [k] handle_mm_fault
+ 25.52% 2.50% gdal_translate [kernel.kallsyms] [k] __handle_mm_fault
+ 12.14% 0.30% gdal_translate [kernel.kallsyms] [k] alloc_pages_vma
+ 11.50% 0.64% gdal_translate [kernel.kallsyms] [k] __alloc_pages_nodemask
+ 7.92% 0.00% gdal_translate [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 7.92% 0.04% gdal_translate [kernel.kallsyms] [k] do_syscall_64
+ 7.31% 0.02% gdal_translate ld-musl-x86_64.so.1 [.] __munmap
+ 7.26% 0.00% gdal_translate [kernel.kallsyms] [k] __x64_sys_munmap
+ 7.26% 0.00% gdal_translate [kernel.kallsyms] [k] __vm_munmap
+ 7.24% 0.01% gdal_translate [kernel.kallsyms] [k] __do_munmap
+ 6.98% 0.00% gdal_translate [kernel.kallsyms] [k] unmap_region
+ 6.83% 6.78% gdal_translate [kernel.kallsyms] [k] clear_page_erms
+ 6.67% 6.67% gdal_translate [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore
Samples: 16K of event 'cpu-clock', Event count (approx.): 165400000000
Children Self Command Shared Object Symbol
+ 56.69% 0.00% gdal_translate libc-2.27.so [.] 0x00007f30814dbf2d
+ 55.94% 55.94% gdal_translate libc-2.27.so [.] 0x000000000018ef2d
+ 6.58% 0.00% gdal_translate [kernel.kallsyms] [k] page_fault
+ 6.58% 0.00% gdal_translate [kernel.kallsyms] [k] do_page_fault
+ 6.58% 1.51% gdal_translate [kernel.kallsyms] [k] __do_page_fault
+ 4.93% 0.10% gdal_translate [kernel.kallsyms] [k] handle_mm_fault
+ 4.77% 0.33% gdal_translate [kernel.kallsyms] [k] __handle_mm_fault
+ 3.75% 0.00% gdal_translate [unknown] [.] 0000000000000000
+ 3.23% 0.06% gdal_translate [kernel.kallsyms] [k] __alloc_pages_nodemask
+ 2.65% 0.00% gdal_translate [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 2.64% 0.11% gdal_translate [kernel.kallsyms] [k] do_syscall_64
+ 2.13% 0.02% gdal_translate [kernel.kallsyms] [k] __alloc_pages_slowpath
+ 2.12% 0.00% gdal_translate libgdal.so.3.1.0 [.] 0x00007f308246fb74
+ 2.11% 0.00% gdal_translate libgdal.so.3.1.0 [.] 0x00007f308246fb63
+ 2.06% 0.00% gdal_translate libgdal.so.3.1.0 [.] 0x00007f308246fb82
+ 1.91% 0.02% gdal_translate [kernel.kallsyms] [k] alloc_pages_vma
+ 1.86% 0.01% gdal_translate [kernel.kallsyms] [k] do_try_to_free_pages
perf record -g -a -F 100 -e cpu-clock -G docker/${CONTAINER_ID} sleep 90
perf report
@blacha
Copy link
Author

blacha commented Jun 14, 2019

A large porition of time is being spent inside of memset inside of alpine, changing the image to ubuntu fixes the problem..

This would imply there is some issue with musl and gdal

@palmerj
Copy link

palmerj commented Jun 14, 2019

Interesting...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment