Create GOES composite imagery products in Python using SatPy and data from public AWS S3 buckets.

GOES Composites with SatPy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e45cfd65-e389-44c7-9aeb-76241f1f22ff",
   "metadata": {},
   "source": [
    "# GOES Composites with SatPy\n",
    "\n",
    "Recently, some GOES composites stopped being distributed over the NOAAPort satellite broadcast. Fortunately, SatPy can readily generate these composites from the data available in the GOES 16-19 AWS S3 buckets. The S3 access here through SatPy will require the use of h5netcdf with xarray."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39a21815-8655-4ea7-b7b8-5850d5529209",
   "metadata": {},
   "source": [
    "## Getting data\n",
    "\n",
    "One option passes directly to SatPy a glob for filenames within the correct path in the S3 bucket. For this method, you need to specify an exactly correct time corresponding to the data, since it's using pattern matching with the wildcards to find the appopriate data files in the bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "143e6715-6517-4690-a99e-26c2a91e539e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "dt = datetime(2025, 4, 11, 15, 1)\n",
    "fnames = [f's3://noaa-goes19/ABI-L1b-RadC/{dt:%Y/%j/%H}/*_s{dt:%Y%j%H%M}*.nc']\n",
    "read_kw = {'storage_options': {'anon': True}}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aed97308-0113-4142-9519-f5048dfd1b38",
   "metadata": {},
   "source": [
    "Another option, with MetPy 1.7.0 (as of this writing coming any day now), is to use the archive client to find the URLs to the various channel data. We grab URLs for all the channels, though the needed data will only be fully loaded based on the requested composite."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0d8a67e1-51d4-4de4-af41-ad72f3dbcf77",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "from metpy.remote import GOESArchive\n",
    "\n",
    "dt = datetime(2025, 4, 10, 23, 30)\n",
    "fnames = [GOESArchive(19).get_product('ABI-L1b-RadC', dt=dt, band=band).url\n",
    "          for band in range(1, 17)]\n",
    "read_kw = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54722b57-e937-4a68-90b9-84be4df470af",
   "metadata": {},
   "source": [
    "With the data in hand, we can get SatPy to load them into its `Scene` object, request the composite (which triggers a load of all needed bands), and resample to a common desired grid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "dc192dac-06ca-485c-97e8-b9358456f7cc",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/rmay/miniconda3/envs/py313/lib/python3.13/site-packages/satpy/readers/generic_image.py:83: UserWarning: The specified chunks separate the stored chunks along dimension \"x\" starting at index 4096. This could degrade performance. Instead, consider rechunking after loading.\n",
      "  data = xr.open_dataset(self.finfo[\"filename\"], engine=\"rasterio\",\n",
      "cannot convert float NaN to integer\n",
      "The following datasets were not created and may require resampling to be generated: DataID(name='geo_color')\n",
      "Fill value incompatible with integer data using 255 instead.\n",
      "/Users/rmay/miniconda3/envs/py313/lib/python3.13/site-packages/trollimage/xrimage.py:1503: RuntimeWarning: divide by zero encountered in scalar divide\n",
      "  scale_factor = 1.0 / (cmap_max - cmap_min)\n",
      "/Users/rmay/miniconda3/envs/py313/lib/python3.13/site-packages/trollimage/xrimage.py:1504: RuntimeWarning: invalid value encountered in scalar multiply\n",
      "  offset = -cmap_min * scale_factor\n"
     ]
    }
   ],
   "source": [
    "from satpy import Scene\n",
    "\n",
    "scn = Scene(filenames=fnames, reader='abi_l1b', reader_kwargs=read_kw)\n",
    "\n",
    "# Name of requested composite. Full list: https://github.com/pytroll/satpy/blob/main/satpy/etc/composites/abi.yaml\n",
    "# Notable options: true_color, geo_color, airmass, dust, snow_fog\n",
    "composite = 'geo_color'\n",
    "\n",
    "# Trigger loading of all channels needed\n",
    "scn.load([composite])\n",
    "\n",
    "# Resample channels to common GOES East Conus grid at 1km resolution.\n",
    "# More grids available at https://satpy.readthedocs.io/en/latest/resample.html#area-definitions-included-in-satpy\n",
    "new_scn = scn.resample('goes_east_abi_c_1km')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50caed98-413b-44f7-b39b-2b714c1788bb",
   "metadata": {},
   "source": [
    "After this we have the option to save the composite to a PNG image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "168f2b78-f6dd-425e-ae48-0a2711dccd17",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/rmay/miniconda3/envs/py313/lib/python3.13/site-packages/dask/_task_spec.py:755: RuntimeWarning: invalid value encountered in sin\n",
      "  return self.func(*new_argspec)\n",
      "/Users/rmay/miniconda3/envs/py313/lib/python3.13/site-packages/dask/_task_spec.py:755: RuntimeWarning: invalid value encountered in cos\n",
      "  return self.func(*new_argspec)\n"
     ]
    }
   ],
   "source": [
    "# Save true color to PNG image\n",
    "new_scn.save_dataset(composite, filename=f'GOES-{composite}.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "318623bc-ca05-4dd2-b999-3746b45d0540",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "Image(f'GOES-{composite}.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a7934ca-000b-4a8a-a846-5c0e510739d5",
   "metadata": {},
   "source": [
    "Or we can also generate a CF-metadata-compliant netCDF-4 file with zlib compression applied."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "dd40f8be-eade-488d-8275-eb5ba8c59831",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[None]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Save to CF-compliant netCDF file with zlib compression\n",
    "new_scn.save_dataset(composite, filename=f'GOES-{composite}.nc',\n",
    "                     encoding={composite: {'compression': 'zlib', 'complevel': 1}})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75b24cbc-4019-4f2f-9f04-71f2580c8c81",
   "metadata": {},
   "source": [
    "Written by Ryan May for Unidata 4/11/2025"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:py313]",
   "language": "python",
   "name": "conda-env-py313-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}