danabauer

Customer Problem

Because Overture transportation is heavily-reliant on OpenStreetMap, we need a schema that is flexible enough to model the different data conventions OSM uses to model sidewalks.

OSM has two such data conventions:

1. A sidewalk can itself be a standalone way.
2. Or a sidewalk can be implicitly or explicitly tagged onto a highway=* type way.

As of 2024-03-01, our schema model supports data convention #1, but not data convention #2. Consequently, if a highway=* way has sidewalk information tagged to it, we lose that information. Therefore, Overture customers miss out completely on these sidewalks.

danabauer

#!/bin/bash
# A simple script for converting files between CSV and Parquet formats using DuckDB. Requires DuckDB installation.

convert_file() {
    local input_file="$1"
    local output_extension="$2"

    # Extracting the filename without extension
    local base_name=$(basename -- "$input_file")
    local name="${base_name%.*}"

danabauer

COPY(
SELECT hierarchies
 FROM read_parquet('s3://overturemaps-us-west-2/release/2024-07-22.0/theme=divisions/type=division/*', filename=true, hive_partitioning=1)
WHERE subtype IN ('macrohood', 'neighborhood', 'microhood')
   AND country = 'US'
   AND region = 'US-NY'
   AND names.primary = 'SoHo'
   ) TO 'soho.json';

danabauer

SELECT 
    id,
    names.primary AS primary_name,
    sources[1].dataset AS primary_source,
    ST_GeomFromBinary(geometry) AS geometry
FROM v2024_04_16_beta_0
WHERE type = 'building'
	AND ST_INTERSECTS(
 ST_GeomFromBinary(geometry),

danabauer

Breaking change in Overture's 2024-03-12-alpha.0 release

In previous releases of Overture data, we used camelCase for attribute names throughout the schema. A duckdb query to find Pennsylvania in our Admins dataset looked like this:

LOAD spatial;
LOAD httpfs:

SELECT 
   id, 

danabauer

This query lets us grab the locality (point geometry and attributes) that represents the United States and its associated area, in this case a multipolygon geometry.

CREATE VIEW admins_view AS (
    SELECT
        *
    FROM
        read_parquet('s3://overturemaps-us-west-2/release/2024-02-15-alpha.0/theme=admins/type=*/*', filename=true, hive_partitioning=1)COPY (
COPY (
SELECT

danabauer

Author: Sean Gillies Version: 1.0

Abstract

This document describes a GeoJSON-like protocol for geo-spatial (GIS) vector data.

Introduction

danabauer

Hey folks! Yesterday I listened to the Overture Maps Birds of a Feather session at Mapping USA and typed up some quick impressions. Jennings led the session and Ben Clark and Marc participated. It was a good discussion that drew lots of interesting questions, and I wanted to share those here.

---

Jennings spoke during the second day of Mapping USA, a free, virtual conference organized by OpenStreetMap US. There were about 40 people listening on Zoom, a mix of OpenStreetMap members and contributors as well as other folks with a more casual interest in OSM. Jennings stepped through slides, but the session was very interactive, and people asked questions throughout. A different audience might not care so much about how the sausage is made, but this group wanted to hear how Overture assembles its data products and how OSM data flows into Overture’s processing pipelines.

Throughout the discussion, Jennings hit on three main selling points:

• Overture is doing the hard work of data

danabauer

/* In the initial implementation of the admins schema the "locality" feature type was 
allowed to have Point, Polygon, or MultiPolygon geometry. There was no "locality area." So back
in July, this is how you would get country borders. See Simon Willison's post: 
https://til.simonwillison.net/overture-maps/overture-maps-parquet. Note the geometry type checking.*/
   
COPY (
    SELECT
           type,
           subType,
           localityType,

danabauer

INSTALL spatial;
INSTALL httpfs;

LOAD spatial;
LOAD httpfs;
SET s3_region='us-west-2';


COPY(
    SELECT