Skip to content

Instantly share code, notes, and snippets.

@oneryalcin

oneryalcin/call_center_monitoring.md

Last active November 9, 2025 22:40
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save oneryalcin/f7fce2184b046285fbf8a02aa9e1f9ff to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save oneryalcin/f7fce2184b046285fbf8a02aa9e1f9ff to your computer and use it in GitHub Desktop.
Download ZIP
Call Center Monitoring Report
Raw
call_center_monitoring.md

Network Latency Analysis Report

Call Center Connectivity to AWS US-East-2 (Ohio)

Monitoring Period: November 4-9, 2025

Executive Summary

Call center users in Pristina (Kosovo) and Diber (North Macedonia) reported app quality degradation on Saturday, November 8, 2025 at 6:00 PM UTC, with brief improvement around 6:30 PM, followed by recurring issues around 8:00 PM. This report analyzes network latency data collected from RIPE Atlas probes monitoring both call center network paths, identifying the specific network segments causing degradation.

Key Finding: Two distinct network paths showed congestion during Saturday evening peak hours:

  • Pristina paths: Cogent European backbone congestion (Akton exit) and AWS-side routing issues (IPKO exit)
  • Diber paths: RETN transatlantic link congestion (Makedonski exit), clean performance (Akton exit)

Call Center Network Configuration

Pristina Call Center

Location: Kosovo Public IPs: 185.191.167.236 (TelKos), 46.99.212.83 (IPKO) ISP Exits: Dual-homed

Monitoring Probes:

  • Probe 61581 (XK) - AS21246 IPKO (direct monitoring)
  • Probe 27588 (AL) - AS42313 ONE Albania (proxy for TelKos path - no RIPE probe available, BGP shows IPKO peers with Albanian providers)

Diber Call Center

Location: North Macedonia Public IPs: 62.162.126.10 (Makedonski), 82.214.85.253 (Akton) ISP Exits: Dual-homed

Monitoring Probes:

  • Probe 62906 (MK) - AS6821 Makedonski Telekom (direct monitoring)
  • Probe 53811 (HR) - AS25467 Akton (proxy via Croatia - only live RIPE probe on Akton network)

Measurement Setup

RIPE Atlas Measurements

Ping Measurements (5-minute interval)

  • ID 135679657: US-East-1 (Virginia) - 54.239.28.168
  • ID 135679662: US-East-2 (Ohio) - 52.95.16.123

Traceroute Measurements (15-minute interval)

  • ID 135679668: US-East-1 (Virginia) - 54.239.28.168
  • ID 135679669: US-East-2 (Ohio) - 52.95.16.123

HTTP Measurement (30-minute interval)

  • ID 135684053: Leesburg VA (Wikimedia anchor) - 208.80.155.69

Data Access

Raw measurement data is publicly accessible via RIPE Atlas API:

https://atlas.ripe.net/api/v2/measurements/{MEASUREMENT_ID}/results/
  ?start={UNIX_TIMESTAMP_START}
  &stop={UNIX_TIMESTAMP_STOP}
  &probe_ids={PROBE_ID}
  &format=json

Example: Fetch Albania probe traceroute data for Nov 6, 17:00-18:00 UTC:

https://atlas.ripe.net/api/v2/measurements/135679668/results/
  ?start=1762448400&stop=1762452000&probe_ids=27588

Saturday Evening Analysis (November 8, 6PM-9PM UTC)

Observed User Impact Timeline

Time (UTC) User Report Probe Detection
18:00 App quality degradation begins HR probe: 240ms spike (2x baseline)
18:30 Brief improvement reported HR probe: 123ms (normal)
19:53-20:08 Second wave of issues HR probe: 268ms spike, XK: 256ms spike
21:00 Issues resolved All probes return to baseline

Spike Statistics by Probe

Probe Location Call Center ISP Exit Spikes Baseline Peak Path
53811 HR Diber Akton 3 123ms 268ms Akton→Cogent→AWS
61581 XK Pristina IPKO 1 148ms 256ms IPKO→Telekom SI→DTAG→AWS
62906 MK Diber Makedonski 0 122ms 160ms* Makedonski→RETN→AWS
27588 AL Pristina TelKos (proxy) 0 130ms 130ms ONE Albania→Sparkle→AWS

*Sustained +38ms elevation, no discrete spikes

Detailed Analysis by Probe

1. Probe 53811 (Croatia/HR) - Monitoring Diber Akton Exit

Call Center: Diber, North Macedonia ISP Exit: 82.214.85.253 (AS25467 Akton) Probe Location: Croatia (only live RIPE probe on Akton network)

Network Path:

Diber (Akton AS25467, Slovenia) → Cogent (AS174, Frankfurt)
  → Cogent European Backbone → Cogent North America → AWS Ohio

Congestion Location Identified:

Hop Location Network Baseline RTT Spike RTT Delta
7-9 Cogent Entry AS174 2.8ms ~3ms Stable
10 Warsaw, PL AS174 Cogent 16.2ms ~16ms Stable
11 Frankfurt, DE AS174 Cogent 21.9ms 30.3ms +8.4ms
12 Amsterdam, NL AS174 Cogent 28.5ms ~29ms Stable
13 Liverpool, GB AS174 Cogent 37.8ms ~38ms Stable
14 Montreal, CA AS174 Cogent 107.2ms 107.4ms Stable
15 Plattsburgh NY AS174 Cogent 107.9ms ~108ms Stable

Root Cause: Cogent European backbone congestion in Frankfurt-Amsterdam segment (Hop 11-12) during European evening peak hours (7-9 PM CET / 6-8 PM UTC).

Technical Details:

  • Transatlantic crossing (Liverpool→Montreal) is stable at 69ms, ruling out submarine cable issues
  • The additional +140ms spike occurs before the transatlantic segment (Frankfurt / Amsteerdam leg most likely)
  • Spikes are transient (5-10 minute duration) indicating capacity saturation, not routing changes
  • HTTP measurements to Wikimedia Leesburg VA showed no degradation (208ms stable), confirming path-specific issue

Supporting Evidence:

  • Zero routing changes observed across 200+ traceroutes
  • Path consistent: 100% via Cogent AS174
  • Cogent is known low-cost provider with frequent capacity issues
  • Industry reports document Cogent evening congestion patterns

Access Raw Data:

# Baseline (Nov 7, 18:00-19:00)
https://atlas.ripe.net/api/v2/measurements/135679669/results/?start=1762354800&stop=1762358400&probe_ids=53811

# Spike (Nov 8, 19:53-20:00)
https://atlas.ripe.net/api/v2/measurements/135679669/results/?start=1762447980&stop=1762448400&probe_ids=53811

2. Probe 61581 (Kosovo/XK) - Monitoring Pristina IPKO Exit

Call Center: Pristina, Kosovo ISP Exit: 46.99.212.83 (AS21246 IPKO) Probe Location: Kosovo (direct monitoring on IPKO network)

Network Path:

Pristina (IPKO AS21246) → Telekom Slovenije (AS5603)
  → Deutsche Telekom (AS3320) → AWS Ohio

Congestion Location Identified:

Segment Baseline Peak Delta Location
Germany exit 31ms 31ms 0ms Europe
Transatlantic crossing 95-97ms 95-97ms 0ms DTAG cable
NYC → AWS Ohio 21ms 43ms +22ms North America

Root Cause: AWS-side routing congestion between Deutsche Telekom's NYC landing point and AWS Ohio data center during North American evening peak (6-8 PM EST / 11PM-1AM UTC).

Technical Details:

  • European and transatlantic segments completely stable (No issues on Deutsche Telecom)
  • Delay occurs entirely in US segment (Hop 9 → Destination)
  • Only 1 spike detected (20:08 UTC, 256ms)
  • Quick recovery within 15 minutes

Why Less Impact than HR Probe:

  • Superior transit provider (Deutsche Telekom vs Cogent)
  • Congestion occurs at destination end, not source/transit
  • Different peering arrangement with AWS

Access Raw Data:

# Baseline (Nov 4, 18:00-22:00)
https://atlas.ripe.net/api/v2/measurements/135679669/results/?start=1762095600&stop=1762110000&probe_ids=61581

# Spike window (Nov 8, 20:00-21:00)
https://atlas.ripe.net/api/v2/measurements/135679669/results/?start=1762448400&stop=1762452000&probe_ids=61581

3. Probe 62906 (North Macedonia/MK) - Monitoring Diber -> Makedonski Exit

Call Center: Diber, North Macedonia ISP Exit: 62.162.126.10 (AS6821 Makedonski Telekom) Probe Location: North Macedonia (direct monitoring on Makedonski network)

Network Path:

Diber (Makedonski Telekom AS6821) → Novatel Bulgaria (AS41313)
  → RETN Sofia (AS9002) → RETN NYC → AWS Ohio

Congestion Location Identified:

Segment Nov 7 (Baseline) Nov 8 (Evening) Delta
Sofia entry (Hop 7) 9.45ms 12.79ms +3.34ms
Sofia → NYC transatlantic 93.26ms 127.24ms +33.98ms
NYC → AWS Ohio 19.51ms 20.43ms +0.92ms

Root Cause: RETN (AS9002) transatlantic link congestion on Sofia→NYC route during evening peak hours.

Technical Details:

  • Path is 100% consistent: analyzed 1,369 traceroutes over 6 days, zero routing changes
  • Single-homed to RETN with no alternative routes
  • RETN uses Route #1: Sofia → Budapest → Frankfurt → Amsterdam → London → AEConnect submarine cable → NYC
  • Measured RTT matches RETN's published 92ms baseline exactly
  • +34ms increase specifically on transatlantic segment visible in traceroute data
  • No discrete spikes, but sustained +38ms elevation from 122ms to 160ms baseline

RETN Network Context:

  • Tier 2 ISP (purchases transit from Telia, Level3, GTT, NTT, TATA, Orange)
  • Uses AEConnect submarine cable (100+ Tbps capacity)
  • No major outages in 2024-2025, generally reliable
  • Likely congestion at European aggregation points (Frankfurt/Amsterdam) before transatlantic crossing

Impact:

  • No user-reported issues (sustained 160ms within acceptable VoIP threshold)
  • Lower priority than HR/XK probe spikes which reached 250-270ms

Access Raw Data:

# Full day comparison
# Nov 7: https://atlas.ripe.net/api/v2/measurements/135679669/results/?start=1762290000&stop=1762376400&probe_ids=62906
# Nov 8: https://atlas.ripe.net/api/v2/measurements/135679669/results/?start=1762376400&stop=1762462800&probe_ids=62906

4. Probe 27588 (Albania/AL) - Monitoring Pristina TelKos Exit (Proxy)

Call Center: Pristina, Kosovo ISP Exit: 185.191.167.236 (AS206262 TelKos) Probe Location: Albania (proxy - no RIPE probe available on TelKos, BGP shows IPKO peers with Albanian providers)

Network Path:

Albania (ONE Albania AS42313) → Sparkle Milan (AS6762)
  → Sparkle Ashburn VA (AS6762) → AWS Ohio

Observed Performance:

  • Nov 8 evening: 130ms baseline maintained, zero spikes to US-East-2 (Ohio)
  • Nov 5/6 evening: Severe issues to US-East-1 (Virginia) - separate incident

Note: This probe serves as proxy for Pristina TelKos exit path based on BGP peering relationships.

US-East-2 (Ohio) Path Analysis:

Segment RTT Status
Milan (Hop 6) 20ms Stable
Transatlantic (Milan→Ashburn) 107ms Stable
Ashburn → AWS Ohio ~3ms Stable

Why Clean to Ohio:

  • Direct Sparkle network path, minimal peering points
  • Sparkle (AS6762) is Tier 1 ISP with good AWS peering
  • Different destination than problematic Virginia endpoint

Note on US-East-1 Issue (Nov 5-6):

  • Separate analysis required
  • Nov 5: 4.5-hour degradation (17:00-21:30), 130ms → 285-382ms
  • Nov 6: 4.8-hour degradation (16:58-21:48), 118ms → 300-370ms
  • Root cause: ICMP deprioritization in Sparkle network to Virginia
  • TCP traffic unaffected (traceroute showed 125ms during 300ms ping spikes)
  • Path: Ashburn VA → AWS Virginia had queuing issues for ICMP packets

Access Raw Data:

# Clean performance to Ohio (Nov 8)
https://atlas.ripe.net/api/v2/measurements/135679662/results/?start=1762376400&stop=1762462800&probe_ids=27588

# Problematic US-East-1 data (Nov 5)
https://atlas.ripe.net/api/v2/measurements/135679657/results/?start=1762204000&stop=1762290400&probe_ids=27588

Conclusions and Recommendations

Root Cause Summary

Pristina Call Center (Saturday 6-8 PM):

  • TelKos exit (via AL probe proxy): Clean performance, Sparkle path - 130ms stable ✅
  • IPKO exit (XK probe): Minor AWS-side routing congestion - 1 spike to 256ms
  • Impact: Minimal on TelKos path, minor intermittent issues on IPKO path

Diber Call Center:

  • Akton exit (HR probe): Severe Cogent European backbone congestion - 3 spikes to 240-268ms ❌
  • Makedonski exit (MK probe): RETN transatlantic sustained elevation (+38ms to 160ms)
  • Impact: Diber users experienced degradation primarily on Akton path

Recommendations

Immediate (Diber):

  1. Prioritize Makedonski exit during evening hours (160ms RETN better than 268ms Cogent spikes)
  2. Avoid Akton exit during 6-9 PM UTC (Cogent congestion window)

Immediate (Pristina):

  1. Prioritize TelKos exit (best performance, 130ms stable via Sparkle, though verify the path with traceorute from call center)
  2. IPKO exit as backup (generally good, minor AWS-side issues)

Path Quality Ranking (Best to Worst)

For AWS US-East-2 (Ohio) during evening hours:

Pristina Call Center:

  1. TelKos exit (AL probe proxy) - Sparkle path: 130ms, zero issues ✅
  2. IPKO exit (XK probe) - Deutsche Telekom: 148ms, 1 minor spike

Diber Call Center:

  1. Makedonski exit (MK probe) - RETN: 160ms, sustained elevation
  2. Akton exit (HR probe) - Cogent: 123-268ms, recurring spikes ❌

Data Methodology

Collection Period: November 4-9, 2025 (6 days)

Measurements:

  • Ping: 1,728 measurements per probe per day (5-min interval)
  • Traceroute: 96 measurements per probe per day (15-min interval)
  • Total dataset: ~40,000+ measurements across 4 probes

Analysis Tools:

  • RIPE Atlas Cousteau Python library
  • DuckDB for time-series analysis
  • Parquet data format for efficient querying
  • ipinfo.io for IP geolocation

Verification:

  • All findings cross-referenced with traceroute hop-by-hop RTT data
  • Multiple time windows analyzed to confirm patterns
  • Baseline comparisons with previous days
  • HTTP measurements used to validate path-specific issues

Appendix: Accessing Raw Data

Quick Start

Install Python dependencies:

pip install ripe.atlas.cousteau ripe.atlas.sagan

Fetch and analyze measurement:

from ripe.atlas.cousteau import AtlasResultsRequest
from datetime import datetime, timezone

# Nov 8, 19:53 spike (HR probe to Ohio)
start = int(datetime(2025, 11, 8, 19, 50, tzinfo=timezone.utc).timestamp())
stop = int(datetime(2025, 11, 8, 20, 10, tzinfo=timezone.utc).timestamp())

req = AtlasResultsRequest(
    msm_id=135679669,  # Traceroute to Ohio
    start=start,
    stop=stop,
    probe_ids=[53811]   # HR probe
)
status, results = req.get()

if status:
    print(f"Found {len(results)} traceroutes during spike window")

All Measurement URLs

Pristina - HR Probe (53811)

  • Ping Ohio: https://atlas.ripe.net/api/v2/measurements/135679662/results/?probe_ids=53811
  • Traceroute Ohio: https://atlas.ripe.net/api/v2/measurements/135679669/results/?probe_ids=53811

Pristina - XK Probe (61581)

  • Ping Ohio: https://atlas.ripe.net/api/v2/measurements/135679662/results/?probe_ids=61581
  • Traceroute Ohio: https://atlas.ripe.net/api/v2/measurements/135679669/results/?probe_ids=61581

Debar - MK Probe (62906)

  • Ping Ohio: https://atlas.ripe.net/api/v2/measurements/135679662/results/?probe_ids=62906
  • Traceroute Ohio: https://atlas.ripe.net/api/v2/measurements/135679669/results/?probe_ids=62906

Debar - AL Probe (27588)

  • Ping Ohio: https://atlas.ripe.net/api/v2/measurements/135679662/results/?probe_ids=27588
  • Traceroute Ohio: https://atlas.ripe.net/api/v2/measurements/135679669/results/?probe_ids=27588

Add time filters: &start={unix_timestamp}&stop={unix_timestamp}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment