Freenet Telemetry Monitor skill for Claude Code

SKILL.md

name

description

freenet-telemetry-monitor

Access and analyze Freenet peer telemetry from the central OpenTelemetry collector on nova.locut.us. This skill should be used when debugging network issues, analyzing peer behavior, investigating operation failures, or monitoring network health across the Freenet network. Invoke when the user asks about network telemetry, peer events, operation traces, or needs to debug issues like LEDBAT death spiral, connection problems, or contract operation failures.

Freenet Telemetry Monitor

This skill provides access to the centralized Freenet telemetry system for debugging network issues and monitoring peer behavior.

Overview

Freenet peers send operation events to a central OpenTelemetry collector. This enables debugging distributed issues by correlating events across multiple peers.

Server Infrastructure

• Host: nova.locut.us (5.9.111.215)
• OTLP/HTTP endpoint: port 4318 (used by peers)
• Health check: port 13133
• Service: freenet-telemetry.service (systemd)
• Log directory: /mnt/media/freenet-telemetry/ (primary, where otel collector writes)
• Legacy log directory: /var/log/freenet-telemetry/ (stale — no longer written to since config change)

Accessing Telemetry

Check Server Health

curl http://5.9.111.215:13133/health

Access Log Files

Note: If running on nova already, use local commands. Otherwise use SSH.

Log files are owned by freenet user/group with 640 permissions. Members of the freenet group (ian, nacho) can read them directly:

# List log files (on nova)
ls -la /mnt/media/freenet-telemetry/

# View recent logs (logs.jsonl is the main file)
tail -100 /mnt/media/freenet-telemetry/logs.jsonl

# If accessing remotely via SSH:
ssh nova "tail -100 /mnt/media/freenet-telemetry/logs.jsonl"

Log Format

Logs are in OTLP JSON Lines format. Each line is a batch of log records with this structure:

{
  "resourceLogs": [{
    "resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "freenet-peer"}}]},
    "scopeLogs": [{
      "scope": {"name": "freenet.telemetry"},
      "logRecords": [
        {
          "timeUnixNano": 1735500000000000000,
          "body": {"stringValue": "{...event_data...}"},
          "attributes": [
            {"key": "peer_id", "value": {"stringValue": "4abc123..."}},
            {"key": "transaction_id", "value": {"stringValue": "01ABC..."}},
            {"key": "event_type", "value": {"stringValue": "put_request"}}
          ]
        }
      ]
    }]
  }]
}

Event Types

Connection Events

• connect - Peer connection events (start_connection, connected, finished)
• disconnect - Peer disconnection

Contract Operations

• put_request - PUT operation initiated

• put_success - PUT completed successfully

• put_broadcast_emitted - PUT broadcast sent to subscribers

• put_broadcast_received - PUT broadcast received

• get_success - GET operation completed

• update_request - UPDATE operation initiated

• update_success - UPDATE completed successfully

• update_broadcast_emitted - UPDATE broadcast sent

• update_broadcast_received - UPDATE broadcast received

• subscribed - Subscription registered

Other Events

• timeout - Operation timed out
• route - Routing events
• ignored - Ignored events

Common Analysis Queries

Find all events for a specific transaction

ssh nova "cat /mnt/media/freenet-telemetry/logs.jsonl | jq -c '.resourceLogs[].scopeLogs[].logRecords[] | select(.attributes[] | select(.key==\"transaction_id\" and .value.stringValue==\"01ABC123...\"))'"

Count events by type

ssh nova "cat /mnt/media/freenet-telemetry/logs.jsonl | jq -r '.resourceLogs[].scopeLogs[].logRecords[].attributes[] | select(.key==\"event_type\") | .value.stringValue' | sort | uniq -c | sort -rn"

Find events for a specific peer

ssh nova "cat /mnt/media/freenet-telemetry/logs.jsonl | jq -c '.resourceLogs[].scopeLogs[].logRecords[] | select(.attributes[] | select(.key==\"peer_id\" and (.value.stringValue | contains(\"4abc\"))))'"

Find timeout events

ssh nova "grep '\"event_type\":\"timeout\"' /mnt/media/freenet-telemetry/logs.jsonl | jq ."

Get events from last N minutes

# Get events from last 10 minutes (adjust timestamp calculation)
ssh nova "cat /mnt/media/freenet-telemetry/logs.jsonl | jq -c '.resourceLogs[].scopeLogs[].logRecords[] | select(.timeUnixNano > (now * 1000000000 - 600000000000))'"

Find failed operations (no success after request)

To diagnose stuck operations, look for request events without corresponding success events by transaction ID.

Using the Analysis Script

A convenience script is provided for common queries:

# Run the analysis script
/home/ian/.claude/skills/freenet-telemetry-monitor/scripts/analyze-telemetry.sh <command> [args]

# Commands:
#   health        - Check server health
#   recent [N]    - Show last N events (default 20)
#   count         - Count events by type
#   tx <id>       - Find events for transaction ID
#   peer <prefix> - Find events for peer ID prefix
#   timeouts      - List timeout events

CRITICAL: Avoid Analyzing Stale Historical Data

The telemetry log accumulates ALL events since collection started. When analyzing network state, you MUST filter to recent data or you will get wildly misleading results.

Common Mistake

# WRONG - analyzes ALL historical data (could show 1000+ "peers" that are long gone)
ssh nova "cat /mnt/media/freenet-telemetry/logs.jsonl | grep 'some_contract'"

# RIGHT - filter to recent data first
ssh nova "tail -5000 /mnt/media/freenet-telemetry/logs.jsonl | grep 'some_contract'"

Best Practices

1. Always determine current network size first: Query recent events to find how many peers are CURRENTLY active (seen in last 10-15 minutes)

2. Filter by timestamp: Events have a timestamp field in the body. Filter to recent timeframes:

   import time
   now = time.time()
   if now - event_timestamp < 3600:  # Last hour only
       process(event)

3. Use tail not cat: Start with tail -N to get recent logs, not the full historical file

4. Validate peer counts: Freenet typically has 10-25 active peers. If your analysis shows hundreds or thousands of peers, you're looking at cumulative historical data, not current state.

Example: Current Network State

# Find currently active peers (last 10 minutes)
ssh nova "tail -5000 /mnt/media/freenet-telemetry/logs.jsonl" | python3 -c "
import json, sys, time
peers = {}
now = time.time()
for line in sys.stdin:
    # ... parse and filter to now - timestamp < 600
"

Debugging Workflow

For Operation Failures

1. Get the transaction ID from peer logs or error messages
2. Query telemetry for all events with that transaction ID
3. Trace the operation path: request -> forwarding -> success/failure
4. Check for timeouts or missing response events

For Connection Issues

1. Filter for connect/disconnect events
2. Look for patterns: rapid connect/disconnect cycles, specific peer failures
3. Correlate with LEDBAT or transport layer issues

For Contract Propagation Issues

1. Find the initial put_request event
2. Trace broadcast_emitted and broadcast_received events
3. Verify the subscription tree is properly receiving updates
4. Check for missing nodes in the broadcast chain

Privacy Notes

Telemetry includes:

• Peer addresses (public info - needed for debugging)
• Contract keys (not contract content)
• Operation timing

Telemetry does NOT include:

• Contract state/delta content
• Private keys or addresses

Related Skills

• freenet-telemetry-dashboard: Web dashboard at nova.locut.us:3133 for visual real-time monitoring. Use this for interactive exploration, topology visualization, and time-travel through events. This skill (telemetry-monitor) is for command-line access to raw logs.
• freenet-diagnostic-reports: Handle user diagnostic reports from freenet service report. Different data source (user-submitted reports vs. network telemetry).