Skool Scraper API Documentation

API_DOCS.md

Skool Scraper API Documentation

A REST API for scraping member data from Skool communities.

Base URL

https://skool-scraper-production-c8a8.up.railway.app

Overview

Endpoint	Method	Description
/health	GET	Health check
/jobs	POST	Create a scraping job
/jobs/:id	GET	Get job status
/jobs/:id/results	GET	Get scraping results

---

Endpoints

Health Check

GET /health

Simple liveness probe for monitoring.

Response

{
  "status": "ok",
  "timestamp": "2024-01-31T12:34:56.789Z"
}

---

Create Scraping Job

POST /jobs

Creates a new job to scrape member data from a Skool community.

Request Body

Field	Type	Required	Description
communityUrl	string	Yes	Skool community URL (must contain "skool.com")
cookies	string | array	Yes	Authentication cookies

Cookie Formats

Option 1: HTTP Header String

{
  "communityUrl": "https://www.skool.com/my-community",
  "cookies": "auth_token=xyz123; session_id=abc456"
}

Option 2: Cookie Editor JSON Export

{
  "communityUrl": "https://www.skool.com/my-community",
  "cookies": [
    { "name": "auth_token", "value": "xyz123", "domain": ".skool.com" },
    { "name": "session_id", "value": "abc456" }
  ]
}

Response (202 Accepted)

{
  "jobId": "550e8400-e29b-41d4-a716-446655440000",
  "status": "pending",
  "statusUrl": "/jobs/550e8400-e29b-41d4-a716-446655440000"
}

Errors

Status	Description
400	Invalid URL or missing cookies

{
  "error": "Validation failed",
  "details": [
    { "path": "communityUrl", "message": "Must be a valid URL" },
    { "path": "cookies", "message": "Cookies are required" }
  ]
}

---

Get Job Status

GET /jobs/:id

Retrieve the current status of a scraping job.

Path Parameters

Parameter	Type	Description
id	UUID	Job identifier

Response (200 OK)

{
  "jobId": "550e8400-e29b-41d4-a716-446655440000",
  "status": "complete",
  "createdAt": "2024-01-31T12:00:00.000Z",
  "startedAt": "2024-01-31T12:00:05.000Z",
  "completedAt": "2024-01-31T12:05:30.000Z",
  "error": null,
  "hasResults": true
}

Status Values

Status	Description
pending	Job queued, not started
running	Scraping in progress
complete	Finished successfully
failed	Encountered an error

Errors

Status	Description
400	Invalid UUID format
404	Job not found

---

Get Job Results

GET /jobs/:id/results

Retrieve scraped member data from a completed job.

Path Parameters

Parameter	Type	Description
id	UUID	Job identifier

Response (200 OK)

{
  "members": [...],
  "totalCount": 42,
  "scrapedAt": "2024-01-31T12:05:30.000Z",
  "partial": false,
  "error": null
}

Member Object Schema

{
  "name": "John Doe",
  "username": "johndoe",
  "profileUrl": "https://www.skool.com/community/members/johndoe",
  "avatarUrl": "https://cdn.example.com/avatar.jpg",
  "memberSince": "2023-06-15",
  "bio": "Software engineer",
  "location": "San Francisco, CA",
  "websiteUrl": "https://johndoe.com",
  "level": 5,
  "tier": "premium",
  "socialLinks": {
    "twitter": "https://twitter.com/johndoe",
    "linkedin": "https://linkedin.com/in/johndoe",
    "instagram": "https://instagram.com/johndoe",
    "facebook": "https://facebook.com/johndoe",
    "youtube": "https://youtube.com/@johndoe"
  },
  "email": "john@example.com",
  "role": "group-member",
  "points": 1250,
  "ltv": 49.99,
  "invitedBy": {
    "userId": "user123",
    "name": "Jane Smith",
    "username": "janesmith"
  },
  "surveyAnswers": [
    { "question": "What is your profession?", "answer": "Software Engineer" }
  ],
  "billing": {
    "currency": "USD",
    "amount": 49.99,
    "interval": "month",
    "model": "subscription"
  }
}

Member Fields

Field	Type	Description
name	string	Display name
username	string	Skool username
profileUrl	string	Profile URL
avatarUrl	string?	Avatar image URL
memberSince	string?	Join date (ISO format)
bio	string?	User bio
location	string?	Location
websiteUrl	string?	Website URL
level	number?	Engagement level (1-9)
tier	string?	Membership tier: free, standard, premium, vip
socialLinks	object?	Social media URLs

Admin-Only Fields (requires admin cookies)

Field	Type	Description
email	string?	Member's email
role	string?	Role: group-owner, group-admin, group-moderator, group-member
points	number?	Engagement points
ltv	number?	Lifetime value (dollars)
invitedBy	object?	Referrer info
surveyAnswers	array?	Survey responses
billing	object?	Payment details (amount in dollars)

Errors

Status	Description
400	Job not complete, no results, or invalid UUID
404	Job not found

{ "error": "Job not yet complete" }

---

Usage Example

# 1. Create a scraping job
curl -X POST https://skool-scraper-production-c8a8.up.railway.app/jobs \
  -H "Content-Type: application/json" \
  -d '{
    "communityUrl": "https://www.skool.com/my-community",
    "cookies": "auth_token=xyz123; session_id=abc456"
  }'

# Response: { "jobId": "abc-123", "status": "pending", "statusUrl": "/jobs/abc-123" }

# 2. Poll job status
curl https://skool-scraper-production-c8a8.up.railway.app/jobs/abc-123

# 3. Get results when complete
curl https://skool-scraper-production-c8a8.up.railway.app/jobs/abc-123/results

---

Error Format

Standard Error

{
  "error": "Error message"
}

Validation Error

{
  "error": "Validation failed",
  "details": [
    { "path": "fieldName", "message": "Validation message" }
  ]
}

---

Configuration

Variable	Default	Description
PORT	3000	Server port
NODE_ENV	development	Environment
LOG_LEVEL	info	Log level
JOB_TTL_SECONDS	3600	Job expiration (1 hour)

---

Notes

• Authentication: No API authentication. Secure via reverse proxy if needed.
• Storage: In-memory only. Jobs lost on restart.
• Rate Limiting: Internal scraping delays (300ms + jitter) prevent overloading Skool.
• Timeouts: Jobs expire after 1 hour by default.