Skip to content
For training pipelines network, not dataset

LLM Training Data Collection at Scale and in Volume

A pretraining corpus is a throughput problem, a diversity problem and a provenance problem. ProxyPerk supplies the egress layer: 200-country residential, 35-country mobile, named ISPs and datacenter pools behind one endpoint, with sticky sessions and an API that lets you pin one identity per domain.

corpus_env.py Python
import os
import requests

# session string from your dashboard
SESSION = "<paste BASE64_CONFIG>"
GATE = "gate.proxyperk.com:8080"
AUTH = "user:" + SESSION
PROXY = f"http://{AUTH}@{GATE}"

# requests, httpx and most loaders
os.environ["HTTPS_PROXY"] = PROXY
os.environ["HTTP_PROXY"] = PROXY

r = requests.get(
    "https://example.com",
    timeout=30
)
print(len(r.text))
standard proxy env vars no SDK
200 countries
35 mobile countries
99.99% uptime SLA
1 Gbps+ ports
7-day instant cash back Volume discounts to 12% Live 24h abuse monitoring

Works with your stack

Python Node.js JavaScript TypeScript Go PHP Ruby Rust C++ Kotlin Swift Selenium Scrapy cURL Pytest Docker Kubernetes GitHub PostgreSQL Redis

The difference

Why a Training Crawl Is Not a Normal Crawl

A product page scrape runs for minutes. A corpus run goes for weeks, across millions of URLs, and has to be reproducible six months later. That changes what “good” means.

Sustained volume

Hundreds of gigabytes, not megabytes. Banned IPs mid-run cost more than the bandwidth does.

Geographic spread

A corpus collected from one country encodes one market. Diversity needs real exits in many countries.

Reproducibility

Pin one identity per domain so a re-crawl sees the same thing and your diffs mean something.

Provenance

You store the data, so you keep the licence and retention decisions. We supply the network only.

Modelling

Size the Run Before You Start

Rotating pools are billed per GB, so bandwidth is the budget line. A rough planning figure is about four bytes of transfer per token of raw text — enough to model the order of magnitude, not to quote a bill.

Corpus size Approx. transfer Residential $3.95 Datacenter $1.90
10 million tokens ~40 GB $158 $76
100 million tokens ~400 GB $1,580 $760
1 billion tokens ~4 TB $15,800 $7,600
10 billion tokens ~40 TB $158,000 $76,000

Estimates only, at list price before discounts. Real transfer is higher than the text itself once retries, HTML overhead and images are counted. Stacking 12 months (15%) with the 100 GB tier (12%) takes 27% off the top.

Diversity

Coverage You Can Actually Select

Diversity is not a slogan if you can name the country, the city and the carrier in the session string. These are the pools available for a corpus run.

200

Residential countries

Country targeting plus sticky sessions. $3.95/GB

35

Mobile countries

Carrier NAT, the hardest class to fingerprint. $9.90/GB

23

Named ISP carriers

Static, city-level, unlimited bandwidth. from $2.00/IP

Practical pattern: run the bulk of the corpus on rotating datacenter where the target is an open API or documentation mirror, take protected long-tail domains on rotating residential, and hold static ISP IPs for the handful of sites that require a logged-in or returning-visitor identity. Route per domain, then reconcile the metadata at write time.

Workload routing explained

Integration

Bring Your Own Loader

We do not ship a LangChain or LlamaIndex plugin, and you do not need one. The gateway is a standard HTTP or SOCKS5 proxy, so anything that accepts a proxy mapping — or honours the usual environment variables — works unchanged. Set the variables once and the whole process routes through it.

loaders_any.py Python
import os
from langchain_core.documents import Document

SESSION = "<paste BASE64_CONFIG>"
GATE = "gate.proxyperk.com:8080"
AUTH = "user:" + SESSION
PROXY = f"http://{AUTH}@{GATE}"

os.environ["HTTP_PROXY"] = PROXY
os.environ["HTTPS_PROXY"] = PROXY

def load(url, text):
    return Document(
        page_content=text,
        metadata={"source": url}
    )
requests · httpx · most loaders env vars

Note on the snippet: it shows the shape, not a full pipeline. Whatever you use to fetch and clean the page stays yours; the proxy only changes where the request leaves from. Because requests and httpx read the environment by default, most loaders pick the setting up without a code change.

Env-var routing

HTTP_PROXY plain HTTP targets
HTTPS_PROXY TLS targets
NO_PROXY hosts that stay direct

Set them once per process — requests and httpx read them by default.

Guarantee

Prove It on One Domain First

Nobody should commit a corpus budget on a spec sheet. Buy the smallest quantity that covers one target domain, run it for a day, and measure success rate and latency.

Step 1

Buy 15 GB — enough for one domain and one full crawl.

Step 2

Run the crawl and record success rate per URL.

Step 3

Not delivered or not as described? Open a ticket within 7 days.

Every ProxyPerk product carries a 7-day instant cash back guarantee — datacenter included, and datacenter IPs are additionally trial-eligible. The request is validated in real time against the published terms.

Covers non-delivery and service issues. Third-party software incompatibility and prohibited uses are excluded — Terms of Refund.

Help center

Training-data questions