LLM Training Data Collection at Scale and in Volume
A pretraining corpus is a throughput problem, a diversity problem and a provenance problem. ProxyPerk supplies the egress layer: 200-country residential, 35-country mobile, named ISPs and datacenter pools behind one endpoint, with sticky sessions and an API that lets you pin one identity per domain.
import os
import requests
# session string from your dashboard
SESSION = "<paste BASE64_CONFIG>"
GATE = "gate.proxyperk.com:8080"
AUTH = "user:" + SESSION
PROXY = f"http://{AUTH}@{GATE}"
# requests, httpx and most loaders
os.environ["HTTPS_PROXY"] = PROXY
os.environ["HTTP_PROXY"] = PROXY
r = requests.get(
"https://example.com",
timeout=30
)
print(len(r.text))
Works with your stack
The difference
Why a Training Crawl Is Not a Normal Crawl
A product page scrape runs for minutes. A corpus run goes for weeks, across millions of URLs, and has to be reproducible six months later. That changes what “good” means.
Sustained volume
Hundreds of gigabytes, not megabytes. Banned IPs mid-run cost more than the bandwidth does.
Geographic spread
A corpus collected from one country encodes one market. Diversity needs real exits in many countries.
Reproducibility
Pin one identity per domain so a re-crawl sees the same thing and your diffs mean something.
Provenance
You store the data, so you keep the licence and retention decisions. We supply the network only.
Modelling
Size the Run Before You Start
Rotating pools are billed per GB, so bandwidth is the budget line. A rough planning figure is about four bytes of transfer per token of raw text — enough to model the order of magnitude, not to quote a bill.
| Corpus size | Approx. transfer | Residential $3.95 | Datacenter $1.90 |
|---|---|---|---|
| 10 million tokens | ~40 GB | $158 | $76 |
| 100 million tokens | ~400 GB | $1,580 | $760 |
| 1 billion tokens | ~4 TB | $15,800 | $7,600 |
| 10 billion tokens | ~40 TB | $158,000 | $76,000 |
Estimates only, at list price before discounts. Real transfer is higher than the text itself once retries, HTML overhead and images are counted. Stacking 12 months (15%) with the 100 GB tier (12%) takes 27% off the top.
Diversity
Coverage You Can Actually Select
Diversity is not a slogan if you can name the country, the city and the carrier in the session string. These are the pools available for a corpus run.
Residential countries
Country targeting plus sticky sessions. $3.95/GB
Mobile countries
Carrier NAT, the hardest class to fingerprint. $9.90/GB
Named ISP carriers
Static, city-level, unlimited bandwidth. from $2.00/IP
Practical pattern: run the bulk of the corpus on rotating datacenter where the target is an open API or documentation mirror, take protected long-tail domains on rotating residential, and hold static ISP IPs for the handful of sites that require a logged-in or returning-visitor identity. Route per domain, then reconcile the metadata at write time.
Workload routing explainedIntegration
Bring Your Own Loader
We do not ship a LangChain or LlamaIndex plugin, and you do not need one. The gateway is a standard HTTP or SOCKS5 proxy, so anything that accepts a proxy mapping — or honours the usual environment variables — works unchanged. Set the variables once and the whole process routes through it.
import os
from langchain_core.documents import Document
SESSION = "<paste BASE64_CONFIG>"
GATE = "gate.proxyperk.com:8080"
AUTH = "user:" + SESSION
PROXY = f"http://{AUTH}@{GATE}"
os.environ["HTTP_PROXY"] = PROXY
os.environ["HTTPS_PROXY"] = PROXY
def load(url, text):
return Document(
page_content=text,
metadata={"source": url}
)
Note on the snippet: it shows the shape, not a full pipeline. Whatever you use to fetch and clean the page stays yours; the proxy only changes where the request leaves from. Because requests and httpx read the environment by default, most loaders pick the setting up without a code change.
Env-var routing
Set them once per process — requests and httpx read them by default.
Guarantee
Prove It on One Domain First
Nobody should commit a corpus budget on a spec sheet. Buy the smallest quantity that covers one target domain, run it for a day, and measure success rate and latency.
Buy 15 GB — enough for one domain and one full crawl.
Run the crawl and record success rate per URL.
Not delivered or not as described? Open a ticket within 7 days.
Every ProxyPerk product carries a 7-day instant cash back guarantee — datacenter included, and datacenter IPs are additionally trial-eligible. The request is validated in real time against the published terms.
Covers non-delivery and service issues. Third-party software incompatibility and prohibited uses are excluded — Terms of Refund.