This page shows how to upload your scraped cars to our server. Start with the four steps. The rest of the page is detail you can look up when you need it.
Every upload is one HTTPS request with three headers and your cars in the body.
application/json.curl -sS https://push.rossauto.ca/v1/push \
-H "Authorization: Bearer $INGEST_KEY_COPART_CA" \
-H "Content-Type: application/x-ndjson" \
-H "Idempotency-Key: copart-ca-test-1" \
--data-binary @test-cars.ndjson
Use --data-binary, not -d. With -d, curl removes the line breaks and the upload fails.
| Reply | Meaning | What you do |
|---|---|---|
| 202 or 200 | We received it. | Nothing. Keep the batchId from the reply. |
| 400 | The file is broken. The reply says which line. | Fix the file, then send it again. |
| 401 | The key is wrong or missing. | Check you used the key of this website. |
| 413 | The upload is bigger than 100 MB. | Split it into parts. |
| 503 | We are busy with other uploads. | Wait 30 seconds. Send it again with the same name. |
| 500 | A problem on our side. | Wait a few minutes. Send it again with the same name. |
A 202 means we received the upload. It does not mean every car in it was good: we check the cars after that. If we find a problem, we tell you, with the batchId and some example cars.
A website with more than 100 MB of cars is sent in parts. This Python example splits a file, names each part, and sends it again when our reply says to.
import os, time
from datetime import datetime, timezone
import requests
URL = "https://push.rossauto.ca/v1/push"
WEBSITE = "copart-ca"
KEY = os.environ["INGEST_KEY_COPART_CA"]
RUN = datetime.now(timezone.utc).strftime("%Y-%m-%d-%H%M") # when this scrape run started
PART_SIZE = 90 * 1024 * 1024 # under the 100 MB limit
def send(body, part):
name = f"{WEBSITE}-{RUN}-part-{part}" # same name on every retry
for attempt in range(1, 9):
try:
reply = requests.post(URL, data=body, timeout=600, headers={
"Authorization": f"Bearer {KEY}",
"Content-Type": "application/x-ndjson",
"Idempotency-Key": name,
})
except requests.RequestException:
time.sleep(30 * attempt) # no reply: try again
continue
if reply.status_code in (200, 202):
return reply.json()["batchId"]
if reply.status_code == 503 or reply.status_code >= 500:
time.sleep(30 * attempt) # busy, or our problem: try again
continue
raise RuntimeError(f"{reply.status_code}: {reply.text}") # 400, 401, 413: fix first
raise RuntimeError(f"part {part}: no success after 8 tries")
def send_file(path):
part, lines, size = 1, [], 0
with open(path, "rb") as file:
for line in file:
if lines and size + len(line) > PART_SIZE:
print(part, send(b"".join(lines), part))
part, lines, size = part + 1, [], 0
lines.append(line)
size += len(line)
if lines:
print(part, send(b"".join(lines), part))
The key values come in a private message, not on this page. This table only shows which key belongs to which website.
| Website | Key name |
|---|