Bulk-loading content: batches, not entries

Applies to: Any content you are bringing in from somewhere else You will learn: why a large import should send batches rather than entries, how to give entries ids you can compute, how to order a load so links land on entries already written, and how to make a load you can stop and rerun at any point

The short version

When you have more than a few hundred entries to bring in, do not create them one at a time. Write them into batch documents instead, and send each batch to your branch in a single request:

	One request per entry	One request per batch
Requests for 100,000 entries	100,000	400, at 250 entries a batch
Typical speed	Thousands of entries an hour	Thousands of entries a minute
Counts against your write limit	Once per entry	Once per batch
If it stops halfway	Some unknown set of entries exists	Every batch is either fully in or not in at all
Running it twice	Duplicates, unless you check before each write	Nothing changes the second time

Five ideas make this work, and the rest of this article takes them one at a time:

Stage your source data as JSONL, so the slow part (getting data out of the old system) happens once.
Give every entry an id you compute from its source id, so the same record becomes the same entry every time.
Link entries by those computed ids, so you never look anything up.
Order batches so referenced entries come first: categories and authors before the articles that point at them.
Keep a record of the batches that have landed, so a rerun picks up where the last one stopped.

A complete, working loader built on these ideas is in the Coho samples, under python/openalex-research-catalog. It loads tens of thousands of entries from the OpenAlex research catalog. The examples here are smaller and use a made-up export of authors, categories, and articles, so the shape of the technique stays clear.

Why one entry at a time is slow

The obvious import looks like this:

for record in records:
    create_entry(record)   # one HTTP request, one transaction, one round trip

Each call has to cross the network, authenticate, open a transaction, validate one entry, commit, and come back. Most of that time is spent on overhead, not on your content. Even with nothing else in the way, one request at a time manages a few dozen entries a second at best.

Something else is in the way, by design. Coho limits how fast a project's tokens can write, so that one script cannot crowd out the people editing in the same project:

	Free	Site	Enterprise
Individual writes	5 a second	20 a second	50 a second
Bulk operations in flight at once	1	2	4

At 5 writes a second, 100,000 entries take five and a half hours on the Free plan. On Site they take well over an hour.

A batch is one request. It counts once against your limits, however many entries it carries, and Coho writes all of those entries in one transaction on the server, next to the database. The per-request overhead is paid once per batch instead of once per entry, and the rate limit is spent once per batch too. That is the whole difference between thousands an hour and thousands a minute.

What a batch looks like

A batch is a single JSON document with a format marker and a list of nodes:

{
  "format": "coho-backup",
  "version": 1,
  "nodes": [
    {
      "id": "37d22b78-6941-5494-b12d-6cf40841fffd",
      "kind": "entry",
      "type": "author",
      "slug": "author-1842",
      "body": {
        "_name": "Maya Okafor",
        "bio": "Writes about infrastructure and the people who run it."
      }
    },
    {
      "id": "5e9018a1-3b32-5433-932f-1a3948f7621a",
      "kind": "entry",
      "type": "article",
      "slug": "article-9031",
      "body": {
        "_name": "Running Postgres without a pager",
        "summary": "What we changed so that nobody gets woken up.",
        "author": "37d22b78-6941-5494-b12d-6cf40841fffd"
      }
    }
  ]
}

It is the same format Coho uses for a branch's backup, which is why the endpoint is called …/backup. Each node has:

Field	What it is
`id`	The entry's id: a lowercase UUID. You choose it, and the next section shows how
`kind`	`entry` for content. (A backup can also carry types, traits, and assets; this article assumes your content types already exist)
`type`	The content type's slug, such as `article`
`slug`	The entry's slug: lowercase letters, digits, and single hyphens, unique across the whole branch
`body`	The entry's fields, exactly as you would send them when creating the entry, with its internal name in `_name`

The body is validated against the content type just as an ordinary save is. An unknown field, a missing required one, or a value of the wrong shape refuses the batch with 400 VALIDATION_FAILED, and the error names the entry and the field. So does an item nested more than five deep. An entry is 2 MB at most, every locale included, and a larger one refuses the batch with 400 ENTRY_TOO_LARGE, naming the entry.

One batch is one transaction. If any entry in it is refused, none of the batch is written. That sounds strict, and it is what makes everything later in this article possible: a batch is always either completely in or not in at all, so "which batches landed?" always has a simple answer.

Step 1: stage your source data as JSONL

Getting data out of the old system is usually the slowest and least reliable part of a migration. It might be paged API calls, a database dump, or a pile of CSV files. Do that once, and save what you get to disk.

JSON Lines (one JSON object per line) is the right format for this staging step:

import json
from pathlib import Path

STAGED = Path("data/staged")

def stage(kind: str, records):
    """Write every record of one kind to data/staged/<kind>.jsonl, one per line."""
    STAGED.mkdir(parents=True, exist_ok=True)
    with (STAGED / f"{kind}.jsonl").open("w") as f:
        for record in records:
            f.write(json.dumps(record) + "\n")

stage("authors", old_cms.list_authors())
stage("categories", old_cms.list_categories())
stage("articles", old_cms.list_articles())

JSONL earns its place here for a few reasons:

You can write it as you go. If the extract dies on page 4,000, everything before it is already on disk.
You can read it without holding it all in memory. One line is one record.
It is easy to inspect. head -1 data/staged/articles.jsonl shows you a real record, and wc -l counts them.

def staged(kind: str):
    """Every record of one kind, read one line at a time."""
    with (STAGED / f"{kind}.jsonl").open() as f:
        for line in f:
            yield json.loads(line)

From here on, nothing touches the old system again. You can change how records become entries and rerun the conversion as often as you like, in seconds, without fetching anything twice.

JSONL is for staging. Batches are JSON. Coho's batch endpoint reads one JSON document per request, the shape in the previous section. Your staged files are the input to a conversion step that writes those documents.

Step 2: give every entry an id you can compute

This is the most important idea in this article.

When you create an entry the ordinary way, Coho picks its id. To link to that entry later, you have to remember what id Coho picked: keep a mapping table, look the entry up by slug, or query for it. That mapping becomes one more thing to keep correct, and a rerun that loses it creates duplicates.

Instead, choose the id yourself, computed from something that never changes in your source data: the record's primary key in the old system. A name-based UUID (version 5) does exactly this. The same inputs always give the same UUID, and different inputs give different ones:

import uuid

# Pick this once for your project and never change it. Any UUID will do.
NAMESPACE = uuid.UUID("9a3f6c1e-2b4d-4e8a-9c71-5d0e8f2b6a43")

def entry_id(kind: str, source_id: str | int) -> str:
    """The Coho id for a source record: the same record always gives the same id."""
    return str(uuid.uuid5(NAMESPACE, f"{kind}:{source_id}"))

>>> entry_id("author", 1842)
'37d22b78-6941-5494-b12d-6cf40841fffd'
>>> entry_id("author", 1842)                  # tomorrow, on another machine
'37d22b78-6941-5494-b12d-6cf40841fffd'
>>> entry_id("category", 1842)                # same number, different kind
'c19e98df-616b-5b8f-ad93-0fd5bc7bacc2'

The kind prefix matters. Old systems often number each table from 1, so author 1842 and category 1842 are different records that share a number. Putting the kind into the name keeps their ids apart.

What you get from this:

Rerunning is safe. Sending a node whose id already exists updates that entry instead of creating a second one (Step 5 covers this).
Links need no lookups. You can compute the id of any entry, including one you have not loaded yet (Step 3).
Nothing needs to be stored. No mapping table and no "already imported" flag in the old system. The id is the mapping.

Keep the namespace UUID in your code, under version control. If you change it, every id changes, and a rerun creates a whole new set of entries beside the old ones.

Slugs and names: also from the source

The same thinking applies to slugs. A slug must be unique across the whole branch, whatever the type, so build it from the kind and the source id rather than from a title:

def slug_of(kind: str, source_id: str | int) -> str:
    """'author-1842': unique on the branch, and the same on every run."""
    return f"{kind}-{source_id}".lower()

A slug made from a title (running-postgres-without-a-pager) reads better, and you can switch to one later. But titles repeat ("Introduction", "Errata", "Untitled") and change between runs, and either one fails a batch with SLUG_CONFLICT or quietly renames an entry. Start with slugs that cannot collide.

The internal name, _name, must be unique among entries of the same base type. Real data repeats names: two authors called "J. Smith", two articles called "Introduction". Settle that before you write a batch, and settle it the same way on every run:

from collections import Counter

def disambiguate(nodes: list[dict]) -> None:
    """Give every holder of a repeated name its source id, so no run depends on order."""
    counts = Counter((n["type"], n["body"]["_name"]) for n in nodes)
    for n in nodes:
        if counts[(n["type"], n["body"]["_name"])] > 1:
            n["body"]["_name"] = f"{n['body']['_name']} ({n['slug']})"

Notice that every holder of a repeated name gets the suffix, not just the second one. If only the second got it, which entry kept the plain name would depend on which record happened to be read first, and a rerun could swap them.

If your types use inheritance, "the same base type" means the root of the family. A blogPost and a newsArticle that both extend article share one set of names.

Step 3: link entries by their computed ids

A reference field holds the id of the entry it points at. A field that takes many references holds a list of ids. Because you can compute any entry's id from its source id, writing a link is just a function call:

def article_node(record: dict) -> dict:
    return {
        "id": entry_id("article", record["id"]),
        "kind": "entry",
        "type": "article",
        "slug": slug_of("article", record["id"]),
        "body": drop_empty({
            "_name": record["title"].strip(),
            "summary": record.get("summary"),
            "content": record.get("content_html"),
            "author": entry_id("author", record["author_id"]) if record.get("author_id") else None,
            "categories": [entry_id("category", c) for c in record.get("category_ids", [])],
        }),
    }

def drop_empty(body: dict) -> dict:
    """Leave out what the source did not have, rather than sending empty values."""
    return {k: v for k, v in body.items() if v not in (None, "", [])}

Look at what this function does not do: it does not ask Coho whether author 77 exists, or what id it was given. It does not need to. Author 77's id is entry_id("author", 77), now and on every future run.

What Coho checks about a link

When an entry is written, Coho looks at each link in it:

If the target exists on the branch, it must be an entry of a type the field allows (or a type that extends one). A link to the wrong kind of entry is refused with VALIDATION_FAILED.
If the target does not exist yet, the link is accepted as it stands. Coho calls this a dangling link: the entry is still valid and still renders, just without that link, until the target arrives.

Coho allows dangling links on purpose, because on a branch, a link's target may simply not have been merged in yet. But that means a link written before its target exists is never type-checked. If you got a source id wrong and the link happens to point at a category instead of an author, you only find out when that link is read. Which brings us to ordering.

Step 4: order the load so referenced entries come first

Sort your entries so that everything an entry links to is written before it is:

categories    ─┐
authors       ─┼─▶  articles

Then every link is checked when it is written, and a wrong link fails loudly, at load time, with the entry and field named. As a bonus, if a load stops partway, everything that landed is complete. You never have a run of articles pointing at authors that are not there yet.

Write this order down as tiers:

# Entries that others link to come first. Within a batch, Coho writes nodes in the order you list them.
TIERS = ["category", "author", "article"]

Then convert every staged record, sort, and cut the result into batch files:

import json
import math
from pathlib import Path

BATCHES = Path("data/batches")
BATCH_SIZE = 250

def convert():
    nodes = []
    nodes += [category_node(r) for r in staged("categories")]
    nodes += [author_node(r) for r in staged("authors")]
    nodes += [article_node(r) for r in staged("articles")]
    disambiguate(nodes)

    # sorted() is stable, so records keep their source order within a tier, and the
    # batches come out the same on every run.
    nodes = sorted(nodes, key=lambda n: TIERS.index(n["type"]))

    BATCHES.mkdir(parents=True, exist_ok=True)
    for old in BATCHES.glob("*.json"):
        old.unlink()
    for i in range(0, len(nodes), BATCH_SIZE):
        document = {"format": "coho-backup", "version": 1, "nodes": nodes[i : i + BATCH_SIZE]}
        (BATCHES / f"{i // BATCH_SIZE + 1:04d}.json").write_text(json.dumps(document))

    print(f"{len(nodes):,} entries in {math.ceil(len(nodes) / BATCH_SIZE)} batches")

The numbered file names (0001.json, 0002.json, …) are the load order. Sending them in name order sends categories first and articles last.

A batch may cross a tier boundary, holding the last authors and the first articles. That is fine: within a batch, Coho writes nodes in the order they are listed, so the authors are written before the articles that follow them.

When entries link to their own kind

Sometimes a record links to another record of the same kind: an article's "related articles", or a category's parent category. Tiers cannot put every article before every other article. You have two good options:

Accept the dangling link. Write the link on the first pass anyway. Links to articles already loaded are checked, links to later ones dangle, and they all resolve once the load finishes. The cost is that the forward links are never type-checked, which is usually acceptable when the field only allows one type.
Load in two passes. Load every article without the self-links, then load every article again with them. The second pass checks every link. Because ids are stable, the second pass updates the entries the first pass created rather than adding new ones, and an article with no self-links comes back unchanged (Step 5 explains how).

For a tree, such as categories with parents, you can also sort parents before children within the tier, so every link is checked on the first pass:

def depth(category: dict, by_id: dict) -> int:
    d = 0
    while category.get("parent_id"):
        category = by_id[category["parent_id"]]
        d += 1
    return d

categories = list(staged("categories"))
by_id = {c["id"]: c for c in categories}
categories.sort(key=lambda c: depth(c, by_id))   # roots, then their children, then theirs

Step 5: make every load safe to rerun

A load you can safely run twice is called idempotent. A load that can stop at any point and resume where it left off is called re-entrant. You want both, because real loads get interrupted: a laptop lid closes, a network drops, one bad record fails a batch in the middle of the night.

You already have most of what you need.

Coho does the idempotent part

When a batch's node names an id that already exists on the branch, Coho compares the incoming entry with the one already stored:

What it finds	What it does	Counted as
No entry with that id	Creates it	`created`
An entry with the same id, with a different body or slug	Saves a new version of it	`updated`
An entry with the same id, identical	Writes nothing at all	`unchanged`

Every batch request answers with those counts:

{ "created": 0, "updated": 12, "unchanged": 988, "missingBlobs": [] }

"Writes nothing" is meant literally. An unchanged entry gets no new version, so rerunning a load does not grow your project's history. You can reload your whole catalogue every night from a fresh extract, and only the records that actually changed in the source become new versions in Coho.

This works only if your conversion is deterministic. The same staged record must turn into exactly the same node, byte for byte, on every run. Common ways to break that without noticing:

Timestamps of "now". "importedAt": datetime.now() makes every entry different on every run. Use the source's own timestamps, or leave the field out.
Random ids anywhere. uuid4() for an entry id or a repeater item's id. Derive them instead (next section).
Unstable ordering. A list of references built from a set, or from a database query with no ORDER BY, can come back in a different order. Sort it.

A quick check: convert twice and compare.

python load.py convert && cp -r data/batches /tmp/first
python load.py convert && diff -r /tmp/first data/batches && echo "deterministic"

Repeater items need stable ids too

If a type has a repeater field (a list of components, such as the sections of a page), each item carries an _id. If you leave it out, Coho stamps a fresh one on every write. Fresh ids mean the item is "different" every time, so a rerun writes a new version of every entry with a repeater, even when nothing changed.

Derive item ids the same way you derive entry ids:

def section_items(record: dict) -> list[dict]:
    return [
        {
            "_id": entry_id(f"article-section:{record['id']}", i),
            "_type": "textBlock",
            "heading": section["heading"],
            "text": section["text"],
        }
        for i, section in enumerate(record.get("sections", []))
    ]

If your source gives each section an id of its own, use that rather than its position, so inserting a section does not shift the ids of every section after it.

You do the re-entrant part

Because a batch lands whole or not at all, resuming is easy: remember which batch files landed, and skip them next time. Append each one's name to a file the moment its request succeeds:

import json
import os
import time
import urllib.error
import urllib.parse
import urllib.request
from pathlib import Path

DONE = Path("data/loaded.txt")

def load(base_url: str, account: str, project: str, ref: str):
    token = os.environ["COHO_TOKEN"]
    url = f"{base_url}/api/v1/accounts/{account}/projects/{project}/refs/{urllib.parse.quote(ref, safe='')}/backup"
    done = set(DONE.read_text().split()) if DONE.exists() else set()

    for batch in sorted(BATCHES.glob("*.json")):
        if batch.name in done:
            continue
        result = post_batch(url, token, batch.read_bytes())
        with DONE.open("a") as f:
            f.write(batch.name + "\n")
        print(f"{batch.name}: {result['created']:,} created, {result['updated']:,} updated, "
              f"{result['unchanged']:,} unchanged")

The two halves cover for each other. loaded.txt makes the rerun quick, because it skips the batches already in. Stable ids make the rerun correct even when loaded.txt is wrong. Suppose the request succeeded but your script died before writing the file name: the next run sends that batch again, and it comes back as unchanged. No duplicates, either way.

That is also why it is safe to clear loaded.txt whenever you convert again. New batches may divide entries differently from old ones, so old file names mean nothing. Clear it, and the next load sends everything. Entries that did not change come back unchanged and cost no history.

Handling refusals and retries

Three kinds of answer need handling:

def post_batch(url: str, token: str, body: bytes) -> dict:
    request = urllib.request.Request(url, data=body, method="POST", headers={
        "Authorization": f"Bearer {token}",
        "Content-Type": "application/json",
    })
    for attempt in range(6):
        try:
            with urllib.request.urlopen(request, timeout=120) as response:
                return json.load(response)
        except urllib.error.HTTPError as e:
            if e.code == 429:
                # Over the rate limit: wait as long as Coho asks, then send the same batch again.
                time.sleep(int(e.headers.get("Retry-After", "5")))
            elif e.code in (502, 503, 504):
                # The request may still have landed. Sending it again is safe: whatever it
                # wrote comes back as unchanged.
                time.sleep(2 ** attempt)
            else:
                # A 400 is about the batch itself, and sending it again will not help.
                raise SystemExit(f"Refused ({e.code}): {e.read().decode(errors='replace')[:2000]}")
    raise SystemExit("Gave up after six attempts; run the load again to resume.")

429 RATE_LIMITED means slow down. Wait for the number of seconds in Retry-After, then send the same batch again.
502, 503, or 504 mean the answer did not reach you. The batch may or may not have landed. With stable ids you do not need to know which: send it again.
400 means something in the batch is wrong, and the response says what: VALIDATION_FAILED with the entry and field, ENTRY_TOO_LARGE with the entry, SLUG_CONFLICT, INTERNAL_NAME_CONFLICT, and so on. Stop, fix the conversion, convert again, and load again. Everything already in comes back unchanged, so a fix costs you nothing but the batches that actually change.

Putting it together

The whole loader is four steps, each one resumable and each one writing its result to disk:

extract   old system  ──▶  data/staged/*.jsonl      slow, once
convert   staged      ──▶  data/batches/0001.json   seconds, as often as you like
load      batches     ──▶  Coho                     thousands of entries a minute
verify    Coho        ──▶  you                      diff, then merge

Getting a token

The batch endpoint is called with a project token: a credential for one project and one role, with an expiry. Creating entries needs nothing above the author role, so give the token no more than that:

coho token create --role author --label "Content import" --expires 7
export COHO_TOKEN=<the token, shown once>

When the import is done, revoke it:

coho token list
coho token revoke <token-id>

Finding where to send batches

The base URL, account, and project come from the CLI's current context:

import json
import subprocess

def current_place() -> dict:
    """The URL, account, project, and ref that `coho status` shows."""
    out = subprocess.run(["coho", "-o", "json", "status"], capture_output=True, text=True, check=True).stdout
    return json.loads(out)

place = current_place()
load(place["url"].rstrip("/"), place["account"], place["project"], "import/legacy-cms")

Load onto a branch, then merge

Notice the ref in that last line: import/legacy-cms, not the trunk. Load onto a branch of its own. Creating a branch copies nothing and takes a moment, whatever the size of your project, so there is no reason not to:

coho branch create import/legacy-cms
python load.py load
coho diff v0.0.x                                          # everything the import would change
coho merge --into v0.0.x -m "Import from the legacy CMS"  # lands whole, or not at all

Your editors keep working on the trunk while the import runs, and they never see half of it. You review the result as a diff before anything reaches them. If something is wrong, fix the conversion and load onto the same branch again. If it is badly wrong, create a fresh branch from the trunk and load there instead.

For an ongoing sync, the same pattern repeats: branch, load the latest extract, review the diff (which now holds only what changed in the source), and merge.

How fast, and why

Here is the difference, worked through for 100,000 entries on the Free plan:

	One request per entry	One request per batch
Requests	100,000	400
What limits it	5 writes a second	1 batch in flight at a time
Time	About 5½ hours	Typically under an hour
Transactions	100,000	400
A failure leaves	An unknown set of entries	A known number of complete batches

Three things make the batched load fast:

Overhead is paid per request, not per entry. Authentication, routing, the network round trip, and opening and committing a transaction happen 400 times instead of 100,000.
The work happens next to the database. Coho validates and writes a batch's entries inside one transaction on the server, rather than waiting on your network between each one.
Rate limits count requests, not entries. A batch of 250 counts as one call. Bulk operations are limited by how many run at once (one on Free, two on Site, four on Enterprise), not by how many entries they carry.

Choosing a batch size

250 entries is a good starting point. Larger batches mean fewer requests, but each one is a single transaction that has to finish within the gateway's time limit of just under a minute. If a batch takes longer than that, you get a 504. The batch may still land, and your retry will find it as unchanged, but it is a sign to use smaller batches.

Entries with large bodies, such as long rich text or many repeater items, are slower to validate and write. If your entries are heavy, start at 100. If they are small, try 500. Watch how long each batch takes, and keep each one well inside the minute.

Why not send batches in parallel?

It is tempting, and it rarely helps. Your plan allows only a few bulk operations in flight at once, so extra concurrent requests are refused with 429. Parallel batches also break the ordering from Step 4: an article batch could land before the author batch it links to. Send batches one after another. At thousands of entries a minute, the load is seldom the slow part of a migration.

Common questions

Does a batch delete entries that are no longer in my source? No. A batch only creates and updates. To remove entries, delete them through the API or the CLI, preferably on the import branch so the deletions appear in the diff before you merge. Because ids are computed, you can find exactly which entries to remove: compute the ids of everything still in your source, and compare them with the entries on the branch.

Can I change the content model and reload? Yes. Change the types first, then convert and load again. Entries whose new form differs are updated, and the rest are unchanged. A batch can also carry the type definitions themselves, but designing and migrating a content model is a topic of its own.

What about images and other files? A batch carries entries, not file bytes. Upload files as assets first, then link to them from your entries by the asset's id, just as you link to other entries.

What if two source records really are the same thing? Decide which source id wins, and use that id for both. For example, keep a small table of merged duplicates and look it up before calling entry_id. Because the result is deterministic, the decision holds on every rerun.

Can I use these ids with the ordinary entry API too? Yes. Creating an entry with an id you choose behaves just like a batch node: the same id converges on the same entry, and an identical request writes nothing. That is useful for a small trickle of changes after the bulk load. But for anything more than a handful of entries, batch it.

Is a batch the same as a backup? It is the same format. Downloading a branch's backup gives you a document in this exact shape, with every type and entry and their ids, and loading it into another project or branch restores them. That makes a backup download a good way to see real examples of what your nodes should look like.

Bulk-loading your content: batches, not entries