n8n Pagination: Stop Duplicate Records When APIs Use Cursors

Why cursor loops duplicate in n8n

Most APIs that “page” today use a cursor, not a page number. You send cursor (or next_token) in the request, the response gives you next_cursor, and you send that back until next_cursor is null or missing. The shape is simple. The n8n implementation is where it breaks.

Three failures show up in every solopreneur stack:

The cursor is never passed back. The HTTP Request node has a static query parameter, the Loop node feeds the items back but not the cursor from the last response, so the API returns page 1 repeatedly. The workflow “works” — it imports the same 100 rows nine times.
No termination check. The response body has next_cursor on every page except the last, but the workflow keeps looping on any non-empty response. When the API is done, next_cursor is null — the If node that should exit still lets it through.
No dedupe, no cap. Even with a correct cursor, a retry or a slow page can return the same record twice. Without a seen set and a max-page budget, one stuck cursor becomes an infinite loop that fills your database and your API quota.

None of these need a newer node. They need the loop to carry state — cursor, seen ids, and page count — on every iteration.

The five-state loop

Carry one state object through the loop: cursor, page, seen, items, and done. The Split In Batches node is the loop primitive; a Code node before it owns the state.

// Code node: "Init pagination state"
// Runs once before the loop. One item in, one state object out.
return [{
  json: {
    cursor: null,      // null = first page
    page: 0,
    seen: [],          // ids already returned (dedupe)
    items: [],         // accumulated rows (or write to a table instead)
    done: false,
    max_pages: 50,     // hard cap — never loop forever
    page_size: 100
  }
}];

Fetch, dedupe, advance — in that order

The HTTP Request node uses the state's cursor. The Code node after it does not just pass items through — it filters them, appends to state, and decides whether to loop again.

// Code node: "Process page, dedupe, next state"
// Input: one item per API record (from HTTP Request response)
//        + the previous state carried in $json.state
const prev = $json.state || {};
const seen = new Set(prev.seen || []);
const items = [...(prev.items || [])];
const page = (prev.page || 0) + 1;

for (const it of $input.all()) {
  const rec = it.json;
  const id = rec.id ?? rec._id ?? rec.email;
  if (id && seen.has(String(id))) continue; // duplicate from retry/slow page
  if (id) seen.add(String(id));
  items.push(rec);
}

const nextCursor = $node['HTTP Request'].json.next_cursor
                ?? $node['HTTP Request'].json.next_token
                ?? null;

return [{
  json: {
    state: {
      cursor: nextCursor,
      page,
      seen: [...seen],
      items,
      done: !nextCursor || page >= (prev.max_pages ?? 50),
      max_pages: prev.max_pages ?? 50,
      page_size: prev.page_size ?? 100
    }
  }
}];

Two points matter. First, read the cursor from the HTTP node's output, not from the current input item — n8n's $input.all() here is data rows, not pagination metadata. Second, dedupe before you write — if the response was a retry of the last page, the seen set is what stops the duplicate from reaching Supabase or Sheets.

Write after the loop, not inside it

Do not upsert each page as it arrives. Accumulate in state, exit the loop on done === true, then write once. If you crash on page 7 of 20, you have lost nothing durable — the next run starts from page 1 and the seen set is rebuilt. If you write inside the loop, a crash on page 7 leaves 600 rows in the table and no record of where you stopped. The dedupe set protects the final write — it does not make partial writes safe.

The If node that ends the loop

Connect the Process node to an If node: {{ $json.state.done === true }}. True branch goes to your final write (or a Set node that unwraps items into rows). False branch loops back to the HTTP Request — and a Code node there injects {{ $json.state.cursor }} as the query parameter.

Wire the Loop branch back to the Init node only if the Split In Batches index is the loop driver. In practice the cleanest shape is:

[Init state]
  |
[HTTP Request: query cursor = {{ $json.state.cursor }}]
  |
[Code: process page, dedupe, next state]
  |
[If done?]
  |- true  -> [Set: unwrap items] -> [Write once: Supabase/Sheets/CRM]
  |- false -> [Wait 500ms (politeness for tight rate limits)] -> back to [HTTP Request]

The Wait is optional but cheap. If you already rate-limit globally, you can drop it — the point is the loop-back edge exists and the done flag is the only exit.

Offset pagination is the same skeleton

If the API uses offset / limit instead of a cursor, the loop is identical: state carries offset, each page advances it by page_size, and done is items.length < page_size || page >= max_pages. The dedupe and cap do not change. A page that returns fewer rows than requested is the API telling you it is empty — treat short page as done, not as a reason to advance.

A short final page also catches a classic bug: APIs that return 200 OK with an empty array. An empty array with a next_cursor still set is a stuck cursor — the page cap is what stops you. Never trust the cursor alone.

Where this ships today

The init-state, dedupe, and loop-back pattern above is bundled as a ready-to-import workflow on the store. Point the HTTP node at your API, set page_size and max_pages, and the loop terminates on its own — duplicates filtered, no infinite run, one write at the end.