collections.diff
Compare two lists of records by key: which were added, which removed, and which changed and in what fields.
1.0.0 · published 2026-10-03 by charlie · Anterra
Pinned by 18 tests, run in TypeScript, Python and Rust.
What it does
Compares two lists of records matched by `key` and says what happened between them: records that were added, records that were removed, and records present in both whose contents changed, with the names of the fields that differ. It is the heart of a sync job, an audit log or a "review changes before import" screen.
ORDER: `added` and `changed` follow the order of the `after` list, `removed` follows the order of the `before` list. `unchanged` is only a count, because a caller showing a diff never wants the untouched rows back.
For example
diff_by_key(before ×3, after ×3, id)→ added ×1, removed ×1, changed ×1, unchanged 1 one of each: an addition, a removal, a change and an untouched recorddiff_by_key(before ×2, after ×2, k)→ added , removed , changed , unchanged 2 identical lists: nothing added, removed or changeddiff_by_key(before ×2, after ×2, k)→ added , removed , changed , unchanged 2 reordering is not a change
The function
The same function in TypeScript, Python and Rust, pinned by the same tests. Pick your language; the choice follows you around the registry.
def diff_by_key(before: Sequence[Mapping[str, Any]], after: Sequence[Mapping[str, Any]], key: str) -> RecordDiff[Mapping[str, Any]]
| before | record[] | the old list; every record needs a unique value at key |
| after | record[] | the new list; every record needs a unique value at key |
| key | string | the field that identifies a record in both lists |
| returns | RecordDiff<record> |
The types it declares, generated into your project
@dataclass(frozen=True)
class RecordDiff(Generic[T]):
"""What changed between two lists of records matched by key."""
#: in after but not before, in after's order
added: List[T]
#: in before but not after, in before's order
removed: List[T]
#: in both but different, in after's order
changed: List[RecordChange[T]]
#: how many records are in both and identical
unchanged: int
@dataclass(frozen=True)
class RecordChange(Generic[T]):
"""One record present in both lists whose contents differ."""
#: the key value, rendered as text
key: str
before: T
after: T
#: names of the fields that differ, sorted by code point
fields: List[str]
Your code names it in one line, in the file that uses it
from fune.collections.diff import diff_by_key # collections.diff@^1
from typing import Any, Dict, List, Mapping, Optional, Sequence
from .collections_diff_types import RecordChange, RecordDiff
# Largest integer JavaScript can hold exactly; beyond it the three languages disagree.
SAFE_INTEGER = 9007199254740991
def _key_of(value: Any, key: str) -> Optional[str]:
"""A record's key as text, rendered the way collections.group-by-key renders
a group name so the two agree on what "the same key" means. None when absent.
"""
if value is None:
return None
if isinstance(value, str):
return value
# bool before int: in Python True is an int.
if isinstance(value, bool):
return "true" if value else "false"
if isinstance(value, int):
if abs(value) > SAFE_INTEGER:
raise ValueError('cannot diff by the out-of-range number %d at "%s"' % (value, key))
return str(value)
if isinstance(value, float):
raise TypeError('cannot diff by the fractional number %r at "%s"' % (value, key))
raise TypeError('cannot diff by the list or map at "%s"' % (key,))
def _same(a: Any, b: Any) -> bool:
"""Deep JSON equality, spelled out so all three languages agree: numbers by
value (1 equals 1.0), no coercion between types (True is not 1, "1" is not
1), lists in order, and a missing map field equal to a None one.
"""
if a is None or b is None:
return a is None and b is None
# Python's == says True == 1; JSON does not.
if isinstance(a, bool) or isinstance(b, bool):
return isinstance(a, bool) and isinstance(b, bool) and a == b
if isinstance(a, (int, float)):
return isinstance(b, (int, float)) and a == b
if isinstance(a, str):
return isinstance(b, str) and a == b
if isinstance(a, (list, tuple)):
return (
isinstance(b, (list, tuple))
and len(a) == len(b)
and all(_same(x, y) for x, y in zip(a, b))
)
if isinstance(a, dict) and isinstance(b, dict):
return all(_same(a.get(k), b.get(k)) for k in set(a) | set(b))
return False
def _index(records: Sequence[Mapping[str, Any]], key: str, side: str) -> Dict[str, int]:
"""Key every record of one list, refusing missing and duplicate keys."""
by_key: Dict[str, int] = {}
for i, record in enumerate(records):
k = _key_of(record.get(key) if isinstance(record, dict) else None, key)
if k is None:
raise TypeError('record %d in %s has no value at "%s"' % (i, side, key))
# Picking one of two duplicates would report changes that never happened.
if k in by_key:
raise ValueError('duplicate key "%s" in %s' % (k, side))
by_key[k] = i
return by_key
def diff_by_key(
before: Sequence[Mapping[str, Any]],
after: Sequence[Mapping[str, Any]],
key: str,
) -> RecordDiff[Mapping[str, Any]]:
"""Compare two lists of records matched by ``key``: what was added, what
was removed, and what changed and in which fields.
"""
for side in (before, after):
if isinstance(side, (str, bytes)) or not isinstance(side, (list, tuple)):
raise TypeError("diff_by_key needs two lists of records")
if not isinstance(key, str) or key == "":
raise TypeError("diff_by_key needs a non-empty key name")
before_keys = _index(before, key, "before")
after_keys = _index(after, key, "after")
added: List[Mapping[str, Any]] = []
changed: List[RecordChange[Mapping[str, Any]]] = []
unchanged = 0
for k, i in after_keys.items():
nxt = after[i]
j = before_keys.get(k)
if j is None:
added.append(nxt)
continue
prev = before[j]
names = set(prev) | set(nxt)
# sorted() on str is code point order, the order all three pin.
fields = sorted(name for name in names if not _same(prev.get(name), nxt.get(name)))
if fields:
changed.append(RecordChange(key=k, before=prev, after=nxt, fields=fields))
else:
unchanged += 1
removed = [before[j] for k, j in before_keys.items() if k not in after_keys]
return RecordDiff(added=added, removed=removed, changed=changed, unchanged=unchanged)Install
fune build
With that line in your source, in a Python project (language python in fune.project), fune build resolves it and nothing else, pins them in fune.lock, downloads only the Python package of each, and builds the code above into your project’s .fune/build, one readable file per capability with a header linking back here. Or pin a range in fune.project and build in one step:
fune add collections.diff
The manifest, vectors and README with only the Python implementation. Install it without the registry with fune add ./collections.diff-1.0.0-python.fune, or fetch it from a terminal with fune pull collections.diff@1.0.0:python.
The whole function, every language, is one file too: collections.diff-1.0.0.fune, 27,655 bytes, sha256 0416ed8558f592216df058b45d0fc42a83ef3f616cee47d44df8507dcd1a0546. It installs into a project of any language.
Customise it in your app
The seams this capability offers. Put a marker directly above a function of your own and fune build wires it into the built code; the package on the registry is not changed, the built file’s header lists it under CUSTOMISED, and fune hooks lists every hook in the project. How hooks work.
before — your function gets the arguments and returns them, changed or not, or throws to refuse the call.
# fune: before collections.diff
after — your function gets the result and the arguments, and returns the final result.
# fune: after collections.diff
replace — it requires no other capability, so there is no dependency to replace.
step — your function runs at a numbered point inside the function’s body, receives the in-scope values it names as parameters, and may return replacements. List the points with fune show collections.diff --steps.
# fune: step collections.diff after <n|label>
Tests
A version published now needs at least 8 tests for every function, and one that expects the error for each function that throws; the registry refuses it otherwise. fune verify --all runs each case in TypeScript, Python and Rust, and a project runs them again with fune verify. This page lists the cases; it does not run them. The exact JSON is vectors.json.
| Case | Arguments | Expected | |
|---|---|---|---|
| one of each: an addition, a removal, a change and an untouched record | before ×3, after ×3, id | → | added ×1, removed ×1, changed ×1, unchanged 1 |
| identical lists: nothing added, removed or changed | before ×2, after ×2, k | → | added , removed , changed , unchanged 2 |
| reordering is not a change | before ×2, after ×2, k | → | added , removed , changed , unchanged 2 |
| from nothing: everything is added, in after's order | , after ×2, k | → | added ×2, removed , changed , unchanged 0 |
| to nothing: everything is removed, in before's order | before ×2, , k | → | added , removed ×2, changed , unchanged 0 |
| changed fields are sorted by code point, including a field added and a field removed | before ×1, after ×1, id | → | added , removed , changed ×1, unchanged 0 |
| changed records follow after's order | before ×2, after ×2, k | → | added , removed , changed ×2, unchanged 0 |
| 1 and 1.0 are the same number, so a float round trip is not a change | before ×1, after ×1, k | → | added , removed , changed , unchanged 1 |
| a number that became a string did change | before ×1, after ×1, k | → | added , removed , changed ×1, unchanged 0 |
| true is not 1 | before ×1, after ×1, k | → | added , removed , changed ×1, unchanged 0 |
Show the other 8 tests
| Case | Arguments | Expected | |
|---|---|---|---|
| a null field and a missing field are the same, so dropping nulls is not a change | before ×1, after ×1, k | → | added , removed , changed , unchanged 1 |
| nested lists and maps are compared deeply; list order matters | before ×1, after ×1, k | → | added , removed , changed ×1, unchanged 0 |
| a key of 1 and a key of "1" match; the key field itself is reported as changed type | before ×1, after ×1, id | → | added , removed , changed ×1, unchanged 0 |
| a duplicate key in before is an error, not a guess | before ×2, , k | → | error: duplicate key "a" in before |
| a duplicate key in after is an error | , after ×2, k | → | error: duplicate key "7" in after |
| a record with no value at the key is an error | before ×1, after ×1, k | → | error: record 0 in after has no value at "k" |
| a fractional key is an error | before ×1, , k | → | error: cannot diff by the fractional number |
| an empty key name is an error | , , | → | error: needs a non-empty key name |
More from the author
CHANGED FIELDS are listed by name, sorted by Unicode code point. Sorting, rather than keeping the records' own field order, is deliberate: JavaScript reorders integer-like object keys ("2" before "a") and the three languages would otherwise disagree about the same records.
EQUALITY is deep JSON equality, spelled out the same way in all three languages rather than inherited from each one's `==`:
- numbers compare by value, so `1` and `1.0` are equal (JSON does not distinguish them, and a round trip through a float column must not show up as a change); - `true` is not `1`, and `"1"` is not `1`: a value that changed type did change; - lists are equal when they have the same length and equal items in the same order; - maps are equal when every field is equal, where a missing field and a null field count as the same. That is the house rule across the collections capabilities, and it stops a serialiser that drops nulls from making every record look changed. It applies to the top-level fields too: a field that goes from null to absent is not reported.
KEYS are rendered the way `collections.group-by-key` renders them: strings are themselves, whole numbers their digits, booleans `true`/`false`. So a record keyed `1` in one list and `"1"` in the other is the same record (and the key field itself is then reported as changed, because its type did change). A fractional number, a list or a map at the key is an error.
ERRORS, not guesses: a record with no value (or null) at the key, and a key that appears twice in the same list. A diff that silently picked one of two duplicates would report changes that never happened. Also an empty key name and a non-list argument. Inputs are never modified; the records in the result are the records passed in.
The result types are generic (`RecordDiff<T>`, `RecordChange<T>`) and this function returns them over `record`. That keeps the door open for a typed variant, and it is also how the manifest gets a record-valued field past the Python type generator, which does not yet import `Any` for a bare `record` field.
Files
| Path | Bytes |
|---|---|
| README.md | 2,642 |
| impl/python.py | 4,233 |
| impl/rust.rs | 5,898 |
| impl/typescript.ts | 4,816 |
| vectors.json | 5,433 |