Functional Weave
Code in Rust

collections.dedupe-by-key

Remove records that share a key value, keeping the first or the last of each, in input order.

1.0.0 · published 2026-10-03 by charlie · Anterra

Pinned by 14 tests, run in TypeScript, Python and Rust.

What it does

Removes records that share the value at `key`, keeping either the first or the last record of each set of duplicates. `first` is "the original wins" (an import that must not overwrite); `last` is "the latest wins" (a change feed where later rows supersede earlier ones).

WHERE SURVIVORS SIT: every survivor keeps its own position relative to the other survivors. With `keep = last`, the surviving record sits where the last occurrence was, not where the first one was:

For example

  • dedupe_by_key(records ×3, id, first) → ×2 keep first: the earliest record of each key survives, in its own place
  • dedupe_by_key(records ×3, id, last) → ×2 keep last: the latest record survives, at the latest record's position
  • dedupe_by_key(records ×5, id, last) → ×3 three copies, keep last, interleaved with others

The function

The same function in TypeScript, Python and Rust, pinned by the same tests. Pick your language; the choice follows you around the registry.

pub fn dedupe_by_key(records: &[Value], key: &str, keep: &str) -> Vec<Value>
recordsrecord[]open JSON-ish maps; shapes may differ between records
keystringthe field whose value identifies a record
keepDedupeKeepwhich of a set of duplicates survives
returnsrecord[]a new list; each survivor stays at its own position in the input

The type it declares, generated into your project

// DedupeKeep is a string in Rust, one of: "first", "last".
// Parameters take it as &str and results hold it as String.

Your code names it in one line, in the file that uses it

fune!(collections.dedupe-by-key@^1);  // then call dedupe_by_key(…)
impl/rust.rs · 79 lines · open · raw

Imports name this capability’s declared dependencies, which fune builds next to it in your project; each one links to its page.

use super::funejson::Value;  ← the fune runtime: the JSON value the test vectors use; fune build keeps it only where a signature takes one
use std::collections::HashSet;

/// Largest integer JavaScript can hold exactly; beyond it the three languages disagree.
const SAFE_INTEGER: i64 = 9007199254740991;

/// The identity of a record, or `None` when it has none.
///
/// Rendered exactly as collections.group-by-key renders a group name, so the
/// two capabilities agree on what "the same key" means.
///
/// # Panics
/// Panics on a float, a list or a map, or an integer outside the safe range.
fn key_of(value: &Value, key: &str) -> Option<String> {
    match value {
        // A record with no key is not a duplicate of anything: two records
        // that both lack an id are two unknowns, not one thing seen twice.
        Value::Null => None,
        Value::Str(s) => Some(s.clone()),
        Value::Bool(b) => Some((if *b { "true" } else { "false" }).to_string()),
        Value::Int(i) => {
            if i.abs() > SAFE_INTEGER {
                panic!("cannot dedupe by the out-of-range number {} at \"{}\"", i, key);
            }
            Some(i.to_string())
        }
        Value::Float(f) => panic!("cannot dedupe by the fractional number {} at \"{}\"", f, key),
        _ => panic!("cannot dedupe by the list or map at \"{}\"", key),
    }
}

/// Remove records that share the value at `key`, keeping the first or the
/// last of each set.
///
/// Survivors keep their own positions: with "last", the survivor sits where
/// the last occurrence was, which is what "latest wins" means in a change feed.
///
/// # Panics
/// Panics on an empty key, an unknown `keep`, or an ungroupable key value.
pub fn dedupe_by_key(records: &[Value], key: &str, keep: &str) -> Vec<Value> {
    if key.is_empty() {
        panic!("dedupe_by_key needs a non-empty key name");
    }
    if keep != "first" && keep != "last" {
        panic!("keep must be \"first\" or \"last\", received \"{}\"", keep);
    }

    // Every key is rendered up front, so a bad value panics whichever mode runs.
    let keys: Vec<Option<String>> = records.iter().map(|r| key_of(r.get(key), key)).collect();

    let mut seen: HashSet<String> = HashSet::new();
    let mut survives = vec![false; records.len()];
    let mut order: Vec<usize> = (0..records.len()).collect();
    // Walking backwards for "last" makes the last occurrence the first one seen.
    if keep == "last" {
        order.reverse();
    }
    for i in order {
        match &keys[i] {
            None => survives[i] = true,
            Some(k) => {
                if seen.insert(k.clone()) {
                    survives[i] = true;
                }
            }
        }
    }

    records
        .iter()
        .enumerate()
        .filter(|(i, _)| survives[*i])
        .map(|(_, r)| r.clone())
        .collect()
}

pub fn fune_vector(args: &[Value]) -> Value {
    Value::Arr(dedupe_by_key(args[0].as_arr(), args[1].as_str(), args[2].as_str()))
}

Install

fune build

With that line in your source, in a Rust project (language rust in fune.project), fune build resolves it and nothing else, pins them in fune.lock, downloads only the Rust package of each, and builds the code above into your project’s .fune/build, one readable file per capability with a header linking back here. A crate’s build.rs runs it before every compile. Or pin a range in fune.project and build in one step:

fune add collections.dedupe-by-key
Download for Rust collections.dedupe-by-key-1.0.0-rust.fune · 10,163 bytes sha256 bbb4c9cb43cf4f848058e2515d610372d19b0f4e756400e7cf2fe9b7c6dcf5b8

The manifest, vectors and README with only the Rust implementation. Install it without the registry with fune add ./collections.dedupe-by-key-1.0.0-rust.fune, or fetch it from a terminal with fune pull collections.dedupe-by-key@1.0.0:rust.

The whole function, every language, is one file too: collections.dedupe-by-key-1.0.0.fune, 16,235 bytes, sha256 05480a9dc85245dd71b28201102ce3496e166eff75901d8ba8c05fb5ae139f26. It installs into a project of any language.

Customise it in your app

The seams this capability offers. Put a marker directly above a function of your own and fune build wires it into the built code; the package on the registry is not changed, the built file’s header lists it under CUSTOMISED, and fune hooks lists every hook in the project. How hooks work.

before — your function gets the arguments and returns them, changed or not, or throws to refuse the call.

// fune: before collections.dedupe-by-key

after — your function gets the result and the arguments, and returns the final result.

// fune: after collections.dedupe-by-key

replace — it requires no other capability, so there is no dependency to replace.

step — your function runs at a numbered point inside the function’s body, receives the in-scope values it names as parameters, and may return replacements. List the points with fune show collections.dedupe-by-key --steps.

// fune: step collections.dedupe-by-key after <n|label>

Tests

A version published now needs at least 8 tests for every function, and one that expects the error for each function that throws; the registry refuses it otherwise. fune verify --all runs each case in TypeScript, Python and Rust, and a project runs them again with fune verify. This page lists the cases; it does not run them. The exact JSON is vectors.json.

CaseArgumentsExpected
keep first: the earliest record of each key survives, in its own place records ×3, id, first → ×2
keep last: the latest record survives, at the latest record's position records ×3, id, last → ×2
three copies, keep last, interleaved with others records ×5, id, last → ×3
three copies, keep first records ×5, id, first → ×3
no duplicates leaves the list as it was records ×3, id, first → ×3
an empty list stays empty , id, last →
records without the key are always kept, never collapsed into one records ×5, id, first → ×4
the number 1 and the string "1" are the same key, as in group-by-key records ×2, id, last → ×1
case matters: "A" and "a" are different keys records ×2, id, first → ×2
booleans are keys too records ×3, flag, last → ×2
Show the other 4 tests
CaseArgumentsExpected
an unknown keep mode is an error records ×1, id, middle → error: keep must be "first" or "last"
an empty key name is an error records ×1, , first → error: needs a non-empty key name
a fractional number at the key is an error, not a guess records ×1, id, first → error: cannot dedupe by the fractional number
a list at the key is an error records ×1, id, first → error: cannot dedupe by the list or map

More from the author

[a#1, b#1, a#2] keep first -> [a#1, b#1] [a#1, b#1, a#2] keep last -> [b#1, a#2]

That is what a change feed means by "latest wins": the survivor is the latest row, in the latest row's place. A caller who wants the latest values in the original slot can dedupe with `last` and re-sort.

WHAT COUNTS AS THE SAME KEY: values are compared by the same rendering `collections.group-by-key` uses, so the two agree on what a key is. Strings are themselves, whole numbers are their decimal digits, booleans are `true` and `false`. That means the number `1` and the string `"1"` are the same key, as they are in a CSV, a query string and every JSON API that is loose about types. A fractional number, a list or a map at the key is an error, because there is no rendering of them all three languages agree on; so is a whole number beyond 2^53, which JavaScript cannot hold exactly.

MISSING KEYS ARE NEVER DUPLICATES: a record whose key is absent or null is always kept. Two records that both lack an id are two things we know nothing about, not one thing seen twice, and collapsing them would silently lose data. (This is where dedupe deliberately differs from group-by-key, which puts them in one "" group.)

Errors: a non-list, an empty key name, a `keep` other than `first` or `last`, and an ungroupable value at the key. The input is never modified.

Files

PathBytes
README.md1,856
impl/python.py2,851
impl/rust.rs2,902
impl/typescript.ts2,951
vectors.json2,969