feat(functions): add string distance/similarity functions

[nish2292]

Changes Made

Add four pairwise string distance/similarity functions as pure Rust scalar UDFs:

• levenshtein_distance - minimum edit distance (Int64)
• jaro_similarity - similarity score 0.0-1.0 (Float64)
• jaro_winkler_similarity - Jaro with prefix bonus (Float64)
• damerau_levenshtein_distance - Levenshtein + transpositions (Int64)

Follows the existing hamming_distance_str pattern. No external dependencies. Exposed via daft.functions API and as Expression methods. Null-safe (returns null when either input is null).

Related Issues

Fixes #6794

Test Plan

24 pytest test cases in tests/functions/test_string_distance.py:

• Levenshtein (6 tests): basic edit distance, empty strings, null handling, identical strings, single-char edits (substitution/insertion/deletion), expression method
• Jaro (6 tests): identical strings, completely different strings, known reference values (martha/marhta = 0.944444), null handling, empty vs nonempty, expression method
• Jaro-Winkler (6 tests): identical strings, prefix bonus >= Jaro invariant, known reference values (martha/marhta = 0.961111), no common prefix (JW == Jaro), null handling, expression method
• Damerau-Levenshtein (6 tests): basic transposition, transposition vs standard Levenshtein (ab/ba = 1 vs 2), empty strings, identical strings, null handling, expression method

DAFT_RUNNER=native pytest tests/functions/test_string_distance.py -v
24 passed in 0.06s

Rust compilation verified:

cargo check --workspace  # zero errors
cargo clippy -p daft-functions-utf8 --no-deps  # zero warnings on new code

AI Disclosure

AI-assisted implementation (Claude Opus 4.6).

[nish2292]

Seems like the failure in the PRBs are unrelated like in #7063 caused by a non-deterministic seeded shuffle..

[greptile-apps]

Greptile Summary

This PR adds four pairwise string distance/similarity functions — levenshtein_distance, jaro_similarity, jaro_winkler_similarity, and damerau_levenshtein_distance — as Rust scalar UDFs, following the established hamming_distance_str pattern. A shared generic helper binary_str_distance in utils.rs eliminates per-function boilerplate for null tracking, broadcasting, and array construction.

• Four new Rust UDFs in daft-functions-utf8, each delegating evaluation and field inference to shared helpers in utils.rs.
• Python exposure via daft.functions and Expression methods; damerau_levenshtein_distance is correctly documented as the OSA variant with a disclaimer that results may diverge from true Damerau-Levenshtein.
• 24 pytest cases covering correctness, null propagation, empty-string edge cases, scalar broadcasting, and expression-method equivalence.

Confidence Score: 5/5

Safe to merge — all four algorithms are correctly implemented, null propagation and scalar broadcasting work as intended, and the OSA distinction for Damerau-Levenshtein is disclosed at both the Rust and Python documentation levels.

The shared binary_str_distance helper in utils.rs eliminates the previously flagged boilerplate duplication. The Jaro algorithm was manually verified against the martha/marhta reference (0.9444). Levenshtein's two-row space optimization is correct and symmetric. Jaro-Winkler's mul_add formula matches the standard. The OSA variant disclaimer is present in the Python docstring. Tests cover all critical edge cases including null propagation and scalar broadcasting in both directions.

No files require special attention.

Important Files Changed

Filename	Overview
src/daft-functions-utf8/src/utils.rs	Adds binary_str_distance and binary_str_distance_to_field shared helpers; correctly handles null tracking via NullBufferBuilder, scalar broadcasting, full-null short-circuit, and empty array cases.
src/daft-functions-utf8/src/levenshtein.rs	Correct Wagner-Fischer implementation using two-row O(min(n,m)) space; safely swaps shorter/longer strings for inner-loop efficiency.
src/daft-functions-utf8/src/jaro.rs	Correct Jaro implementation; match_distance uses saturating_sub to avoid underflow; transposition counting loop verified against martha/marhta reference (0.9444).
src/daft-functions-utf8/src/jaro_winkler.rs	Correct Jaro-Winkler formula via mul_add; prefix-length capped at 4 characters; reuses compute_jaro_similarity to avoid duplication.
src/daft-functions-utf8/src/damerau_levenshtein.rs	Implements OSA variant; correctly documented at both Rust and Python docstring level including the concrete CA→ABC divergence example.
daft/functions/str.py	Four new Python wrappers with complete docstrings, doctests, type annotations, and OSA disclaimer; follows hamming_distance_str pattern exactly.
daft/expressions/expressions.py	Adds four Expression methods using inline imports, consistent with the existing hamming/regexp_count/length_bytes pattern in the same file.
tests/functions/test_string_distance.py	24 tests covering all four functions: correctness, empty strings, null propagation, known reference values, scalar broadcasting both directions, and expression-method equivalence.

_{Reviews (3): Last reviewed commit: "Merge branch 'main' into feat/string-dis..." | Re-trigger Greptile}

[cckellogg]

@greptileai

[cckellogg]

Thanks for the contribution. I think binary_str_distance might need to use the existing broadcast-aware UTF-8 input handling here.

Right now it sets len = left.len() and then indexes both arrays with left.get(i) / right.get(i). That works for column-column inputs with equal lengths, but it breaks valid scalar broadcast cases like:

levenshtein_distance(col("a"), "kitten")
levenshtein_distance("kitten", col("a"))

For col("a"), "kitten", right has length 1, so right.get(i) can go out of bounds once i > 0. For "kitten", col("a"), the helper uses left.len() as the output length, so it can produce only one row instead of broadcasting the left scalar across the column.

Could we reuse the same pattern as the nearby UTF-8 helpers, parse_inputs to compute/validate the expected output size, then create_broadcasted_str_iter for both sides? find_impl does something similar. Roughly it would looks like this:

left.with_utf8_array(|left| {
  right.with_utf8_array(|right| {
      let (is_full_null, expected_size) = parse_inputs(left, &[right])
          .map_err(|e| DaftError::ValueError(format!("Error in {name}: {e}")))?;

      if is_full_null {
          return Ok(
              DataArray::<T>::full_null(name, &return_dtype, expected_size).into_series(),
          );
      }
      if expected_size == 0 {
          return Ok(DataArray::<T>::empty(name, &return_dtype).into_series());
      }

      let left_iter = create_broadcasted_str_iter(left, expected_size);
      let right_iter = create_broadcasted_str_iter(right, expected_size);

      let mut values = Vec::with_capacity(expected_size);
      let mut validity = NullBufferBuilder::new(expected_size);

      for (l, r) in left_iter.zip(right_iter) {
          match (l, r) {
              (Some(l), Some(r)) => {
                  values.push(compute(l, r));
                  validity.append_non_null();
              }
              _ => {
                  values.push(T::Native::default());
                  validity.append_null();
              }
          }
      }

      let result = DataArray::<T>::from_field_and_values(field.clone(), values)
          .with_nulls(validity.finish())?;
      Ok(result.into_series())
  })
})

[nish2292]

Good catch, I've addressed this as per your suggestions. I confirmed that it resolved the OOB and length issues. Also added regression tests for scalar broadcasts.

[codspeed-hq]

Merging this PR will not alter performance

✅ 40 untouched benchmarks
⏩ 10 skipped benchmarks¹

---

_{Comparing nish2292:feat/string-distance-functions (1651d2e) with main (afa221d)}

Footnotes

1. 10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[cckellogg]

@greptileai