Skip to content

feat(sink): add HarnessSink for Harness AI Evals#4

Open
sunilgattupalle wants to merge 1 commit into
mainfrom
feat/harness-sink
Open

feat(sink): add HarnessSink for Harness AI Evals#4
sunilgattupalle wants to merge 1 commit into
mainfrom
feat/harness-sink

Conversation

@sunilgattupalle

Copy link
Copy Markdown
Collaborator

Summary

  • Adds HarnessSink — pushes evaluation scores directly into a Harness AI Evals run
  • Follows the same credentials-only constructor pattern as LangfuseSink; required context (harness_run_id, harness_dataset_item_id) comes from eval_case.metadata
  • Adds examples/harness_sink_example.py showing end-to-end usage

Design

The sink takes only credentials in the constructor (from params or env vars). Two metadata keys are required on each EvalCase:

  • harness_run_id — the run to write into (created externally by the user)
  • harness_dataset_item_id — the dataset item this case corresponds to

write() POSTs each item immediately. finalize() PATCHes all seen runs to completed.

Test Plan

  • Set HARNESS_ACCOUNT_ID, HARNESS_API_KEY, HARNESS_PROJECT_ID env vars
  • Create a dataset + eval + run in Harness AI Evals
  • Run python examples/harness_sink_example.py with real IDs
  • Verify scores appear in the Harness UI under the run

Reads harness_run_id and harness_dataset_item_id from eval_case.metadata,
posts items immediately in write(), and marks the run complete in finalize().
Follows the same credentials-only constructor pattern as LangfuseSink.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

AI-Session-Id: ccc01b30-2601-497f-9554-dece2a213c53
AI-Tool: claude-code
AI-Model: unknown
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants