[#708] Fix OutOfMemory during replication initialize with JDBC backend

[vharseko]

Fixes #708

Problem

Initializing a large replica (~40M entries) with the JDBC/PostgreSQL backend fails with OutOfMemoryError during the index initialization phase, with all indexes stuck at 0% complete and PostgreSQL transferring huge amounts of data.

Root cause: Storage.CursorImpl opened every tree cursor as a scrollable ResultSet over the whole table (select h,k,v from ... order by k). The postgres driver streams rows only for TYPE_FORWARD_ONLY statements with a fetch size — scrollable result sets are always materialized entirely in client memory. Import phase two opens such a cursor over id2entry (MostlyOrderedChunk.flip() → ImporterToChunkAdapter.flip()), so the driver tried to buffer all 40M entries in the heap. The giant query also monopolized the single connection shared by all phase-two index writers, which is why every index reported 0% complete. The same materialization happened for any cursor in normal operation (e.g. unindexed searches), and no configuration workaround exists (defaultRowFetchSize is ignored for scrollable result sets).

Fix

• CursorImpl now iterates with keyset-paginated forward-only batches (where k>? order by k + offset ? rows fetch next ? rows only, limit ?,? for mysql), fully draining and closing each batch statement. Cursor memory is O(batch) instead of O(tree). Batch size is configurable via org.openidentityplatform.opendj.jdbc.fetchsize (default 1000).
• Cursor positioning is resolved server-side: positionToKey is a point lookup on (h,k), positionToKeyOrNext → where k>=?, positionToLastKey → order by k desc, positionToIndex → offset. Semantics mirror the JE backend cursor (failed positioning leaves the cursor undefined).
• delete() issues a regular delete where h=? and k=? instead of CONCUR_UPDATABLE/deleteRow().
• For postgres, openTree creates create index if not exists k_<hash> on <table> (k): the (h,k) primary key cannot serve order by k, so without it every batch would be a full-table top-N sort. Existing deployments get the index on next backend open.
• For mysql, the old k tinyblob dialect cannot serve the keyset batches at all: the optimizer refuses prefix indexes (k(255)) for order by k and even for the range predicate — the plan stays full scan + filesort (verified on mysql:9.2). The dialect now uses k varbinary(255) (same 255-byte cap) with primary key(h,k), and openTree creates the full k_<hash> index through a metadata check, since mysql has no create index if not exists. The batch query then runs as an index range scan without filesort, same as postgres. Tables of the previous tinyblob dialect are not migrated and must be re-created.

Tests

PgSqlTestCase 33/33, MySqlTestCase 33/33, EncryptedTestCase (jdbc) 32/32 — including testExportLDIFAndImportLDIF and testRebuildAllIndex/testRebuildDegradedIndex, which exercise the exact import phase-two and ID2EntrySource cursor paths that previously OOMed, plus the new testCursorCrossesFetchSizeBatches: a cursor with fetchsize=2 over 7 keys checks iteration order across batch boundaries, positionToKey/positionToKeyOrNext/positionToIndex/positionToLastKey, delete() through the cursor with continued iteration, and the read-only guard. Verified in postgres and mysql that k_* indexes are created on all trees and serve the keyset batches with index scans.

Bundled test-infra fixes so the classes also pass in one TestNG fork (98/98) and over a dirty database: test_issue_496 accepts attributeOrValueExists when another backend test class already added the schema definitions to the shared in-JVM server, and the jdbc TestCase drops stale opendj_% tables before setUp — PgSql and Encrypted map the same tree names to the same tables, and leftovers encrypted with a lost cipher key broke the next run's cleanup.

[maximthomas]

I guess the migration from tinyblob to varbinary(255) fo MySQL could potentially cause data loss