Bulk Study Update parser stalls¶
Confirmed execution defect¶
The parser previously partitioned records into bounded channels of 100 items, then drained the screening channel before starting the simple-update channel. A file with 101 simple-update rows filled the second channel and blocked the producer while the first consumer waited for completion. The error partition had the same problem because its reader started only after both update readers.
The failure is reproducible through the actual CSV parser: 100 simple-update rows pass, while 101, 119 and 401 rows fail to complete. The deployed project-management tag 11.43.0 contains the same partitioning calls. Parsing progress is reported before persistence, so a complete parsed-row count does not prove any corresponding batch committed.
The fix consumes one stream in batches of at most 400 records. It processes errors without a separate channel and awaits the screening and simple writes in each batch before admitting another batch. This also prevents a failed parser invocation from leaving detached write tasks behind. The first ten error rows and up to 100 progress errors remain bounded; result counts come from repository matched/modified results, not parsed progress.
Writes within one job are now sequential. This deliberately trades the former eight concurrent batches for bounded memory, deterministic batch ordering and a clear completion boundary. Future parallelism needs separate throughput evidence and failure-draining tests. Different jobs can still execute concurrently under Job Service's existing allocation policy.
The caller propagates its cancellation token through parsing. The existing repository methods do not accept that token: an in-flight write is awaited, then cancellation is observed before another write starts. This is cooperative cancellation, not transactional rollback, attempt fencing, or proof that an interrupted job made no writes.
Flag decision: this correctness fix is unflagged. It repairs a reproducible deadlock in an existing operation and adds no recovery endpoint, retry, cancellation publisher or production enablement. Deployment remains subject to the separate production approval boundary.
Current incident evidence and limits¶
Read-only diagnostics on 8 September found seven active slots: two old attempts on an absent instance and five newer attempts on the registered instance. MassTransit 8.4.0's default strategy counts active jobs per registered instance; no global concurrency limit was configured. Seven total rows against a per-instance limit of five therefore does not establish a global-limit violation.
An exact SQL command-to-Mongo embedded-job join found seven active jobs still Parsing with positive parsed counts. The one WaitingForSlot job also retained positive parsing progress from an earlier attempt. Private IDs were used only inside the read-only queries; reports retain ordinals and counts. Counters do not reconstruct committed writes, and a fresh service heartbeat does not prove that a particular attempt is alive or dead.
This fix does not recover existing attempts. Keep them quarantined pending exact-attempt liveness, input identity and committed-write reconciliation. In particular, never retry solely because the deadlock is fixed. Simple field assignment may be idempotent for unchanged input, but concurrent edits, screening effects and unknown partial commits still require reconciliation.
Verification¶
Run the focused parser/service tests:
dotnet test src/libs/project-management/SyRF.ProjectManagement.Core.Tests/SyRF.ProjectManagement.Core.Tests.csproj \
--filter 'FullyQualifiedName~BulkStudyUpdateParserRecoveryTests|FullyQualifiedName~StudyReferenceFileParserTests|FullyQualifiedName~ProjectManagementServiceStatisticsFenceTests|FullyQualifiedName~ProjectManagementServiceTests'
The regression suite covers the 100/101-row boundary, 119/401-row simple updates, more than 100 error rows, screening writes, concurrent jobs, cancellation during a write, write failure, and preservation of matched/modified counts on a simple-update replay. The replay test is not authorization for production retries.
Run the synthetic full-pipeline test in this worktree's isolated local environment:
It uploads 119 PDF-path updates through the normal UI/S3/job pipeline, requires CompletedParsing, and verifies every persisted PDF path. The isolated local rehearsal passed on 8 September 2026: the upload journey completed in 23.4 seconds and all four Playwright tests passed (three authentication setup tests and the bulk update journey). The local setup now configures the scoped LocalStack main-container name and enables upload-bucket versioning, both required by the notifier. Preview/staging rehearsal is still pending; this local result does not establish recovery safety for existing attempts.
Related recovery work: PR 2887 remains dry-run-only; PR 2900 contains the unproven authoritative recovery design. Stock CancelJob cannot carry an expected attempt or retry number. No production control event is authorized by these diagnostics or tests.