What it does
The SFDataWarehouseSendQueue stateful service in TulaERP pushes ERP rows to the TulaDW ingestion API (http://localhost:8132 by default). It runs three modes side-by-side so both per-row updates and full reloads are covered.
Three push modes
flowchart LR
subgraph ERP[TulaERP processes]
A[Operational code<br/>e.g. OrdersController]
L[Background loop<br/>every 15 min]
F[On-demand<br/>SendFullDatabaseAsync]
end
subgraph SF[Stateful service]
Q[(Reliable Queue<br/>DataWarehouseQueue)]
DL[(Dead-letter Queue)]
R[Sender]
end
DW[(TulaDW API<br/>:8132)]
A -->|EnqueueAsync| Q
L -->|paginated dim refresh| R
F -->|full DB walk| R
Q -->|batch up to 100| R
R -->|POST /api/dataload/load| DW
R -->|after 5 retries| DL
classDef erp fill:#0a0f1a,stroke:#00b4c5,color:#c8d0dc
classDef sf fill:#0c1120,stroke:#1cd4e6,color:#c8d0dc
classDef dw fill:#0a0f1a,stroke:#ed8936,color:#c8d0dc
class A,L,F erp
class Q,DL,R sf
class DW dw
| Mode | Trigger | What it sends |
|---|---|---|
| Incremental queue | App code calls EnqueueAsync(DataWarehouseMessage) |
Per-row updates as they happen (orders shipped, invoices generated, …) |
| Periodic dimension sync | Every 15 minutes | Members, Clients, Items, Invoices, MemberCalls — keeps slowly-changing dims fresh even if no enqueue happens |
| Full database sync | SendFullDatabaseAsync() over SF remoting |
Every dimension + every fact + dynamic dump of ConnectProd.*, BiometricsProd.*, GatewayProd.* |
Reliability
sequenceDiagram
autonumber
participant App as ERP App Code
participant SQ as Send Queue
participant Snd as Sender Loop
participant DW as TulaDW API
participant DLQ as Dead Letter Queue
App->>SQ: EnqueueAsync(message)
Note over Snd: Every ~2s
Snd->>SQ: dequeue up to 100
Snd->>Snd: group by TargetTable
Snd->>DW: POST /api/dataload/load
alt success
DW-->>Snd: 200 OK
else failure
DW-->>Snd: error
Snd->>Snd: backoff 2s → 60s
Snd->>DW: retry (up to 5x)
alt still failing
Snd->>DLQ: move to dead letter
end
end
- Batching: up to 100 messages per dequeue cycle, grouped by destination table to minimize round-trips.
- Retry: 5 attempts with exponential backoff (2s base, 60s ceiling).
- Dead-letter: failures past 5 retries land in
DataWarehouseDeadLetterQueuefor inspection — not auto-replayed. - HTTP timeout: 120s per call.
Tables registered on startup
The service registers schemas with the warehouse so it can warn on unmapped fields:
| Dimensions | Facts |
|---|---|
Dim_Date, Dim_Members, Dim_Client, Dim_Clinician |
Fact_Invoice |
ERP_Orders, ERP_OrderShipments, ERP_BillingEvents, ERP_Inventory |
For the full set of valid target tables, query GET /api/dataload/tables against TulaDW.
Triggering a full sync
From any caller with a Service Fabric remoting reference to IDataWarehouseSendQueueService:
await proxy.SendFullDatabaseAsync();
This sets a flag the loop checks at the top of each iteration — no parameters, always re-syncs everything. Expect a multi-minute run for full ERP datasets.
Where to look in code
| File | What's in it |
|---|---|
SFDataWarehouseSendQueue/SFDataWarehouseSendQueue.cs |
The service — queue, sender, retry, full-sync orchestration |
Service Fabric/TulaSF.Interfaces/IDataWarehouseSendQueueService.cs |
The remoting contract |
Contract/TulaERP.Contract/Models/DataWarehouseMessage.cs |
The message shape |
TulaERP/Documentation/DataIngestionGuide.md |
The DW-side ingestion API reference |