Skip to content

Aggregator Integration Process Flows

This page follows an SMS chunk from the router outbox through Infozillion submission, DLR checks, reporting, and refunds. It describes the current behavior of aggregator-integration-service. See the architecture page for service roles and decisions, and the configuration reference for runtime settings.

End-to-end direct SMS flow

sequenceDiagram
    autonumber
    actor C as API consumer
    participant G as gateway-service
    participant R as sms-router-service
    participant L as client-service
    participant DB as PostgreSQL
    participant K as Kafka
    participant A as aggregator-integration-service
    participant I as Infozillion

    C->>G: POST /api/sms/messages
    G->>R: body + trusted client/user headers
    R->>R: validate, segment, route, price
    R->>L: idempotent balance deduction
    L-->>R: deducted
    R->>DB: commit request, chunk, recipients, outbound/reporting outbox
    R-->>C: accepted with messageId
    R->>K: OutboxPublisher sends sms.outbound.mnoUserId
    K->>A: SmsChunkEvent
    A->>DB: commit DISPATCHING claim
    A->>I: POST send-sms
    I-->>A: server and MNO response
    A->>DB: commit chunk/recipient result and audit
    alt terminal at send time
        A->>DB: insert MESSAGE_FINALIZED into router outbox
    else MNO accepted and DLR enabled
        A->>DB: recipient SUBMITTED, first poll near +3m
        loop until terminal or ninth attempt
            A->>I: POST check-delivery-report
            I-->>A: recipient delivery classifications
        end
        A->>DB: terminal recipient + MESSAGE_FINALIZED outbox
    end
    opt eligible direct failure
        A->>L: POST internal idempotent refund
    end
    R->>K: publish MESSAGE_FINALIZED from shared outbox
    K->>L: sms.reporting
    L->>DB: upsert reporting fact
Hold "Ctrl" to enable pan & zoom

The client request finishes before the provider receives the SMS. A successful API response means the router accepted and charged the request. It does not mean Infozillion or an MNO delivered it.

Outbound event publication and consumption

Router side

For each chunk whose recipients use the same route and message settings, the router saves:

  • sms_request_chunk with the selected route, charge, owner, channel, and optional campaign UUID;
  • one sms_recipient row per MSISDN;
  • an outbox_event for sms.outbound.<mnoUserId> with the chunk ID as its Kafka key;
  • a separate MESSAGE_SUBMITTED reporting event.

OTP and transactional chunks have one recipient. Promotional chunks have no more than 999 recipients. The aggregator does not split the chunk again or calculate a new price.

Aggregator side

flowchart TD
    Poll[Kafka polls outbound topics] --> Parse{JSON parses?}
    Parse -- no --> ImmediateDlt[IllegalStateException, DLT immediately]
    Parse -- yes --> Required{chunkId, mnoId, mnoUserId and recipients present?}
    Required -- no --> ImmediateDlt
    Required -- yes --> Dispatch[Invoke dispatcher]
    Dispatch --> Result{Unhandled exception?}
    Result -- no --> CommitOffset[Listener completes and offset can commit]
    Result -- yes --> Retry[5s exponential retry, max 60s interval]
    Retry -->|recovers within about 5m| Dispatch
    Retry -->|exhausted| Dlt[Publish original record to topic.dlt]
    Dlt --> DltConsumer[Separate DLT group logs payload]
Hold "Ctrl" to enable pan & zoom

The main regular expression accepts exactly one part after sms.outbound.. DLT records cannot return to the send consumer. When the recoverer writes to .dlt, it keeps the source partition number. The DLT topic must therefore have that partition.

The DLT is currently for investigation only. The consumer can read the event but does not call RefundService. If it cannot parse a DLT record, it logs the error and continues.

Dispatch claim and duplicate prevention

Dispatch has three parts because the database and the Infozillion HTTP call cannot share one transaction.

flowchart TD
    Event[SmsChunkEvent] --> Lock[SELECT chunk FOR UPDATE]
    Lock --> Exists{Chunk exists?}
    Exists -- no --> Skip[Return without provider call]
    Exists -- yes --> State{Stored status}
    State -- DISPATCHED --> Skip
    State -- DISPATCHING --> Reconcile[Skip duplicate and log reconciliation required]
    State -- other --> Route[Validate and resolve effective route]
    Route -->|invalid| ConfigFail[Mark chunk and recipients FAILED]
    Route -->|valid| Claim[Set DISPATCHING and commit]
    Claim --> Send[Call Infozillion with no DB transaction]
    Send --> Record[New transaction: audit and apply outcome]
    ConfigFail --> Direct{campaignRef is null?}
    Direct -- yes --> Refund[Register chunk refund]
    Direct -- no --> RouterRefund[Router campaign process owns refund]
Hold "Ctrl" to enable pan & zoom

If two consumers receive the same event, the chunk row lock lets only one continue. The first saves DISPATCHING. The second sees that state and skips the send. A repeated Kafka event after the result is saved sees DISPATCHED and also skips.

If the service stops after claiming the chunk but before saving the result, the chunk remains DISPATCHING. The service does not retry it because the provider may already have accepted the SMS. An operator must compare the provider transaction, audit data, and database record. No worker currently handles this state automatically.

Effective route selection

flowchart TD
    Start[senderConfigId from event] --> Legacy{Null?}
    Legacy -- yes --> LegacyRoute[Use event MNO, MNO-user and CLI]
    Legacy -- no --> Match{Equals chunk.senderConfigId?}
    Match -- no --> Fail[Configuration failure]
    Match -- yes --> Exact[Load exact CLI and wildcard rules]
    Exact --> Select{Exact exists?}
    Select -- yes --> ExactRule[Select exact, even if later invalid]
    Select -- no --> Wildcard[Select wildcard]
    ExactRule --> Active{activeRoute}
    Wildcard --> Active
    Active -- PRIMARY --> Parent[Validate active public sender graph]
    Active -- SECONDARY --> Secondary[Validate generated secondary, same tenant/operator, fallback CLI]
    Active -- invalid --> Fail
    Parent --> Persist[Persist effective route on chunk]
    Secondary --> Persist
    LegacyRoute --> Send[Build provider request]
    Persist --> Send
Hold "Ctrl" to enable pan & zoom

If an exact rule exists but is invalid, the service does not try the wildcard rule. The exact rule always takes priority. A secondary route is a manual configuration choice, not load balancing or automatic failover based on a response code.

Infozillion send construction

The service builds the request as follows:

  1. Read the username and password for the selected credential. The service decrypts the password.
  2. Use the credential's billMsisdn for MNO. For IPTSP, use the final CLI as the billing identity.
  3. Add the one service-wide Infozillion API key.
  4. Map PROMOTIONAL to transaction type P; map OTP/transactional to T.
  5. Map UCS2 to message type 3; all other current encoding values become 1.
  6. Add the selected route's long-SMS setting, CLI, normalized recipients, message, and operator-approved campaign ID.
  7. POST to the MNO or IPTSP send-sms URL selected by mno_type.

The service waits for the call to finish, up to one configured timeout. If the call fails, the circuit-breaker fallback creates a response with code 9099, localFallback=true, and cause TIMEOUT or NOT_SENT.

Send result decision table

flowchart TD
    Response[Send response] --> Code{serverResponseCode}
    Code -- 9099 --> Charged[Chunk DISPATCHED, recipients DELIVERED, no refund or DLR]
    Code -- other non-9000 --> Reject[Chunk/recipients FAILED]
    Code -- 9000 --> Toggle{dlrEnabled exactly false?}
    Toggle -- no --> Type{Effective mnoType}
    Type -- IPTSP --> Delivered[Recipient DELIVERED now]
    Type -- MNO --> Submitted[Recipient SUBMITTED, poll at +3m]
    Toggle -- yes --> SmsType{PROMOTIONAL?}
    SmsType -- yes --> Delivered
    SmsType -- no --> Ack{mnoResponseCode equals 1000?}
    Ack -- yes --> Delivered
    Ack -- no --> AckFail[Recipient FAILED]
    Reject --> Direct{Direct message?}
    AckFail --> Direct
    Direct -- yes --> ChunkRefund[Refund chunk cost]
    Direct -- no --> CampaignOwner[Router owns campaign refund]
Hold "Ctrl" to enable pan & zoom

The service also does the following:

  • An accepted response copies serverTxnId and mnoTxnId to recipients.
  • Code 9099 usually has no transaction reference, so it cannot be polled.
  • Send-terminal DELIVERED/FAILED recipients get dlr_received_at and one reporting outbox event immediately.
  • A rejected send with a code other than 9099 saves the server code and message on every recipient.
  • A DLR-disabled OTP/transactional acceptance without MNO code 1000 records MNO did not acknowledge send (no ANS 1000 ack).

Why 9099 is special

The current rule assumes that Infozillion charges for every 9099. Both a provider 9099 and a locally created fallback therefore become DELIVERED without a refund. The audit records whether the cause was TIMEOUT or NOT_SENT. Operators should monitor these results separately because DELIVERED represents the billing rule, not a confirmed DLR.

DLR scheduling and claiming

The DLR worker runs when dlr.poller.enabled=true. On each run, it:

  1. Selects up to dlr.poller.batch-size SUBMITTED recipients whose dlr_poll_next_at time has passed.
  2. Lock them with FOR UPDATE SKIP LOCKED, allowing another instance to claim other rows.
  3. Excludes records without serverTxnId from the request groups. Normal dispatch does not schedule these records.
  4. Group by serverTxnId, which normally corresponds to one promotional chunk or one single-recipient transactional chunk.
  5. Loads the MNO and MNO-user credential used to dispatch the chunk.
  6. POST one DLR request per transaction group.
  7. Increment the poll count, apply per-recipient results, and either finalize or schedule the next offset.

The whole scheduled method, including provider calls, runs inside one database transaction. When choosing the batch size, allow for the number of transaction groups multiplied by the provider timeout.

Nine-attempt curve

Attempt Target elapsed time from dispatch Delay after previous target
1 3 minutes 3 minutes from send
2 10 minutes 7 minutes
3 30 minutes 20 minutes
4 1 hour 30 minutes
5 2 hours 1 hour
6 6 hours 4 hours
7 12 hours 6 hours
8 18 hours 6 hours
9 23 hours 5 hours

Scheduler and processing time may make each check slightly late. If the ninth check still has no final result, the status becomes NOT_AVAILABLE, dlr_poll_next_at is cleared, and the final reporting event is created. This state does not receive an automatic refund.

DLR response classification

If the provider's serverResponseCode is not 9000, the service does not trust any recipient result. It schedules every recipient for another check or changes it to NOT_AVAILABLE after the last attempt.

For code 9000, precedence is:

  1. MSISDN in dndMsisdn becomes REJECTED with error code DND.
  2. MSISDN in invalidMsisdn becomes REJECTED with error code INVALID_MSISDN.
  3. Flat MSISDN-Delivered becomes DELIVERED.
  4. Flat MSISDN-UnDelivered becomes FAILED, using the MNO response code/message.
  5. A missing or unknown status remains SUBMITTED and is scheduled for the next check.

Every final result creates MESSAGE_FINALIZED. For a direct message, FAILED and REJECTED also register a recipient refund in a new transaction. If refund registration fails, reporting still completes and the log identifies the recipient for a manual refund.

Direct refund processes

Whole-chunk send failure

sequenceDiagram
    participant A as Dispatcher
    participant DB as sms_aggregator
    participant C as client-service

    A->>A: Determine terminal direct chunk failure
    A->>DB: Find chunk:id idempotency record
    alt not registered and positive chunk cost
        A->>DB: Insert PENDING refund
    end
    A->>C: POST refund with userId, amount, key, reason
    alt accepted or replayed
        A->>DB: REFUNDED, refundedAt, attempts+1
    else call fails
        A->>DB: remain PENDING, attempts+1, lastError
    end
Hold "Ctrl" to enable pan & zoom

The first refund attempt happens after the dispatch result is saved. A refund error therefore cannot roll back the send state.

Per-recipient DLR failure

The DLR process calls registerRecipientRefund in a REQUIRES_NEW transaction. It only creates the refund record and does not make an HTTP call while finishing the DLR batch. The scheduler sends the refund later.

Retry state machine

stateDiagram-v2
    [*] --> PENDING: refund registered
    PENDING --> REFUNDED: client-service accepts or replays key
    PENDING --> PENDING: HTTP attempt fails and attempts < 10
    PENDING --> FAILED: next scan sees attempts >= 10
    REFUNDED --> [*]
    FAILED --> [*]
Hold "Ctrl" to enable pan & zoom

The local record and client-service use the same idempotency key. With the default 30-second scan, a refund that always fails usually becomes FAILED about five minutes after the first attempt. FAILED records are not retried automatically.

Campaign correlation

Campaign messages and direct promotional messages may both have an operator-approved campaignId, so this field cannot identify the refund owner. Use only campaignRef, the internal campaign UUID.

flowchart LR
    Agg[Aggregator terminal recipient update] --> Shared[(sms_router.sms_recipient)]
    Shared --> Rollup[Router CampaignDlrRollupPoller]
    Rollup --> Map[Map result to campaign_recipient]
    Map --> Retry{Transient and attempts remain?}
    Retry -- yes --> RetryPending[RETRY_PENDING]
    Retry -- no --> CampaignRefund[Router refund flow]
    Map --> Final[Recount campaign status and cost]
Hold "Ctrl" to enable pan & zoom

Aggregator refund methods do nothing when the chunk has campaignRef. This stops the aggregator and campaign refund processes from refunding the same charge.

Terminal reporting process

For every recipient with a final status, the aggregator creates MESSAGE_FINALIZED and adds it to the router outbox. This applies whether the status was set during sending, a DLR check, the last DLR attempt, or a configuration failure.

sequenceDiagram
    participant A as Aggregator transaction
    participant D as sms_router database
    participant O as Router OutboxPublisher
    participant K as Kafka sms.reporting
    participant C as client-service

    A->>D: Update recipient terminal state
    A->>D: Insert MESSAGE_FINALIZED in same transaction
    D-->>A: Commit both or neither
    O->>D: Claim pending report event
    O->>K: Publish keyed by recipientId
    K->>C: Consume event
    C->>C: Upsert reporting fact by recipientId
Hold "Ctrl" to enable pan & zoom

If the service cannot convert or insert the outbox event, it rolls back the related final-status transaction. Audit insertion is different: it is best-effort and does not block dispatch or finalization.

Operator failure scenarios

Symptom Likely state Safe action
Kafka lag grows but provider is healthy Too few partitions or consumers, slow provider calls, or topic information has not refreshed Check group assignment, metadata.max.age.ms, poll size, and provider response time. Do not publish the event again without checking the chunk state.
Chunk remains DISPATCHING Service stopped after saving the claim but before saving the result Check Infozillion and the audit first. Changing it to FAILED without checking can send the SMS twice.
Record appears in .dlt Invalid event or an error continued through about five minutes of retries Check the first error and chunk state. The DLT consumer does not refund or replay the event.
Many recipients remain SUBMITTED past due time DLR worker disabled, DB locks/pool pressure, provider circuit open, or missing effective configuration Check worker flag, due-row age, provider/audit, circuit state, and transaction duration.
Many NOT_AVAILABLE records No usable final DLR after all nine attempts Treat delivery as unknown and investigate provider coverage. The current policy does not refund these records.
Refund rows remain PENDING Client-service discovery/auth/network/ledger failure Verify Eureka, shared admin key, client-service health, and last error.
Refund records are FAILED All ten HTTP attempts failed Check the idempotency key in client-service before applying any manual credit.
9099 volume increases Provider failures, timeout, circuit-open, or connection problems Split real versus local fallback using audit provenance; remember all are currently billed/delivered by policy.
Reporting facts lag after recipients terminal Router outbox publisher or sms.reporting consumer backlog Inspect router outbox state and client reporting consumer, not aggregator Kafka send.

Process invariants

  • Do not send a chunk already marked DISPATCHED or DISPATCHING.
  • Use campaignRef, not campaignId, to identify campaign work.
  • Do not issue an aggregator refund for campaign work.
  • Do not check a DLR without a provider serverTxnId.
  • Do not treat NOT_AVAILABLE as a confirmed delivery failure.
  • Commit terminal recipient state and its reporting outbox event together.
  • Use the same refund idempotency key for registration, retry, and client-service.
  • Preserve effective MNO/MNO-user/CLI on the chunk so DLR uses the route actually dispatched.