Skip to content

Troubleshooting: Infozillion, Operator, and Delivery Reports

Use this guide for SMS failures after the router creates a request. The outbound flow uses these tables:

  • sms_router.sms_request stores the client submission;
  • sms_router.sms_request_chunk stores operator and route chunks;
  • sms_router.sms_recipient stores each recipient and delivery status;
  • sms_router.outbox_event stores events waiting for Kafka;
  • sms_aggregator.aggregator_call_audit stores Infozillion calls;
  • sms_aggregator.balance_refund_log stores direct-SMS refund attempts.

Start with the public message ID. The aggregator architecture and aggregator process flows provide more detail.

SMS is failed or has not completed

The public message status comes from sms_recipient. Do not use only sms_request.status to diagnose the problem.

Find the SMS request

SELECT id AS sms_request_id,
       message_id,
       client_id,
       system_user_name,
       sms_type,
       total_recipients,
       total_cost,
       submitted_at
FROM sms_router.sms_request
WHERE message_id = :message_id;

If the query returns no row:

  • confirm the message ID;
  • confirm the environment;
  • check the original HTTP response and router logs.

Check chunks and recipients

SELECT c.id AS chunk_id,
       c.status AS chunk_status,
       c.mno_id,
       c.mno_user_id,
       c.sender_cli,
       c.channel_id,
       c.channel_name,
       c.recipient_count,
       c.server_txn_id,
       c.dispatched_at,
       r.id AS recipient_id,
       RIGHT(r.msisdn, 4) AS msisdn_last4,
       r.status AS recipient_status,
       r.dlr_poll_count,
       r.dlr_poll_next_at,
       r.dlr_received_at,
       r.error_code,
       r.error_message
FROM sms_router.sms_request_chunk c
JOIN sms_router.sms_recipient r ON r.chunk_id = c.id
WHERE c.sms_request_id = :sms_request_id
ORDER BY c.id, r.id;

Check the result:

State What it means
Chunk and recipient are PENDING Check the router outbox and Kafka. The aggregator has not completed the chunk.
Chunk is DISPATCHING The provider call may have happened. Check audit and provider records before changing the status.
Chunk and recipient are FAILED Check routing, credentials, recipient error fields, and the Infozillion audit.
Chunk is DISPATCHED and recipient is SUBMITTED Infozillion accepted the send. DLR polling owns the next update.
Recipient is REJECTED Check error_code and error_message for DND or invalid-number details.
Recipient is FAILED The delivery report says the SMS was not delivered. Check the DLR audit and refund.
Recipient is NOT_AVAILABLE All DLR checks ended without a known final result. It is not automatically refunded.
Recipient is DELIVERED with code 9099 Current policy treats it as delivered and charged, but it is not a confirmed handset delivery.

For one recipient, add AND r.msisdn = :normalized_msisdn to the query. Avoid exporting full recipient lists. Treat phone numbers, message content, provider responses, and credentials as restricted data.

Check the Infozillion audit

SELECT id,
       chunk_id,
       mno_user_id,
       api_endpoint,
       response_status,
       aggregator_code,
       server_txn_id,
       latency_ms,
       error_class,
       error_message,
       created_date
FROM sms_aggregator.aggregator_call_audit
WHERE chunk_id = :chunk_id
ORDER BY created_date;

The api_endpoint column identifies the type of Infozillion call:

api_endpoint value Call
send-sms SMS submission through an MNO route.
send-sms-iptsp SMS submission through an IPTSP route.
check-delivery-report Delivery-report check for a submitted SMS.

Use response_status, aggregator_code, error_class, and error_message to identify the result or failure.

The response_body column contains the response as JSON:

  • For send-sms and send-sms-iptsp, it contains the server and MNO transaction IDs, response codes, response messages, and local fallback details when the response was created by timeout or circuit-breaker handling.
  • For check-delivery-report, it contains the server result, overall delivery state, per-recipient MSISDN-Delivered or MSISDN-UnDelivered values, DND and invalid-MSISDN lists, and ANS/MNO response codes and messages.

The audit is best-effort. A missing row does not prove that no provider call occurred. Read response_body only when required because a delivery-report response can contain full phone numbers and other provider data.

Check the router outbox

If the chunk did not reach the aggregator, check its outbound event:

SELECT id, topic, status, retry_count, last_error,
       kafka_partition, kafka_offset, sent_at, created_date
FROM sms_router.outbox_event
WHERE aggregate_type = 'SMS_REQUEST_CHUNK'
  AND aggregate_id = CAST(:chunk_id AS text)
ORDER BY id;

Check status, retry_count, and last_error. A SENT row includes the Kafka partition and offset.

For a direct-SMS failure, use the refund row's idempotency_key to match sms_aggregator.balance_refund_log with client_service.refund_log. Campaign refunds are handled by sms-router-service. See Client billing and balance.

Messages remain PENDING

PENDING normally means the router created the chunk and recipients, but the aggregator has not completed them.

Check the following:

  1. The router outbox row is SENT, not PENDING or FAILED.
  2. Kafka group aggregator-integration has an assigned consumer for sms.outbound.<mnoUserId>.
  3. The consumer group has no unexpected lag.
  4. For a new MNO-user topic, allow up to metadata.max.age.ms, which is 30 seconds by default, for subscription refresh.
  5. The event contains chunkId, mnoId, mnoUserId, and at least one recipient.
  6. sms.outbound.<mnoUserId>.dlt does not contain the event.

DLT records are logged but are not replayed or refunded automatically. Do not republish an event until checking the chunk and provider state. Infozillion may already have received it.

Chunk remains DISPATCHING

The aggregator sets a chunk to DISPATCHING before calling Infozillion. The chunk can remain there if the service stops during the call or before saving the result.

SELECT id, sms_request_id, status, server_txn_id, mno_id, mno_user_id,
       sender_cli, dispatched_sender_config_id, dispatched_at
FROM sms_router.sms_request_chunk
WHERE id = :chunk_id;

The consumer does not retry a DISPATCHING chunk because Infozillion may already have accepted it.

Before changing the status, compare:

  • aggregator logs;
  • sms_aggregator.aggregator_call_audit;
  • Infozillion records;
  • recipient, timestamp, and transaction reference.

Do not reset the chunk to FAILED or PENDING without confirming the provider outcome. A reset can send the SMS twice. There is currently no automatic DISPATCHING recovery task or /internal/admin/poll-now endpoint.

Recipients remain SUBMITTED

SUBMITTED means Infozillion accepted the send and DLR polling is waiting for a final result. The first check is scheduled about three minutes after dispatch.

SELECT id, chunk_id, msisdn, status, server_txn_id,
       dlr_poll_count, dlr_poll_next_at, dlr_received_at,
       error_code, error_message
FROM sms_router.sms_recipient
WHERE chunk_id = :chunk_id
ORDER BY id;

Check the following:

  • dlr.poller.enabled is true on at least one healthy aggregator instance.
  • server_txn_id has a value.
  • dlr_poll_next_at is due.
  • The chunk has valid MNO and MNO-user configuration.
  • Infozillion DLR audit rows contain a response code or error.
  • The mnpspDlr circuit is not open.
  • The database connection pool is healthy and no long DLR transaction is blocking work.

DLR checks run near 3 minutes, 10 minutes, 30 minutes, 1 hour, 2 hours, 6 hours, 12 hours, 18 hours, and 23 hours. If the ninth check still has no known final result, the recipient becomes NOT_AVAILABLE.

Recipient becomes NOT_AVAILABLE

This means all nine DLR checks ended without a known final result.

Common causes:

  • repeated non-9000 responses from Infozillion;
  • missing chunk or credential data;
  • missing phone-number classification;
  • an unknown delivery-status value.

NOT_AVAILABLE is final but is not a confirmed failure. Current policy sends the final report and does not refund it. Before taking financial action, check the Infozillion or MNO record and compare the audit response with the expected MSISDN-Status format.

Elevated 9099 outcomes

Infozillion and local fallback handling can both produce code 9099. Current policy marks the chunk DISPATCHED, marks its recipients DELIVERED, and does not refund them.

Use the audit response_body and error fields to identify the source:

  • Infozillion returned 9099.
  • A local TIMEOUT occurred and the request may have reached Infozillion.
  • A local NOT_SENT error occurred, such as connection refusal or an open circuit.

The source does not change the current billing rule. Alert on a spike because DELIVERED in this case does not mean a confirmed handset delivery.

High outbound Kafka lag

Common causes include slow provider calls, too few partitions or consumers, provider incidents, listener errors, and repeated Kafka rebalances.

Check the following:

  1. Compare lag for each MNO-user topic in group aggregator-integration.
  2. Check Infozillion latency and response codes in aggregator_call_audit.
  3. Keep max.poll.records × worst-case processing time below max.poll.interval.ms. Defaults are 10 records, a 30-second provider timeout, and a 600-second poll interval.
  4. Check consumer membership changes and repeated rebalances.
  5. Confirm the main listener is not consuming .dlt topics.
  6. Scale partitions and consumers only within provider credential limits.

The current dispatch path does not call TpsLimiter, so tps_limit is not enforced there.

Sender route or credential failure

Routing and credential errors mark the chunk and recipients FAILED. They also create final reporting and a whole-chunk refund for direct SMS.

Check the following:

  • Event senderConfigId matches sms_request_chunk.sender_config_id.
  • The sender, selected MNO-user, and MNO are active.
  • An exact or wildcard CLI rule exists.
  • active_route is PRIMARY or a valid SECONDARY fallback.
  • A secondary route belongs to the same user and operator and has fallback_sender_cli.
  • The Jasypt master key can decrypt the stored credential.
  • MNO credentials have bill_msisdn. IPTSP routes use the effective CLI instead.

If an exact CLI rule exists but is invalid, routing does not fall back to the wildcard rule.

Provider rejects send-sms

A code other than 9000 or 9099 is a final submission failure.

Use the Infozillion audit query above and check the latest send row. Validate:

  • MNO-user credentials and the global API key;
  • billing MSISDN and effective CLI;
  • SMS type and long-SMS support;
  • normalized recipient limits;
  • the operator-approved campaignId for promotional SMS.

For a direct SMS, aggregator-integration-service creates the refund. For a campaign, sms-router-service owns the refund.

DLR says DND, invalid, or undelivered

These are final delivery results:

Provider classification Recipient state Direct refund
dndMsisdn REJECTED, code DND Recipient cost
invalidMsisdn REJECTED, code INVALID_MSISDN Recipient cost
MSISDN-UnDelivered FAILED with MNO code/message Recipient cost
MSISDN-Delivered DELIVERED None

For direct SMS, aggregator-integration-service registers a refund for the failed recipient. If refund registration fails, the status and report may still be saved; check logs and refund records before taking action. Campaign refunds are handled by sms-router-service.

Reporting does not reflect terminal status

Final reporting follows this path:

terminal sms_recipient
  -> sms_router.outbox_event
  -> sms.reporting
  -> client-service reporting data

Check the following in order:

  1. Confirm the recipient has a final status.
  2. Find a router outbox row with aggregate type REPORT and event type MESSAGE_FINALIZED.
  3. Check the outbox status, retry count, and error.
  4. Check Kafka lag and DLT records for sms.reporting and the client reporting group.
  5. Check client-service reporting errors and the report fact for the recipient ID.

The current code saves the terminal status and outbox event in one transaction. If the status exists without an outbox row, check for a manual database update or an older service version.

Prefix routing and mobile number portability

The router chooses the Bangladesh operator from configured phone-number prefixes. The aggregator does not call an MNP lookup service. A ported number may therefore use its original-prefix route unless an earlier process corrects it. This is a routing limitation, not a DLR failure.