Platform Reliability and Operations

The platform accepts and saves requests before sending messages in the background. This gives clients a quick response and allows saved work to continue after a temporary interruption. The gateway, service directory, databases, message queue and background tasks work together to keep the platform available.

How API requests reach services

gateway-service is the public entry point. It sends each request to the service responsible for that feature:

Path	Destination
/auth/**, /api/me/**	Identity/session capabilities
/api/sms/**	Direct messaging
/api/campaigns/**	Campaign management
/admin/clients/**	Client/user/balance administration
/admin/reports/**	Reporting and exports
/admin/sms-config/**	Routing and pricing administration
/internal/clients/**	Internal balance operations

Several gateway and backend instances can run at the same time. They share session information through Redis and find available services through the service directory.

Finding available services

Each service tells discovery-service its name, address, port and health. It regularly sends a heartbeat to show that it is still available. If the heartbeat stops, that service instance is removed from the available list. Other services use this list to find client-service, sms-router-service and other dependencies.

This allows services to restart or add more instances without changing public URLs. Operators should monitor the service directory and heartbeat age because outdated information can temporarily send requests to unavailable instances.

Saving work before background processing

The platform saves the business record and a pending background event together:

flowchart TB
    Request[Accept an SMS, campaign or report update]
    Save[Save the work and its pending event together]
    Response[Return a response to the client]
    Publish[Send the event to the internal message queue]
    Process[Delivery or reporting service processes it]
    Retry[Wait and try publishing again]

    Request --> Save
    Save --> Response
    Save --> Publish --> Process
    Publish -->|Temporary failure| Retry --> Publish

Hold "Ctrl" to enable pan & zoom

The internal message queue does not need to be available at the exact moment a request is accepted. Once the work is saved, publishing can continue later. OTP events are sent before transactional, promotional and reporting events.

An event may be published more than once. Receiving services check message, recipient or campaign IDs and saved status to avoid repeating completed work.

Preventing duplicate or conflicting work

Risk	How the platform reduces it
Two requests spend the same balance	The balance is locked while each charge is checked and applied
A client repeats a submission	An idempotency key identifies the retry and prevents a second charge or message
A queue event tries to send the same SMS again	The saved sending status shows whether the message is already being sent or has finished
Several service instances try to run the same scheduled task	A shared database lease allows one instance to run it at a time
Several workers try to process the same records	Each worker claims different records
The same reporting update arrives again	Recipient and campaign IDs prevent duplicate report records
The same refund or returned charge is requested again	A unique key prevents the balance from being returned twice
The same channel failure is counted again	Each failed recipient is counted once, and only one alert is created during the waiting period

The platform cannot always save a database update at the exact same moment that an external provider accepts a message. If the provider result is unclear, processing stops so an operator can check it before risking a duplicate message or refund.

Retrying failed work

The platform retries temporary failures without retrying forever:

• failed message-queue publishing waits and tries again for a limited number of rounds;
• background checks resume campaigns stuck while being created, built, sent or cancelled;
• temporary campaign delivery failures return to the queue after a configured delay;
• campaign refunds retry with the same key so the balance is not returned twice;
• direct-message refunds try up to ten times before requiring an operator;
• alert delivery pauses after five failed attempts and can restart for up to three more rounds;
• unreadable or repeatedly failing queue events are moved to a dead-letter topic (DLT) for review.

When all automatic attempts are used, the NOC must review the failed work, dead-letter events, unclear provider results and unresolved balance changes.

Health checks and diagnostics

Services provide checks that show whether they are running and ready to receive traffic. Internal diagnostic pages can also show:

• overall dependency health;
• database and migration status;
• performance measurements;
• scheduled tasks and their ownership leases;
• logging settings;
• environment and configuration values;
• available request paths;
• thread and heap dumps.

Restarting and adding service instances

• During shutdown, services allow active work a limited time to finish.
• More gateway instances can be added behind the load balancer because sessions and service locations are shared.
• Router instances use the database to ensure scheduled tasks and pending events are not processed by several instances at once.
• Delivery capacity depends on the number of message-queue partitions and the provider's response speed.
• Reporting capacity depends on message-queue partitions and database performance.
• Alert workers claim different pending alerts so they do not deliver the same work at the same time.

Rules to follow during an incident

1. Check the saved message status before placing sending work back on the queue.
2. Check the provider result before resetting a message stuck in DISPATCHING.
3. Check client-service using the idempotency key before applying a manual refund.
4. Use the internal campaignRef, not the operator-approved campaignId, when checking which service handles a refund.
5. Do not expose internal identity headers or direct service ports to external users.
6. Do not delete configuration or history directly from the database to bypass normal safety checks.
7. Do not assume a dead-letter event was replayed or refunded automatically.
8. An accepted request, response code 9099 or status NOT_AVAILABLE does not confirm delivery to the recipient.

What to monitor

Outcome	Monitor
API availability	Gateway success rate, errors and response time; session-check response time
Reliable request acceptance	Router database errors, oldest pending or failed event, message-queue publishing errors
Timely sending	Queue delay by MNO account and the oldest campaign waiting to be sent
Delivery-status updates	Overdue delivery checks, number of SUBMITTED messages and NOT_AVAILABLE volume
Correct balances	Charge and refund errors, age of unresolved refunds and number of returned charges
Report updates	sms.reporting delay, reporting dead-letter count and latest report update time
Channel health	Failure alerts, 9099 rate and provider failure causes
Scheduled tasks	Task lease age, stuck campaign count and latest task run

See the system architecture, installation and configuration, and troubleshooting playbooks.