Docs/Platform/Reliability & Recovery

Reliability & Recovery

Message states, failure handling, restart recovery, and administrative retry.

FireSonic is designed around persist-before-process semantics. Once an HL7 message is accepted, its state is tracked explicitly so processing, delivery failures, restart recovery, and administrative retry remain visible and recoverable.

Reliability model

Persist firstAccepted messages are stored before asynchronous processing begins.

Explicit statesProcessing outcomes remain visible for operations and troubleshooting.

Controlled retryRetriable failures use bounded attempts and backoff rather than uncontrolled loops.

Restart recoveryInterrupted work can be recovered from persisted state after a restart.

Normal flow

NORMAL FLOWpersist first · process asynchronously · deliver downstream

ReceiveHL7 accepted

→

PersistStored locally

→

ProcessHooks + mapping

→

ValidateSemantic rules

→

DeliverFHIR request

→

PROCESSEDCompleted

Failure states

FireSonic distinguishes processing, delivery, and semantic-validation failures because each has a different operational cause and recovery path.

Status	When it is used	Typical recovery
`PROCESSING_FAILED`	Conversion, script execution, mapping, or another internal processing step fails.	Correct the processing cause, then retry the message.
`DELIVERY_FAILED`	The generated Bundle cannot be delivered to the configured FHIR endpoint.	Restore downstream availability or credentials, then retry delivery.
`REJECTED`	Semantic validation or workflow policy intentionally prevents delivery.	Correct the underlying data or rule condition before retrying.

RECOVERY PATHfailed state → inspect → correct cause → retry

Failed stateFailure remains persisted

→

InspectReview message and diagnostics

→

Correct causeFix data, script, config, or downstream service

→

RetryReturn eligible message to processing

→

DeliveredProcessing completes

Automatic retry

Retriable processing and delivery failures use bounded retry attempts with backoff. This prevents transient failures from immediately becoming permanent while avoiding uncontrolled retry loops.

processor.max.retry.attempts=5
processor.base.backoff.ms=1000

When the configured retry policy is exhausted, the message remains in its failed state for inspection and later administrative retry.

FHIR delivery circuit breaker Commercial

The FHIR delivery circuit breaker protects the processing pipeline when the downstream FHIR server repeatedly fails, times out, becomes overloaded, returns HTTP 429/5xx responses, or has an authentication problem.

Downstream outage behavior

When the breaker opens, FireSonic continues accepting and persisting new HL7 messages while outbound FHIR delivery is paused. After the cooldown period, delivery is tested again automatically.

fhir.delivery.breaker.enabled=true
fhir.delivery.breaker.failure.threshold=3
fhir.delivery.breaker.recovery.success.threshold=2
fhir.delivery.breaker.cooldown.ms=30000

Processing circuit breaker Commercial

The processing circuit breaker protects the runtime from repeatedly executing the same failing processing path, for example after a broken Groovy script, mapper defect, or repeated runtime processing error.

When repeated identical failures cross the configured threshold, active workers are allowed to finish and new processing pauses. Newly accepted messages remain safely persisted in RECEIVED state until processing resumes after the cooldown.

processing.breaker.enabled=true
processing.breaker.failure.threshold=5
processing.breaker.window.ms=60000
processing.breaker.cooldown.ms=300000

Processing statuses

Status	Meaning
RECEIVED	Message accepted and persisted.
PROCESSING	Message currently being processed.
PROCESSED	Successfully converted and delivered.
FILTERED	Intentionally ignored by workflow logic.
REJECTED	Workflow or semantic validation prevented delivery.
DELIVERY_FAILED	FHIR delivery failed after the applicable retry policy.
PROCESSING_FAILED	Internal processing failed after the applicable retry policy.

Administrative retry

After the underlying problem is corrected, eligible failed messages can be returned to processing without requiring the upstream system to resend the original HL7 payload.

Capability	Notes
Single-message retry	Retry an eligible failed message from the Administration Console.
Batch retry Commercial	Retry multiple eligible failed messages in one administrative operation.

Restart recovery

Persisted records survive a FireSonic restart. If work was interrupted while a message was in PROCESSING, the runtime recovery process identifies stale processing records and returns eligible work to processing according to the configured recovery policy.

This prevents interrupted messages from remaining indefinitely locked in an in-progress state and allows recovery without resending the source HL7 message.