Skip to content

Communication

In distributed systems, services need to exchange information reliably and efficiently. The choice between synchronous and asynchronous communication impacts -
- Consistency Model (Strong vs Eventual)
- Latency SLA (P99, P95 tail latencies)
- System Coupling (degree of interdependence)
- Failure Recovery (fault isolation)
- Scalability (horizontal scaling capabilities)

Synchronous = Tight Coupling + Availability Amplification → Use for critical path only
Asynchronous = Loose Coupling + Operational Complexity → Use for background jobs
gRPC is the future → 10x faster than REST, but less accessible
Kafka is the standard → High throughput, replay capability, event sourcing
Idempotency is non-negotiable → Design all async handlers to be idempotent
Observability is critical → Monitor latencies, lag, error rates, queue depth
CAP theorem is reality → Partition tolerance mandatory, choose CP or AP
Complexity scales with scale → Simple systems can be synchronous, complex ones need async

Synchronous Communication

A caller service makes a request and blocks/waits for a response before proceeding. The request-response pattern is inherently synchronous.

Implementation Mechanisms

REST APIs (HTTP/HTTPS)

  • Protocol: HTTP/1.1 or HTTP/2
  • Serialization: JSON (text-based, 2-3 KB overhead)
  • Latency: 50-500ms per call depending on network/processing
  • Use Case: Simple CRUD operations, public APIs, browser clients

Considerations: - Stateless by design (easier horizontal scaling) - Connection overhead: TCP 3-way handshake (~5-10ms for distant regions) - Polling complexity: consumer must repeatedly ask for updates - Retry logic requires idempotency keys to prevent duplicates

GraphQL

  • Advantage: Eliminates over-fetching and under-fetching of data
  • Latency: Similar to REST but can reduce round-trips
  • Complexity: Query parsing and execution overhead (~2-5ms additional)
  • Use Case: Complex data aggregation, mobile clients with limited bandwidth

Trade-offs: - Solves data fetching problems but doesn't solve inherent synchronous coupling - Deeper query complexity can lead to N+1 problems in resolvers

Feign (Spring Cloud) with Service Discovery (Eureka).

  • Purpose: Client-side HTTP client with declarative interface
  • Discovery: Automatic service location via Eureka registry
  • Circuit Breaker Integration: Works with Hystrix/Resilience4j
  • Latency: Similar to REST (~50-500ms) + discovery lookup (~5-20ms) 
    Service A (Feign Client) → Eureka Registry (lookup ServiceB instance)
                        → HTTP call to ServiceB
                        → Blocking wait

Failure Scenario: - Service B instances crash → Eureka marks them unhealthy (takes 30-60 seconds) - New requests fail → Circuit breaker opens (fail-fast) - Cascading failures can occur if timeout not configured properly

gRPC (Remote Procedure Call)

  • Protocol: HTTP/2 with multiplexing
  • Serialization: Protocol Buffers (binary, ~300 bytes overhead)
  • Latency: 5-50ms (10x faster than REST due to binary format)
  • Performance: 1000+ concurrent streams per connection
  • Throughput: 100K+ requests per second per instance

Advantages: - Bi-directional streaming (client→server, server→client, both directions) - Server push capabilities - Native deadline/timeout management - Load balancing aware (gRPC can balance per request)

Use Cases: - Inter-service communication in microservices (no public API) - Real-time data streaming - High-frequency data updates (time-series, metrics) - Internal services where schema evolution is controlled

Example: 

service OrderService {
  rpc CreateOrder(OrderRequest) returns (OrderResponse);
  // Bi-directional streaming
  rpc StreamOrders(stream Order) returns (stream OrderStatus);
}

Key Issues with Synchronous Communication.

  • Service A directly depends on Service B's availability
  • Service B's latency becomes Service A's latency
  • Schema changes in B require coordinated deployments

Cascading Failure Example.

Order Service → Payment Service → Bank Service
     ↓              ↓                  ↓
1s timeout    + 1s timeout        (down)
     ↓              ↓
Result: 3 second timeout for user request
If 100 requests/sec → 100 connection threads blocked for 3 seconds
→ Rapid thread pool exhaustion → Denial of Service
Availability Math for the Same Call Chain - For synchronous serial dependencies, end-user success needs all required services to be available at the same time - A_total = A_order * A_payment * A_bank.

  • A_order = 99.99% = 0.9999
  • A_payment = 99.9% = 0.999
  • A_bank = 99.9% = 0.999

A_total = 0.9999 * 0.999 * 0.999 = 0.997901099 = 99.7901%

So failure probability is:

1 - 0.997901099 = 0.002098901 = 0.2099%

Annual downtime (525,600 minutes/year):

Downtime = (1 - A_total) * 525,600 = 0.002098901 * 525,600 ~ 1,103 minutes/year (~18.4 hours/year)

Availability Amplification.

  • Concept: If each service has 99.9% uptime, the composite system availability decreases
  • Math: 99.9% × 99.9% × 99.9% = 99.7% (3 hops)
  • Impact: At 99.7%, you have ~26 hours of downtime per year
  • Mitigation: Circuit breakers (fail fast), timeouts, fallbacks

3. Distributed Transaction Problem

  • ACID transactions impossible across network boundaries
  • Network partitions prevent atomic all-or-nothing outcomes
  • Compensation logic (saga pattern) required for consistency

Example - Distributed Transaction: 

Service A writes: user balance -= $100  [Success]
Network fails
Service B writes: order record += 1     [Fails]

Result: Inconsistent state - money taken but no order created

4. Timeout Configuration

Timeouts should be based on latency SLOs, not guesswork.

  • Too aggressive (for example 100ms): healthy but slow tail requests fail.
  • Too lenient (for example 60s): threads/connections stay blocked, increasing blast radius.
  • Practical rule: start with downstream P95 or P99 and add safety margin + jitter.

What P95 means (and other P values):

P means percentile of latency distribution. - P50 (median): 50% of requests finish at or below this latency. - P90: 90% finish at or below this latency. - P95: 95% finish at or below this latency. - P99: 99% finish at or below this latency (tail latency).

Example (10,000 Payment requests in 1 minute): - P50 = 40ms - P90 = 80ms - P95 = 120ms - P99 = 300ms

Interpretation: 95% of calls complete within 120ms, but the slowest 1% can still take around 300ms.

So when we set timeout to 250ms, we are intentionally allowing normal tail (P95) while still cutting off pathological slow requests before they block resources too long.

Senior guidance for choosing timeout target: - User-facing synchronous call: usually start near P99 (or P95 + safety margin) and tune using error budget. - High-QPS internal hop: prefer tighter timeout near P95 + margin to protect capacity. - Always combine timeout with retries (bounded), jitter, circuit breaker, and idempotency.

Example (timeout budget): - End-to-end API SLO: 800ms - Call chain: API Gateway -> Order -> Payment -> Bank - Budget split: Gateway(80) + Order(120) + Payment(250) + Bank(250) + buffer(100) - Set Payment timeout near observed tail (say P95=120ms) with margin, e.g. 250ms, not 60s.

This keeps failures fast and recoverable, and prevents thread pool lockup.

5. Retry Storms

A retry storm happens when many clients retry at the same time after a transient failure, multiplying load on an already unhealthy dependency.

  • If 10k requests fail and each client retries 3 times immediately, backend sees up to 40k attempts (1 original + 3 retries).
  • This can turn a short outage into sustained overload (thundering herd).

Mitigation pattern: - Exponential backoff + jitter - Exponential backoff with jitter is a retry strategy where clients wait progressively longer between retries (backoff), but add randomness (jitter) to avoid synchronized “retry storms” that can overwhelm a recovering service. It’s widely used in distributed systems, including AWS SDKs, to improve resilience and fairness.
- Retry only idempotent operations - Cap attempts and total retry time - Combine with circuit breaker and rate limiting.

Exponential backoff
On each retry, wait longer than the previous attempt (commonly 2x) to reduce pressure on an unhealthy dependency.

Example sequence: 1s -> 2s -> 4s -> 8s

Formula:

wait_interval = base * multiplier^n

Where: - base = initial delay (for example 1s) - multiplier = growth factor (commonly 2) - n = retry attempt number (0, 1, 2, ...)

Jitter
Add randomness to retry delay so clients do not retry at the exact same time.

Example:

retry_delay = 2000ms + random(0-200ms)

Without jitter, many clients retry in lockstep and create a thundering herd.
With jitter, retries are spread over time, reducing synchronized traffic spikes and improving recovery chances.

Formula:

delay = min(max_delay, base_delay * 2^attempt) + random(0, jitter)

Example delays (base=100ms, max=2s, jitter=0-100ms): - Retry 1: ~100-200ms - Retry 2: ~200-300ms - Retry 3: ~400-500ms - Retry 4: ~800-900ms - Retry 5: ~1600-1700ms

This spreads retries over time, reduces synchronized spikes, and improves recovery probability.

When to Use Synchronous Communication

Query Operations (Read-Only) - Fetching product catalog, user profiles, inventory status - No side effects → Safe to retry without concern

Transactional Workflows (Atomic Operations) - Payment processing: Must validate before committing - Inventory reservation: Must check stock before confirming order - Requirement: Immediate feedback needed for validity Strongly Consistent Operations - Distributed transactions using 2-phase commit (rare, use carefully) - When eventual consistency is not acceptable (compliance requirements)

Asynchronous Communication

A sender publishes a message to a broker and continues immediately without waiting for processing. Processing happens later by one or more consumers.

Architecture Overview

Producer → Message Broker → Consumer
         (fire & forget)    (processes when ready)

Key Properties: - Producer decoupled from Consumer - Broker guarantees message delivery (if configured) - Consumer processes at own pace - Enable backpressure (broker buffers messages)

Message Broker Responsibilities

  1. Message Persistence: Store messages durably (disk, replicated)
  2. Routing: Direct messages to correct consumers
  3. Redelivery: Retry if consumer fails to acknowledge
  4. Retention: Keep messages for configured duration
  5. Consumer Offset Management: Track what each consumer has read
  6. Fault Tolerance: Replicate data across multiple brokers

Asynchronous Patterns

Pattern 1: Point-to-Point (Queue)

Producer → Queue (Broker) (Message stays until consumed Only ONE consumer processes it Message deleted after acknowledgment)→ Consumer

Characteristics: - Delivery Guarantee: Exactly-once (with idempotent processing) - Consumers: One message, one handler - Ordering: Per-queue ordering (if single partition) - Retention: Until consumed and acknowledged - Examples: RabbitMQ, ActiveMQ, AWS SQS

Use Cases: - Task distribution: Job queues (background processing) - Work distribution: Order processing across multiple workers - Rate limiting: Decouple fast producer from slow consumer

Failure Handling: 

Message published → Consumer processes → Crashes before ACK
Broker detects no ACK → Retries with another consumer instance

Pattern 2: Publish-Subscribe (Topic)

Publisher → Topic (Broker) ─��→ Subscriber1
                            ├→ Subscriber2
                            └→ Subscriber3
Message goes to ALL active subscribers

Characteristics: - Delivery Guarantee: At-least-once per subscriber - Consumers: Message goes to all subscribers - Ordering: Per-partition ordering (Kafka) - Retention: Time-based or size-based - Examples: Kafka, RabbitMQ (with fanout), AWS SNS

Use Cases: - Event broadcasting: User signed up → multiple services react - Data replication: Update propagated to read replicas - Real-time notifications: Push to multiple clients

Critical Issue - Subscriber Availability: 

Scenario 1: Subscriber offline at publish time
→ Message lost (if not persistent topic)

Scenario 2: Subscriber offline but topic is persistent
→ Subscriber can replay from offset
→ Recovery is possible

Pattern 3: Event-Driven Architecture

Difference from Messaging: - Messaging: Explicit message format, tight coupling on schema - Events: Publish intent/state change, not contracts - Subscribers react to events based on their own business logic

Example: 

Event: UserRegisteredEvent {
  userId: "123",
  email: "user@example.com",
  timestamp: "2024-09-13T22:02:37Z"
}

Subscribers:
- Email Service: Send welcome email
- Analytics Service: Track registration metrics
- Recommendation Service: Initialize user profile

Advantages: - Loose coupling: Event structure changes don't affect all subscribers - Scalability: New subscribers can be added without modifying publisher - Auditability: Event log provides complete history

Network Partitions & Byzantine Failures

Scenario: Service A can't reach Service B

Service A      Network Partition      Service B
(Active)                              (Active)

A can't call B, so:
- Timeout occurs
- Circuit breaker opens
- Fallback logic executes (stale cache, default value)

B processes messages from queue independently
Result: Temporarily inconsistent state (A thinks B failed, B is fine)

Recovery: - When network heals, systems resync - Data reconciliation may be needed - Audit logs help identify inconsistencies

Cascading Failures

Scenario: Service B slow due to database overload

Service A → Service B (100ms, times out at 200ms)
Service C → Service A (100ms, times out at 200ms)
Service D → Service C (100ms, times out at 200ms)

A's 100 threads waiting for B → Exhausted
C's 100 threads waiting for A → Exhausted
D's 100 threads waiting for C → Exhausted
Entire system grinds to halt

Mitigation: - Circuit Breaker: Stop calling failing service, fail fast 

Closed (normal) → Threshold errors exceeded → Open (failing)
→ After timeout → Half-Open (test)
→ If success → Closed
→ If failure → Open
- Bulkheads: Isolate thread pools (don't use shared pool) 
Service A
├── Thread Pool 1 (for Service B calls) [10 threads max]
├── Thread Pool 2 (for Service C calls) [10 threads max]
└── Thread Pool 3 (for Service D calls) [10 threads max]

If B fails: Only Pool 1 exhausted, Pool 2 & 3 work fine
- Timeouts: Kill long-running requests - Fallbacks: Graceful degradation - Retry with Exponential Backoff 
Attempt 1: Immediate
Attempt 2: 100ms + random(0-50ms)
Attempt 3: 200ms + random(0-100ms)
Attempt 4: 400ms + random(0-200ms)
Attempt 5: 800ms + random(0-400ms) [then give up]
- Saga Pattern (Distributed Transactions) 
Order Service → Payment Service → Inventory Service
              → if Payment fails → Compensation (refund)
              → if Inventory fails → Compensation (cancel payment)

Production Considerations

Parameter Synchronous Asynchronous
Monitoring - P99, P95, P50 latencies
- Error rate (4xx, 5xx)
- Circuit breaker state changes
- Timeout occurrences		 - Consumer lag (offset - latest offset)
- End-to-end latency (publish to process)
- Message loss rate
- Queue depth (buildup indicates slow consumers)
- Redelivery rate (indicates failure)
Alerting - Latency SLO breaches
- Error rate spikes
- Circuit breaker opens
- Timeout spikes		 - Consumer lag > threshold
- End-to-end latency > SLO
- Message loss detected
- Queue depth > threshold
- Redelivery rate > threshold
Scaling Horizontal scaling: Add more instances
- Caching: Reduce repeated calls
- Database optimization: Faster responses → shorter timeout needed
 Partition keys: Distribute load evenly
- Consumer group scaling: Add instances
- Retention tuning: Balance disk space vs. replay capability
Security Considerations Authentication/Authorization -
- gRPC: mTLS (mutual TLS) for service-to-service
- REST: OAuth2, JWT tokens
- Kafka: ACLs, authentication plugins
 Data Encryption -
- In-transit: TLS/SSL
- At-rest: Disk encryption, broker-side
- End-to-end: Application-level encryption