Mastering Distributed Transactions with the Saga Pattern
A 3am Page: The Distributed Transaction Dilemma
It was 3 a.m. when the phone rang at the fintech startup's headquarters. The CTO, Alex, was jolted awake by the urgent call. The company's transaction system had failed, leaving hundreds of customer transactions in a state of limbo. As Alex rushed to his computer, he knew the stakes were high. Inconsistent transactions meant that some customers were charged without receiving their services, while others received services without being charged. The financial loss was mounting by the minute, and customer trust was hanging by a thread.
The Fallout
The startup had been growing rapidly, and with growth came complexity. Their once-reliable monolithic system was now a web of microservices, each handling different parts of the transaction process. But this distributed architecture had a critical flaw: without a robust transaction management system, inconsistencies were inevitable. The team had relied on traditional methods, but they were no match for the demands of a distributed system.
As the sun rose, the team gathered in the conference room, bleary-eyed but determined. They needed a solution, and fast. The financial implications were severe, with potential losses running into the hundreds of thousands. More importantly, the company's reputation was at risk. Customers were already voicing their frustrations on social media, and the pressure was on to restore trust.
The Search for a Solution
The team knew they needed a new approach to handle distributed transactions. The traditional ACID (Atomicity, Consistency, Isolation, Durability) properties that worked well in their old monolithic system were not feasible in their new architecture. They needed a pattern that could ensure consistency across multiple services without sacrificing performance or reliability.
As they brainstormed, the term "Saga Pattern" kept coming up. It promised a way to manage distributed transactions by breaking them into a series of smaller, manageable steps, each with its own compensating action in case of failure. Could this be the answer to their dilemma? The team was about to embark on a journey to master the Saga Pattern, hoping it would be the key to restoring their system's integrity and their customers' trust.
Before the Saga: Handling Transactions in a Monolithic World
In the early days of the fintech startup, the team relied on a monolithic architecture to manage their transactions. This approach was straightforward and effective for a single, unified application. At the heart of this system were ACID transactions, a set of properties that ensured reliable processing of database operations.
ACID Transactions in Monolithic Architectures
ACID stands for Atomicity, Consistency, Isolation, and Durability. These properties are crucial for maintaining data integrity:
- Atomicity ensures that a series of database operations are completed entirely or not at all. If one part of the transaction fails, the entire transaction is rolled back.
- Consistency guarantees that a transaction will bring the database from one valid state to another, maintaining all predefined rules.
- Isolation ensures that concurrent transactions do not interfere with each other, providing the illusion that each transaction is occurring in isolation.
- Durability means that once a transaction is committed, it will remain so, even in the event of a system failure.
In a monolithic architecture, these properties were relatively easy to enforce because all components of the application were tightly coupled and operated within a single database. The team could rely on the database management system to handle transaction integrity, which was crucial for their financial operations.
Challenges Faced with Scaling and Microservices
As the startup grew, so did the complexity of their system. The monolithic architecture began to show its limitations. Scaling the application to handle increased load became cumbersome, as any change required redeploying the entire application. This lack of flexibility hindered the team's ability to innovate and respond to market demands quickly.
The team decided to transition to a microservices architecture to overcome these challenges. This approach allowed them to break down the monolithic application into smaller, independent services, each responsible for a specific business function. However, this shift introduced new challenges in transaction management.
In a microservices architecture, each service typically has its own database, leading to the need for distributed transactions. The traditional ACID properties were difficult to maintain across multiple databases, as they were designed for a single, centralized database system. The team faced issues with data consistency and integrity, as transactions could span multiple services and databases.
The Emergence of Distributed Systems
The move to microservices was part of a broader trend towards distributed systems, which offered greater scalability and flexibility. However, managing transactions in such an environment required a new approach. The team needed a solution that could handle the complexities of distributed transactions without sacrificing the reliability and consistency that ACID transactions provided in their monolithic system.
This is where the Saga Pattern began to emerge as a potential solution. It offered a way to manage distributed transactions by breaking them down into a series of smaller, manageable steps, each with its own compensating action in case of failure. This approach promised to address the challenges the team faced with their new architecture, paving the way for more reliable and efficient operations in a distributed world.
Introducing the Saga Pattern: A New Hope
As the fintech startup team gathered in their conference room, the air was thick with urgency. Their system had recently suffered a failure due to inconsistent transactions, and the team was determined to find a solution. Enter the Saga Pattern—a beacon of hope in the world of distributed transactions.
What is the Saga Pattern?
The Saga Pattern is a design pattern that provides a way to manage distributed transactions in a microservices architecture. Unlike traditional ACID transactions, which are difficult to implement across distributed systems, the Saga Pattern breaks down a transaction into a series of smaller, manageable steps. Each step is a local transaction that updates a service and publishes an event or message to trigger the next step.
The core principles of the Saga Pattern include:
- Atomicity through Compensation: Instead of rolling back a transaction, the Saga Pattern uses compensating transactions to undo the effects of previous steps if something goes wrong.
- Isolation through Eventual Consistency: Each step in the saga is isolated, and the system eventually reaches a consistent state.
- Durability through Persistence: Each step's state is persisted, ensuring that the saga can resume after a failure.
Choreography vs. Orchestration
As the team delved deeper into the Saga Pattern, they encountered two distinct approaches: choreography and orchestration. Understanding the difference between these two is crucial for implementing the pattern effectively.
-
Choreography: In this approach, each service involved in the saga listens for events and decides when to act based on those events. There is no central coordinator; instead, services communicate through a series of events. This approach is decentralized and can lead to more resilient systems, but it can also become complex as the number of services grows.
-
Orchestration: Here, a central orchestrator manages the saga's flow, directing each service on what to do and when. This approach provides more control and can simplify the saga's logic, but it introduces a single point of failure and can become a bottleneck.
Addressing Distributed Transaction Challenges
The fintech team realized that the Saga Pattern could address several challenges they faced with distributed transactions:
-
Scalability: By breaking down transactions into smaller steps, the Saga Pattern allows systems to scale more effectively. Each service can handle its part of the transaction independently, reducing bottlenecks.
-
Reliability: With compensating transactions, the system can recover from failures without requiring a complete rollback. This increases the overall reliability of the system.
-
Flexibility: The Saga Pattern supports both choreography and orchestration, allowing teams to choose the approach that best fits their needs and architecture.
To visualize how the Saga Pattern works, consider the following diagram illustrating a simple saga using choreography:
In this diagram, each service listens for events and performs its local transaction. If a failure occurs, compensating transactions are triggered to maintain consistency.
Armed with this new understanding, the fintech team felt a renewed sense of hope. The Saga Pattern offered a structured way to manage distributed transactions, promising to restore their system's reliability and regain customer trust.
Implementing Your First Saga: A Step-by-Step Guide
The fintech startup team, still reeling from their recent system failure, gathered in the conference room. They knew they needed a robust solution to handle distributed transactions across their microservices architecture. The Saga Pattern seemed promising, but how would they implement it? Let's walk through the steps they took to set up their first Saga.
Setting Up a Simple Distributed System
Before diving into the Saga Pattern, the team needed a basic distributed system. They decided to use a simple e-commerce scenario involving three microservices: Order, Payment, and Inventory. Each service would be responsible for its own data and operations.
- Order Service: Handles order creation and updates.
- Payment Service: Manages payment processing.
- Inventory Service: Updates stock levels.
They chose to use Spring Boot for each microservice, with communication facilitated by Apache Kafka, a distributed event streaming platform. This setup allowed services to communicate asynchronously, a key requirement for implementing the Saga Pattern.
Implementing a Saga with Choreography
The team opted for the choreography-based Saga approach, where each service listens for events and reacts accordingly. This decentralized method suited their need for flexibility and scalability.
- Order Creation: When a new order is created, the Order Service publishes an
OrderCreatedevent to Kafka.
java
// OrderService.java
public void createOrder(Order order) {
// Save order to database
orderRepository.save(order);
// Publish event
kafkaTemplate.send("order-events", new OrderCreatedEvent(order.getId()));
}
- Payment Processing: The Payment Service listens for
OrderCreatedevents. Upon receiving one, it processes the payment and publishes aPaymentProcessedevent.
java
// PaymentService.java
@KafkaListener(topics = "order-events")
public void handleOrderCreated(OrderCreatedEvent event) {
// Process payment
processPayment(event.getOrderId());
// Publish event
kafkaTemplate.send("payment-events", new PaymentProcessedEvent(event.getOrderId()));
}
- Inventory Update: The Inventory Service listens for
PaymentProcessedevents. It updates the stock and publishes anInventoryUpdatedevent.
java
// InventoryService.java
@KafkaListener(topics = "payment-events")
public void handlePaymentProcessed(PaymentProcessedEvent event) {
// Update inventory
updateStock(event.getOrderId());
// Publish event
kafkaTemplate.send("inventory-events", new InventoryUpdatedEvent(event.getOrderId()));
}
Handling Failures and Compensations
The team knew that handling failures was crucial. They implemented compensation logic to revert actions if something went wrong.
- Compensation in Payment Service: If payment processing fails, the Payment Service publishes a
PaymentFailedevent, triggering a compensation action in the Order Service.
java
// PaymentService.java
public void processPayment(String orderId) {
try {
// Attempt payment processing
} catch (Exception e) {
// Publish failure event
kafkaTemplate.send("payment-events", new PaymentFailedEvent(orderId));
}
}
- Order Compensation: Upon receiving a
PaymentFailedevent, the Order Service marks the order as failed and initiates any necessary compensations.
java
// OrderService.java
@KafkaListener(topics = "payment-events")
public void handlePaymentFailed(PaymentFailedEvent event) {
// Mark order as failed
Order order = orderRepository.findById(event.getOrderId());
order.setStatus(OrderStatus.FAILED);
orderRepository.save(order);
}
By implementing these steps, the team successfully set up their first Saga. They could now handle distributed transactions with greater reliability, ensuring that their system remained consistent even in the face of failures. This foundational implementation paved the way for more complex Sagas and further optimizations.
Real-World Applications: Sagas in Action
As the fintech startup team delved deeper into the world of distributed transactions, they found themselves inspired by industry giants like Netflix and Uber, who had successfully implemented the Saga Pattern to tackle similar challenges. These companies provided a roadmap for how the Saga Pattern could be leveraged to enhance scalability and reliability in complex systems.
Netflix: Orchestrating Media Services
Netflix, a leader in streaming services, faced the daunting task of managing distributed transactions across its vast array of microservices. With millions of users streaming content simultaneously, ensuring a seamless experience was paramount. Netflix adopted the Saga Pattern, specifically using orchestration, to manage transactions across its services.
In Netflix's architecture, a central orchestrator service coordinates the sequence of operations required to fulfill a user request, such as starting a video stream. If any step in the process fails, the orchestrator triggers compensating actions to revert the system to a consistent state. This approach allows Netflix to maintain high availability and reliability, even when individual services encounter issues.
Benefits Observed:
- Scalability: By decoupling services and managing transactions through orchestration, Netflix can scale individual components independently, accommodating fluctuating user demand.
- Reliability: The ability to handle failures gracefully ensures that users experience minimal disruption, maintaining trust in the platform.
Uber: Choreographing Ride Requests
Uber, the global ride-sharing giant, employs the Saga Pattern using choreography to manage the complex interactions between its services. When a user requests a ride, multiple services—such as driver matching, fare calculation, and payment processing—must work in concert to complete the transaction.
In Uber's choreography-based implementation, each service autonomously listens for events and performs its part of the transaction. For example, once a driver is matched, the fare calculation service automatically calculates the cost, and the payment service processes the transaction. If any service fails, compensating actions are triggered to ensure the system remains consistent.
Benefits Observed:
- Decentralization: By allowing services to operate independently, Uber reduces the risk of a single point of failure, enhancing system robustness.
- Flexibility: The event-driven nature of choreography allows Uber to easily integrate new services and features without disrupting existing workflows.
Lessons Learned from Industry Implementations
The fintech startup team gleaned several valuable lessons from these industry examples:
-
Choose the Right Approach: The decision between orchestration and choreography depends on the specific needs of the system. Orchestration provides centralized control, which is beneficial for complex workflows, while choreography offers flexibility and resilience through decentralization.
-
Prioritize Fault Tolerance: Implementing compensating actions is crucial for maintaining system consistency. Both Netflix and Uber emphasize the importance of designing robust compensation logic to handle failures effectively.
-
Embrace Observability: Monitoring and logging are essential for diagnosing issues in distributed systems. Both companies invest heavily in observability tools to gain insights into transaction flows and quickly identify bottlenecks or failures.
By studying these real-world applications, the fintech startup team gained confidence in their ability to implement the Saga Pattern effectively. They understood that while the journey might be challenging, the potential benefits in scalability and reliability were well worth the effort.
Advanced Saga Techniques: Optimizing for Performance
As the fintech startup team delved deeper into the Saga Pattern, they realized that while it solved many of their distributed transaction issues, there were still challenges to address, particularly around performance. The team needed to ensure that their Saga implementations were not only reliable but also optimized for speed and resilience. Here, we explore advanced techniques they used to enhance performance.
Strategies for Reducing Latency
Latency can be a significant bottleneck in distributed systems, and the team knew that minimizing it was crucial for maintaining a seamless user experience. They focused on two main strategies:
- Parallel Execution of Independent Transactions: By identifying independent transactions within a Saga, the team could execute them in parallel, reducing the overall time taken. For instance, if a transaction involved updating a user's account balance and sending a notification, these could be processed simultaneously.
```python
import asyncio
async def update_balance():
# Simulate balance update
await asyncio.sleep(1)
return "Balance updated"
async def send_notification():
# Simulate sending notification
await asyncio.sleep(1)
return "Notification sent"
async def execute_saga():
results = await asyncio.gather(update_balance(), send_notification())
print(results)
asyncio.run(execute_saga())
```
In this Python example, update_balance and send_notification are executed concurrently, reducing the total execution time from 2 seconds to just over 1 second.
- Optimizing Network Calls: The team also optimized network calls by using asynchronous I/O operations and reducing the payload size. This was particularly effective in scenarios where the Saga involved multiple microservices communicating over the network.
Techniques for Improving Fault Tolerance
Fault tolerance is critical in ensuring that a system can recover gracefully from failures. The team implemented several techniques to enhance the fault tolerance of their Saga implementations:
-
Idempotency: Ensuring that operations are idempotent was a key strategy. This means that even if a transaction is executed multiple times due to retries, the outcome remains the same. For example, updating a user's account balance should be designed to handle duplicate requests without causing inconsistencies.
-
Timeouts and Retries: The team implemented timeouts and retry mechanisms to handle transient failures. By setting appropriate timeout values and retry limits, they could prevent the system from getting stuck on a failed transaction.
```python
import requests
from requests.exceptions import Timeout
def perform_transaction_with_retry(url, max_retries=3):
retries = 0
while retries < max_retries:
try:
response = requests.get(url, timeout=5)
if response.status_code == 200:
return response.json()
except Timeout:
retries += 1
print(f"Retrying... ({retries}/{max_retries})")
raise Exception("Transaction failed after retries")
perform_transaction_with_retry("https://api.example.com/transaction")
```
In this example, the transaction is retried up to three times in case of a timeout, improving the system's resilience to temporary network issues.
Balancing Consistency and Availability
The team faced the classic trade-off between consistency and availability, especially in a distributed environment. They adopted the following approaches to strike a balance:
-
Eventual Consistency: By embracing eventual consistency, the team allowed the system to remain available even if all nodes were not immediately consistent. This approach was suitable for scenarios where immediate consistency was not critical, such as updating user profiles.
-
Quorum-Based Approaches: For transactions requiring stronger consistency, the team used quorum-based approaches. This involved ensuring that a majority of nodes agreed on a transaction before it was considered committed. While this increased latency slightly, it provided a higher level of consistency.
The fintech startup's journey with the Saga Pattern was a testament to the power of thoughtful design and optimization. By implementing these advanced techniques, they not only improved the performance of their distributed transactions but also ensured that their system was robust and reliable, ultimately restoring customer trust and operational efficiency.
When to Use the Saga Pattern: Making the Right Choice
As the fintech startup team delved deeper into distributed transactions, they faced a critical decision: when should they employ the Saga Pattern? This choice was pivotal, as it would determine the reliability and efficiency of their operations. Understanding the scenarios where the Saga Pattern excels, its limitations, and how it compares to other patterns like Two-Phase Commit (2PC) was essential.
Scenarios Where the Saga Pattern Excels
The Saga Pattern shines in environments where:
-
High Availability and Scalability are Crucial: In distributed systems, especially those with microservices architectures, maintaining high availability is paramount. The Saga Pattern allows for asynchronous processing, which can enhance system responsiveness and scalability.
-
Long-Running Transactions: When transactions span multiple services and take a considerable amount of time, the Saga Pattern's ability to manage these without locking resources is invaluable. This is particularly beneficial in e-commerce platforms where order processing involves multiple steps across different services.
-
Complex Business Processes: For workflows that require multiple steps and involve various services, such as booking a trip (involving flights, hotels, and car rentals), the Saga Pattern provides a structured way to handle these complex transactions.
Limitations and Trade-offs
While the Saga Pattern offers numerous advantages, it also comes with trade-offs:
-
Eventual Consistency: Unlike ACID transactions, which guarantee immediate consistency, the Saga Pattern operates on eventual consistency. This means that while the system will eventually reach a consistent state, there might be temporary inconsistencies.
-
Complexity in Compensation Logic: Designing compensation actions for each step in the saga can be complex and error-prone. The team at the fintech startup had to meticulously plan these compensations to ensure that any failed transaction could be rolled back effectively.
-
Increased Latency: The asynchronous nature of the Saga Pattern can introduce latency, as each step in the saga may involve waiting for events to be processed by other services.
Comparisons with Two-Phase Commit (2PC)
The team also considered the Two-Phase Commit protocol, a traditional approach for ensuring atomicity in distributed transactions. Here's how the Saga Pattern compares:
-
2PC Guarantees Atomicity: 2PC ensures that all parts of a transaction either commit or roll back together, providing strong consistency. However, it can be a bottleneck in distributed systems due to its synchronous nature and the need for a coordinator.
-
Saga Offers Flexibility: Unlike 2PC, the Saga Pattern allows for more flexibility and resilience in distributed environments. It doesn't require a central coordinator, reducing the risk of a single point of failure.
-
Performance Considerations: While 2PC can lead to performance issues due to its locking mechanism, the Saga Pattern's asynchronous approach can improve throughput, albeit at the cost of immediate consistency.
For the fintech startup, the decision to use the Saga Pattern was clear. Their need for high availability, scalability, and the ability to handle complex, long-running transactions made the Saga Pattern the right choice. However, they remained mindful of its limitations, ensuring they designed robust compensation logic and prepared for eventual consistency.
Common Pitfalls and How to Avoid Them
As the fintech startup team delved deeper into implementing the Saga Pattern, they quickly realized that the path to mastering distributed transactions was fraught with potential pitfalls. Here are some common mistakes they encountered and how they learned to avoid them.
Misunderstanding Compensation Logic
One of the first challenges the team faced was grasping the concept of compensation logic. In a distributed transaction, when a step fails, it's crucial to undo the previous steps to maintain consistency. However, the team initially misunderstood this concept, thinking that compensation was merely a rollback, similar to traditional ACID transactions.
To clarify, compensation in the Saga Pattern involves executing specific actions to counteract the effects of a previous step. For example, if a payment is processed but the subsequent inventory update fails, the compensation logic might involve refunding the payment rather than simply reversing the transaction.
The team learned to:
- Clearly define compensation actions for each step in the Saga.
- Ensure that compensation logic is idempotent, meaning it can be applied multiple times without adverse effects.
- Test compensation scenarios thoroughly to ensure they work as intended.
Overcomplicating the Saga Design
In their eagerness to implement a robust system, the team initially overcomplicated their Saga design. They attempted to account for every possible edge case and failure scenario, leading to a convoluted and difficult-to-maintain system.
To streamline their design, the team adopted the following strategies:
- Start with a simple Saga and gradually add complexity as needed.
- Use choreography for simpler Sagas where services can react to events independently, and orchestration for more complex scenarios requiring a central coordinator.
- Regularly review and refactor the Saga design to eliminate unnecessary complexity.
By focusing on simplicity and clarity, the team was able to create a more maintainable and efficient Saga implementation.
Ignoring Monitoring and Observability
Initially, the team underestimated the importance of monitoring and observability in their Saga implementation. Without proper monitoring, it was challenging to detect and diagnose issues, leading to prolonged downtimes and frustrated customers.
To address this, the team implemented comprehensive monitoring and observability practices:
- Use distributed tracing tools like OpenTelemetry to track the flow of transactions across services.
- Set up alerts for anomalies and failures in the Saga execution.
- Implement logging at each step of the Saga to capture detailed information about the transaction state.
These practices enabled the team to quickly identify and resolve issues, ensuring a more reliable and resilient system.
By learning from these pitfalls, the fintech startup team was able to refine their approach to the Saga Pattern, ultimately achieving a robust and efficient distributed transaction system.
The Future of Distributed Transactions: What's Next for Sagas?
As the fintech startup team continues to refine their distributed transaction system, they find themselves pondering the future of the Saga Pattern. With the rapid evolution of technology, they wonder how emerging trends might influence their implementation and what innovations could further enhance their system's reliability and efficiency.
Emerging Technologies and Frameworks
The landscape of distributed transactions is continuously evolving, with new technologies and frameworks emerging to support more robust and scalable implementations. One such development is the rise of cloud-native platforms like Kubernetes, which offer built-in support for microservices orchestration. These platforms can simplify the deployment and management of Saga-based systems by providing tools for service discovery, load balancing, and scaling.
Additionally, frameworks like Spring Cloud Data Flow and Apache Camel are gaining traction for their ability to streamline the implementation of complex workflows, including Sagas. These frameworks offer pre-built components and connectors that can reduce the time and effort required to set up and manage distributed transactions.
Potential Improvements in Saga Implementations
The team is also excited about potential improvements in Saga implementations that could address some of the current limitations. One area of focus is enhancing the fault tolerance of Sagas. By leveraging techniques such as event sourcing and CQRS (Command Query Responsibility Segregation), developers can create more resilient systems that can recover from failures more gracefully.
Another promising development is the integration of distributed tracing tools like OpenTelemetry. These tools provide deep insights into the flow of transactions across microservices, enabling teams to identify bottlenecks and optimize performance. By incorporating distributed tracing into their Saga implementations, the fintech team can gain a clearer understanding of how their transactions are executed and where improvements can be made.
The Role of AI and Machine Learning in Transaction Management
As the team explores the future of distributed transactions, they recognize the potential of AI and machine learning to revolutionize transaction management. Machine learning algorithms can be employed to predict transaction failures and automatically trigger compensating actions, reducing the need for manual intervention.
Moreover, AI-driven analytics can help identify patterns and anomalies in transaction data, providing valuable insights for optimizing system performance. By integrating AI into their Saga-based system, the team can enhance its ability to adapt to changing conditions and improve overall reliability.
In conclusion, the future of distributed transactions and the Saga Pattern is bright, with numerous opportunities for innovation and improvement. By staying abreast of emerging technologies and leveraging advancements in AI and machine learning, the fintech startup team can continue to refine their system and ensure it meets the demands of an ever-evolving digital landscape.
Key Takeaways: Mastering the Saga Pattern
As the fintech startup team wraps up their journey with the Saga Pattern, they reflect on the transformative impact it has had on their distributed transaction management. Here are the key takeaways from their experience:
Recap of the Saga Pattern's Benefits and Challenges
The Saga Pattern emerged as a beacon of hope for the team, offering a robust solution to the challenges of distributed transactions. Its primary benefits include:
- Scalability: By breaking down transactions into smaller, manageable steps, the Saga Pattern allows systems to scale efficiently across microservices.
- Fault Tolerance: With built-in compensation mechanisms, the pattern ensures that failures in one part of the transaction don't cascade, maintaining system integrity.
- Flexibility: The choice between choreography and orchestration provides flexibility in implementation, catering to different architectural needs.
However, the team also faced challenges:
- Complexity: Designing compensation logic and managing state across distributed systems can be intricate.
- Consistency: Achieving eventual consistency requires careful planning and monitoring.
Practical Tips for Successful Implementation
The team's success with the Saga Pattern was not without its lessons. Here are some practical tips they found invaluable:
- Start Simple: Begin with a basic Saga implementation to understand the flow and nuances before scaling up.
- Design for Failure: Anticipate potential points of failure and design compensating transactions early in the process.
- Monitor and Observe: Implement robust monitoring and observability to track the state of transactions and quickly identify issues.
- Iterate and Improve: Continuously refine the Saga logic based on real-world performance and feedback.
Encouragement to Experiment and Iterate
The journey with the Saga Pattern is ongoing. The team encourages others to experiment with different configurations and approaches. The landscape of distributed transactions is ever-evolving, and staying adaptable is key. By iterating on their implementations and learning from each iteration, teams can harness the full potential of the Saga Pattern, ensuring reliable and efficient operations in their distributed systems.
