Event-driven architecture (EDA) is a software architecture pattern that offers flexibility, scalability, and potential for vastly improved performance within High Performance (HPC) and High Throughput Computing (HTC) clusters compared to traditional polling-based architecture.
An event-driven architecture is primarily composed of event emitters, event consumers, and channels. Emitters are responsible for collecting and transmitting events on one or more channels. Consumers are responsible for receiving and forwarding and/or reacting to events on those channels. Channels are simply the pathways through which events are transmitted.
There are numerous ways to model an event-driven architecture. Some models guarantee event/message delivery (analogous to TCP acknowledgements/retransmission), while others are designed to fire and forget (analogous to UDP). It may be a requirement for some components to utilize a hybrid model. In a hybrid model, perhaps the method of transmission is chosen based on the event/message being collected/emitted? Regardless of the chosen model, components interoperate using loosely coupled services and well defined messages.
The asynchronous, loosely coupled nature of an event-driven model makes it possible for components to interact in a more efficient, non-blocking manner at scale. Blocking operations may be necessary, but should be avoided as much as possible. When events/messages are well defined, identical (multiple instances), similar, or disparate components within HPC/HTC systems can intelligently interoperate with relatively low latency/overhead.
Adaptive Computing’s HTC solution, Nitro, is an implementation of an event-driven architecture. The nitty gritty details are beyond the scope of this blog, but comparing and contrasting a small piece of the whole using a polling-based vs. an event-driven implementation for obtaining Nitro job status, illustrates the advantages of EDA over polling.
Nitro Job Status – Polling vs. Event-driven
For sake of illustration, let us run a Nitro job comprised of 100K tasks on a cluster of 100 compute nodes. That will average out to 1000 tasks per node. Please understand that this is a completely hypothetical, high-level description. Nitro implements some deep-dark magic that simply cannot be covered (NDA and complexity) in the scope of this blog post. However, discussing one event-driven feature within Nitro allows us to compare and contrast a Polling vs. event-driven implementation.
In a polling-based approach, each Nitro coordinator would be responsible for polling each of the 100 Nitro Workers to obtain status for each assignment and/or task on-demand or during each polling-interval. The response might contain state of the compute node (power, temp, load average, etc.), and the state of completed and/or running Nitro assignments/tasks. As a measure of time, it would take roughly JOB_STATUS_TIME = <number of workers> * (time per request/response) to obtain the job status.
- Easy to conceptualize, debug, and maintain.
- Relatively simple compared to event-driven models.
- Lossless messaging. You either get a response or the node is presumed unreachable.
- The response effectively serves as a heart-beat indicating that the Nitro Worker process is reachable.
- Blocking – A reasonably accurate status is only obtained after all Workers have a chance to respond.
- Only the Nitro Coordinator (requestor) gets a chance to react to the data in the response, unless the Coordinator persists/relays/republishes the response.
- Only after JOB_STATUS_TIME would the Nitro Coordinator be able to report a fairly accurate job status. I say “fairly accurate”, because at scale (i.e. 1,000 – 10,000 nodes), due to latency, the status of the nodes that responded early in the polling process may/will have changed by the time all Workers respond, because Nitro jobs, by nature are generally shorter/smaller than HPC jobs.
- Large messages/response.
In an event-based approach, each Nitro Worker would be responsible for collecting and emitting events which the Nitro Coordinator consumes to maintain the state of the Nitro job within the Coordinator.
The following represent examples of the granularity that can be achieved in an event-driven model on each compute node:
- Changes in load, cpu temperature, power consumption, etc., could generate events emitted only when load/temp/power exceeds or drops below a configurable threshold.
- Nitro tasks may complete in sub-second time. Worker/task events could be collected and emitted in several ways:
- Emit a task completion event immediately upon completion if, and only if a large number of task events are desired.
- Emit a task completion event based on a configurable time interval or number of tasks completed. The message associated with the event would identify tasks completed.
- Emit an assignment completion event upon completion of all assigned tasks within an assignment (batch of tasks).
- Potential for a fully-asynchronous non-blocking solution.
- Messages/Events are small and represent very specific state changes within each Nitro Worker and/or task.
- Messages/Events can be collected at any level and relayed (emitted individually or together in batch)
- Any number of consumers can react to events.
- Highly scalable when using high performance channels (i.e. ZMQ)
- Highly flexible workflow orchestration.
- Highly optimizable. How events are collected, emitted, consumed can be tuned.
- Relative complexity (implementation, debugging).
- Potential loss of events/messages/state. As far as I know, no one in this space guarantees lossless transmission of events/messages. Even ZMQ has a high-water mark. Some sort of state reconciliation routine may be needed for certain applications.
I do not have empirical data to prove my theories/opinions, because Nitro was originally designed as an event-driven architecture. I’m confident that if Adaptive Computing had time and resources to implement Nitro both ways, the resulting metrics would be compelling in favor of the event-driven implementation solely based on runtime metrics. All of the other advantages are just more layers and icing on an already delicious cake.