ArmoniK documentation

What is ArmoniK ?

ArmoniK is an open-source, cloud-native platform for orchestrating compute-intensive distributed workloads. You break your application down into tasks — with dependencies between them — and ArmoniK runs them across a cluster, handling scheduling, data movement between tasks, autoscaling, and fault recovery for you. Client SDKs are available for several languages (C#, C++, Python, Java, Rust), so you can submit and orchestrate tasks without writing low-level distributed-systems code.

Why ArmoniK ?

Many organizations need to run large numbers of computational tasks — simulations, risk calculations, machine learning pipelines, batch analytics, rendering — reliably and at scale, without building and maintaining their own task-scheduling infrastructure.

ArmoniK provides this as a modular, customizable platform: you adapt it to your workload’s data transfer patterns, task durations, and resource requirements, rather than reshaping your application around a one-size-fits-all scheduler. It is already used in production by financial services and insurance companies to run distributed workloads made up of millions of tasks.

Key Features of ArmoniK

Goals

  • Simplifies Development: Makes it easier to build distributed applications without deep expertise in distributed systems.

  • Accessible Interface: Usable without requiring specialized parallel-programming knowledge.

  • Performance Optimization: Helps applications achieve good performance at scale.

Architecture

  • Computational Kernels: Users can define their computational tasks.

  • Data Management: Manages data reading, writing, and inter-process communication.

  • Task-Based Programming: Breaks down complex operations into smaller tasks, allowing for more manageable development and execution. This includes task distribution, load balancing, and dependency resolution through a distributed scheduler.

Built-in Features

  • Open Source: Available for users to modify and contribute to via GitHub.

  • Fault Tolerance: Ensures continuation of operations even with node failures.

  • Advanced Data Management: Supports overlapping computations and communications, checkpointing, and prefetching.

  • Observability: Provides extensive monitoring tools including GUIs and APIs for tracking application performance.

  • Portability and Malleability: Supports multiple programming languages (C#, C++, Python, Java, Rust, etc.) and dynamic resource allocation during execution.

  • Production Ready: Focused on stability and security, making it suitable for critical systems.

Use Cases

ArmoniK is well suited for any workload that benefits from high concurrency and task distribution, including:

  • Scientific computation and simulations

  • Machine learning pipelines

  • Batch analytics pipelines

  • Real-time distributed processing

  • Financial simulations (e.g. risk calculation) and combinatorial optimization

  • Scalable algorithmic processing (e.g. bioinformatics, Monte Carlo, rendering)

Note

This project is an Open Source (Apache 2.0 License).

When should I use ArmoniK

ArmoniK should be used when the following criteria are met:

  1. A high task throughput is required (from 250 to 10,000+ tasks per second).

  2. The tasks are loosely coupled.

  3. Variable workloads (tasks with heterogeneous execution times) are expected and the solution needs to dynamically scale with the load.

Versions

The current version of ArmoniK uses the tags listed in versions.tfvars.json where:

  • armonik is the tag of the main repository of ArmoniK

  • infra is the tag of ArmoniK.Infra repository which host the infrastructure modules

  • core is the tag of ArmoniK.Core repository used for container images of Control plane, Polling agent and Metrics exporter

  • api is the tag of ArmoniK.Api repository (informative only)

  • gui is the tag of ArmoniK.Admin.GUI repository used for container images of ArmoniK AdminGUI

  • extcsharp is the tag of ArmoniK.Extensions.Csharp repository used for container images of the DLL worker

  • samples is the tag of ArmoniK.Samples repository

Quick Start

Detailed Installation

Launching & Examples

User guide

Administration & Operations

Benchmarking & Performance

Reference

Contributing & Development

Appendices