ElastiCluster

elasticluster is a command line tool to create, manage and setup computing clusters hosted on cloud infrastructures (like Amazon's Elastic Compute Cloud EC2, Google Compute Engine or a private OpenStack cloud).

Its main goal is to get your own private computational cluster up and running with just a few commands; the video demoes the basic features of elasticluster.

The following configurations are available, tested, and supported:

  • SLURM/Ubuntu: Run a SLURM batch-queueing system on top of the Ubuntu operating system.
  • GridEngine: Run a GridEngine 6.2u5 batch-queueing system.
  • Hadoop: Run an Hadoop cluster to start your favorite map/reduce job.
  • IPython cluster: Run an IPython cluster to execute your python code in parallel over multiple virtual machines.
  • GlusterFS/Ceph/OrangeFS: Configure a distributed storage like GlusterFS or Ceph, or a parallel filesystem like OrangeFS (formerly known as PVFS2). You can also combine these storage with one of the previous cluster types!
  • TORQUE+MAUI: Run the TORQUE batch-queueing system together with the MAUI scheduler on top of a CentOS Linux distribution. (But note: due to license limitations of TORQUE and MAUI, this configuration relies on a private package repository. We can provide the source RPM so you can rebuild the repository, but we cannot distribute the binaries publicly.)

elasticluster can deploy cluster software on vanilla Ubuntu or CentOS virtual machines: the only requirement is that the VMs can be accessed via SSH and that Python >= 2.4 is installed.

Instructions on the deployment and configuration of the cluster software is encoded in Ansible playbooks, which are re-played on each newly-spawned VM. This allows running clusters on different cloud infrastructures or on top of different versions of the base OS, for instance. There is nothing special about the playbooks used by elasticluster -- you can modify them to customize your cluster, or re-use any Ansible playbook that you wrote for your local, "bare metal", HPC cluster.

If you are interested in elasticluster, please find out more at:
http://gc3-uzh-ch.github.io/elasticluster/

Documentation for elasticluster is available on Read The Docs

Source code is available on GitHub