The system consists of the following elements:
– Central server
– Docker registry compatible image registry managed by Megamind
– Computing nodes
– Developers
– End users
- Central server. Megamind Central server consists of the following parts:
- Server REST API. Provides a top-level API for task management and connecting computing nodes. The task in this context is a docker image deployed to container(s) and run on a computing node. REST API server consists of an industrial-grade web server (apache / nginx) with https configured and accounts proxying calls to Megamind REST API server. The server supports the following commands:
- Public part
- get a list of free computing nodes (id, hardware characteristics, computation cost)
- create a task by running an image from the registry on a particular node
- stop the task by id, destroying it and removing it from the list. It removes port forwarding from NAT and from the list.
- get a list of running tasks for the account (with id) and their state (started/pending/completed)
- pause / resume a task
- Non-public part
- get parameters to connect the node to the system (parameters for entering swarm – token and ip)
- node registration: transfer to the server of its characteristics, ip, user:password
- The server also contains a manager node, a null / manager node docker swarm. The REST API server executes requests by managing the internal infrastructure using docker swarm (currently using parser of command line output, in the future versions – through the Docker API):
- getting a list of worker nodes: sudo docker node ls
For for more information on node:
- task creation: sudo docker service create
- deleting the task: sudo docker service rm
- get a list of working tasks for the account: for each node getting all tasks via docker node ps and filtering by id for of this account
- pause / resume task
With port-forwarding:
- it performs port forwarding to each created container (command for iptables);
- keeps track of already forwarded ports;
- removes port forwarding if necessary;
Or (option with direct channels to each node):
If your nodes are on a white IP and accessible from the Internet, then you need to open a port in the firewall (and don’t forget port forwarding in docker)
- Server of the web interface for setting tasks;
- Registry of images managed by Megamind GMbH
Fully compatible with standard docker registry. Needed authentication with htpasswd – every account registration on server from item 1 should add an entry to htpasswd. Allows to:
- unload (push) images
- download (pull) images
- get a list of images associated with this account
- delete images
All this using the standard docker registry interfaces.
- Nodes with computing equipment. High performance computers running MindOS.
Docker is installed on the node and the node is added into the swarm.
- Developers
They work under operating systems that support docker images, creating images for the Image Registry and load them into the Registry. They don’t work with Megamind-specific code or
Software — all work is done using docker tools.
- End users
They perform task management using direct API requests from clause 1.1 or using web interface: start new tasks, view running ones, stop, etc. They can also administer the counterpart of the computing Software (what requests / receives results from the nodes). Interface for accessing the task is ssh. If necessary, port forwarding is also done using ssh.