
How to manage cloud resources using Python? Raise clusters on demand through a few lines of code
- Tutorial
The Simple Azure Library lets you manage your cloud resources, including creating, managing, and deleting virtual machines in a cloud environment. You can use this library for any purpose: from deploying a sandbox for Dev & Test purposes to hosting and managing solutions in industrial operation.
Using Simple Azure, you can easily raise ipython notebook and ipython clusters in a cloud environment, deploy ready-made virtual machines to choose from hundreds presented in the VMDepot directory .
The following is a brief introduction and examples of using Simple Azure for simple tasks and the tasks of deploying an IPython cluster in the cloud.
Video :
Change the virtual machine to choose from:
In this code example, an OpenSUSE-based Azure Data Science Core virtual machine is selected from the VMDepot community directory with a customized environment for working with big data, HPC and the following packages: cython, ipython, matplotlib, networkx, nltk, nodejs, numpy, pandas, pytables, redis, scikit-image, scikit-learn, scipy, statsmodels, sympy.
Simple Azure supports commands in the Linux shell. For example, you can create virtual machine clusters like StarCluster with the following command:
This guide describes how to deploy an IPython cluster in the cloud in a few steps.
Step one:
Step Two - Create Three Nodes for a Cluster with ADSC:
Step Three - Import the IPython Plugin:
SSH setup:
Step four - configure the master and engine nodes:
Name binding in IPython plugin:
Setting up an SSH connection to hosts:
Create IPython Profile:
Running IPController on master:
Configuring engine nodes:
The final step is to run ipengine on each engine node so that they start interacting with master:
You can find more tutorials at the following link . The subtleties of installing and configuring the library are described here . You can get the source code of the library and join the development of the project on GitHub . You can always find the latest news and information about the project on the official Simple Azure website .
Using Simple Azure, you can easily raise ipython notebook and ipython clusters in a cloud environment, deploy ready-made virtual machines to choose from hundreds presented in the VMDepot directory .
The following is a brief introduction and examples of using Simple Azure for simple tasks and the tasks of deploying an IPython cluster in the cloud.
Installation
$ pip install simpleazure
$ simpleazure-cluster start mycluster
Virtual Machine Deployment
from simpleazure.simpleazure import SimpleAzure as saz
azure = saz()
azure.get_config()
azure.create_vm()
Video :
Change the virtual machine to choose from:
azure.set_image(label="Ubuntu Server 12.04.2 LTS")
Placement of several cars
azure = saz()
azure.get_config()
azure.create_cluster(num=5)
my-cluster-vm-0-87412
{'request_id': '88c94c00288d42acaf877783f09c4558'}
my-cluster-vm-1-61293
{'request_id': 'abfd563c2c4f4926872b6b1dba27a93b'}
my-cluster-vm-2-96085
{'request_id': '29b55f6cb5e94cfdbf244a7c848c854d'}
my-cluster-vm-3-46927
{'request_id': 'b1a3446ebafe47a295df4c9d1b7d743c'}
Hosting machines from the VMDepot community directory
In this code example, an OpenSUSE-based Azure Data Science Core virtual machine is selected from the VMDepot community directory with a customized environment for working with big data, HPC and the following packages: cython, ipython, matplotlib, networkx, nltk, nodejs, numpy, pandas, pytables, redis, scikit-image, scikit-learn, scipy, statsmodels, sympy.
azure = saz()
azure.get_config()
q = azure.get_registered_image(name="Azure-Data-Science-Core")
azure.set_image(image=q)
azure.create_vm()
Command line support
Simple Azure supports commands in the Linux shell. For example, you can create virtual machine clusters like StarCluster with the following command:
$ simpleazure-cluster start mycluster
Enabling IPython Cluster
This guide describes how to deploy an IPython cluster in the cloud in a few steps.
Step one:
from simpleazure.simpleazure import SimpleAzure as saz
azure = saz()
azure.get_config()
Step Two - Create Three Nodes for a Cluster with ADSC:
adsc = azure.get_registered_image(name="Azure-Data-Science-Core")
azure.set_image(image=adsc)
azure.set_location("West Europe")
azure.create_cluster(3)
Step Three - Import the IPython Plugin:
from simpleazure.plugin import ipython
ipy = ipython.IPython()
SSH setup:
ipy.set_username(azure.get_username())
ipy.set_private_key(azure.get_pkey())
Step four - configure the master and engine nodes:
from simpleazure import config
master = config.get_azure_domain(azure.results['master'])
engines = [ config.get_azure_domain(x) for x in azure.results.keys()]
Name binding in IPython plugin:
ipy.set_master(master)
ipy.set_engines(engines)
Setting up an SSH connection to hosts:
ipy.init_ssh()
ipy.connect_nodes()
Create IPython Profile:
ipy.create_profile()
Running IPController on master:
ipy.run_ipcontroller()
Configuring engine nodes:
ipy.copy_pkey_to_nodes() # <- Temporary function to distribute id_rsa private key to node(s)
ipy.copy_json2engines()
The final step is to run ipengine on each engine node so that they start interacting with master:
ipy.run_ipengine()
ipy.apply_ipcluster(azure)
Additional Information
You can find more tutorials at the following link . The subtleties of installing and configuring the library are described here . You can get the source code of the library and join the development of the project on GitHub . You can always find the latest news and information about the project on the official Simple Azure website .
Sitelinks
Try Azure for 30 days for free
Take training courses at Microsoft Virtual Academy (MVA)
- Microsoft Application Development
- Microsoft Azure and SQL Azure Databases Security Overview
- Azure Software Management
- Dive into Azure for application development and infrastructure services
- An Introduction to Developing Modern Azure Applications for Beginners
- or Azure cloud development courses
- or many other free MVA courses !
Only registered users can participate in the survey. Please come in.
When working with Python, which topics are you more interested in?
- 58.6% Running Python Applications in the Cloud 34
- 62% Python Management of Cloud Resources 36
- 43.1% Examples of scientific and other Python projects in the cloud 25
- 60.3% Subtleties, best practices, Python examples in the cloud 35