Sunday, 31 July 2016

Back to blogging after two long years!!

What have I been upto ?

Once I left Magarpatta(June 20, 2014), Pune, I was excited to know that I was capable of doing great work in technology. I worked on four projects at the same time, used to work with multiple managers at the same time. I learned that I can contribute to Opensource ( Openstack ). By contributing to opensource, you gain wealth of knowledge. I think if you put in one hour of your time to contribute to FOSS, you gain 10 x knowledge and exposure. As a fresh graduate from Bits Pilani Goa, it was a tremendous learning. I was talking directly to several developers, architects, industry experts, read about what companies work and their business. At that point, I got a whole new perspective of how things work in the industry, how opensource is powering the world.

Coming to present, un-employed :P :P as of June 10, 2016.

For the past 2 years, I was working as Research and Development engineer at IPsoft Labs, Bangalore.

Working there, I had got exposure to lot of new technologies and to a whole new world of Artificial Intelligence. I used to learn, program, document, test, prepare PPTS, talk to customers. I am glad now that things worked out very well that I had said 'yes' when my VP and CTO of the team said 'there is going to be a lot of work'.

I had a chance to work and learn from some of the best minds. What still amazes me is their way of thinking and how to solve a technical problem. Approach is so subtle and the steps are so comprehensive.

Right now, I am watching youtube series Deep learning tutorials and once I am done with the tutorials, I would like to share my learning here.

I have also signed up for Machine learning course at coursera taught by Andrew Ng, which is best for starters. By the end of October, I would be done with the coursework.

My next projects which I have in mind are IMDBWhatsapp, that's what my friends used to call :P. It's a andriod project on an interesting idea. I would like to apply machine learning and see how it can make it better.

Other pet projects in my mind are create a website from ground up so that I get exposure to Web development, which could be useful for freelancing in future.

That's all for today. I am trying to make blogging a habit, so that I could share my learning :)

Have a great day !!

Friday, 15 August 2014

Infinite recursion ;)

My blog post has been featured in Openstack super user, Mirantis tutorials, Opensource.com

My first attempt at blogging and it has been featured in Openstack Superuser, Mirantis tutorials and Opensource.com. It gives me a great motivation to learn more and more about openstack and blog about the latest innovations happening around Openstack. 

Openstack Super user : http://superuser.openstack.org/articles/how-to-contribute-to-openstack

Mirantis Tutorials(14th in Additional) : http://www.mirantis.com/openstack-portal/external-tutorials/tutorials-week-may-19-2014-may-26-2014/

Opensource.com(How to contribute to Openstack) : http://opensource.com/business/14/7/openstack-news-july-21

In future, I would like to learn more and more about Openstack and I would simultaneously update my blog. Stay tuned ;)

Sunday, 6 July 2014

My contribution to Openstack

Contribution to Openstack

Stackalytics:

Stackalytics is one stop for looking at some one's contribution to Openstack. It has great user interface where we can find the modules user has contributed to, timeline of his contributions. Here is my Stackalytics page link. I have mainly contributed for Openstack-manuals, documentation for Openstack projects about installation, user guides, admin guides and Openstack-API which hosts the documentation related to the API of several Openstack components.

Monday, 26 May 2014

Deploying Devstack with Ironic

Devstack with Ironic

Openstack

 Openstack is a cloud operating system which manages pools of resources like storage, newtorking, computing and provisions cloud solutions which is massively scalable. Openstack is an free and open source project. Openstack's mission is to provide ubiquitous cloud computing platform which serves as Infrastructure As A Service (IAAS) to meet the needs of public, private cloud and hybrid cloud.

Devstack

Devstack is an all in one installer for Openstack. Openstack is getting bigger and bigger day by day. Many new projects have been added to it and it is getting more complicated. To make things easy, Devstack offers a way to install all components of Openstack with out much fuss. These are the key components of Openstack :
  1. Compute ( Code name 'Nova') : Nova is responsible for creation and management of virtual machines. It has support for different types of Hypervisors like KVM, Xen, QEMU, Vmware-vsphere, Hyper-V, Baremetal
  2. Object Storage (Code name 'Swift) : Swift provides Object storage. Swift is used to store Objects and files and Openstack is responsible for data replication and ensuring integrity in cluster.
  3.  Block Storage (Code name 'Cinder') : Cinder is used to provide storage for compute instances. It manages creation, attaching, detaching of the block storage to the servers. In addition to this, it can use many storage like Ceph, IBM Storage, NetApp etc
  4. Networking ( Code name ' Neutron'): Neutron is used to provide network as a service to the compute instances. It is responsible for allocating IP addresses and for managing networks inside openstack
  5. Dashboard (Code name 'Horizon') : Horizon is a web interface to manage all the services in openstack. It can be used to spin up a virtual machine , assign a floating IP to it, access its console.
  6. Identity(Code name 'Keystone') : Keystone is the identity service which provides authentication to all the services running inside openstack. It has capabilities to integrate with LDAP.
  7. Image service(Code name 'Glance') : Glance is used to provide images for the virtual machines. It can also be used to store backups. It has the capability to use stored images as templates.
  8. Telemetry(Code name 'Ceilometer') : Ceilometer provides features like Billing for customers. 
  9. Orchestration(Code name 'Heat') : Heat is used to orchestrate different cloud applications using templates
  10. Database(Code name 'Trove'): Trove is relatively new project as of today. It is used to provide Database As A Service. It uses openstack as a black box to provide database as a service
Openstack also consists of many other projects like Ironic which is now a separate project for provisioning of Baremetal instances, Sahara which is used to provision Hadoop cluster on top of Openstack, Fuel which is an all-in-one control plane for automated hardware discovery, network verification, operating systems provisioning and deployment of OpenStack, Marconi which is message queue as a service, TripleO which is openstack on top of openstack, Rally which is benchmarking as a service and many more will be added in future.

Deploying Devstack with Ironic

Devstack can be used to configure all the services that we are interested in deploying in openstack. To know more about devstack, you can refer here.

Lets begin the deployment process. I would prefer trying out this deployment in virtual box which gives me the flexibility to take snapshots of working devstack,  create new virtual machines to try out different configurations. You can download virtual box and follow this guide for creating a clean installation of Ubuntu 12.04 LTS 64 bit vm with atleast 2 gigs of ram. If in case you dont see 64 bit option in the second step, enable virtualization in your Bios.
Now that you have your vm with Ubuntu 12.04 LTS, its time to install Devstack. Start the virtual machine. Open a terminal (ctrl + shift + t) , run an update

sudo apt-get update

All the code of openstack is hosted on github so install git using this command,

sudo apt-get install git

Check if git has installed successfully by checking its version,

git --version

Once git has been installed, its time to clone the devstack from openstack github repository. You can have a look at the repo that we are going to clone here . Click on the branch, you will notice stable/havana and stable/icehouse . To install openstack havana clone the devstack by checking out the stable/havana branch. To install openstack icehouse (latest as of now), clone the devstack by checking out the stable/icehouse. 

For devstack with Icehouse use this command

git clone https://github.com/openstack-dev/devstack.git -b stable/icehouse

For devstack with Havana use this command

git clone https://github.com/openstack-dev/devstack.git -b stable/havana

If you are facing errors with ssl certificate verification , use http instead of https. If this doesn't work , change sslVerify to false using 

git config --global http.sslVerify false
   
Once the download is done , change to the directory, devstack using
cd devstack

Check the contents of that folder using 
ls -la

You can notice few shell scripts like stack.sh, clean.sh, unstack.sh, rejoin-stack.sh, run_tests.sh, exercise.sh. stack.sh is a shell script which downloads all the components from the respective openstack repositories and clients in /opt/stack. Besure to run this command as a normal user not root . unstack.sh is used to stop the openstack services. rejoin-stack.sh is used to join the already running openstack , it will start all the stopped services and screen logs will be appended. clean.sh does its best to remove mysql, rabbitmq and other openstack configuration files.

Important step

create a new file while you are in devstack's directory named localrc using your favorite editor. I am using vi editor so I would do something like this

vi localrc

Paste this contents to enable all the services and Ironic.

######## From here #######
 RECLONE=yes
LOGFILE=/opt/stack/logs/stack.sh.log
SCREEN_LOGDIR=/opt/stack/logs/screen
# Enable Ironic API and Ironic Conductor
enable_service ironic
enable_service ir-api
enable_service ir-cond

# Enable Neutron which is required by Ironic and disable nova-network.

disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service neutron


# You can replace "password" with your own one
DATABASE_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_PASSWORD=password
ADMIN_PASSWORD=password
SERVICE_TOKEN=password

# Create 3 virtual machines with 512M memory and 10G disk to
# pose as Ironic's baremetal nodes.
IRONIC_BAREMETAL_BASIC_OPS=True
IRONIC_VM_COUNT=3
IRONIC_VM_SPECS_RAM=512
IRONIC_VM_SPECS_DISK=10
IRONIC_VM_SSH_PORT=22

# Size of the ephemeral partition in GB. Use 0 for no ephemeral partition.
IRONIC_VM_EPHEMERAL_DISK=0

VIRT_DRIVER=ironic

# By default, DevStack creates a 10.0.0.0/24 network for instances.
# If this overlaps with the hosts network, you may adjust with the
# following.
NETWORK_GATEWAY=10.1.0.1
FIXED_RANGE=10.1.0.0/24
FIXED_NETWORK_SIZE=256

# Log all devstack output to a log file

LOGFILE=$HOME/devstack.log
######### Till here########

Save the file. 

Now run 

./stack.sh

This will take around 20 minutes. I will clone all the required repositories of all the components that are enabled in the localrc file. As it takes place, let understand what this file means.

 Reclone = yes makes stack.sh to clone the repo at each run. If this is set to no, stack.sh is going to clone only if its not in the $dest directory. Second line signifies to make a log of the stack.sh. Since it will run for several minutes, it would be handy to check whats going on and in case of failures to track which component has failed.Once stack.sh completes, we can see screen with terminals of all the services. Its better to log the screens for the same purpose as stated above. enable_service ironic tells stack.sh to enable ironic and download the necessary files related to it. Similarly other enable_service commands do the same. Ironic requires neutron so we have to disable the nova network. 

Subsequent lines deal with the password for the database, rabbitmq server, admin password, service token password.

Lines starting with Ironic are used to create 3 vm's with ram 512 mb and with 10 gb space. These three vm's will act as fake baremetal nodes for Ironic.

In case of normal openstack installation, the harware is virtualized using different hypervisors which create virtual machines. Ironic on the other side picks the bare metal servers and PXE (Pre Execution Environment) boots them and deploys the image. Hardware is not virtualized in this case. With Ironic we get the flexibility of cloud with bare metal. Imagine you have 5 servers and you want them to spin up just like you can do spin up a vm in openstack. Ironic does that. 

Rest of the file deals with the network configuration for vm's. What class of IP address will be allocated, what would be their gateway.

Once the stack.sh completes succesfully (I know it wont happen the first time, if it does, you are very lucky
:-D ), its time to login to Horizon.

check your network interface using ifconfig and note down the Ip address in eth0 interface. Open your browser and go to http://IP. My Ip adress of eth0 interface was x.x.x.x  so my horizon would be http://x.x.x.x  Login credentials would be admin/password . If you have changed the password for ADMIN_PASSWORD, use that password to login.

Once you login, you can see collapsible tiles on the left hand side if you have installed Icehouse. Familiarize yourself with it. We haven't created any instances yet.

Time to check the screens. use "screen -x stack" to see the screens which are consists of different parts of openstack interacting with each other with the help of API. To jump to the next screen use "ctrl +a +n" and to jump to previous screen use "ctrl + a + p". To detach from the screens use "ctrl + a +d".
To list the screens use ctrl + "

Change your working directory to devstack and source the openrc file using this command

source openrc admin admin

This helps us to use the clients python-novaclient. You can also do this using the API's. Get a token from keystone, use curl to get instances using Nova API. Lets stick to the first method.

Query the image id of the default cirros-0.3.1-x86_64-uec image using this command.

nova image-list

Cirros is the OS that we are going to boot in our baremetal instance. You should get something like this

+--------------------------------------+---------------------------------+--------+--------+
| ID                                                                    | Name                                        | Status | Server |
+--------------------------------------+---------------------------------+--------+--------+
| a5627443-6d18-4719-866e-62dcde0dd2b9   | bm-deploy.initramfs                   | ACTIVE |        |
| 38d47e4a-13cc-40d8-b16c-b52933a8558b    | bm-deploy.kernel                      | ACTIVE |        |
| 2224b2f2-8a71-4755-a90a-f2c7945a5357     | cirros-0.3.1-x86_64-uec           | ACTIVE |        |
| 919d95f7-8517-4759-b901-cba302d038ba    | cirros-0.3.1-x86_64-uec           | ACTIVE |        |
| 14760c82-d960-448d-80d6-d83302f0234c    | cirros-0.3.1-x86_64-uec-kernel | ACTIVE |        |
| ce8b75e6-94e5-42bf-8a08-140a54468ec4     | cirros-0.3.1-x86_64-uec-kernel | ACTIVE |        |
| 8df7830c-f979-4ee3-b065-f8126f69fdc1        | cirros-0.3.1-x86_64-uec-ramdisk| ACTIVE |        |
| b46e49c3-7a31-4c2e-b870-7570e0dd4ba5    | cirros-0.3.1-x86_64-uec-ramdisk | ACTIVE |        |
+--------------------------------------+---------------------------------+--------+--------+

Copy the ID of cirros-0.3.1-x86_64-uec . Create a keypair using ssh. Use ssh-keygen -t rsa to create ssh keys. Follow the onscreen instructions. The keys will be stored in a hidden directory starting with .ssh. It would contain the public and private keys, know hosts .

Add the keypair to nova so that we can boot the baremetal instance.

nova keypair-add default --pub-key ~/.ssh/id_rsa.pub

Lets boot the instance

nova boot --flavor baremetal --image "paste the Id of cirros here" --key-name default testing

The vm we are booting is of flavour baremetal with the image corresponding to the image id of cirros and the keypair corresponding to the default. You would get something like this:

+--------------------------------------+-------------+---------+------------+-------------+-------------+
| ID                                   | Name        | Status  | Task State | Power State | Networks          |
+--------------------------------------+-------------+---------+------------+-------------+-------------+
| 2aabb512-796b-4178-a67f-cfbdce058190 | testing | Build | Spawing      | No State    |    |
+--------------------------------------+-------------+---------+------------+-------------+-------------+

Since we have created three vm which were acting as three fake baremetals, we have powered on one baremetal out of them. Now check the state of baremetal instances using 

ironic node-list

+--------------------------------------+--------------------------------------+-------------+------------+
| UUID                                 | Instance UUID                        | Power State | Provisioning State |
+--------------------------------------+--------------------------------------+-------------+------------+
| a89a107d-63b4-4412-b3f1-0241fb3b0edb | None                                 | power off   | None               |
| f51ca8a4-c262-40b2-ad6c-288ec2a0b71a | None                                 | power off   | None               |
| add9a19c-6991-4bc5-8744-fd43e9f0720a | 2aabb512-796b-4178-a67f-cfbdce058190 | power on   |wait call-back       |
+--------------------------------------+--------------------------------------+-------------+------------+

Note that one of the baremetal instance is powered on and its in wait call-back state.

Check your baremetal instance status by 

ironic node-list


+--------------------------------------+-------------+---------+------------+-------------+-------------+
| ID                                   | Name        | Status  | Task State | Power State | Networks          |
+--------------------------------------+-------------+---------+------------+-------------+-------------+
| 2aabb512-796b-4178-a67f-cfbdce058190 | testing | Active | -    | No State    | Running   | private = 10.1.0.4
+--------------------------------------+-------------+---------+------------+-------------+-------------+

Problem with SSH
If you are facing any problem with ssh, edit the /etc/sshd_config file and uncomment the ListenAddress. Save the file and restart the service ssh.

You have succesfully deployed Ironic with devstack :) 


Saturday, 24 May 2014

Google Summer Of Code Intern in Openstack

Yay!! Google Summer Of Code Intern In Openstack



The Google Summer of Code (GSoC) is an annual program, first held from May to August 2005, in which Google awards stipends (of US$5,500, as of 2014) to all students who successfully complete a requested free and open-source software coding project during the summer. The event draws its name from the 1967 Summer of Love (of the 1960s counterculture), and the idea for the SoC came directly from Google's founders, Sergey Brin and Larry Page.

For the first time Openstack was selected as mentoring organization in GSOC. Here's the list of students whose proposal got accepted for Openstack.

Here's my proposal link (link) and link to the proposed idea (idea) that I would be working on. I am so thankful to my mentor (Sriram) who has helped me a lot.

I am really excited to be a part of such awesome community. 

How to contribute to Openstack

How to contribute to Openstack

My contribution : link

First of all lets answer the question who can contribute to Openstack. Anyone . Yes, anyone can contribute to Openstack. Whether you are interested in developing new feature in Openstack  or in Documentation or in fixing Bugs , you are welcome. That's how Open source projects work.

Lets answer another question . Why should anyone contribute to Openstack. The answer would be :  To learn more about the project. By contributing you learn a lot of things. You are making the system better and helping others all over the world who use Openstack.

Let's begin

This  is where you should start. The link has all the information on How to contribute. All the commands used here are from that link. In case if you want more info please use the link provided in the beginning. My mentor suggested me to fix a Bug in Openstack. Bug can be a very small one like fixing a typo in the code message or it can be a critical  one. Both are considered as contribution to Openstack. You are not fixing a typo error but you are fixing a bug in Openstack and making it better. I went with a very small bug in Sahara (obviously I don't have adequate knowledge to fix a critical Bug but I made my contribution with what I can do) .I had to change all the instances in the code with the word "components" with "component(s)". Seems simple ? yes it is! 


Where to begin?

Launchpad Account

Launchpad has all the information about bugs, overview etc. Get a launchpad account by registering here. Click on Openstack and you will be redirected to its main launchpad page. All the projects in Openstack, links to documentation, irc, mailing list etc can be found there. 

Join Openstack Foundation

Fill out the details and join the foundation link

Log Into Gerrit Review System

Login here with your Launchpad account. This is where all the code reviews happen. Sign the  Openstack individuals contributor License Agreement. And upload SSH keys .

Uploading SSH keys

Once you are logged into review.openstack.org, click here to upload your SSH keys. Follow this link to generate SSH keys. Once you have generated SSH keys, copy the Public key and upload it in review.openstack.org. Change to the directory where you have created your SSH keys, then list the contents of the folder and look for something ending in .pub which would be your public key. 

Install git 

I am using Ubuntu 12.04 LTS so you can begin with opening terminal ;)

Open Terminal (ctrl + alt +t ) and type "sudo apt-get install git" . After installing git, use this command to check for successful installation "git --version". If you can see the version of git, then it's installed. Next set your username and email id . Type these commands in terminal.

git config --global user.name "Firstname Lastname"
git config --global user.email "your_email@youremail.com"
 
To check if you have configured properly check your configuration by using this command

git config --list
 
You would get information about your email id ,username and other details. Learn more about git here

Install git-review

Install git review. Follow this link for different linux distros. In ubuntu it would be "sudo apt-get install git-review"

Lets track some Bugs

Login to your launchpad account and click on bugs. Out of all those bugs, low-hanging-fruit would be easy bugs to fix for beginners. Click on the low-hanging-fruit tag and select a bug which is easy to fix. Click on it, you can see its description, who reported it, which part of Openstack it is affecting and other details. Look for the Bugs which are Triaged and Confirmed and which are not assigned to any one. Triaged bugs will contain information on how to fix them in most of the cases. If you need any help understanding the bug or trying to find a fix for the bug, you can comment in the bug page or you can directly reach out to the reporter in IRC.

Once you have selected a bug and if you feel you could fix it assign it to yourself. This is the bug I assigned to myself. Next thing would be to clone the part of Openstack which has this bug.

I have cloned Sahara since the bug was in it. To clone a project to your local machine, open terminal and then use "git clone https://github.com/openstack/urproject.git" .You would get a message cloning into <whateverproject>, receiving objects, receiving deltas then done. In my case, I would clone sahara project from github using this command : "git clone https://github.com/openstack/sahara.git". Change the directory to project " cd sahara". Next type the command "git review -s". This basically checks if you can login to Gerrit with your SSH keys. If your git username is different from your gerrit (review.openstack.org) username , use this command:

git config --global gitreview.username yourgerritusername
 
Your gerrit username would be the one in your profile tab in review.openstack.org. To verify your configuration use git config --list.

Once again verify your SSH keys , gerritusername in git config. I you get an error "We don't know where your gerrit is.", you will need to add a new git remote. The url should be in the error message. Copy that and create the new remote. 

git remote add gerrit ssh://<username>@review.openstack.org:29418/openstack/urproject.git

Next list the contents of the folder by using "ls -la" which would also list the hidden folders and files. You should see a .git hidden folder and .gitreview hidden file . Now try again "git review -s"

Everyone continuously make changes to the master branch in Github. To get the most updated code with all the changes use the following commands. 

git remote update
git checkout master
git pull --ff-only origin master
 
To learn more about what origin, remote, master mean use this link . It has nice way of teaching about all the useful commands.

***Important Steps***

Create a Topic Branch . Since I am trying to fix a bug I would do git checkout -b "bug/1312908" where 1312908 is the bug id. Each bug is associated with a bug id which can be found in its page. You would see a message "switched to a new branch 'bug/<bugid>' ". Now use "git status" to check your branch.

Fixing the Bug

Now that we have everything in place and properly configured, lets fix the bug. To fix the bug in sahara all I had to do was follow the link Sergey gave in comments of the bug page which had all the occurrences of the word components. I opened those files and changed all the occurrences. Now if you remember we are currently in branch "bug/<bugId>". Follow this link to know how to run test cases. You have to run unit tests to check if everything is working.

Fixed the bug, what to do next?

Next thing would be to commit the changes that you have made to fix the bug. This is where I made a mistake and had to commit twice. Please follow strictly the pattern that is given in the Commit messages  so as to same your time.

Note that in most cases the Change-Id line should be automatically added by a Gerrit commit hook that you will want to install. See Project Setup for details on configuring your project for Gerrit. If you already made the commit and the Change-Id was not added, do the Gerrit setup step and run:

git commit --amend
 
The commit hook will automatically add the Change-Id when you finish amending the commit message, even if you don't actually make any changes.
Make your changes, commit them, and submit them for review:

git commit -a
git review
 
Caution: Do not check in changes on your master branch. Doing so will cause merge commits when you pull new upstream changes, and merge commits will not be accepted by Gerrit.

Submitted for review, Whats next

Once you have submitted it appears on https://review.openstack.org and wait for the code reviewers to review. More info about the review process can be found here .Follow the comments and make necessary changes according to their comments. Once everything is correct it will be reviewed by core developers team and Jenkins will test all the components. You can check the status of gate jobs of your review at http://status.openstack.org/zuul/. After that it will be merged to the master branch.


This was my experience of first bug fix. I have learned a lot of new things by doing this. Hope this helps :-)

Friday, 21 March 2014

About me


Hey There!


Hello everyone, I am an Undergraduate student from India pursuing MSC.(Tech.) Information Systems in BITS Pilani K K Birla Goa Campus. Currently I am working as a intern in Avaya Pune, India on Openstack, Nagios. Cloud computing ,Andriod App development, Data Mining interests me. I will be updating this soon .This is my first Blog. Feel free to hate it ;)

I will blog about the following things soon

1.Deploying Openstack Ironic with Devstack
2.Deploying Nagios for IT Monitoring


Stay tuned :)

You can reach me at
Mail       :  shanker.mani0@gmail.com
Twitter   :  Manishankert
Skype    :  manishanker2012