tag:blogger.com,1999:blog-14797806011423373242024-03-17T20:00:00.474-07:00Byte Musings"Articles on Cloud, Architecture, Java, Artificial Intelligence/Machine Learning" By Raghuveer Bhandarkar, @raghumbrRaghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.comBlogger54125tag:blogger.com,1999:blog-1479780601142337324.post-23438437413874064642020-10-04T07:18:00.003-07:002020-10-05T20:41:28.148-07:00Cross Validation and its Importance<p> </p><p class="MsoNormal"><span style="font-family: arial;">In this post, we will take a look at Cross validation and
it’s importance in building machine learning models.<o:p></o:p></span></p>
<h2 style="text-align: left;"><span style="font-family: arial;">Overview:</span></h2>
<p class="MsoNormal"><span style="font-family: arial;">There are multiple approaches to split a dataset for
training and estimating the accuracy of the model. One of the common approaches
is to split the dataset into 3 sets, a training set, a validation set and a
test set. While the training set is used for training the model, the validation
set is used for choosing the best model from amongst many models. The test set
is finally used to estimate the model performance on unseen data.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family: arial;">Now, what if our split resulted in some important data
points being grouped in the validation and test set. Our model would miss those
important data points from being trained on and will result in a poorer model. To
alleviate this, we use cross validation. There are multiple flavors of cross
validation, viz., K-fold cross validation, Stratified cross validation and
Leave one out cross validation (LOOC).<o:p></o:p></span></p>
<h3 style="text-align: left;"><span style="font-family: arial;">K-Fold cross validation:</span></h3><div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-bAvG1XnGR5Y/X3nZurmFadI/AAAAAAAABBQ/_sMt4TciWUQSBjvjHKsAckHC8SgXbJVsQCLcBGAsYHQ/s1290/cv.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="562" data-original-width="1290" src="https://1.bp.blogspot.com/-bAvG1XnGR5Y/X3nZurmFadI/AAAAAAAABBQ/_sMt4TciWUQSBjvjHKsAckHC8SgXbJVsQCLcBGAsYHQ/s320/cv.png" width="320" /></a></div><br /><span style="font-family: arial;"><br /></span></div>
<p class="MsoNormal"><span style="font-family: arial;">In this approach, the training and validation sets are
combined and then split into k smaller subsets. The model is then trained on
k-1 subsets and then validated on the remaining one subset. This process is
carried out k times, allowing all the model to get trained on all possible
combinations of datasets. The process is illustrated in the image above.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family: arial;">How do we estimate the performance of the model:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family: arial;">The model performance could be estimated as the average of
the k estimates.<o:p></o:p></span></p>
<h3 style="text-align: left;"><span style="font-family: arial;">Stratified cross validation:</span></h3>
<p class="MsoNormal"><span style="font-family: arial;">Let us consider a scenario, where we are building a model to
predict whether a loan applicant will default or not. If our dataset contains
1000 datapoints and only 20 of them are defaulters. Thus, only 2% of our
dataset contains the defaulter applicants. When we split our data into
training, validation, test (or cross validation), there is no guarantee that
our training set will contain these datapoints. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family: arial;">To resolve this, we can split in such a way that each
dataset gets an equal proportion of these classes. Stratified cross validation
does exactly this and provides us a proportional training data to work with.<o:p></o:p></span></p>
<h3 style="text-align: left;"><span style="font-family: arial;">Leave one out cross validation:</span></h3>
<p class="MsoNormal"><span style="font-family: arial;">In this approach, if there are N datapoints available for
training and validation combined, n-1 datapoints are used for training and one
datapoint is used for validation. This is repeated N times, allowing all
different combinations. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family: arial;">This approach is a very compute intensive approach, since if
we have 1000 datapoints, the training will happen 1000 times. This is generally
used only we don’t have enough data to work with.</span><o:p></o:p></p>Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com1tag:blogger.com,1999:blog-1479780601142337324.post-46135876101143123212020-03-27T20:55:00.000-07:002020-03-27T20:55:37.263-07:00My External Publications<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Arial, Helvetica, sans-serif;">Here is a list of my Publications in external websites:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">1. Article on Policy Iteration Algorithm explained using a simple game involving a pirate ship with rewards and benefits. This algorithm is a part of ReInforcement Learning, an offshoot of Machine Learning.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Title: </b></span><span style="background-color: white; color: rgba(0, 0, 0, 0.84); font-family: medium-content-title-font, Georgia, Cambria, "Times New Roman", Times, serif; font-size: large; font-weight: inherit;">Policy Iteration in RL: A step by step Illustration</span><br />
<span style="background-color: white; color: rgba(0, 0, 0, 0.84); font-family: medium-content-title-font, Georgia, Cambria, "Times New Roman", Times, serif;"><b>Link: </b></span><a href="https://towardsdatascience.com/policy-iteration-in-rl-an-illustration-6d58bdcb87a7">https://towardsdatascience.com/policy-iteration-in-rl-an-illustration-6d58bdcb87a7</a><br />
<br />
<br /></div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com0tag:blogger.com,1999:blog-1479780601142337324.post-34568301156387251702020-02-09T06:35:00.002-08:002020-02-09T06:45:25.326-08:00Difference between port and targetPort in Kubernetes<div dir="ltr" style="text-align: left;" trbidi="on">
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Overview:</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">In the previous <a href="https://www.bytemusings.com/2020/01/deploying-spring-boot-app-on-kubernetes.html">post</a>, we deployed a simple microservice to Kubernetes. In this post, we would take a look at the various port mappings available for an application in Kubernetes and the differences between them.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-tauI-FOUoqE/XkAZQpXiObI/AAAAAAAAA3U/xqa75u2gsWkriLN3huf4D6JcM13p0o_mQCLcBGAsYHQ/s1600/portdiff.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="900" data-original-width="1600" height="225" src="https://1.bp.blogspot.com/-tauI-FOUoqE/XkAZQpXiObI/AAAAAAAAA3U/xqa75u2gsWkriLN3huf4D6JcM13p0o_mQCLcBGAsYHQ/s400/portdiff.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Port Mappings:</span></h4>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Let us take a look at the service yaml file created in previous post. </span></div>
<div>
<script src="https://gist.github.com/raghumb/f647009b51b8f21558bf6fcee2948550.js"></script>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">You must be wondering that there are so many port mappings available in the yaml file, and what each of those means.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">There are two port mappings available, port and targetPort. There is also a third option., when we use Service type as nodePort, we can also provide a specific nodePort which could be used.</span></div>
<div>
<br />
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Explanation of Ports:</span></h4>
<div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>NodePort:</b></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">When we use nodePort Service type, Kubernetes assigns a static port through which external clients can access the service. In our example, it is 31869.</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Port:</b></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Port (8086), is the port through which other services in the same node or other nodes within the cluster can access the service. </span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<div>
<b>TargetPort:</b></div>
<div>
TargetPort (8085), is the port through which the actual Pod is accessible.</div>
</span></div>
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Explanation of Traffic Flow:</span></h4>
</div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The diagram above shows the traffic flow when a request is made by an external client. When the external client makes a call using Node Port (31869), the kube proxy will handle this request and will forward the request to the port exposed by the service which is 8086 in our case. Finally, this request gets forwarded to the POD which is exposed on port 8085.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">We can observe that, Service 2 can access Service 1 directly using the Port 8086.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Conclusion:</span></h4>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">I hope we have learnt about the various port mappings used and next time, we deploy an application to Kubernetes, we know the meaning of each port mapping.</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com0tag:blogger.com,1999:blog-1479780601142337324.post-24812154010737594132020-01-30T23:32:00.002-08:002020-01-30T23:38:10.157-08:00Deploying a Spring Boot App on Kubernetes<div dir="ltr" style="text-align: left;" trbidi="on">
<h3 style="text-align: left;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Overview</span></h3>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">In this post, I will be explaining how to deploy a simple Spring Boot Application onto Kubernetes. I will be using MicroK8s for the Kubernetes environment. For a quick guide on how setup MicroK8s refer <a href="https://microk8s.io/docs/">here</a>.</span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<h4 style="text-align: left;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Spring Boot App Docker Image:</span></h4>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Let us create a simple Spring Boot application and dockerize it. The source could be downloaded from github repo <a href="https://github.com/raghumb/basic-ms">here</a>.</span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Build the jar file and then dockerize it. I will be using the docker image from local, we can also push/pull this image from a Docker Repository.</span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Docker command to dockerize the app is:</span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">docker build -t basic-microservice:local .</span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Run the command 'docker images' and you should be able to see the docker image '</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">basic-microservice:local'. I have tagged the image as 'local' and not as latest. This is because, since we are using local images for Kubernetes, MicroK8s has a workaround for using local images.</span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Follow the instructions <a href="https://microk8s.io/docs/registry-images">here</a>, to upload docker image to MicroK8s cache.</span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<h4 style="text-align: left;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Create Deployment in Kubernetes:</span></h4>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Let us create a deployment.yaml file which is an abstraction which specifies how many instances of the app should be there, which docker image to be used etc.</span></div>
<div>
<script src="https://gist.github.com/raghumb/0a227bb6d2c90d55b57cf18b59712da8.js"></script>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Run the following kubectl command to create the deployment:</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">microk8s.kubectl apply -f basic-ms-deployment.yml</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">If you observe the deployment yml, we can observe that we have mentioned the docker image name created above and also the 'imagePullPolicy' is set to 'Never', which means the docker image would not be pulled from any repository. We have also configured the 'replicas' to 1, which means only one instance will be created. </span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<h4 style="text-align: left;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Create Service in Kubernetes:</span></h4>
</div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Let us create a service.yaml file which is an abstraction which specifies how should we access the application deployed in terms of what IP/Port should be used.</span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Run the following kubectl command to create the service:</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">microk8s.kubectl apply -f basic-ms-service.yml</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<script src="https://gist.github.com/raghumb/f647009b51b8f21558bf6fcee2948550.js"></script>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">If you observe the service yml, we can observe that it has a selector node with apps tag, which points to 'basic-ms'. This should match the name we gave to our deployment. We have also provided type as 'NodePort', which means, the application is accessible from outside the cluster using a Node IP and a static port. We have provided the port as 8086, which is the port accessible from outside.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Run the following commands to observe the Pods/Deployment/Service which are created:</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">microk8s.kubectl get pods</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">microk8s.kubectl get deployments</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">microk8s.kubectl get services</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-0KxBX_F6O_E/XjPWkPfNUVI/AAAAAAAAA2g/PwnVE72IS5YpTHtKx6Yr_TdBZMK0bU1cwCLcBGAsYHQ/s1600/3.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="304" data-original-width="1167" height="83" src="https://1.bp.blogspot.com/-0KxBX_F6O_E/XjPWkPfNUVI/AAAAAAAAA2g/PwnVE72IS5YpTHtKx6Yr_TdBZMK0bU1cwCLcBGAsYHQ/s320/3.PNG" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-gmzdQFTCVoY/XjPWpYl2PzI/AAAAAAAAA2k/hElO3RiV438yLaCErHYM6_zLm0QicoxZACLcBGAsYHQ/s1600/4.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="209" data-original-width="1169" height="57" src="https://1.bp.blogspot.com/-gmzdQFTCVoY/XjPWpYl2PzI/AAAAAAAAA2k/hElO3RiV438yLaCErHYM6_zLm0QicoxZACLcBGAsYHQ/s320/4.PNG" width="320" /></a></div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h4 style="text-align: left;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Testing the Application:</span></h4>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Finally, let us test our application. When we run 'get services' command, we can see that the port is 31869 which is the port assigned by Kubernetes and 8086 is the port assigned by us.</span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Let us try to do a curl to our application:</span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">curl 10.152.183.96:8086/hello/message</span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">We can see the REST API providing a response: "From Spring Boot!"</span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">This concludes a very simple application being deployed to Kubernetes with relative ease. Let us look at more features in upcoming posts.</span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com0tag:blogger.com,1999:blog-1479780601142337324.post-75565351148348051482020-01-14T07:59:00.001-08:002020-01-14T08:05:25.399-08:00Kubernetes Basics<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Overview:</span></h2>
<div>
<hr />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Kubernetes has become the leader in Container Orchestration and a working knowledge of Kubernetes is essential for every developer.</span></div>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">In this post I am going to cover some very basic concepts of Kubernetes.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-5oMtyEVltiM/Xh3ioDImuOI/AAAAAAAAA1o/0BUfWxC_IO4FeeoEbzT1OUcKbPCmJgEDACLcBGAsYHQ/s1600/overview.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="680" data-original-width="1600" height="272" src="https://1.bp.blogspot.com/-5oMtyEVltiM/Xh3ioDImuOI/AAAAAAAAA1o/0BUfWxC_IO4FeeoEbzT1OUcKbPCmJgEDACLcBGAsYHQ/s640/overview.png" width="640" /></a></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Concepts:</span></h2>
<hr />
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Pod </b>is a logical unit containing one or more containers (if you use a side car). It is the smallest execution unit.</span></div>
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Replica Set</b> represents number of instances of pods which are running. Used for scaling of pods.</span></div>
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Stateful Set</b> represents unique pods which retain state while running. We use Stateful Set while working with stateful workloads like a database cluster or a master slave configuration of workload.</span></div>
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Deployment</b> represents abstraction used to represent and update Pods and ReplicaSets. We use deployment as a "kind" in the yaml files while deploying an app in Kubernetes.</span></div>
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Service</b> is used for defining how we expose/access an application within a Pod.</span></div>
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Generally when we expose an application in Kubernetes using Service, there are four options:</span></div>
<div style="text-align: left;">
</div>
<ol style="text-align: left;">
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Cluster IP - Apps are exposed internally within the cluster.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">NodePort - Apps are exposed via a static port and the IP of the node.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">External IP - Apps are exposed via an externally assigned static IP.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Load Balancer - Apps are exposed via cloud provider provided Load balancer.</span></li>
</ol>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">References:</span></h3>
<span style="font-family: "arial" , "helvetica" , sans-serif;">1. Kubernetes Concepts. Retrieved from </span><a href="https://kubernetes.io/docs/concepts/">https://kubernetes.io/docs/concepts/</a></div>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com0tag:blogger.com,1999:blog-1479780601142337324.post-22913097882223509882019-08-10T07:47:00.002-07:002019-08-10T07:47:18.928-07:00Quick Introduction to Matplotlib<div dir="ltr" style="text-align: left;" trbidi="on">
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Overview:</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">In Datascience and Machine learning, plots and graphs play a very important role. To understand the data and its characteristics, it is very important to plot the data and make sense out of it. Matplotlib is a powerful Python library which is very handy in building effective plots. We will take a look at a brief intro to Matplotlib.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Scatter Plot:</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Scatter Plot plots two variables one each along X and Y axis and provides a view of the relationship or correlation between the two variables. Let us create two variables, where one variable has an exponential relationship with the other (i.e logarithmic relationship).</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Let us hypothetically assume that age and salary are related exponentially. So as age increases, salary increases exponentially. Let us create two dataframes with this in mind and plot them. Note how we added labels for X and Y axis and also a title for the plot.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /><script src="https://gist.github.com/raghumb/94f0a2f43968796aee8cad783e9bcade.js"></script></span></div>
<br />
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-ii_YNcqFFWY/XU7SSvSRhEI/AAAAAAAAAxE/z9xUoYzoEM8H-xNdXdMeTVrboRMvn7RYACLcBGAs/s1600/1.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="699" data-original-width="1468" height="304" src="https://1.bp.blogspot.com/-ii_YNcqFFWY/XU7SSvSRhEI/AAAAAAAAAxE/z9xUoYzoEM8H-xNdXdMeTVrboRMvn7RYACLcBGAs/s640/1.PNG" width="640" /></a></div>
<br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Line Plot:</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">We can also plot two variables and draw a line to reflect the relationship between them. Matplotlib works both with Numpy arrays and with Pandas dataframe. Let us create two variables one representing weight in Kilogram and the other in pounds. Obviously, the relationship between these two is linear. Let us plot this. Note how we provided a legend to the line. If there are multiple lines, we can provide a legend to each of the lines plotted.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span><script src="https://gist.github.com/raghumb/7f128d0903b5da1c6fced23cc39aa8e5.js"></script></div>
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-MetrDLw7OlU/XU7TsMFwzsI/AAAAAAAAAxQ/ZrmKQLSAXvkg_LkB-EkmDQmL5O7lkW0KQCLcBGAs/s1600/2.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="687" data-original-width="1502" height="292" src="https://1.bp.blogspot.com/-MetrDLw7OlU/XU7TsMFwzsI/AAAAAAAAAxQ/ZrmKQLSAXvkg_LkB-EkmDQmL5O7lkW0KQCLcBGAs/s640/2.PNG" width="640" /></a></div>
<br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Multiple Line Plots in a Graph:</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">We can draw multiple lines in a graph. This will be help us visualize how multiple variables are moving against each other. Let us plot the same age and salary which we plotted earlier, but instead of plotting age vs salary, let us plot age line and salary line independently. We will use date as an index just to see how to plot indexed dataframes.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Note how we used legend for each of the lines and also the position of the legend could be customized. Also note the line color has been customized as well.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span><script src="https://gist.github.com/raghumb/f5a4ba0741569c18e9c7432f9a191ba4.js"></script></div>
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-Ky3xBZiBOyc/XU7YI1ySuFI/AAAAAAAAAx4/VYBGspo29GEitEyrfnggT42hBrpI8i8RwCLcBGAs/s1600/3.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="759" data-original-width="1265" height="384" src="https://1.bp.blogspot.com/-Ky3xBZiBOyc/XU7YI1ySuFI/AAAAAAAAAx4/VYBGspo29GEitEyrfnggT42hBrpI8i8RwCLcBGAs/s640/3.PNG" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<h4 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Note:</span></h4>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">A final note while using Matplotlib in unix systems. Typically we might want to save images to file system using plt.savefig(). However, when you invoke this multiple times for different plots, the plots tend to overwrite on each other. To avoid this, we would need to clear the canvas after plotting each graph. This could be done by using plt.clf()</span></div>
<br /></div>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com0tag:blogger.com,1999:blog-1479780601142337324.post-81712143035735436142019-08-06T07:28:00.001-07:002019-08-06T07:28:38.962-07:00Quick Introduction to Pandas Library<div dir="ltr" style="text-align: left;" trbidi="on">
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Overview:</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Pandas is a python library which provides very powerful tools to perform complex data manipulations and analysis. In this article, few of the commonly used operations are explained.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">A common datastructure is a 'Dataframe'. Dataframe is a matrix type of datastructure with easy access to rows and columns.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Creation of Dataframe:</span></h3>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-avCqBwES_Wc/XUmHc0BfhaI/AAAAAAAAAvo/GTIOq-L-9sYFp3Z0qSZKzcNTMVkLrOcWwCLcBGAs/s1600/1.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="475" data-original-width="1600" height="188" src="https://1.bp.blogspot.com/-avCqBwES_Wc/XUmHc0BfhaI/AAAAAAAAAvo/GTIOq-L-9sYFp3Z0qSZKzcNTMVkLrOcWwCLcBGAs/s640/1.PNG" width="640" /></a></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<script src="https://gist.github.com/raghumb/23c1b60961c9bbf249c2f91f34dc0840.js"></script>
<br />
<div>
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Accessing data from Dataframe:</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Data in Dataframe could be accessed in various ways.</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-s4iq28lL5UU/XUmLNLL7rmI/AAAAAAAAAwA/5-rgZotXiWcK5Bonm9Kv9J8A7rhUmIOYgCLcBGAs/s1600/2.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="601" data-original-width="1600" height="240" src="https://1.bp.blogspot.com/-s4iq28lL5UU/XUmLNLL7rmI/AAAAAAAAAwA/5-rgZotXiWcK5Bonm9Kv9J8A7rhUmIOYgCLcBGAs/s640/2.PNG" width="640" /></a></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<br />
<script src="https://gist.github.com/raghumb/84fc40478b8ef0b4fb403824c12532a5.js"></script>
</div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Data can also be accessed by slicing the dataframe instead of loops. It is the most efficient way of accessing data.</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-Jxezz9Q3LLY/XUmMYs8K3NI/AAAAAAAAAwI/eo_fLUTGrusmqzwwneRPlYyQTVwgAMr8gCLcBGAs/s1600/3.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="267" data-original-width="1600" height="106" src="https://1.bp.blogspot.com/-Jxezz9Q3LLY/XUmMYs8K3NI/AAAAAAAAAwI/eo_fLUTGrusmqzwwneRPlYyQTVwgAMr8gCLcBGAs/s640/3.PNG" width="640" /></a></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<script src="https://gist.github.com/raghumb/24225fc9781324a68b9d80f8702d6706.js"></script></div>
<div>
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Creating Dataframe with Index:</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">We can create Dataframe with an index. Let us create a dataframe with date as index.</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-kvjzvqVRxUM/XUmNKzZuq2I/AAAAAAAAAwQ/cPE_Vj_ZW0cQYxdFyAEDxVdhTIfyd1wMACLcBGAs/s1600/4.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="502" data-original-width="1600" height="200" src="https://1.bp.blogspot.com/-kvjzvqVRxUM/XUmNKzZuq2I/AAAAAAAAAwQ/cPE_Vj_ZW0cQYxdFyAEDxVdhTIfyd1wMACLcBGAs/s640/4.PNG" width="640" /></a></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<br />
<script src="https://gist.github.com/raghumb/4827993d05fb35bd52e30a435fbbda09.js"></script></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">As we can see from above, the values are filled as NaN (Not a Number). We can replace NaN with zero using fillna() function. </span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-czugPP3EH4E/XUmOWLKy0QI/AAAAAAAAAwc/cB463qYVJYksolgr8C-HGZD3ONsxQBUYQCLcBGAs/s1600/5.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="332" data-original-width="1600" height="132" src="https://1.bp.blogspot.com/-czugPP3EH4E/XUmOWLKy0QI/AAAAAAAAAwc/cB463qYVJYksolgr8C-HGZD3ONsxQBUYQCLcBGAs/s640/5.PNG" width="640" /></a></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<script src="https://gist.github.com/raghumb/7af746a5e3d38d5845c28135989bacee.js"></script>
</div>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com0tag:blogger.com,1999:blog-1479780601142337324.post-53339001318245816502019-08-04T07:19:00.002-07:002019-08-04T08:13:41.556-07:00Quick Introduction to Numpy Library<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Overview:</span></h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Numpy is a popular library in Python, which is used in Machine Learning and Statistical analysis. This article provides some of the commonly used operations in Numpy. Numpy array operations are efficient and fast compared to normal array operations.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Jupyter Notebook:</span></h4>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The code is run using Jupyter Notebook and the relevant snapshots are provided.</span></div>
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Note: Numpy could be imported as below:</span></h4>
<span style="font-family: "arial" , "helvetica" , sans-serif;">import numpy as np</span><br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Array Operations:</span></h3>
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Basic Operations:</span></h4>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Array could be created using np.array(). Shape property provides a tuple of number of rows and columns. We can apply various mathematics functions like mean, median etc on the array.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-2yrRcnv8TGA/XUbrB-QTmnI/AAAAAAAAAtc/7Jev5OFw79cbhH5M-OW8_AoixQx8pF20ACLcBGAs/s1600/array.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="550" data-original-width="1600" height="217" src="https://1.bp.blogspot.com/-2yrRcnv8TGA/XUbrB-QTmnI/AAAAAAAAAtc/7Jev5OFw79cbhH5M-OW8_AoixQx8pF20ACLcBGAs/s640/array.PNG" width="640" /></a></div>
<br /></div>
<div>
<script src="https://gist.github.com/raghumb/1776b3f75ce32a0485824e5661aa3b3d.js"></script>
</div>
<div>
<br />
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Comparison in Array:</span></h4>
</div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">We can apply various techniques to compare one row against other, which would be required when we perform computational analysis.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-xcUeAAL4g2k/XUbtlbTRO0I/AAAAAAAAAt8/tBrgmFuFn7INuJeoJpswVjSJFpQUoNOBACLcBGAs/s1600/comp.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="548" data-original-width="1600" height="218" src="https://1.bp.blogspot.com/-xcUeAAL4g2k/XUbtlbTRO0I/AAAAAAAAAt8/tBrgmFuFn7INuJeoJpswVjSJFpQUoNOBACLcBGAs/s640/comp.PNG" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<script src="https://gist.github.com/raghumb/51bb446796f52843c210659dbb2a0d4b.js"></script>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Array Data Retrieval:</span></h4>
</div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">We can retrieve data from an array, in terms of rows, columns. We can also, retrieve specific rows or columns or exclude specific rows or columns.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-aYQcYXC2vLI/XUbxmCTZTRI/AAAAAAAAAu0/uls-sRZgWNcGqSjgvzJHCdUFnSnMiDDBgCLcBGAs/s1600/retreival.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="627" data-original-width="1600" height="250" src="https://1.bp.blogspot.com/-aYQcYXC2vLI/XUbxmCTZTRI/AAAAAAAAAu0/uls-sRZgWNcGqSjgvzJHCdUFnSnMiDDBgCLcBGAs/s640/retreival.PNG" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
<script src="https://gist.github.com/raghumb/b8440e6873938e3bef4dc2bca8966953.js"></script>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Max Min Operations:</span></h4>
</div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">We can find out various maximum and minimum values from the array. We can find out index of maximum value in the array and also retrieve maximum across an row or column.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-yNS3vADTWgY/XUb1E_C_krI/AAAAAAAAAvM/N2_AJeiRi68YPWt9TIqLgoG5YJzTXAPHACLcBGAs/s1600/max%2Bmin.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="701" data-original-width="1600" height="280" src="https://1.bp.blogspot.com/-yNS3vADTWgY/XUb1E_C_krI/AAAAAAAAAvM/N2_AJeiRi68YPWt9TIqLgoG5YJzTXAPHACLcBGAs/s640/max%2Bmin.PNG" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<script src="https://gist.github.com/raghumb/dfb70da38bd699bdffa5d794f88e7cf0.js"></script>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<br /></div>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com0tag:blogger.com,1999:blog-1479780601142337324.post-71125733183273564162019-04-11T05:14:00.001-07:002019-04-11T05:14:12.704-07:00Serverless Technology Spectrum<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
In this article, I am going to introduce you to Serverless technology in various flavors along with a usecase and its benefits.</div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; min-height: 8pt; padding: 0px; vertical-align: baseline;">
<br /></div>
<h2 style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 20px; margin: 0px; padding: 0px; vertical-align: baseline;">
What is Serverless?</h2>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
First of all, Serverless does not mean 'No Servers!!' (It is a misnomer). There are various definitions for Serverless, we will try to look at what Serverless means in various contexts.</div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; min-height: 8pt; padding: 0px; vertical-align: baseline;">
<br /></div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
Let me illustrate through an analogy.</div>
<h3 style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 16px; margin: 0px; padding: 0px; vertical-align: baseline;">
Analogy:</h3>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
Let us consider a scenario where, there are a number of doctors (dentists, chiropractors etc) who want to serve their patient, but they do not want to invest in real estate for the clinics, equipment and also do not want the burden of managing/repairing their equipment.</div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
To cater to their needs, an investor comes forward and builds a huge building and also procures all the necessary tools and instruments needed by various doctors. When a patient arrives, the doctor will be provided any vacant room and the room will be quickly setup with all the instruments. Once the consultation completes, the room is available for other doctors. The doctors only pay for the hourly consumption of the room and the instruments.</div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; min-height: 8pt; padding: 0px; vertical-align: baseline;">
<br /></div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
This is the Serverless model, except that, rooms and instruments could be related to servers and the infrastructure required, whereas the incoming patients could be related to the incoming service requests. Application developers can just focus on writing application logic and the code will be run on servers provisioned on the fly by the cloud provider.</div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
<br /></div>
<h2 style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 20px; margin: 0px; padding: 0px; vertical-align: baseline;">
How does Serverless work?</h2>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
Serverless has its roots in 'Function As a Service' paradigm. When we write a function which adheres to a particular definition, the function is deployed and served by the underlying platform when a request arrives.</div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
Serverless Technology works in a similar way, wherein, we have to implement a function which provides a particular behavior and define trigger points which could invoke the function. The trigger point could be a HTTP invocation, or any other event, like a message in a queue.</div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
When the function gets the first request, an infrastructure (which could be a server/container depending on provider), is provisioned and the function is deployed, so that it can serve the request. This period is known as the ‘Cold Start’. After a certain period of time, if there are no more requests for the function, the infrastructure is tore down, thus saving money.</div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
Thus if we have a service which would be serving requests for lets say 5 hours in a day, then we are only paying for 5 hours!. The service would be running roughly for 5 to 6 hours (some cloud providers keep the function warm for a little longer after the requests are served). This provides huge cost benefits.</div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
<br /></div>
<h2 style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 20px; margin: 0px; padding: 0px; vertical-align: baseline;">
Business Use case:</h2>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
A logistics company which provides courier services for businesses in a particular geography, wants to build a service which provides status of a particular shipment. Since it’s end users are business clients, this service would usually be invoked only during business hours (say 8 AM to 5 PM). If this company does not use Serverless technology, then they would have to pay for the service infrastructure for all 24 hours a day. However, if they use Serverless, then they would just have to pay per use and since the service is not likely to be invoked after business hours, their cost benefits would be proportional to 62%. And in case, if the service gets invoked after business hours (just in case, some client who stays up late and uses the service), it still works!!</div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
<br /></div>
<h2 style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 20px; margin: 0px; padding: 0px; vertical-align: baseline;">
What are the benefits of Serverless Technology?</h2>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-zyQ4xwg1PuM/XK8u3WHBLyI/AAAAAAAAAqY/71JG21LWSuMDFiT6iMCyPyi39SpZ9s9qwCLcBGAs/s1600/benefits1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="318" data-original-width="565" height="180" src="https://4.bp.blogspot.com/-zyQ4xwg1PuM/XK8u3WHBLyI/AAAAAAAAAqY/71JG21LWSuMDFiT6iMCyPyi39SpZ9s9qwCLcBGAs/s320/benefits1.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<ul style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; margin: 0px; padding: 0px 0px 0px 30px; vertical-align: baseline;">
<li style="border: 0px; font-family: inherit; font-style: inherit; font-weight: inherit; list-style-type: inherit; margin: 0.5ex 0px; padding: 0px; vertical-align: baseline;">Serverless Technology enables Event Driven Architecture.</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; font-weight: inherit; list-style-type: inherit; margin: 0.5ex 0px; padding: 0px; vertical-align: baseline;">Since there is no need to provision infrastructure, Serverless Technology boosts developer productivity.</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; font-weight: inherit; list-style-type: inherit; margin: 0.5ex 0px; padding: 0px; vertical-align: baseline;">Serverless functions are scalable (though the definition of scalability depends on the cloud environment on which the function is hosted and might have constraints, depending on various other factors). However, in general, serverless functions offer horizontal scalability based on traffic.</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; font-weight: inherit; list-style-type: inherit; margin: 0.5ex 0px; padding: 0px; vertical-align: baseline;">Serverless Technology saves cost. In terms of cost savings, traditional architectures were on ‘Always Lights On” mode, and hence whether the application is serving requests or not, we had to pay for the infrastructure.</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; font-weight: inherit; list-style-type: inherit; margin: 0.5ex 0px; padding: 0px; vertical-align: baseline;">Serverless also removes the burden of managing infrastructure (This is true only for Public clouds, since the Serverless infrastructure is actually managed by the Cloud provider).</li>
</ul>
<div>
<span style="color: #454545; font-family: Roboto-Light;"><span style="font-size: 14px;"><br /></span></span></div>
<div>
<h2 style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 20px; margin: 0px; padding: 0px; vertical-align: baseline;">
What are the Serverless offerings in Public Cloud landscape?</h2>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
Three of the major public cloud providers shown below support Serverless.</div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-6OQFCYsAzdw/XK8vG3iC6ZI/AAAAAAAAAqc/TlDozXrYMhkCSct_9NMyZtPQEaolagsZACLcBGAs/s1600/public.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="556" data-original-width="1600" height="111" src="https://2.bp.blogspot.com/-6OQFCYsAzdw/XK8vG3iC6ZI/AAAAAAAAAqc/TlDozXrYMhkCSct_9NMyZtPQEaolagsZACLcBGAs/s320/public.png" width="320" /></a></div>
<h2 style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 20px; margin: 0px; padding: 0px; vertical-align: baseline;">
What does Serverless mean in Private Cloud?</h2>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-R5_CJ4ShYgQ/XK8vVwURgYI/AAAAAAAAAqk/gyWRS6KcPacSL1vd7CRDsn0nGQI_KEuXgCLcBGAs/s1600/private.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="459" data-original-width="1558" height="94" src="https://4.bp.blogspot.com/-R5_CJ4ShYgQ/XK8vVwURgYI/AAAAAAAAAqk/gyWRS6KcPacSL1vd7CRDsn0nGQI_KEuXgCLcBGAs/s320/private.png" width="320" /></a></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-VPnwuZrLsuY/XK8vaLwrtrI/AAAAAAAAAqs/Zc3KAnfi-YsVH_iKfwU71TVZZ5kFkhk0QCLcBGAs/s1600/private%252Bcloud.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="900" data-original-width="1600" height="180" src="https://2.bp.blogspot.com/-VPnwuZrLsuY/XK8vaLwrtrI/AAAAAAAAAqs/Zc3KAnfi-YsVH_iKfwU71TVZZ5kFkhk0QCLcBGAs/s320/private%252Bcloud.png" width="320" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
In public cloud, the Serverless infrastructure is managed by the cloud provider (like AWS for example) and it is much like a black box.</div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; min-height: 8pt; padding: 0px; vertical-align: baseline;">
<br /></div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
However, in private cloud, the serverless infrastructure has to be managed on premise and we can use various frameworks which enable Serverless technology.</div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
On private cloud, we can provision infrastructure using products like VMWare VXRail and then deploy and use Kubernetes as a container orchestration platform.</div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
We can then utilize some of the Serverless frameworks available on Kubernetes, which are shown above.</div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
Similar to the Serverless model explained above, in Kubernetes, the Pods would be created on demand and the respective function/service would be deployed on the Pod.</div>
<div style="background-color: white; border: 0px; color: #454545; font-family: Roboto-Light; font-size: 14px; padding: 0px; vertical-align: baseline;">
The above diagram shows some of the prominent Serverless frameworks supported on Kubernetes and also a typical On Premise Serverless Solution.</div>
</div>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com0tag:blogger.com,1999:blog-1479780601142337324.post-73030939105550134672019-01-15T19:45:00.002-08:002019-01-15T22:17:30.028-08:00Terraform Part 2<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "arial" , "helvetica" , sans-serif;">In the previous post '<a href="http://www.raghumb.com/2018/11/quick-introduction-to-terraform.html">Intro to Terraform</a>', I provided an introduction to Terraform. In this post, I will be digging a little bit deeper into how Terraform could be used in real world usecases.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">How to use organize Terraform files?</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">We can organize Terraform files so that, each component which we would provision would go into its own .tf file. That way, we can have a modularized approach.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">provider.tf</span></h4>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">This file would have the provider configuration details, so that any change to provider related config could be done in one place.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /><script src="https://gist.github.com/raghumb/a288f4fa9dc94dd551f2b2576fe89039.js"></script></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">vars.tf</span></h4>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">This file would have the variables used by every different file. We can variabilize different parts of our infrastructure configuration, like for eg, the CIDR ranges which will be used in VPC and subnet. This way, we can parameterize our provisioning code.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /><script src="https://gist.github.com/raghumb/e2253b5068cab586c27bc7f873777f2f.js"></script></span></div>
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">One tf file per Component:</span></h4>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">We can have one .tf file per component, for instance, a vpc.tf file, a subnet.tf file etc. This way we can isolate changes.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /><script src="https://gist.github.com/raghumb/edbee8b67b24a99c86eab5958dec7326.js"></script></span></div>
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Data Selector:</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Another important feature which we can use is the Datasource configuration. Let us say we create a component in one file (say vpc) and in another file we would like to retrieve the Id of this component. One good way to do this is to configure a datasource and use it to filter and retrieve the component.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Example:</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /><script src="https://gist.github.com/raghumb/6269592e8a46fc5e191cf1063ebe8c03.js"></script></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Data Configuration for Policy:</span></h3>
</div>
<div>
<span style="font-family: arial, helvetica, sans-serif;">If we are going to create a IAM policy (let us say for SQS), using terraform, one of the challenges is how to do we create provide all the inputs required for this policy. Generally, AWS policies are in JSON format. Terraform provides a data configuration option for this.</span></div>
<div>
<span style="font-family: arial, helvetica, sans-serif;"><br />Below is an example for creating a policy for SQS. We can provide all the required config data in data node and then use the .json option while creating resource.</span></div>
<div>
<span style="font-family: arial, helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: arial, helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /><script src="https://gist.github.com/raghumb/69a975ac031235bd30428584334cb5cf.js"></script></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com0tag:blogger.com,1999:blog-1479780601142337324.post-91938998838933539742019-01-13T04:37:00.003-08:002019-01-13T04:44:46.198-08:00Bayes Theorem<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "arial" , "helvetica" , sans-serif;">In this article, let us try to understand Bayes Theorem.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">This article has been inspired by the two videos provided in References section. The illustrations used here are my own.</span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">Bayes theorem helps us draw inferences from data. It also challenges our beliefs which could be often biased.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Let us say that we came across a group of athletes from many countries with the athletes from top 2 or 3 popular sports from those countries. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">We will consider a group of athletes who play either soccer or basketball.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The height of one of the athletes is more than 7 feet. What do you think this athlete plays? </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Soccer or Basketball.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Our intuition definitely says that he must be playing basketball.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Now, let us do the math.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The most popular sport played across the globe is soccer (No offense to Basketball!). Let us say we have total of 100 athletes and 90% of them play soccer and 10% play basketball.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">That means 90 athletes play soccer.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Now let us see how many of soccer players are more than 7 feet tall. We might have 20% soccer players who are more than 7 feet tall.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Now let us see how many of basketball players are more than 7 feet tall. Given the nature of the game, we might have 80% of basketball players who are more than 7 feet tall. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The distribution is shown in the below diagram. </span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-ZFTfpqpmINU/XDsrNGYFeAI/AAAAAAAAApE/NovDongjA0wwBkz0Spoy58ZJtKmXsSCvACLcBGAs/s1600/bayes.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><img border="0" data-original-height="900" data-original-width="1600" height="180" src="https://3.bp.blogspot.com/-ZFTfpqpmINU/XDsrNGYFeAI/AAAAAAAAApE/NovDongjA0wwBkz0Spoy58ZJtKmXsSCvACLcBGAs/s320/bayes.png" width="320" /></span></a></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Clearly, we are trying to find out if the player in question lies in Area 1 or in Area 2.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">If we compare the areas, obviously area 2 is larger than area 2, which means the player must be playing soccer.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Why did our intuition go wrong. It went wrong because, we discounted the fact that there are more number of soccer players in the group than basketball players.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">So, our chances that we run into a soccer player is 9 times more than the chance that we run into a basketball player.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">This concept is fundamental to Bayesian Theorem.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Putting this in form of an equation:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">So the probability that the player is a Soccer player, given that he is more than 7 feet tall should be calculated by multiplying the Area 2 by the probability of total number of soccer players and </span><span style="font-family: "arial" , "helvetica" , sans-serif;">dividing by total probability of players who are more than 7 feet tall.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-rKb9Qsf93CE/XDsw1CeT_xI/AAAAAAAAApc/91fn7PRIiE8bBjFbMAdxO42FaNlurOfogCLcBGAs/s1600/eq1n.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><img border="0" data-original-height="648" data-original-width="1152" height="180" src="https://4.bp.blogspot.com/-rKb9Qsf93CE/XDsw1CeT_xI/AAAAAAAAApc/91fn7PRIiE8bBjFbMAdxO42FaNlurOfogCLcBGAs/s320/eq1n.png" width="320" /></span></a></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="white-space: pre;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span><br />
<span style="white-space: pre;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">On similar lines:</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-TR7T5Vcu_g8/XDsw5AyGHUI/AAAAAAAAApg/DRMsvuqsBF8rVDYiEITSlx539J8k74pQgCLcBGAs/s1600/Eqn2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><img border="0" data-original-height="648" data-original-width="1152" height="180" src="https://3.bp.blogspot.com/-TR7T5Vcu_g8/XDsw5AyGHUI/AAAAAAAAApg/DRMsvuqsBF8rVDYiEITSlx539J8k74pQgCLcBGAs/s320/Eqn2.png" width="320" /></span></a></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">We can see that the probability of the player who is more than 7 feet tall being a soccer player is higher. The above formula whose intuition we know now, is known as Bayes Theorem. Bayes theorem is used very heavily in Machine Learning and in Robotics.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Another aspect of Bayes theorem is the prior and posterior probabilities. Prior probability is the probability before we measure something and whose knowledge is assumed. Posterior probability is the probability after measuring something.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">In the above example, probability of Soccer players is prior probability and the probability of a player being a soccer player given that is more than 7 feet tall could be considered as posterior probability.</span><br />
<span style="white-space: pre;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span><br />
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">References:</span></h4>
<span style="font-family: "arial" , "helvetica" , sans-serif;">1. https://www.youtube.com/watch?v=BrK7X_XlGB8&t=8s</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">2. https://www.youtube.com/watch?v=5NMxiOGL39M</span><br />
<br /></div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com1tag:blogger.com,1999:blog-1479780601142337324.post-3359668180589124622018-11-13T20:41:00.000-08:002018-11-13T20:41:10.084-08:00Quick Introduction to Terraform<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Introduction</span></h2>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">There are many tools and frameworks available for provisioning infrastructure on the cloud. One of the most popular cloud agnostic tool is Terraform. </span><span style="font-family: "arial" , "helvetica" , sans-serif;">It is a vital piece of Infrastructure as a Code Service and could be used to tear up or tear down infrastructure.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">We will learn about some basic nuances of Terraform in this post. </span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Installing Terraform:</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Terraform could be installed by finding the appropriate package and following instructions. (Refer https://www.terraform.io/intro/getting-started/install.html). </span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">In this post, I will be using Terraform on CentOS.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Getting Started:</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Let us try to get our hands dirty. Create a directory, 'scripts' and cd into that.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Run 'terraform --version' to confirm if terraform is successfully installed.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Let us create our first terraform file. Terraform files have an extension of .tf. </span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">So, create a file 'first.tf'.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Let us create a variable called 'name' and try to print it out.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span><script src="https://gist.github.com/raghumb/f682b4e2c9c20010a9a694c49f268854.js"></script></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Note the syntax, the variables should be referred with a prefix 'var' and variables can have default values.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Before we can start running this file, we need to initialize terraform. Terraform init is used to initialize a new or existing Terraform configuration.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The command to initialize is 'terraform init'</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Creating a Terraform Plan:</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Now, to run this file, we need to create a plan first. A plan generates and shows an execution plan. It is always advisable to create a plan as it gives an opportunity to review what resources are being created.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">We will save the plan to a file and then apply changes using the plan.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Commands:</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span><br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><i>terraform plan -out first.terraform</i></span></div>
<i><span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
</i><br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><i>terraform apply first.terraform</i></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">When you run the 'apply' command, we will get a result as below:</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><i>"Apply complete! Resources: 0 added, 0 changed, 0 destroyed."</i></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<div>
<i><br /></i></div>
<h3 style="text-align: left;">
Terraform Providers:</h3>
<div>
Providers are responsible in Terraform for managing the lifecycle of a resource. i.e creating a resource or updating it etc. There are a lot of cloud providers which are implemented by Terraform. It supports AWS, Google Cloud to name a few.</div>
<div>
<br /></div>
<h4 style="text-align: left;">
Sample provider configuration(AWS):</h4>
<div>
Get the AWS Secret key and access key and set it into environment variables:</div>
<div>
<br /></div>
<div>
<div>
export AWS_ACCESS_KEY_ID=<YOUR_KEY></div>
<div>
export AWS_SECRET_ACCESS_KEY=<YOUR_KEY></div>
</div>
<div>
<br /></div>
<div>
Create a new file 'second.tf' with the following content:</div>
<div>
<br /></div>
<div>
<br /></div>
</span></div>
</div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><script src="https://gist.github.com/raghumb/244b0e1386fc9251b238ffa0278534d5.js"></script></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span><span style="font-family: "arial" , "helvetica" , sans-serif;">Note that we are trying to create a EC2 instance of type t2.micro and tagged as 'my-vm-automated'.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Note that before creating a plan file, we need to run 'terraform init' again. When we ran it first time, the provider was not there and when we change or add a provider, we will need to initialize again.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Create a plan file and apply the plan file.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><i>terraform plan -out second.terraform</i></span><br />
<i><span style="font-family: "arial" , "helvetica" , sans-serif;"></span></i><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><i>terraform apply second.terraform</i></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><i><br /></i></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">You will see that terraform will show what resources are being created.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Finally you would see:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><i>"Apply complete! Resources: 1 added, 0 changed, 0 destroyed."</i></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><i><br /></i></span>
<br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Terraform State:</span></h3>
</div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span><br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">When you apply changes, you would notice that a 'terraform.tfstate' file is created.</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<div>
This file is used by terraform to save the current state of the infrastructure. This is used when we run subsequent files and hence terraform will know what changes to apply.</div>
<div>
<br /></div>
<div>
Now let us try to rerun the same terraform plan again:</div>
<div>
<i>terraform apply</i></div>
<div>
<i><br /></i></div>
<div>
You will get a message indicating, the plan is older than the current state and cannot be run. Terraform gets this insights through the terraform.tfstate file and verifies if the plan is latest. If not, it will not run the plan.</div>
<div>
Now let us try to simply apply the terraform files. </div>
<div>
<i>terraform apply</i></div>
<div>
<i><br /></i></div>
<div>
We will see that the no resources are added or deleted, since we have not made any changes to terraform file.</div>
<div>
<i>"Apply complete! Resources: 0 added, 0 changed, 0 destroyed."</i></div>
<div>
<i><br /></i></div>
<div>
Now login to AWS console and verify that the EC2 instance is created.</div>
<div>
<br /></div>
<h3 style="text-align: left;">
Destroy resources:</h3>
<div>
Run the command '<i>terraform destroy</i>'.</div>
<div>
It will check the state file and notify you that it is going to destroy those resources as below:</div>
<div>
Once you confirm, it will delete the EC2 instance created.</div>
<div>
<br /></div>
<h3 style="text-align: left;">
Creating an EC2 instance with a key pair:</h3>
<div>
<div>
Create a key pair using puttygen. (Say a file myec2keypair.pem)</div>
<div>
Update the above configuration to refer to that key pair.</div>
</div>
<div>
<br /></div>
<div>
<br />
<script src="https://gist.github.com/raghumb/c1d1034a576e23b3c17fe93b3f0cce95.js"></script></div>
<div>
Next time you want to create an infrastructure on AWS, resist the temptation of using AWS CLI or GUI....Happy Terraforming!!</div>
</span></div>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com0tag:blogger.com,1999:blog-1479780601142337324.post-32450854966005722382018-09-07T04:10:00.002-07:002018-09-07T09:25:03.349-07:00AWS - Basic Security Considerations<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Often the default Security rules in AWS allow highest level of access and this is undesirable in production system. Principle of Least access is the recommended security approach for any secure system. </span></div>
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">This article explains how we can apply some specific Security rules to various resources in AWS.</span></div>
<div>
<br /></div>
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Access to EC2 instances</span></h2>
<hr />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Do not provide SSH access to EC2 instances in public subnet, which are exposed to outside world. Rather, create a separate EC2 instance which will act like a Bastion server. Let us call the security group associated with this Bastion server as ‘BastionSG’. </span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<div style="font-family: Arial, Helvetica, sans-serif;">
The ‘BastionSG’ should have a security group which will allow a specific IP or a range of IP in corporate network via SSH for allowing inbound access.</div>
<div style="font-family: Arial, Helvetica, sans-serif;">
Expose SSH from all EC2 instances only from the Bastion server by providing the ‘BastionSG’ as the source Security Group.</div>
<div style="font-family: Arial, Helvetica, sans-serif;">
<br /></div>
<h2 style="font-family: Arial, Helvetica, sans-serif; text-align: left;">
Access to VPC</h2>
<hr />
<div style="font-family: Arial, Helvetica, sans-serif;">
<div>
By default, a VPC will have a NACL(Network Access control) which allows traffic to all. We can create a custom NACL which will restrict traffic.</div>
<div>
For example following diagram shows a custom NACL which allows only a range of IPs for SSH access.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-OLdljd8DXww/W5JaJy03W6I/AAAAAAAAAnw/ScOeYQQvuJE-ZWXJ8hepObVcCEkig81XgCPcBGAYYCw/s1600/NACL.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="545" data-original-width="1086" height="160" src="https://2.bp.blogspot.com/-OLdljd8DXww/W5JaJy03W6I/AAAAAAAAAnw/ScOeYQQvuJE-ZWXJ8hepObVcCEkig81XgCPcBGAYYCw/s320/NACL.PNG" width="320" /></a></div>
<br /></div>
</div>
<div style="font-family: Arial, Helvetica, sans-serif;">
<br /></div>
<h2 style="font-family: Arial, Helvetica, sans-serif; text-align: left;">
EC2 instance to another EC2 instance</h2>
<hr />
<div style="font-family: Arial, Helvetica, sans-serif;">
<div>
When one EC2 instance communicates with another, we can use Security Groups to restrict access. Security Groups act like firewall to the EC2 instances and will control how is the service accessed? (Source IP, Port, Destination port). We can specify the source as an IP, range of IP or another Security Group.</div>
<div>
<br />
Let us say from EC2 instance we deploy an application and on the another EC2 instance, we deploy a PostGres Database. Now the application EC2 instance will need to connect to the EC2 instance having PostGres.</div>
<div>
<br />
<br />
We can create a security group for the application EC2 instance and create another Security Group for associating with the PostGres EC2 instance. The source for the PostGres EC2 instance should be the application instance Security Group as shown in below diagram.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-RGVsKuvYmlU/W5JbEyMuGhI/AAAAAAAAAn4/Kc1NqvVZ0WwQklW-I5jkxhWB81U2ST-ZgCPcBGAYYCw/s1600/Security%2BGroup.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="684" data-original-width="1287" height="170" src="https://2.bp.blogspot.com/-RGVsKuvYmlU/W5JbEyMuGhI/AAAAAAAAAn4/Kc1NqvVZ0WwQklW-I5jkxhWB81U2ST-ZgCPcBGAYYCw/s320/Security%2BGroup.PNG" width="320" /></a></div>
<br /></div>
</div>
<div style="font-family: Arial, Helvetica, sans-serif;">
<br /></div>
<h2 style="font-family: Arial, Helvetica, sans-serif; text-align: left;">
EC2 instance to other AWS services</h2>
<hr />
<div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: 0.0001pt; text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">IAM roles could be used
to trust entities while communicating between services.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Let us say from EC2
instance, we are communicating to a SQS queue. Instead of saving the SQS access
keys onto the EC2 instance, it is better to create an IAM role specific to this
communication. </span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: 0.0001pt; text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Also, if we have one EC2 instance which will read from the queue
and one EC2 instance which will write to the queue, we can create two IAM roles
to achieve this and attach them to each of the EC2 instances respectively.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">We can create an IAM
role EC2ToSQS Read which will allow the EC2 instance to read from the SQS queue.
Note that we need to create a Custom policy to get fine grained control over
this access. The default policies allow sqs:* which allow all operations on SQS
(The only differentiation is admin or read only).</span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: 0.0001pt; text-align: left;">
<br /></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: 0.0001pt; text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">We can follow below steps to create and associate IAM Roles:</span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: 0.0001pt;">
<div style="text-align: left;">
<h4 style="text-align: left;">
<b>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">Create IAM Role:</span></b></h4>
</div>
<div style="text-align: left;">
<ol style="text-align: left;">
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Create an IAM role ‘EC2ToSQSRead’</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Select EC2 as the Service.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Save the IAM role. We will come back to this role after creating the policy.</span></li>
</ol>
</div>
<h4 style="text-align: left;">
<b>
<span style="font-family: "arial" , sans-serif; font-size: small;">Create Policy:</span></b></h4>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-tmLxdXo10t0/W5JcgH3fiuI/AAAAAAAAAoU/rWRG_2dcZn4lbr589g_AD0WhE_M9kAgWACPcBGAYYCw/s1600/Policy.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="763" data-original-width="1600" height="152" src="https://4.bp.blogspot.com/-tmLxdXo10t0/W5JcgH3fiuI/AAAAAAAAAoU/rWRG_2dcZn4lbr589g_AD0WhE_M9kAgWACPcBGAYYCw/s320/Policy.PNG" width="320" /></a></div>
<span style="font-family: "arial" , sans-serif;"><br /></span>
<br />
<ul style="text-align: left;">
<li><span style="font-family: "arial" , sans-serif;">Create a policy</span></li>
<li><span style="font-family: "arial" , sans-serif;">Choose ‘SQS’ as the Service.</span></li>
<li><span style="font-family: "arial" , sans-serif;">Since we are interested in reading from the queue, let us chose ‘ReceiveMessage’ access level.</span></li>
<li><span style="font-family: "arial" , sans-serif;">Chose ‘Specific’ in the resources and Add an ARN for finer control. Here default is ‘arn:aws:sqs:::’ which again allows access to all accounts. We can provide which account, Region, queue it is allowed to access</span></li>
</ul>
<br />
<span style="font-family: "arial" , sans-serif;">The policy json would like the below:</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-eHmaGaECNZc/W5JcnzRwD-I/AAAAAAAAAoY/1cmsX2bohrUNrJnCmjuMV_Vpf1PpWN0nQCPcBGAYYCw/s1600/Policy%2Bjson.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="398" data-original-width="1098" height="115" src="https://3.bp.blogspot.com/-eHmaGaECNZc/W5JcnzRwD-I/AAAAAAAAAoY/1cmsX2bohrUNrJnCmjuMV_Vpf1PpWN0nQCPcBGAYYCw/s320/Policy%2Bjson.PNG" width="320" /></a></div>
<span style="font-family: "arial" , sans-serif;"><br /></span>
<span style="font-family: "arial" , sans-serif;"><br /></span>
<span style="font-family: "arial" , sans-serif;">Now we can go back to the IAM role and attach the above policy to it. This IAM role could be attached to the EC2 instance.</span><br />
<span style="font-family: "arial" , sans-serif;"></span><br />
<span style="font-family: "arial" , sans-serif;">Similarly, we can create another policy/role for SQS Write access and attach it to another EC2 instance.</span><br />
<div>
<br /></div>
</div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: 0.0001pt;">
<span style="font-family: "arial" , sans-serif;"><br /></span></div>
</div>
<div style="font-family: Arial, Helvetica, sans-serif;">
<br /></div>
<div style="font-family: Arial, Helvetica, sans-serif;">
<br /></div>
</div>
<div>
<br /></div>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com0tag:blogger.com,1999:blog-1479780601142337324.post-87171081525040026442016-10-14T08:50:00.000-07:002016-10-14T09:08:29.906-07:00Pivotal Cloud Foundry (PCF) Integration with Elastic Cloud Storage (ECS)<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Recently, I was involved in integrating Pivotal Cloud Foundry with Elastic Cloud Storage (ECS), an object storage solution from EMC.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">In this post, I'm going to document the hiccups we faced during this integration and how did we resolve this, so that it is easier for other folks who would like to carry out this integration.</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
</span><br />
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">References:</span></h2>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;">We followed the blog post, </span><a href="https://blog.codedellemc.com/2016/01/05/emc-ecs-with-cloud-foundry/" style="font-family: arial, helvetica, sans-serif;">https://blog.codedellemc.com/2016/01/05/emc-ecs-with-cloud-foundry/</a></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">and the service broker code from git hub,<a href="https://github.com/spiegela/ecs-cf-service-broker">https://github.com/spiegela/ecs-cf-service-broker</a>. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
</span><br />
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">1. application.yml file Configuration:</span></h2>
<hr />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">First task would be to update the application.yml file in the broker code to have the correct configuration.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Note the spring profiles created in yml file. The active spring profile is defined in the build.gradle file to be 'development'. So based on that, we need to update the correct section in the yml file.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Under the broker section:</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span>a) Provide a valid ECS namespace name (The namespace name is case sensitive). Under this namespace, PCF would create a bucket to store all the metadata related to this integration.</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span>b) Provide a valid ECS replication group name (case sensitive).</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span>c) Provide a management end point, which would generally be a https end point (with port 4443).</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span>For eg, https://10.20.30.40:4443</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<div>
<br /></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>d) Provide an object end point, which in our case was same as management end point (or /object/bucket suffix in the url)</div>
<div>
<br /></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>e) Add a password property in this section and set it to the ECS password (This attribute is missing in the application.yml file, but you can see this property in the BrokerConfig.java file)</div>
<div>
<br /></div>
<h2 style="text-align: left;">
2. Enable SSL handshake communication:</h2>
<hr />
<div>
<div>
Second task is to enable SSL handshake between the Service Broker and ECS. The broker uses the public key file localhost.pem file which is present in src/main/resources folder. We will need to replace this file with a public key file corresponding to our ECS installation.</div>
<div>
<br /></div>
<div>
Let's export the public certificate from our ECS application. </div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>a) Open the ECS application in a browser (say Chrome)</div>
<div>
<br /></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>b) Follow these steps to save the certificate from ECS to local file system.</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>http://docs.bvstools.com/home/ssl-documentation/exporting-certificate-authorities-cas-from-a-website</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>c) Lets say in step b) above, the file was saved as ecscert.cer</div>
<div>
<br /></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>d) Now , we will need to convert the public key file format from .cert to .pem. We will use Java keytool for this. There could be other tools for performing this step as well.</div>
<div>
<br /></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>e) Run the following commands from command prompt. We are creating a sample keystore temporarily. We have named it as 'mytest', it would be temporary and doesnt matter. While creating this, it would ask for a password which should be remembered as it is required in further steps.</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>In the third command, below provide path to the ecscert.cer. If you are running these commands from same directory as the cert file, then provide the file name, otherwise complete path to the file. </div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>keytool -genkey -alias test -keystore mytest</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>keytool -delete -alias test -keystore mytest</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>keytool -import -trustcacerts -alias test -file ecscert.cer -keystore test.keystore</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>keytool -exportcert -alias test -file localhost.pem -rfc -keystore test.keystore</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>After the fourth command above, it would create a new file localhost.pem file which is what we would need.</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>f) Copy the above localhost.pem to src/main/resources and replace the existing localhost.pem file.</div>
</div>
<div>
<br /></div>
<h2 style="text-align: left;">
3. Service Broker security:</h2>
<hr />
<div>
<div>
The service broker application uses spring security, so it uses a default username called 'user' and a password as defined in the below section (depending on the spring profile which you choose):</div>
<div>
<br /></div>
<div>
security:</div>
<div>
user:</div>
<div>
password: password</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span></div>
<div>
So, with the above config, the broker would be secured using credentials user/password.</div>
</div>
<div>
<br /></div>
<h2 style="text-align: left;">
4. Service Broker API Version:</h2>
<hr />
<div>
Cloud Foundry comes with different Service Broker API versions and the broker application has to be compatible with it. This broker application uses API version 2.8, but your Cloud Foundry might expect a different version. You can use declare a bean to provide a new BrokerAPIVersion(). In our case, we simply set the brokerApiVersion field property in BrokerConfig.java to 2.7. </div>
<div>
<br /></div>
<div>
<br /></div>
<h2 style="text-align: left;">
5. Push Service Broker app to Cloud Foundry:</h2>
<hr />
<div>
<div>
The service broker application should be pushed to Cloud Foundry just like any other application. Sometimes is better to run this application locally to check if it is working fine.</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>a) Build the application using 'gradlew assemble'<br />
<br /></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>b) Run the application using java -jar build/libs/ecs-cf-service-broker-0.0.1-SNAPSHOT.jar to see if it starts without any issues.<br />
<br /></div>
</div>
<div>
c) Push the application to Cloud Foundry. We used memory of 750M for this.<br />
<br /></div>
<div>
<br /></div>
<h2 style="text-align: left;">
6. Register Service Broker with Cloud Foundry:</h2>
<hr />
<div>
<div>
Once the application is pushed successfully, we need to register the broker, so that it would appear in Cloud Foundry marketplace.</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>Run these commands after logging into CF CLI as admin.</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>a) cf create-service-broker ecs-broker user password https://ecs-broker-url</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>Note that the user & password above, are the broker credentials configured in step 3. The url is the service broker application URL, which we get after pushing to Cloud Foundry.</div>
<div>
<br /></div>
<div>
b) cf enable-service-access ecs-namespace</div>
<div>
<br />
c) cf enable-service-access ecs-bucket </div>
<div>
<br />
d) cf marketplace</div>
<div>
</div>
<div>
The fourth command above, 'cf marketplace' should display the ecs-broker service.</div>
</div>
<h2 style="text-align: left;">
<br />7. Verify Bucket Creation in ECS:</h2>
<hr />
<div>
Login to ECS, go the namespace configured in Step 1. We would see a bucket 'ecs-cf-broker-repository'. This bucket was created by Cloud Foundry as part of integration.</div>
<div>
<br /></div>
<h2 style="text-align: left;">
Conclusion:</h2>
<div>
Bingo!, these steps would help us to successfully integrate ECS with Cloud Foundry and ready to rock and get ready to write cool Cloud Native applications using ECS Object Storage!!</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com53tag:blogger.com,1999:blog-1479780601142337324.post-67153746455438718082016-07-08T08:19:00.001-07:002016-07-10T20:55:48.490-07:00Microservices based Cloud Native Application - Part III<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Preview:</span></h2>
<div style="background-color: white; color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.2px; line-height: 18.48px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">This is the third post in the series of Microservices based application development.</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span><br />
<div style="background-color: white; color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.2px; line-height: 18.48px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">The entire series could be found here:</span></div>
<div style="background-color: white; color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.2px; line-height: 18.48px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><a href="http://byteposts.blogspot.in/2016/06/microservices-based-cloud-native.html">Microservices based Cloud Native Application - Part I</a></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><a href="http://byteposts.blogspot.in/2016/07/microservices-based-cloud-native.html">Microservices based Cloud Native Application - Part II</a></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><a href="http://byteposts.blogspot.in/2016/07/microservices-based-cloud-native_8.html">Microservices based Cloud Native Application - Part III</a></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Overview:</span></h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Continuing from previous posts, in this post, I'm going to write about a few challenges which I faced while implementing the Microservices and how did I address them. This might hopefully help other folks who might run into similar issues.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Challenges faced while implementing Microservices:</span></h2>
<hr />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Issue 1:</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span><br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">While using Zuul API, I was getting the following exception, when the angular JS application, invoked the Zuul service.</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">com.netflix.zuul.exception.ZuulException: Forwarding error</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span>at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:132)</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span>at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.handleException(RibbonRoutingFilter.java:157) </span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.run(RibbonRoutingFilter.java:78)</div>
<div>
<br /></div>
<div>
<div>
We also got the following exception:</div>
<div>
com.netflix.discovery.shared.transport.TransportException: Cannot execute request on any known server</div>
</div>
<div>
<br /></div>
<h4 style="text-align: left;">
<b>Root Cause:</b></h4>
<div>
<div>
The root cause of both of the above exceptions were same:</div>
<div>
<br /></div>
<div>
The zuul server was failing to register with Eureka server as an Eureka Client.</div>
<div>
After analyzing the logs, we found that while communicating with the Eureka Server, there was a mismatch in the API interface signatures. Looked like a version mismatch between Eureka client in Zuul and the Eureka server!</div>
<div>
<br /></div>
<div>
And it was indeed. </div>
</div>
<div>
<br /></div>
<h4 style="text-align: left;">
<b>Solution:</b></h4>
<div>
In the pom.xml of individual microservices, we were using Eureka clients as:</div>
<div>
<br /></div>
<pre style="background-color: #d8d8d4;"><dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-eureka</artifactId>
<version>1.0.6.RELEASE</version>
</dependency>
</pre>
<div>
Clearly, this might lead to confusions and issues, if different microservices define different versions of Eureka client, than the one defined in Eureka server.</div>
<div>
<br /></div>
<div>
To fix this and to bring in consistency in Eureka versions in all microservices, we removed the versioning from individual pom's and introduced dependency management in the parent pom (the pom of the parent project for all microservices).</div>
<div>
<br /></div>
<pre style="background-color: #d8d8d4;"><dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>Brixton.RELEASE</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
</pre>
<div>
<div>
As mentioned above, we used Brixton release and it automatically pulls the correct version of the dependencies defined in the child poms.</div>
<div>
<br /></div>
<div>
So the pom.xml of individual microservices for Eureka client will look like the below. Notice there is no version!</div>
</div>
<br />
<pre style="background-color: #d8d8d4;"><dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-eureka</artifactId>
</dependency>
</pre>
<h3 style="text-align: left;">
<hr />
<br />Issue 2:</h3>
<div>
<div>
While inovking the Zuul APIs from the Angular JS application, the API calls were failing with CORS (Cross Origin Resource Sharing)issue as below:</div>
<div>
"No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:63342' is therefore not allowed access".</div>
</div>
<h4 style="text-align: left;">
<br /><b>Root Cause:</b></h4>
<div>
The reason for above issue is that, the angular app was running on a domain 'http://pronet-profile-web.cfapps.io/' and the Zuul App was running on a domain 'http://pronet-edge.cfapps.io/'. Notice that the sub domains are different. This was causing the CORS issue. </div>
<div>
<br /></div>
<h4 style="text-align: left;">
<b>Solution:</b></h4>
<div>
<div>
Add CORS filter in the Zuul server. Generally Spring recommends adding of an annotation "@CrossOrigin" on the spring boot microservice. However, this solution somehow did not work on the Zuul application (having @EnableZuulProxy annotation).</div>
<div>
<br /></div>
<div>
As an alternate fix, we added the the below filter to the Zuul application to enable CORS:</div>
</div>
<div>
<br /></div>
<pre style="background-color: #d8d8d4;">@Bean
public CorsFilter corsFilter() {
final UrlBasedCorsConfigurationSource urlBasedCorsConfigurationSource= new UrlBasedCorsConfigurationSource();
final CorsConfiguration corsConfig = new CorsConfiguration();
corsConfig.setAllowCredentials(true);
corsConfig.addAllowedOrigin("*");
corsConfig.addAllowedHeader("*");
corsConfig.addAllowedMethod("OPTIONS");
corsConfig.addAllowedMethod("HEAD");
corsConfig.addAllowedMethod("GET");
corsConfig.addAllowedMethod("PUT");
corsConfig.addAllowedMethod("POST");
corsConfig.addAllowedMethod("DELETE");
corsConfig.addAllowedMethod("PATCH");
urlBasedCorsConfigurationSource.registerCorsConfiguration("/**", corsConfig);
return new CorsFilter(urlBasedCorsConfigurationSource);
}
</pre>
<div>
Note that you need to add the "OPTIONS" method as well.</div>
<div>
<br /></div>
<hr />
<h3 style="text-align: left;">
Issue 3:</h3>
<div>
While inovking the Zuul APIs from the Angular JS application, the API calls were failing with the below error:</div>
<div>
<div>
The 'Access-Control-Allow-Origin' header contains multiple values 'http://localhost:63342, http://localhost:63342', but only one is allowed.</div>
</div>
<div>
<br /></div>
<div>
<b>Root Cause:</b></div>
<div>
<div>
On analyzing this, we found that the individual microservices had a "@CrossOrigin" annotation. We already have a CORS filter on the Zuul server (Fix for Issue 2 above.)</div>
<div>
Adding another filter on the microservice layer, duplicates the CORS filter and hence we were gettting that error.</div>
</div>
<div>
<br /></div>
<div>
<h4 style="text-align: left;">
<b>Solution:</b></h4>
<div>
Removing the @CrossOrigin annotation from the individual microservices solved the issue.</div>
</div>
<div>
<b><br /></b></div>
</span></div>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com8tag:blogger.com,1999:blog-1479780601142337324.post-57952083031047156872016-07-08T02:50:00.003-07:002016-07-08T08:25:51.139-07:00Microservices based Cloud Native Application - Part II<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Preview:</span></h2>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">This is the second post in the series of Microservices based application development.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The entire series could be found here:</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><div>
<a href="http://byteposts.blogspot.in/2016/06/microservices-based-cloud-native.html">Microservices based Cloud Native Application - Part I</a></div>
<div>
<br /></div>
<div>
<a href="http://byteposts.blogspot.in/2016/07/microservices-based-cloud-native.html">Microservices based Cloud Native Application - Part II</a></div>
<div>
<a href="http://byteposts.blogspot.in/2016/07/microservices-based-cloud-native_8.html"><br /></a></div>
<div>
<a href="http://byteposts.blogspot.in/2016/07/microservices-based-cloud-native_8.html">Microservices based Cloud Native Application - Part III</a></div>
</span></div>
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></h2>
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Overview:</span></h2>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Continuing from my previos post, I'm going to explain in detail, three concepts which are essential ingredients of a Microservices Architecture.</span><br />
<ol style="text-align: left;">
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Service Discovery</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">API Gateway</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Circuit Breaker</span></li>
</ol>
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Service Discovery:</span></h2>
<hr />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">In a Microservices environment, we will have multiple services and when the same is deployed in a Cloud Environment, we will have multiple instances of each service.</span></div>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">In such a scenario, we need services to be self discover-able. This will help in two ways. </span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">First, when one service invokes another service, it needs to know the actual location where it is hosted and which instance it should point to.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Second, in a cloud environment, when we add/remove instances, other services need to know about this transparently.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Using a centralized Service Discovery will help us solve this problem. Spring Cloud Netflix provides a library called 'Eureka' which will allow services to register to and discover other services.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Following are some code snippets for Eureka client and Eureka Server:</span></div>
<div>
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Eureka client: </b></span></h4>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Code:</b></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">The following annotation needs to be placed in all the microservices which would register into Eureka. (Note that I'm using Spring boot app).</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<pre style="background-color: #d8d8d4;"><span style="font-family: "arial" , "helvetica" , sans-serif;">@SpringBootApplication
@EnableAutoConfiguration
@EnableDiscoveryClient
public class AppMain extends SpringBootServletInitializer{
}
</span></pre>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Configuration:</b></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">The microservice registers into Eureka with a specific name (or serviceId). This could be configured in a file bootstrap.yml.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<pre style="background-color: #d8d8d4;"><span style="font-family: "arial" , "helvetica" , sans-serif;">bootstrap.yml:
server:
port: 8090
spring:
application:
name: profile-details</span>
</pre>
<br />
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">In order to locate the Eureka server, the client needs to know the server details. This could be configured in a file application.yml.</span></div>
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<pre style="background-color: #d8d8d4;"><span style="font-family: "arial" , "helvetica" , sans-serif;">eureka:
instance:
hostname: localhost
leaseRenewalIntervalInSeconds: 10
metadataMap:
instanceId: ${vcap.application.instance_id:${spring.application.name}:${server.port:8080}}
client:
serviceUrl:
defaultZone: ${vcap.services.eureka-service.credentials.uri:http://127.0.0.1:8761}/eureka/
</span></pre>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Eureka server:</span></h4>
</div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">This is a Spring boot app with an annotation @EnableEurekaServer. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Code:</b></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<pre style="background-color: #d8d8d4;">@SpringBootApplication
@EnableEurekaServer
public class DiscoveryServerApplication {
public static void main(String[] args) {
SpringApplication.run(DiscoveryServerApplication.class, args);
}
}
</pre>
<br />
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Configuration:</b></span></div>
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">We can configure this application to be a Eureka server in application.yml</span></div>
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<pre style="background-color: #d8d8d4;">eureka:
instance:
hostname: localhost
client:
registerWithEureka: false
fetchRegistry: false
serviceUrl:
defaultZone: http://${eureka.instance.hostname}:${server.port}/eureka/
</pre>
<br />
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">API Gateway:</span></h2>
<hr />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span><br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">A microservices based application can have multiple clients (for eg, Web, Mobile, Partner integrations etc). Note that each of the individual microservices can evolve on its own and can deploy with different versions for different clients. In such scenarios, it will be necessary to provide a centralized interface which will perform the routing and tranformation services required.</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">An API Gateway does exactly this. Spring cloud Netfix provides a library Zuul which acts as an API gateway.</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">To create an API gateway server, use the annotation @EnableZuulProxy on the spring boot app. Note that this application should also register itself with Eureka, since it has to locate other services which it would forward to.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Code:</b></span><br />
<pre style="background-color: #d8d8d4;">@SpringBootApplication
@EnableDiscoveryClient
@EnableZuulProxy
public class EdgeServerApplication {
}
</pre>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Configuration:</b></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">The following configs in application.yml will allow it to register with Eureka and add routing logic for request forwarding.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<pre style="background-color: #d8d8d4; text-align: left;">eureka:
instance:
hostname: localhost
leaseRenewalIntervalInSeconds: 10
metadataMap:
instanceId: ${vcap.application.instance_id:${spring.application.name}:${server.port:8080}}
client:
serviceUrl:
defaultZone: ${vcap.services.eureka-service.credentials.uri:http://127.0.0.1:8761}/eureka/
zuul:
routes:
profile-skills:
path: /**/skill/**
stripprefix: false
serviceId: profile-skills
profile-summary:
path: /**/summary/**
stripprefix: false
serviceId: profile-details
</pre>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Note:</b></span></div>
<br />
<ul style="text-align: left;">
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">The angular web application will point to the edge server. So all microservice invocations from the angular app goes via edge server.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">For eg, if the angular web app has to invoke profile-skills service, then it would invoke /<edge-server-host-url>/<something>/skill/<something>/.</span></li>
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">The edge server would then apply the rule as in the above configuration, and forward it to the "profile-skills" microservice (It would use the host name map obtained from Eureka to resolve the actual host url)).</span></li>
</ul>
</ul>
<ul style="text-align: left;">
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">The "stripprefix" attribute in the above configuration would retain the prefix part of the url before the routing path (like before /skill or before /summary)</span></li>
</ul>
<br />
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></h2>
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
Circuit Breaker:</span></h2>
<hr />
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">When we have a huge number of microservices (which we will, in a typical complex application), it is necessary for the services to be fault tolerant. Since it is common for services to fail in a cloud environment using commodity hardware, we need to design our services in a fault tolerant way.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Circuit Breaker is a pattern used in Microservices which will work pretty much like a Circuit breaker in electric circuits. When a service fails, it creates an open circuit, breaking the flow and to fix that temporarily, we define an alternative service implementation, which will kick in and close the circuit.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">In our use case, Profile-Details service invokes Profile-Recommendation service. When Profile-Recommendation service fails, an alternative implementation kicks in which will just return a dummy default recommendation, so that the entire flow is not broken. Once the Profile-Recommendation service is back online, normal services will resume.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">We use FeignClient (which is a Client side load balancer) along with Hysterix.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Code:</b><br />In the profile-details service, add this annotation to the spring boot app:</span><br />
<pre style="background-color: #d8d8d4;">@SpringBootApplication
@EnableDiscoveryClient
@EnableFeignClients
public class ProfileDetailsApplication {
}
</pre>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
</div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Now to invoke the Profile-Recommendation service, we need not know the host url of the Profile-Recommendation service service. We just need to know the service name (configured in bootstrap.yml of the Profile-Recommendation service). Since we use Eureka, the service will resolve the host url.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Create an interface as below to invoke an API in Profile-Recommendation service.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<pre style="background-color: #d8d8d4;"><span style="font-family: "arial" , "helvetica" , sans-serif;">@FeignClient(name = "profile-recommendation" , fallback = RecommendationClientFallback.class)
public interface RecommendationFeignClient {
@RequestMapping(method = RequestMethod.GET, value = "/api/profile/{userId}/recommendation")
List</span><profilerecommendation><span style="font-family: "arial" , "helvetica" , sans-serif;"> getRecommendations(@PathVariable("userId") String userId);
}</span>
</profilerecommendation></pre>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">The value of the name attribute is the service name which we are calling. The value of the fallback attribute is an alternative implementation, in case the actual call fails.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Invoking another microservice from one is this simple!!!</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Following is the fallback implemntation:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<pre style="background-color: #d8d8d4;">@Component
public class RecommendationClientFallback implements RecommendationFeignClient {
@Override
public List<profilerecommendation> getRecommendations(String userId) {
List<profilerecommendation> recommdList = new ArrayList<profilerecommendation>();
ProfileRecommendation recommend = new ProfileRecommendation();
recommend.setRecommendationText("This is a default recommendation!");
recommdList.add(recommend);
return recommdList;
}
}
</profilerecommendation></profilerecommendation></profilerecommendation></pre>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Thats it!! When profile-recommendation service fails, it would invoke the fallback and would return "This is a default recommendation!".</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span></div>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com0tag:blogger.com,1999:blog-1479780601142337324.post-4239281069052827992016-06-23T01:48:00.000-07:002016-07-08T08:23:31.072-07:00Microservices based Cloud Native Application - Part I<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Preview:</span></h2>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">In this post, I'm going to write about an application based on Microservices Architecture, which I presented in the Cloud Foundry Meetup. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The entire series link could be found here:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><a href="http://byteposts.blogspot.in/2016/06/microservices-based-cloud-native.html">Microservices based Cloud Native Application - Part I</a></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><a href="http://byteposts.blogspot.in/2016/07/microservices-based-cloud-native.html">Microservices based Cloud Native Application - Part II</a></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><a href="http://byteposts.blogspot.in/2016/07/microservices-based-cloud-native_8.html">Microservices based Cloud Native Application - Part III</a></span></div>
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span><span style="font-family: "arial" , "helvetica" , sans-serif;">Application Use Case:</span></h2>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The application is a miniature LinkedIn kind of application, which was built to demonstrate the key aspects of a Cloud Native application.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span><br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
Application Features:</span></h3>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The app allows users to build their profile with experience details, their skills, certifications. </span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The app allows the users to connect with other users and they can endorse skills and also recommend other users.</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<h2 style="text-align: left;">
Application Architecture:</h2>
<div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-oE6CMSBad0M/V2uhEv7BUDI/AAAAAAAAAfM/HaS3egJcQI8KCB1Nn2uZIcDdcYmtOthYwCLcB/s1600/arch.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="274" src="https://3.bp.blogspot.com/-oE6CMSBad0M/V2uhEv7BUDI/AAAAAAAAAfM/HaS3egJcQI8KCB1Nn2uZIcDdcYmtOthYwCLcB/s320/arch.png" width="320" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
The application uses Microservices Architecture, which consists of loosely coupled, independently evolvable/deployable services.</div>
<div>
It uses polyglot persistence. i.e, each service uses the right database which fits the use case. Flavors of Nosql and Sql is used in this application.</div>
<div>
<br /></div>
<div>
In addition to the microservices which implement functional features, the following Cloud Native application features are also implemented. </div>
<div>
<ol style="text-align: left;">
<li>Loosely coupled bounded context</li>
<li>Service Discovery </li>
<li>API Gateway</li>
<li>Centralized Logging</li>
<li>Fault Tolerance</li>
</ol>
<div>
These are explained in more detail in the following sections.</div>
<div>
<br /></div>
<h2 style="text-align: left;">
Microservices:</h2>
<div>
<ol style="text-align: left;">
<li><b>Profile-Basic</b>: This service would allow CRUD of a User Profile, like name, email etc. Since the data is well defined, we used a RDBMS (Mysql) for this.</li>
<li><b>Profile-Skill</b>: This service would allow the user to add new skills to his/her profile and allow other users to endorse skills. Due to the key/value nature of the data, we used a key/value store (Redis) for this</li>
<li><b>Profile-Details</b>: This service would allow CRUD of a Experience details/Certification Details of a user. Since the data is largely huge texts without restricted by a schema, we used a Document store (MongoDB) for this.</li>
<li><b>Profile-Recommendation</b>: This service would allow users to recommend other users. Since the data is largely huge texts without restricted by a schema, we used a Document store (MongoDB) for this. This is implemented as a separate service to demonstrate a key feature (Circuit Breaking). From a usecase perspective, creating a separate service would allow other services to look at the recommendations of a user to rate the profile (future usecase)</li>
<li><b>Profile-Connection</b>: This service would allow users to add other users to their network and also provides a listing of all the users present in their network. In order to maintain a networked graph of all the users, we used a Graph database (Neo4J) for this service</li>
<li><b>Discovery-Server</b>: This service would act a discovery service and would allow all other microservices to register to and discover each other. (including multiple instances of each service). This microservice is implemented by using the Netflix Eureka library.</li>
<li><b>Edge-Server</b>: This service would act like a API gateway, and can be used to provide authentication, routing and transformation of the requests. This service is implemented by using Netflix Zuul library</li>
<li><b>Profile-web</b>: This is an angular application, which consists of the actual client side implementation of the application.</li>
</ol>
</div>
<div>
<b>Continued...<a href="http://byteposts.blogspot.in/2016/07/microservices-based-cloud-native.html">See Part II</a></b></div>
</div>
</div>
</span></div>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com0tag:blogger.com,1999:blog-1479780601142337324.post-62089082731388197102016-05-12T22:04:00.000-07:002016-05-18T03:52:45.425-07:00Cloud Foundry Meetup - Developing Cloud Native Apps<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
<span style="background-color: white; color: #333333; line-height: 22.8571px;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">Developing Cloud Native Apps</span></span></h2>
<div>
<span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white; line-height: 22.8571px;">This post provides a brief about what to expect from the Cloud Foundry meetup session, '</span><span style="line-height: 22.8571px;">Developing Cloud Native Apps on Cloud foundry' </span></span><span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; line-height: 22.8571px;">which is scheduled on June 7th, Bangalore. </span></div>
<div>
<span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; line-height: 22.8571px;"><br /></span></div>
<h3 style="text-align: left;">
<span style="background-color: white; line-height: 22.8571px;"><span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;">Presenters:</span></span></h3>
<div>
<span style="background-color: white; line-height: 22.8571px;"><span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;"></span></span><br />
<div>
<span style="background-color: white; line-height: 22.8571px;"><span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></div>
<span style="background-color: white; line-height: 22.8571px;"><span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;">
</span></span>
<div>
<span style="background-color: white; line-height: 22.8571px;"><span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;">Rajagopalan Sundararajan, Senior Solutions Architect, EMC</span></span></div>
<span style="background-color: white; line-height: 22.8571px;"><span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;">
<div>
Raghuveer Bhandarkar, Solutions Architect, EMC</div>
</span></span></div>
<div>
<h4 style="text-align: left;">
<strong style="background-color: white; color: #333333; font-size: 11pt; line-height: 115%;"><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></strong></h4>
<h4 style="text-align: left;">
<strong style="background-color: white; color: #333333; font-size: 11pt; line-height: 115%;"><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Preview:</span></span></strong></h4>
</div>
<div>
<span style="background-color: white; color: #333333; font-size: 11pt; line-height: 115%;"></span><br />
<div>
<span style="background-color: white; color: #333333; font-size: 11pt; line-height: 115%;"><strong><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></strong></span></div>
<span style="background-color: white; color: #333333;">
</span>
<div>
<div style="font-size: 11pt; line-height: 115%;">
<span style="background-color: white; color: #333333;"><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-size: 11pt; line-height: 115%;"><span style="font-family: "arial" , "helvetica" , sans-serif;">With the rapid
adoption of Cloud computing by organizations, it has become increasingly
imperative to make our application architecture, cloud enabled. This talk which is a part of the Cloud Foundry Meetup, covers Cloud native Software applications, its characteristics, challenges, supported by a demonstration through an app.</span></span></span></div>
<div style="font-size: 11pt; line-height: 115%;">
<span style="background-color: white; color: #333333;"><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-size: 11pt; line-height: 115%;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></span></div>
<h4 style="text-align: left;">
<strong style="background-color: white; color: #333333; font-size: 11pt; line-height: 16.8667px;"><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Abstract:</span></span></strong></h4>
<span style="background-color: white; color: #333333;"><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-size: 11pt; line-height: 115%;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: 14.6667px; line-height: 16.8667px;">Cloud native Software and applications is a buzz word today and enterprises are trying to understand this concept and apply in building software systems which are “Cloud Native”. This talk captures the experiences of developing such cloud native systems for EMC's customers. We will explore the key characteristics of Cloud native software systems, key enablers required to develop them and the key challenges they bring along. </span></span></span><br />
<span style="background-color: white; color: #333333;"><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-size: 11pt; line-height: 115%;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: 14.6667px; line-height: 16.8667px;">It is followed by a demo of how Cloud foundry supports building Cloud Native software. We will touch very briefly on 12 factor app principles, Micro Services using Spring Boot, Spring Cloud among other topics.</span></span></span><br />
<span style="background-color: white; color: #333333;"><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-size: 11pt; line-height: 115%;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: 14.6667px; line-height: 16.8667px;"><br /></span></span>
</span><br />
<h4 style="text-align: left;">
<strong style="background-color: white; color: #333333; font-size: 11pt; line-height: 16.8667px;"><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Meetup details:</span></span></strong></h4>
<div class="separator" style="clear: both; text-align: center;">
<span style="background-color: white; color: #333333;"><a href="https://2.bp.blogspot.com/-VQQvA6sbyO0/VzxJAMqBRNI/AAAAAAAAAe8/RfS69PXkbgwlBg59aOtsVX9hFUxygGXMQCLcB/s1600/Social%2BMedia_BANNER.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://2.bp.blogspot.com/-VQQvA6sbyO0/VzxJAMqBRNI/AAAAAAAAAe8/RfS69PXkbgwlBg59aOtsVX9hFUxygGXMQCLcB/s320/Social%2BMedia_BANNER.JPG" width="311" /></a></span></div>
<span style="background-color: white; color: #333333;"><span style="color: black; font-family: "calibri" , "sans-serif"; font-size: 11.0pt;"><br /></span>
<span style="color: black; font-family: "calibri" , "sans-serif"; font-size: 11.0pt;"><br /></span>
<span style="color: black; font-family: "calibri" , "sans-serif"; font-size: 11.0pt;">REGISTER TODAY FOR THE CLOUD FOUNDRY MEGA
MEETUP! <b>Bangalore </b></span><span class="MsoHyperlink"><span style="font-family: "calibri" , "sans-serif"; font-size: 11.0pt;"><a href="http://bit.ly/1Np3acD">bit.ly/1Np3acD</a></span></span><span style="color: black; font-family: "calibri" , "sans-serif"; font-size: 11.0pt;"> <b>Pune </b></span><span style="font-family: "calibri" , "sans-serif"; font-size: 11.0pt;"><a href="http://bit.ly/1TP6hK1">bit.ly/1TP6hK1</a></span></span><br />
<span style="background-color: white; color: #333333;"><br /></span>
<span style="background-color: white; color: #333333;"><span style="color: black; font-family: "calibri" , "sans-serif"; font-size: 11.0pt;">EMC and KPIT, in association with Cloud Foundry,
present Cloud Foundry Mega Meetup 2016. Be a part of the largest Cloud Foundry
event in India – join hundreds of developers, architects and business leaders
at Cloud Foundry Mega Meetup to help your company deliver apps faster than ever
before. Attend the event to catch industry experts present on topics that are
increasingly relevant in today’s scenario and define your organization’s
multi-cloud strategies.</span></span><br />
<span style="background-color: white; color: #333333;"><span style="color: black; font-family: "calibri" , "sans-serif"; font-size: 11.0pt;"><br /></span></span></div>
<span style="background-color: white; color: #333333;">
</span></div>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com12tag:blogger.com,1999:blog-1479780601142337324.post-9112334268713541902016-04-28T08:04:00.004-07:002016-04-28T20:15:23.164-07:00#GIDS16..The Great Indian Developer Summit, Bengaluru<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">#GIDS16</span></h2>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The Great Indian Developer Summit 2016, is currently under way in Bangalore, India. In this article, I'm briefly touching upon various topics discussed and my first hand experience of the event.</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">There were talks on what are the current industry trends what are the emerging technologies. It is no more sufficient to have the knowledge of one Stack/knowledge, but need to have knowledge of operations (read Devops), Cloud and working with huge data sets (read Big Data).</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">On a lighter note, the classical "Blue Pill or Green Pill" dilemna from 'The Matrix' made its appearance in a couple of presentations.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">A few talks are detailed out below:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">1. Microservice, Microservice, Microservice:</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">There were various talks on Microservice, ranging from how to go about building Microservice, to, why do we need Microservice. One of the interesting talk, went about explaining that Microservice is not a silver bullet, which should be applied to every project, but chosen wisely in order to be agile.</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">There was also a demo on Microservice implementation using Kubernetos, Docker on Open Shift platform. Various Netflix OSS components like Hysterix was used for Circuit Breakers and Load balancing.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">2. Vert.x 3.0:</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">An amazing ecosystem on JVM which supports building reactive, asynchronous applications using langauges like Scala, Groovy, Javascript etc. Scott Davis demonstrated a hands on example of how we can build a http server quickly and deploy on multiple server instances, all of which would communicate to each other using UDP protocol and would act as a cluster. </span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">One statement which summed all of that, was "Vert.X is the Docker for JVM".</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Man, believe me, amazingly cool stuff!</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Vert.x is going to be the next big thing!!!.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">3. Are we computer scientists? Nah...we are story tellers:</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Scott gave an amazing presentation on how we, engineers are more like story tellers, not being just source code producing machine...but rather one who build applications. He also stressed on the need to write Unit tests to validate what we code, drawing analogy to scientists who validate their theory. Spot on!</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">An interesting point was made about how a user story in Agile (the good old, "As a ... I want ... so that ..."), does not describe what is needed most of the time. He made a case to rather use Hypothesis instead of User Story.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">4. Java is NOT DEAD! It's Maturing!!!</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Bob McWhirter's presentation was received with thunderous applause and cheers from the audience, when he went to say that, Java is not dead, but is getting wiser and mature with age. The enhancements in Java 8, Java EE 7, Vert.x and Groovy/Scala hold much promise for Java and he backed it up with statistics to indicate the trend that more and more applications were getting written in Java. Even if the application is not written Java as a langauge itself, JVM will be in the picture somewhere.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Long live Java!!!</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">5. Groovy:</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Paul King introduced Groovy with all its features including Domain Specific langauge (DSL),Closures, functional programming aspects of Groovy. He also explained on how Groovy eliminates the pain points of Java and is piggy backing on the strength of Java. </span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">6. Lean Engineering:</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Bill Scott shared his experience of how bringing the Product engineering development team and the designers (and customers sometimes)together in a room, facilitating collaboration and experimentation lead to building world class product features at Paypal.</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">7. Mobile backend as a Service (MBAAS):</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Mike Keith explained the various architectural components required to build Mobile serving backend system. It included how the ecosystem would interact with legacy enterprise applications, also touching upon, security aspects like OAuth/SAML. Various API management tools like APIGEE were introduced. The challenges of identity management in the wake of different business scenarios like 'Business to Customer (B2C)' or 'Business to Employee' (B2E) were presented.</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">8. Angular 2, Java Lambda expressions:</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">There were talks on how Angular 2 is radically different from Angular 1 with introductions to Web Socket programming. Java Lambda expressions and Streams with a well drafted lessons learned from using these technologies was presented.</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Overall, it was very exciting and the next couple of years appear to be the most turbulent years in technology, with a huge influx of stacks.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">After all, Albert Einstein could not be more apt when he said "Once you stop learning, you start dying".</span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com1tag:blogger.com,1999:blog-1479780601142337324.post-44979691662514900902016-01-18T02:46:00.000-08:002016-01-18T02:49:04.427-08:00Using Maven Dependency Tree in troubleshooting<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "arial" , "helvetica" , sans-serif;">In this blog post, I'm going to explain how to check the maven dependency tree and how it is useful in troubleshooting certain run time exception which we might encounter while running Spring applications.</span><br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></h3>
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Maven Dependency </span><span style="font-family: "arial" , "helvetica" , sans-serif;">resolution:</span></h2>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: arial, helvetica, sans-serif;">When you use maven, it uses its own dependency resolution mechanism to decide which jar to use when there is a conflict.</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Lets say you are using two dependencies, each of which have the same jar. Now, unless you pay attention, maven might end up using the incorrect jar version, and you will start getting exceptions like 'ClassNotFound' or 'NoSuchMethodError'.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">So, it helps to know which version of jar has maven resolved and is added to our application. This is where the Maven dependency tree helps us.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Maven Dependency Tree:</span></h2>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">A simple command, "mvn dependency:tree -Dverbose " will print out the entire maven dependency tree.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">I will show with an example how to analyze this tree. Since is based on a issue I recently faced, I hope it will serve best to understand it better.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Practical Example:</span></h2>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Lets say we have a maven project for a very basic java spring project for illustration purpose.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Lets use power Mockito and Mockito core for unit test cases and the dependency is:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><dependency></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> <groupId>org.powermock</groupId></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> <artifactId>powermock-module-junit4</artifactId></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> <version>1.5.6</version></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> <scope>test</scope></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> </dependency></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> <dependency></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> <groupId>org.powermock</groupId></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> <artifactId>powermock-api-mockito</artifactId></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> <version>1.5.6</version></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> <scope>test</scope></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> </dependency></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Also add a dependency on Mockito core:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><dependency></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> <groupId>org.mockito</groupId></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> <artifactId>mockito-core</artifactId></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> <version>1.9.5</version></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> <scope>test</scope></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"></dependency></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Add a dependency for spring mockito annotations:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><dependency></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span><groupId>org.kubek2k</groupId></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span><artifactId>springockito-annotations</artifactId></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span><version>${springockito-annotations-version}</version></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span><scope>test</scope></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"></dependency></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Create a Junit Test case as below:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">@RunWith(PowerMockRunner.class)</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">public class MyLogicTest{</span><br />
<span class="Apple-tab-span" style="white-space: pre;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span>@Mock</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span>private MyService service;</span><br />
<span class="Apple-tab-span" style="white-space: pre;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span>@Test</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span>public void testMyMethod(){</span><br />
<span class="Apple-tab-span" style="white-space: pre;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">}</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Run the test case. You would get an error:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">java.lang.NoSuchMethodError: org.mockito.internal.creation.MockSettingsImpl.setMockName(Lorg/mockito/mock/MockName;)Lorg/mockito/internal/creation/settings/CreationSettings;</span><br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">You might have encountered similar NoSuchMethodError or ClassNotFound exceptions while running Spring applications. Most probably, those errors would have occurred because, the same jar (with different versions) would have been present in different dependencies. This might cause Spring to pick an incorrect version of the jar and thereby causing run time exceptions.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">What is the actual issue here?</span></h2>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">power-mockito 1.9.5 is declared in pom, but somehow older version is being used and causing problem. We need to find out which dependency is the culprit and causing this issue.</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<h2 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">How to resolve this issue?</span></h2>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span><br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Run the below command which would display the tree structure of dependencies which maven has resolved.</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<div>
<br /></div>
<div>
mvn dependency:tree -Dverbose </div>
<div>
<br /></div>
<div>
It would show a tree structure with parent and child nodes.</div>
<div>
Search for "org.powermock" and you would notice the below:</div>
<div>
<br /></div>
<div>
<div>
[INFO] +- org.powermock:powermock-api-mockito:jar:1.5.6:test</div>
<div>
[INFO] | +- (org.mockito:mockito-all:jar:1.9.5:test - omitted for conflict with 1.9.0)</div>
</div>
<div>
<br /></div>
<div>
<div>
It is clearly saying it is using 1.9.0 instead of 1.9.5</div>
<div>
Now, we need to find out which dependency is using 1.9.0. So search for 1.9.0 in the above output. You would find:</div>
<div>
<br /></div>
<div>
[INFO] +- org.kubek2k:springockito-annotations:jar:1.0.9:test</div>
<div>
[INFO] | \- org.mockito:mockito-all:jar:1.9.0:test</div>
</div>
<div>
<br /></div>
<div>
So it is clear that "org.kubek2k:springockito" is the culprit and it is adding the older version of power-mockito. To fix this issue, we need to tell maven to ignore or exclude "power-mockito" jar from "org.kubek2k:springockito".</div>
<div>
<br /></div>
<h2 style="text-align: left;">
How to instruct Maven to use the right jar version?</h2>
<div>
<br /></div>
<div>
Use exclusions!!</div>
<div>
<div>
So go to the pom and modify to add a exclusions element as below:</div>
<div>
<br /></div>
<div>
<dependency></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span><groupId>org.kubek2k</groupId></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span><artifactId>springockito-annotations</artifactId></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span><version>${springockito-annotations-version}</version></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span><scope>test</scope></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span><exclusions></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span><exclusion></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span> <groupId>org.mockito</groupId> </div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span> <artifactId>mockito-all</artifactId></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span></exclusion></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span></exclusions></div>
<div>
</dependency></div>
</div>
<div>
<br /></div>
<div>
That's it!!. This should resolve the run time issue and the test case should start working. I hope this post was helpful in understanding the significance of Maven Dependency trees.</div>
</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<br /></div>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com4tag:blogger.com,1999:blog-1479780601142337324.post-73382230781432613512015-12-11T00:13:00.001-08:002016-06-23T01:50:13.173-07:00Microservices using Pivotal Cloud Foundry<div dir="ltr" style="text-align: left;" trbidi="on">
<div pid="2" style="background-color: white; box-sizing: border-box; color: #262626; font-family: Georgia, serif; font-size: 19px; line-height: 22.04px; margin-bottom: 15px; margin-top: 5px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Have you ever faced the following issues in your projects:</span></div>
<div pid="3" style="background-color: white; box-sizing: border-box; color: #262626; font-family: Georgia, serif; font-size: 19px; line-height: 22.04px; margin-bottom: 15px; margin-top: 5px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">1. Roll out features quickly to production:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">The customer needs very minor changes to functionality/UI and would like that to be rolled out at the earliest, because they have a higher business impact. However, we are not able to deliver it, because though it is a small change, we have to go through the whole nine yards of release processes involving full regression etc. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br style="box-sizing: border-box;" /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">2. Move your application from one datacenter to another:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">You need to move your application to a different data center in a different geography due to various reasons (like legal, merger and acquisition etc). This might seem to be a herculean task to you. More often than not, you would get caught in a web of discovering (often surprising yourself!), a myriad of system configurations spread across file system, database, even worse within the code!!!. This is only going to make your task even more daunting.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br style="box-sizing: border-box;" /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">3. Scale out your application based on modules:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">You might observe that few modules of your application face a huge load compared to others, and you might want to dedicate more resources to these modules instead of scaling out the entire application. However, you are not able to do this at a module level, thus wasting money/resources by scaling the entire application.</span></div>
<div pid="4" style="background-color: white; box-sizing: border-box; color: #262626; font-family: Georgia, serif; font-size: 19px; line-height: 22.04px; margin-bottom: 15px; margin-top: 5px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Is there a better way out???</span></div>
<div pid="5" style="background-color: white; box-sizing: border-box; color: #262626; font-family: Georgia, serif; font-size: 19px; line-height: 22.04px; margin-bottom: 15px; margin-top: 5px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Think <strong style="box-sizing: border-box;">Micro Services</strong>!!</span></div>
<div pid="6" style="background-color: white; box-sizing: border-box; color: #262626; font-family: Georgia, serif; font-size: 19px; line-height: 22.04px; margin-bottom: 15px; margin-top: 5px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><strong style="box-sizing: border-box;">Micro Services</strong> help us break down the application which was a monolith hitherto, into smaller deployable units. So no more, one huge .ear file having large number of war files. Typcially, they are built as stateless REST services providing a certain modular functionality. And each of these services could be built and deployed individually.</span></div>
<div pid="7" style="background-color: white; box-sizing: border-box; color: #262626; font-family: Georgia, serif; font-size: 19px; line-height: 22.04px; margin-bottom: 15px; margin-top: 5px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">This loosely coupled architecture would allow us to seamlessly develop and deploy individual micro services quickly and be truly agile.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">So Micro Services will try to address issues #1 and #3 above.</span></div>
<div pid="8" style="background-color: white; box-sizing: border-box; color: #262626; font-family: Georgia, serif; font-size: 19px; line-height: 22.04px; margin-bottom: 15px; margin-top: 5px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">So, will just breaking down into smaller Micro Services serve as a magic bullet and solve all the problems?</span></div>
<div pid="8" style="background-color: white; box-sizing: border-box; color: #262626; font-family: Georgia, serif; font-size: 19px; line-height: 22.04px; margin-bottom: 15px; margin-top: 5px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Well, not really. It might introduce complications and more problems, unless we manage it better.</span></div>
<div pid="9" style="background-color: white; box-sizing: border-box; color: #262626; font-family: Georgia, serif; font-size: 19px; line-height: 22.04px; margin-bottom: 15px; margin-top: 5px;">
<h3 style="text-align: left;">
<strong style="box-sizing: border-box;"><span style="font-family: "arial" , "helvetica" , sans-serif;">How to manage micro services?</span></strong></h3>
</div>
<div pid="10" style="background-color: white; box-sizing: border-box; color: #262626; font-family: Georgia, serif; font-size: 19px; line-height: 22.04px; margin-bottom: 15px; margin-top: 5px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Use of a PAAS (Platform As a Service) would help us to manage the Micro Services environment better. In this article, I would dwelve upon Pivotal Cloud Foundry (PCF), which is a PAAS offering from Pivotal Inc. which could be deployed on top of any IAAS offerings like VMWare, AWS etc.</span></div>
<div pid="11" style="background-color: white; box-sizing: border-box; color: #262626; font-family: Georgia, serif; font-size: 19px; line-height: 22.04px; margin-bottom: 15px; margin-top: 5px;">
<h3 style="text-align: left;">
<strong style="box-sizing: border-box;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Why PCF?</span></strong></h3>
<span style="font-family: "arial" , "helvetica" , sans-serif;">1. PCF supports a large number of run times including Java, Python, PHP etc. So there is no vendor lock-in and you can pretty much develop using the language of your choice (and which is supported by PCF).</span></div>
<div pid="11" style="background-color: white; box-sizing: border-box; color: #262626; font-family: Georgia, serif; font-size: 19px; line-height: 22.04px; margin-bottom: 15px; margin-top: 5px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br style="box-sizing: border-box;" /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">2. One of the principles to be followed here is to isolate configuration from code. The same code/deployable artifact (like .war for example) should work in all environments without change.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">(This principle would address issue #2 mentioned earlier in the article). PCF provides a configuration server which would serve as a centralized server providing all the system configurations, thereby allowing the platform to spawn new instances of the application seamlessly.</span></div>
<div pid="11" style="background-color: white; box-sizing: border-box; color: #262626; font-family: Georgia, serif; font-size: 19px; line-height: 22.04px; margin-bottom: 15px; margin-top: 5px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br style="box-sizing: border-box;" /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">3. PCF provides a wide range of services through marketplace. It includes (no sql databases like Mongo DB are provided a service), messaging system (like Rabbit MQ) etc. These services ease the development and deployment/configuration efforts. Even more, these services could be bound to our application with a single command. Look ma, no hands!!</span></div>
<div pid="12" style="background-color: white; box-sizing: border-box; color: #262626; font-family: Georgia, serif; font-size: 19px; line-height: 22.04px; margin-bottom: 15px; margin-top: 5px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Overall, PCF appears to be a suitable solution for quickly building and deploying microservice applications onto the cloud.</span></div>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com0tag:blogger.com,1999:blog-1479780601142337324.post-15247440280058978502015-03-22T07:37:00.001-07:002015-03-22T07:37:49.643-07:00Map Reduce Explained<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Arial, Helvetica, sans-serif;">In this post, I'm going to write about Map Reduce Algorithm. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">Map Reduce has taken the computing world by storm in recent years. It's massive scalability has given an edge to solving complicated problems.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Map Reduce is often explained by the famous counting example (word count more often than not). I would also take the same route, albeit, a different example., let us count the number of books with the same name.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Example:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">Suppose, there are 3 book shelves and each of them have different books.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-Y6yC4x-Af0s/VQ7QNpdTSUI/AAAAAAAAAcM/b2KAi6KbWAY/s1600/book%2Bshelves.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-Y6yC4x-Af0s/VQ7QNpdTSUI/AAAAAAAAAcM/b2KAi6KbWAY/s1600/book%2Bshelves.png" height="131" width="320" /></a></div>
<span style="font-family: Arial, Helvetica, sans-serif;">For brevity, let us say there are 4 different books in each of these shelves.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">First Shelf:</span></h3>
<span style="font-family: Arial, Helvetica, sans-serif;">"Harry Potter and the Half-Blood Prince" (count: 3)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">"The Da Vinci Code" (count: 1)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">"Think and Grow Rich" (count: 2)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">"The Bridges of Madison County" (count: 2)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Second Shelf:</span></h3>
<span style="font-family: Arial, Helvetica, sans-serif;">"Harry Potter and the Half-Blood Prince" (count: 2)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">"The Da Vinci Code" (count: 2)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">"Think and Grow Rich" (count: 3)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">"The Bridges of Madison County" (count: 1)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Third Shelf:</span></h3>
<span style="font-family: Arial, Helvetica, sans-serif;">"Harry Potter and the Half-Blood Prince" (count: 1)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">"The Da Vinci Code" (count: 4)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">"Think and Grow Rich" (count: 2)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">"The Bridges of Madison County" (count: 1)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Map function:</span></h3>
<span style="font-family: Arial, Helvetica, sans-serif;">The map function will create a map with a key and value. In our case, since we are counting books of each name, we can have the book name as the key. Let us put a value of 1 for each book.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Following is the pseudo code for the map function:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">function map(books){</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"> Loop thru all books</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> emit book.name as key, 1 as value.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">}</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Let us apply the map function for the first shelf. </span><span style="font-family: Arial, Helvetica, sans-serif;">Note that since there are 3 "Harry Potter and the Half-Blood Prince" books, there will be 3 key/value pairs for this book each with value 1. The mapper will not (in this example), try to aggregate the counts.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Similarly, a mapper is associated with all the three shelves, and similar key/value pairs are generated for each shelf.</span><span style="font-family: Arial, Helvetica, sans-serif;">The key/value pairs as in the following diagram will be generated.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-sGMq--shIog/VQ7RQ9KRNnI/AAAAAAAAAcU/PsH9oljp2_w/s1600/final%2Bmaps.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-sGMq--shIog/VQ7RQ9KRNnI/AAAAAAAAAcU/PsH9oljp2_w/s1600/final%2Bmaps.png" height="131" width="320" /></a></div>
<br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Shuffling:</span></h3>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">In this step, all the map entries with the same key will be assigned to one reducer. For eg, all "Harry Potter and the Half-Blood Prince" key/value entries will be assigned to one reducer.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Reduce:</span></h3>
<span style="font-family: Arial, Helvetica, sans-serif;"></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">A reduce function will receive a key and a list of aggregated values as input. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">For the first shelf, it will receive "Harry Potter and the Half-Blood Prince" as key input and a list of [1,1,1] as value input. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">function reduce(name as key, List of values){</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Loop thru each value in List of values</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span>sum += value</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span>End Loop</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span>emit name, sum</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">}</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Following diagram shows the 3 Reducers in action:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-6NYARiq8fi8/VQ7SQQ8qEtI/AAAAAAAAAcc/Qjo4DxrwVRo/s1600/final%2Breducers.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-6NYARiq8fi8/VQ7SQQ8qEtI/AAAAAAAAAcc/Qjo4DxrwVRo/s1600/final%2Breducers.png" height="126" width="320" /></a></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">So, we get the book name "Harry Potter and the Half-Blood Prince" and the corresponding count 3 as value.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Overall Architecture:</span></h3>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-BdCTEtdK7Ok/VQ7S73t1iXI/AAAAAAAAAck/8SyNfCjn3ZM/s1600/final%2Barch.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-BdCTEtdK7Ok/VQ7S73t1iXI/AAAAAAAAAck/8SyNfCjn3ZM/s1600/final%2Barch.png" height="126" width="320" /></a></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><br />Depending on requirement, the system can have multiple mappers and reducers, using a master/slave architecture, with the master deciding on shuffling and providing data to each reducer.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">There are different frameworks which provide Map/Reduce programming model, Hadoop being one of them.</span></div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com0tag:blogger.com,1999:blog-1479780601142337324.post-81320029061173186372015-01-31T09:02:00.000-08:002015-02-01T06:05:40.194-08:00Spring Integration - Bulk processing Example<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Arial, Helvetica, sans-serif;">In this post, I'm going to share my experience in using Spring Integration for a bulk processing task. This was the first time I was using Spring Integration and it did not disappoint me. It was pretty robust in terms of error handling and definitely scalable.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Disclaimer:</b></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">The actual task being performed has been modified into a fictitious e – commerce domain task, since I cannot reveal the original task which is customer specific and confidential. However, the nature of the task in terms of processing of clob and xml remain the same and the complexity of the original task has been retained.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Also, the views posted on this post are strictly personal.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<h2 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Objective:</span></h2>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">The objective is to work upon customer information from an ecommerce application and process it and save it into database as clob data.</span></div>
<span style="font-family: Arial, Helvetica, sans-serif;"></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">The task is to perform data processing by reading xml from a file in the file system and then store it as clob into a table.</span><br />
<div>
<br /></div>
<div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Following is the high level of tasks required to be performed:</span></div>
</div>
<div>
<ol style="text-align: left;">
<li><span style="font-family: Arial, Helvetica, sans-serif;">There will be one xml file for each customer and the file would be prefixed with the corresponding customer Id.</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">The xml has customer’s personal information. The file would be named as customerID_personal.xml. For eg, 1000_personal.xml</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Read personal info file (say with name 1000_ personal.xml) from file system.</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Perform some transformation on the xml.</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Perform some more processing on the resulting xml and then save the xml into a database table as a clob data against the customerID.</span></li>
</ol>
<h2 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Constraints:</span></h2>
</div>
<div>
<ol style="text-align: left;">
<li><span style="font-family: Arial, Helvetica, sans-serif;">The volume of files is high running close to more than half a million.</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Need track of which file process succeeded and which one errored out, with reliable/robust error handling.</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">This is not a real time processing application. i.e, files are dumped into the file system by a separate system and the processing can happen offline.</span></li>
</ol>
<h2 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Design Approach:</span></h2>
</div>
<div>
<ol style="text-align: left;">
<li><span style="font-family: Arial, Helvetica, sans-serif;">The customerIDs are initally saved into a table (say ALL_CUSTOMERS) with a status column (STATUS_ID), with values enumerated for NOT_STARTED, IN PROGRESS, ERROR, COMPLETE.</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Instead of polling the files in the file system, we will poll the ALL_CUSTOMERS table to get the records which are in NOT_STARTED status and process them.</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">When we start processing each record, we will update it to IN PROGRESS.</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">If there is any error during processing, we will update it to ERROR.</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">If there is no error during processing, we will update it to COMPLETE.</span></li>
</ol>
</div>
<h2 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Use of Spring Integration to solve this problem:</span></h2>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Following diagram shows the process flow. </span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-d5cLxuVf3-M/VM0JjOmgh6I/AAAAAAAAAbg/9dFCQhWqdgM/s1600/Process.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-d5cLxuVf3-M/VM0JjOmgh6I/AAAAAAAAAbg/9dFCQhWqdgM/s1600/Process.png" height="320" width="305" /></a></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Step 1 - Polling records from database:</span></h3>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Goal is to poll records which are in NOT_STARTED status and place them on a channel</span></div>
<div>
<ol style="text-align: left;">
<li><span style="font-family: Arial, Helvetica, sans-serif;">Create an inbound channel adapter which polls database at regular intervals. The inbound adapter could be a jdbc adapter. Also, we can associate a poller to the adapter to poll the database at regular intervals.</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">The JDBC adapter:</span></li>
<ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">has as a query attribute, where we can specify the query to be run when the poller wakes up.</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Has a rowmapper attribute, where we can specify a POJO, which represents each record retrieved from the database.</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Has a “update” attribute, where we can specify a query which will be run after polling each record. We will use this to update the status of the polled record to “IN PROGRESS”</span></li>
</ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">The inbound channel adapter connects to a channel, “dbInputChannel” where it places the data retrieved from database.</span></li>
</ol>
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Step 2 – Arrange for parallel processing of each of these records</span></h3>
</div>
<span style="font-family: Arial, Helvetica, sans-serif;">Connect the dbInputChannel to a record splitter, which will split each of the records and put it in another channel “storageDataFileReadChannel”. So, here we are creating one thread per record read from the database and then these records are processed in parallel. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Step 3 – Use of Headers</span></h3>
<span style="font-family: Arial, Helvetica, sans-serif;">Place each of the record read as a header data (which is like session context), so that the CustomerID is available for all channels in the workflow till the end.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Step 4 – Read Personal Info file from file system.</span></h3>
<span style="font-family: Arial, Helvetica, sans-serif;">Create a service activator (which is essentially a Java service class), which reads data from a channel, invokes the service method and then places data on a output channel.</span><br />
<br />
<ul style="text-align: left;">
<li><span style="font-family: Arial, Helvetica, sans-serif;">Input channel : richStorageDataFileReadChannel</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Service method: fileReadAdapter.readFile</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Output channel: richStorageDataOutputChannel</span></li>
</ul>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Step 5 – Perform XSLT transfomation on the personal info data</span></h3>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Create a Transformer which will act on the personal info data, apply XSLT transformation and place the output on an output channel.</span></div>
<div>
<ul style="text-align: left;">
<li><span style="font-family: Arial, Helvetica, sans-serif;">Input channel : transactionSourceChannel</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">XSLT :xsl/customTransformation.xsl</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Output channel: transactionOutputChannel</span></li>
</ul>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Step 6 – Perform processing logic on the transformed data</span></h3>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Create a service activator (which is essentially a Java service class), which reads data from a channel, invokes the service method and then places data on an output channel. This service method will also save the data back into database and will also mark the status of the record in ALL_CUSOMERS table as COMPLETE.</span></div>
<div>
<ul style="text-align: left;">
<li><span style="font-family: Arial, Helvetica, sans-serif;">Input channel : transactionOutputChannel</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Service method: transactionService.processData</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Output channel: processedTransactionChannel</span></li>
</ul>
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Error handling:</span></h3>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">A global error handler is used which will set the record status to “ERROR”. This error handler would be invoked if there is any error at any step in the entire workflow.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<h2 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Results, Performance:</span></h2>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Overall, the workflow worked like a charm. There were some errors encountered occasionally, sometimes due to invalid xml in the file system mainly due to presence of special characters. </span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">However, the error handling mechanism took care of it, the record was marked as error and importantly, the process didn’t stop there. </span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"></span><br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">
<div>
The speed was good too, with a processing rate of 120 records in 5 minutes (on a not so high end processor and the original task involved two different file reads per record). Note that I had set the timer to poll the database every 5 minutes. It could process 120 records in parallel, before it could start the next poll.</div>
<div>
<br /></div>
<h2 style="text-align: left;">
Conclusion:</h2>
<div>
I would recommend to use Spring Integration because of its ease of use and robustness. It is well documented too and is a good fit for bulk processing.</div>
<div>
<br /></div>
</span></div>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com10tag:blogger.com,1999:blog-1479780601142337324.post-40230097448199186152014-12-31T08:41:00.000-08:002014-12-31T08:41:30.791-08:00Spring Integration - Introduction<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Arial, Helvetica, sans-serif;">In this blog post and the subsequent post, I'm going to write about Spring Integration and also share my experience in using Spring Integration in a live project.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">To begin with, I will provide a brief introduction to Spring Integration framework and in the next post I will show how to apply it.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Much of the content of this post is taken from the Spring Integration official website <a href="http://docs.spring.io/spring-integration/reference/html">Spring Integration Reference</a></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<h2 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">What is Spring Integration?</span></h2>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Spring </span><span style="font-family: Arial, Helvetica, sans-serif;">Integration</span><span style="font-family: Arial, Helvetica, sans-serif;"> </span><span style="font-family: Arial, Helvetica, sans-serif;">is a Spring driven framework which supports message driven architecture and provides routing, transformation of messages and implements the common Enterprise Integration patterns, more importantly the "Pipes and Filters" pattern.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<h2 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">What is Pipes and Filter Pattern?</span></h2>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">This is an Enterprise Integration pattern, which is applied, if multiple processing units are required to process a message and we need to increase reuse and also achieve decoupling.</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-jae6KR0KK_k/VKQiKECIkOI/AAAAAAAAAaY/XiZEGJQ1FbQ/s1600/pipes%2Band%2Bfilter.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-jae6KR0KK_k/VKQiKECIkOI/AAAAAAAAAaY/XiZEGJQ1FbQ/s1600/pipes%2Band%2Bfilter.jpg" height="98" width="320" /></a></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">The "filters" represent any component that is capable of producing and/or consuming messages, and the "pipes" transport the messages between filters so that the components themselves remain loosely-coupled.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<h2 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Components of Spring Integration Framework:</span></h2>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">There are 3 main components (as in Pipes and Filter pattern):</span></div>
<div>
<ol style="text-align: left;">
<li><span style="font-family: Arial, Helvetica, sans-serif;">Message </span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Message Channel</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Message Endpoint</span></li>
</ol>
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Message:</span></h3>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-TsUjT9KqbYc/VKQlDbhEfLI/AAAAAAAAAak/jXMgzgSnVZM/s1600/message.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-TsUjT9KqbYc/VKQlDbhEfLI/AAAAAAAAAak/jXMgzgSnVZM/s1600/message.jpg" /></a></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Generic wrapper for any Java object combined with metadata used by the framework while handling that object. In the header, any arbitrary key/value could be stored. In other words, any POJO could be used as a Message.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Message Channel:</span></h3>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-ExJo7LyzpGA/VKQlJc0U3wI/AAAAAAAAAas/FFXcSTO5djo/s1600/channel.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-ExJo7LyzpGA/VKQlJc0U3wI/AAAAAAAAAas/FFXcSTO5djo/s1600/channel.jpg" height="50" width="320" /></a></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">A Message Channel represents the "pipe" of a pipes-and-filters architecture. Producers send Messages to a channel, and consumers receive Messages from a channel. The Message Channel therefore decouples the messaging components, and also provides a convenient point for interception and monitoring of Messages.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Types of channels:</span></div>
<div>
<ol style="text-align: left;">
<li><span style="font-family: Arial, Helvetica, sans-serif;">Point to Point: At most one consumer can receive each Message</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Publish/Subscribe: Broadcast each Message to all of its subscribers</span></li>
</ol>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Another category of channels:</span></div>
</div>
<div>
<ol style="text-align: left;">
<li><span style="font-family: Arial, Helvetica, sans-serif;">Pollable Channels: Are capable of buffering Messages within a queue</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">SubscribableChannel: Iimplemented by channels that send Messages directly to their subscribed MessageHandlers</span></li>
</ol>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">There are various implementations available for each of the above channel types. For eg, QueueChannel, DirectChannel etc. </span></div>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<h2 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Message Endpoints:</span></h2>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">A Message Endpoint represents the "filter" of a pipes-and-filters architecture. This is the processing unit which acts on the message.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Types of endpoint:</span></h3>
<div>
<ol style="text-align: left;">
<li><span style="font-family: Arial, Helvetica, sans-serif;">Transformer: Responsible for converting a Message's content or structure and returning the modified Message</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Filter: Determines whether a Message should be passed to an output channel or not</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Router: Responsible for deciding what channel or channels should receive the Message next (if any).</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Splitter: Responsible for splitting a message into multiple messages and then send each of those to its output channel.</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Service Activator: A generic end point. We can configure a service provider and a service method with custom logic.</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Aggregator: Combines multiple messages into one (opposite of Splitter).</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Channel Adapter: Connects a Message Channel to some other system or transport.</span></li>
</ol>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Types of Channel Adapter:</span></div>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-wudpKmuU_zU/VKQlvUmQiPI/AAAAAAAAAa0/xhO3FDYAuP0/s1600/inbound%2Badapter.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-wudpKmuU_zU/VKQlvUmQiPI/AAAAAAAAAa0/xhO3FDYAuP0/s1600/inbound%2Badapter.jpg" height="54" width="320" /></a><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-ULXQ8Xtizgg/VKQlzlXNTQI/AAAAAAAAAa8/uyH0xe12NLE/s1600/outbound%2Badapter.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-ULXQ8Xtizgg/VKQlzlXNTQI/AAAAAAAAAa8/uyH0xe12NLE/s1600/outbound%2Badapter.jpg" height="54" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div>
<ol style="text-align: left;">
<li><span style="font-family: Arial, Helvetica, sans-serif;">Inbound adapter: An inbound "Channel Adapter" endpoint connects a source system to a MessageChannel.</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Outbound adapter: An outbound "Channel Adapter" endpoint connects a MessageChannel to a target system.</span></li>
</ol>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com1tag:blogger.com,1999:blog-1479780601142337324.post-89649727024656565202014-12-27T07:51:00.001-08:002014-12-27T07:56:17.521-08:00HTML5 Canvas - Basics<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Arial, Helvetica, sans-serif;">In this post we will take a look at the HTML5 Canvas element and how to use it.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<h2 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">What is canvas?</span></h2>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">HTML5 canvas element provides platform to draw stuff on it and also animate them.</span></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<h2 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">How to create a Canvas?</span></h2>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Canvas is a html5 element, which is created as below:</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"></span>
<span style="font-family: Arial, Helvetica, sans-serif;"></span>
<br />
<pre><span style="font-family: Arial, Helvetica, sans-serif;"><canvas id="mycanvas" width="578" height="200"></canvas></span></pre>
<span style="font-family: Arial, Helvetica, sans-serif;"></span></div>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<h2 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">How to get a handle to the canvas?</span></h2>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">In order to draw shapes or write text into the canvas, we need to get a handle to the canvas object. This could be done in javascript as below:</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"></span><br />
<div>
<pre><span style="font-family: Arial, Helvetica, sans-serif;"><script></span>
<span style="font-family: Arial, Helvetica, sans-serif;"> var canvas = document.getElementById('myCanvas');</span>
<span style="font-family: Arial, Helvetica, sans-serif;"> var context = canvas.getContext('2d');</span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> //TODO</span>
<span style="font-family: Arial, Helvetica, sans-serif;"></script></span></pre>
</div>
<span style="font-family: Arial, Helvetica, sans-serif;"></span></div>
</div>
<div>
<br /></div>
<h3 style="text-align: left;">
Entire html code:</h3>
<pre><span style="font-family: Arial, Helvetica, sans-serif;"><!DOCTYPE HTML></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><html></span><span style="font-family: Arial, Helvetica, sans-serif;"> </span>
<span style="font-family: Arial, Helvetica, sans-serif;"> <body></span>
<span style="font-family: Arial, Helvetica, sans-serif;"> <canvas id="myCanvas" width="578" height="250"></canvas></span>
<span style="font-family: Arial, Helvetica, sans-serif;"> <script></span>
<span style="font-family: Arial, Helvetica, sans-serif;"> var canvas = document.getElementById('myCanvas');</span>
<span style="font-family: Arial, Helvetica, sans-serif;"> var context = canvas.getContext('2d');</span>
<span style="font-family: Arial, Helvetica, sans-serif;"> //TODO</span><span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> </span>
<span style="font-family: Arial, Helvetica, sans-serif;"> </script></span>
<span style="font-family: Arial, Helvetica, sans-serif;"> </body></span>
<span style="font-family: Arial, Helvetica, sans-serif;"></html> </span>
<span style="font-family: Arial, Helvetica, sans-serif;">The TODO line above is where we start writing javascript code to draw shapes etc and play around with the canvas. The code in the following sections are intended to be placed after the TODO.</span>
</pre>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<h2 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">How to draw a line?</span></h2>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"></span><br />
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-AdtMEFhSLXA/VJ7UnZ46EuI/AAAAAAAAAZw/T0ZUxvhzJ9o/s1600/line.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-AdtMEFhSLXA/VJ7UnZ46EuI/AAAAAAAAAZw/T0ZUxvhzJ9o/s1600/line.png" height="178" width="320" /></a></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Lines are drawn from the current position of the cursor till the destination x and y coordinates provided. Note that the coordinates are relative points from top left corner of the canvas.</span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">
</span>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">To move the cursor around, we can use the moveTo function.</span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">
</span>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">The width of the line could be set using the lineWidth attribute of canvas.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">The color of the line could be set using the strokeStyle attribute of canvas. You can give either rgb values or hex values.</span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">
</span>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<div>
<pre><span style="font-family: Arial, Helvetica, sans-serif;">context.beginPath();</span></pre>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">//move the cursor to position (100,200)</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">context.moveTo(100, 200);</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">//draw line from (100,200) to (400,100)</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">context.lineTo(400, 100);</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">//set line width</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">context.lineWidth = 10;</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">//set line color</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">context.strokeStyle = '##FE2E2E';</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">context.stroke();</span></div>
</div>
</div>
</div>
<span style="font-family: Arial, Helvetica, sans-serif;">
</span>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">
</span>
<br />
<h2 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">
How to draw an arc?</span></h2>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-QOSzosTHjSs/VJ7VOAfYU-I/AAAAAAAAAaI/LrkCf8YcItU/s1600/arc.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-QOSzosTHjSs/VJ7VOAfYU-I/AAAAAAAAAaI/LrkCf8YcItU/s1600/arc.png" height="178" width="320" /></a></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">To draw an arc, we need to specify the coordinates of the center of the circle and also the radius. The start and end angles decide how big or how complete the arc should be relative to the circle. Whether the arc should be drawn in clockwise/counter clockwise is decided by the last parameter (boolean).</span></div>
<div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">context.beginPath();
</span>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">var centerX = 250;</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">var centerY = 150;</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">var startAngle = 0.4 * Math.PI;</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">var endAngle = 1.8 * Math.PI;</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">var radius = 50;</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">var counterClockwise = false;</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">context.lineWidth = 5;</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">context.arc(centerX, centerY, radius, startAngle, endAngle, counterClockwise);</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">context.stroke();</span></div>
</div>
</div>
<br />
<div>
<br /></div>
<h2 style="text-align: left;">
How to draw a circle?</h2>
<div>
Circle is a special case of arc, where the start angle is 0 and end Angle is 2* Math.PI.</div>
<div>
<br /></div>
<h2 style="text-align: left;">
How to draw a rectangle?</h2>
<div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-7qxbbajWZxU/VJ7Ux4yMQoI/AAAAAAAAAaA/HZO21LY_5Z4/s1600/rectangle.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-7qxbbajWZxU/VJ7Ux4yMQoI/AAAAAAAAAaA/HZO21LY_5Z4/s1600/rectangle.png" height="178" width="320" /></a></div>
<br />
To draw a rectangle, use the rect() function, which takes the x and y coordinates of the top left corner of the rectangle, along with width and height.</div>
<div>
<br /></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">context.beginPath();</span>
</div>
<div>
var x = 100;</div>
<div>
var y = 100;</div>
<div>
var width = 300;</div>
<div>
var height = 200;</div>
<div>
context.rect(x, y, width, height);</div>
<div>
context.fillStyle = 'red';</div>
<div>
context.fill();</div>
<div>
context.stroke();</div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
</div>
Raghuveer Bhandarkarhttp://www.blogger.com/profile/08931983865959870700noreply@blogger.com0