{"id":330,"date":"2016-08-27T20:54:33","date_gmt":"2016-08-27T20:54:33","guid":{"rendered":"https:\/\/rejupillai.com\/?p=330"},"modified":"2021-03-10T15:34:11","modified_gmt":"2021-03-10T15:34:11","slug":"a-date-with-docker-swarm-before-k8s","status":"publish","type":"post","link":"https:\/\/rejupillai.com\/index.php\/2016\/08\/27\/a-date-with-docker-swarm-before-k8s\/","title":{"rendered":"A date with Docker (Swarm) before K8s"},"content":{"rendered":"\n<p>This post is about&nbsp;<strong>Docker&nbsp;<\/strong>(Containers), the notoriously disruptive technology that has taken the software and IT industry by storm last year, and continues to rock the foundation of old-style but most practiced ways of developing and running software.<\/p>\n\n\n\n<p>Though I was introduced to&nbsp;Docker&nbsp;2 years back, it was only recently that I started &#8216;playing&#8217; with it, and I kind of love it. I believe developers&nbsp;can implement amazing use-cases stacking and weaving containers together, the extent of which is only limited by&nbsp;one&#8217;s imagination. It&#8217;s like creating a cocktail of software and tools that gives you an instant kick, because, with&nbsp;<strong>Docker<\/strong>, building, shipping, and running software is&nbsp;incredibly fast and surprisingly light and easy.<\/p>\n\n\n\n<p>Sometimes it&#8217;s just not necessary to know every bit of things under the hood to make your &#8216;business&#8217; applications run.&nbsp;Docker<strong>&nbsp;<\/strong>(images)&nbsp;does this perfectly well. The best part is, it&#8217;s just not development.&nbsp;Docker&nbsp;is increasingly becoming popular for production deployments&nbsp;with automated CI\/CD pipelines.<\/p>\n\n\n\n<p>With PaaS solutions like Cloud-foundry and Stackato (Helion) an HPE product using&nbsp;Docker&nbsp;as the default container and Microsoft,&nbsp;offering both Windows and Linux based&nbsp;container services through it&#8217;s Azure Cloud platform,&nbsp;&nbsp;Docker&nbsp;is all set to play a pivotal role in every enterprise&#8217;s IT landscape. In fact, some compare it to what VMware was 10 years back as pioneers&nbsp;in the world of virtualization.<\/p>\n\n\n\n<p>Some facts and figures of&nbsp;Docker&#8217;s&nbsp;rapid adoption can be found here &#8211;&nbsp;<a rel=\"noreferrer noopener\" href=\"https:\/\/www.datadoghq.com\/docker-adoption\/\" target=\"_blank\">8 surprising facts about Docker&#8217;s real adoption<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>The&nbsp;playground&nbsp;<\/strong><\/h2>\n\n\n\n<p>In the world of sleek and slim ultra-books, sometimes having a mobile workstation as a laptop&nbsp;by your side&nbsp;is a very good&nbsp;thing,&nbsp;despite its&nbsp;huge size and&nbsp;friends and colleagues giggling&nbsp;at almost a&nbsp;kilogram of&nbsp;the power adapter, occasionally mistaking it for a home inverter.&nbsp;In hind-sight, a bunch of Ubuntu VMs on any cloud platform would have been just fine for this use case.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>The Use Case<\/strong><\/h2>\n\n\n\n<p>So I decided to implement a simple use-case to evaluate the 3 fundamental needs of every&nbsp;application &#8211; compute, storage and networking, also explore the truth&nbsp;behind&nbsp;Docker<strong>&#8216;s<\/strong>&nbsp;tagline <em><strong>build -&gt; ship -&gt; run<\/strong><\/em><strong>. <\/strong>My use case is so common, that it could be easily called the &#8220;hello world&#8221; of DevOps.<\/p>\n\n\n\n<p>The plan is simple &#8211; A &#8216;confident&#8217; developer commits a change (as a fix for&nbsp;a bug identified in production), which is propagated through various quality gates and deployed to test (or prod) and made available&nbsp;for customers. In other-words an automated CI\/CD pipeline using containers. Now we all know it&#8217;s a far-cry in the real enterprise today, many times because of reasons beyond technical.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/media-exp1.licdn.com\/dms\/image\/C4E12AQF7_j1ZRPuptg\/article-inline_image-shrink_1000_1488\/0\/1520479728871?e=1620864000&amp;v=beta&amp;t=k6h6i9_AsSW7o4u7-XlBqjeaP_kYQNi7ysA-bRvtmOw\" alt=\"No alt text provided for this image\"\/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Ingredients for the Cocktail<\/strong><\/h2>\n\n\n\n<p>&nbsp;Git-lab (SCM), Jenkins (CI), Sonar (Quality), Nexus (Distribution),&nbsp;Rundeck (CD), Tomcat (Application),&nbsp;a pinch of Swarm and Consul &#8220;image&#8221; ( for garnish ) all of&nbsp;which can be found here in the&nbsp;<a rel=\"noreferrer noopener\" href=\"https:\/\/hub.docker.com\/r\/reju\/\" target=\"_blank\">Docker Hub Repository<\/a><\/p>\n\n\n\n<p>From the&nbsp;<strong>Docker<\/strong>&nbsp;world, the following tools are used to help realize the use-case.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Docker-Machine<\/li><li>Docker-Swarm<\/li><li>Docker-Engine<\/li><li>Docker-Compose<\/li><li>Docker Images&nbsp;( pushed to Docker Hub )&nbsp;<\/li><li>DockerFile ( Optional to build an image from scratch <\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>The solution<\/strong><\/h2>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/media-exp1.licdn.com\/dms\/image\/C4E12AQGX-40XKwmsDQ\/article-inline_image-shrink_1000_1488\/0\/1520479728694?e=1620864000&amp;v=beta&amp;t=SMNHCvrSg_0iHBC6g5xrhhEgkIU2Ub9rMCbLnamHIvo\" alt=\"No alt text provided for this image\"\/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>The essentials: Compute<\/strong><\/h2>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/media-exp1.licdn.com\/dms\/image\/C4E12AQH3RAPOUGB45w\/article-inline_image-shrink_1000_1488\/0\/1520479728470?e=1620864000&amp;v=beta&amp;t=8lrXbq2jnpBLAuaxWjkuZ082lCMl5i7Dpw4RhKHst-c\" alt=\"No alt text provided for this image\"\/><\/figure>\n\n\n\n<p>Containers are the compute nodes that run on machines. Containers are instances of images just like objects are Instances of classes. You &#8220;spin&#8221; containers when you need them, and you can spin many containers from 1 image. Containers are extremely lightweight and you never think of it as more than a compute engine with ephemeral memory ( data vanishes when containers are removed ).&nbsp;Just sheer processing power.&nbsp;This gives us the freedom of destroying and creating containers at will, in Cloud&#8217;s terminology &#8211; <em>elasticity<\/em>.&nbsp;<\/p>\n\n\n\n<p>The swarm cluster decides where ( which machine ) to spin the Container, but this decision can be influenced using constraints and affinities. Container&#8217;s&nbsp;primary memory (RAM) is limited by the Machine&#8217;s secondary memory.&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>The essentials:&nbsp;Network<\/strong><\/h2>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/media-exp1.licdn.com\/dms\/image\/C4E12AQE11v-J8ERQZw\/article-inline_image-shrink_1000_1488\/0\/1520479728577?e=1620864000&amp;v=beta&amp;t=ByMfk0TGkKfotwXZHCxUApOvltHZs0V_59XCZcZRdS8\" alt=\"No alt text provided for this image\"\/><\/figure>\n\n\n\n<p>There are various network types for containers, which are either local (to the machine or host ) or global to the entire swarm which can be multi-host ( across different machines) as used in this example. To implement any use case it&#8217;s important that Compute nodes are able to &#8220;talk&#8221; to each other in a secure way.<\/p>\n\n\n\n<p>There are 2 levels of networking in this example.&nbsp;Machine-2-Machine and Container-2-Container.&nbsp;C2C&nbsp;connectivity&nbsp;happens across an &#8220;overlay&#8221; network driver, while M2M relies on the way hypervisor (Virtual box) is configured, like in this example there are 2 virtual network adapters NAT and Host-only&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>VBox NAT is used for internet connectivity&nbsp;( 172.x.x.x series )<\/li><li>VBox Host-Only is used to connect M2M ( 192.x.x.x series )<\/li><li>Swarm Overlay driver&nbsp;to connect C2C ( 10.x.x.x series )&nbsp;<\/li><\/ul>\n\n\n\n<p>The Container Ports is mapped to the machine available port to access it from Machine to do SSH and SCP.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>The essentials:&nbsp;Storage<\/strong><\/h2>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/media-exp1.licdn.com\/dms\/image\/C4E12AQHukGCnyf9w0g\/article-inline_image-shrink_1000_1488\/0\/1520479728659?e=1620864000&amp;v=beta&amp;t=yYSnSzlXfjsY79R0VqTKc23TwEvgtzI-q1l5tOa6UR4\" alt=\"No alt text provided for this image\"\/><\/figure>\n\n\n\n<p>Containers have&nbsp;ephemeral memory, the life of which is only till the Containers are running. So it&#8217;s crucial to save the state to a more &#8220;persistent&#8221; store. Docker volumes come to our rescue for this.&nbsp;Docker&nbsp;volumes are typically an \u201cR\/W&#8221; mount on a machine file system which is &#8220;mapped&#8221; to a containers file system [&nbsp;machine\/var: \/container\/var]. As part of our use&nbsp;case implementation, a \u201cData Container&#8221; is used to map C2M, though it&#8217;s not mandatory.&nbsp;Data containers are an elegant way to store container data (ever from multiple Containers) on to the machine.<\/p>\n\n\n\n<p>But it&#8217;s also interesting to know what data needs to be stored in ephemeral memory and what to permanent store.&nbsp;There is no thumb rule, but ideally, all data generated by the primary process (PID = 1) which is also called as application data is typically stored on volumes so that the state of the application is recovered after a reboot of the container or even machine. One of the other ways to store data (and persist across reboots) is to use a&nbsp;Docker&nbsp;commit subcommand, which saves the current state of Container into a higher version of the image (always tagged)<\/p>\n\n\n\n<p><strong>Docker<\/strong>&nbsp;images use a differential layered approach to storing images, this is an excellent way to store and also transfer only delta changes across machines and even network (or internet, a push to&nbsp;<strong>Docker<\/strong>&nbsp;Hub). This makes&nbsp;<strong>Docker<\/strong>&nbsp;pull and push extremely light and lightning-fast in a&nbsp;development cycle.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Docker Compose<\/strong><\/h2>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter\"><img decoding=\"async\" src=\"https:\/\/media-exp1.licdn.com\/dms\/image\/C5612AQGVblvyR-sB1g\/article-inline_image-shrink_1000_1488\/0\/1520148248162?e=1620864000&amp;v=beta&amp;t=XAyxQp4HQUM8z6NBbTCKvk41cYSYQGcxWcE3xQUgC1s\" alt=\"No alt text provided for this image\"\/><\/figure><\/div>\n\n\n\n<p>Docker-Compose, the orchestrator helps&nbsp;weave containers&nbsp;together and build complex service dependencies for ex: app-db dependency. The services (containers) are defined in a simple YAML file called docker-compose.yml.&nbsp;It&#8217;s a really simple yet extremely powerful file, service definition attributes like the image to use, port mapping, volume mapping, machine affinity and constraints, environment variables to be used inside containers, the order of container spin, and networks.&nbsp;Docker<strong>&#8211;<\/strong>Compose has an equivalent attribute for every&nbsp;Docker&nbsp;run command.&nbsp;This greatly reduces the need of running different containers individually.&nbsp;One can create an entire complex environment (incl. this example) from a single compose file. Now that&#8217;s called Infrastructure as code.<br><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>DockerFile<\/strong><\/h2>\n\n\n\n<p>The images can also be built by using&nbsp;DockerFile, which has instructions to install the libraries, packages, and applications <em>FROM<\/em> scratch. Often this is considered the best way to create an image in the first place. The other option of course is to save the state of a running (or stopped) container to an image.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Why should you consider (<\/strong>Docker<strong>)  Containers?<\/strong><\/h2>\n\n\n\n<ul class=\"wp-block-list\"><li>very&nbsp;fast and light-weight<\/li><li>scales wonderfully<\/li><li>promotes micro-services architecture<\/li><li>distributed computing made easy<\/li><li>massive&nbsp;collection of both official distribution and reworked user images in Docker Hub<\/li><li>LB and HA built into the Swarm cluster<\/li><li>a perfect tool to bridge the Dev-Ops gaps&nbsp;<\/li><li>&#8220;highly secure&#8221; as per&nbsp;<strong>Docker<\/strong>&nbsp;Inc, but customer concerns&nbsp;(like image-scan) still valid.<\/li><li>fast adoption in the industry and a vibrant community of contributors.<\/li><li>Production-ready. Experts&nbsp;predict&nbsp;in the next 5 years 40% of customers globally will move to container-based solutions in production.<\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"<p>This post is about&nbsp;Docker&nbsp;(Containers), the notoriously disruptive technology that has taken the software and IT industry by storm last year, and continues to rock the foundation of old-style but most practiced ways of developing and running software. Though I was introduced to&nbsp;Docker&nbsp;2 years back, it was only recently that I<\/p>\n","protected":false},"author":1,"featured_media":469,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-330","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized","ct-col-2"],"_links":{"self":[{"href":"https:\/\/rejupillai.com\/index.php\/wp-json\/wp\/v2\/posts\/330","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rejupillai.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rejupillai.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rejupillai.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rejupillai.com\/index.php\/wp-json\/wp\/v2\/comments?post=330"}],"version-history":[{"count":3,"href":"https:\/\/rejupillai.com\/index.php\/wp-json\/wp\/v2\/posts\/330\/revisions"}],"predecessor-version":[{"id":471,"href":"https:\/\/rejupillai.com\/index.php\/wp-json\/wp\/v2\/posts\/330\/revisions\/471"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rejupillai.com\/index.php\/wp-json\/wp\/v2\/media\/469"}],"wp:attachment":[{"href":"https:\/\/rejupillai.com\/index.php\/wp-json\/wp\/v2\/media?parent=330"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rejupillai.com\/index.php\/wp-json\/wp\/v2\/categories?post=330"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rejupillai.com\/index.php\/wp-json\/wp\/v2\/tags?post=330"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}