Requires DevOps Java

The DevOps movement

Java magazine

01/16

Written by:
Patrick Peschlow

Links & literature

[1] http://www.gartner.com/technology

[2] http://www.forrester.com/rb/research

[3] http://puppetlabs.com

[4] http://www.opscode.com/chef

[5] http://cfengine.com

[6] http://github.com/capistrano

[7] http://controltier.org

[8] http://fabfile.org

[9] http://rundeck.org

[10] http://docs.puppetlabs.com/mcollective

[11] http://cukes.info

[12] http://auxesis.github.com/cucumber-nagios

[13] http://hudson-ci.org

[14] http://jenkins-ci.org

[15] http://fedorahosted.org/cobbler

[16] http://vagrantup.com

[17] http://twitter.com/#!/DEVOPS_BORAT

[18] http://teddziuba.com/2011/03/devops-scam.html

[19] http://replaysolutions.com/2011-DevOpsSurveyResults

[20] http://rww.to/qiAxu1

[21] http://www.jedi.be/blog

[22] http://agilesysadmin.net/

[23] http://dev2ops.org

[24] http://kief.com

[25] http://www.agileweboperations.com/devops-series

[26] http://agileoperations.net/index.php?frontpage

[27] http://theagileadmin.com

[28] http://www.planetdevops.net

[29] Udo Pracht: "Becoming productive together", in WIRTSCHAFTSINFORMATIK & MANAGEMENT, issue 04/2011

[30] http://devopsdays.org

[31] http://campdevops.com

[32] http://velocityconf.com/velocity2011

[33] http://puppetlabs.com/community/puppet-camp

[34] http://devopscafe.org

[35] http://devopsweekly.com

The term “DevOps” is currently on everyone's lips, but there is still no common understanding of what exactly the goal of DevOps is and how it can best be achieved. Nevertheless, it can also happen to us that we will soon have to “do” DevOps. This not only increases curiosity, but also concern. Reason enough to put things in order.

When Patrick Debois organized a conference in Belgium a good two years ago and was looking for a name for it, he had no idea that the name would soon spread like wildfire. The conference was called DevOpsDays and was the first in a series of similar conferences. Since those first DevOpsDays, the term “DevOps” has been used to an ever greater extent and is now widely recognized. Well-known analysts such as Gartner [1] or Forrester [2] have DevOps on their radar and provide critical assessments. Various companies offer integrated “DevOps solutions”, well-known manufacturers give their products the DevOps stamp and you can read headlines like “VMware goes DevOps”. Interestingly, despite the current hype, there is no common understanding of what DevOps is actually about. The aim of this article is therefore to give a clear picture of what DevOps is and what it means for our daily work.

Dev and Ops become DevOps

The term is made up of "Dev", which represents the software developer (Developers), and "Ops", which stands for IT operations (Operations). The combination to the common "DevOps" symbolizes intuitively a solidarity between software developers and IT operations. And in fact, that is the basic idea of ​​DevOps and the trigger of the associated movement: a bringing together of the two areas of software development and IT operations, which are fundamentally different in traditional perception. This short explanation has the advantage that we can spontaneously imagine something under DevOps. On the other hand, it allows a wide range of interpretations, which can easily lead to misunderstandings. Currently, the DevOps movement sees its main task in channeling the many interpretations and formulating a clear definition of DevOps. In the following we will clarify how the term “DevOps” is to be understood today. First, let's look at the underlying conflict between Dev and Ops.

The traditional conflict between development and operation

In a somewhat simplified way, the task of software developers is to implement the functions requested by the client as quickly as possible. When a new function becomes available, there is potential added value for the end user. This added value is often recognized by the client at the first acceptance. The more frequently new features are completed, the more positively the developers are perceived. For the developers, it is largely irrelevant whether the new features are actually available on the production system.

Put simply, the task of IT operations is to make the software supplied by development available to end users in the productive environment. This includes the deployment of new software releases and ensuring ongoing operations under certain quality requirements. The company bears direct responsibility for the availability of the application, and its success is measured by the extent to which the given quality requirements are achieved. As a rule, users expect full availability and security of the application. If, for example, the availability is impaired, this falls back directly on the operation. The result is a very negative perception by the client, especially if the users of the application report a problem before the monitoring systems used sound the alarm. In order to minimize the probability of unexpected failures, the company often does everything in its power to protect the state of a stable application from changes.

This comparison of the tasks of development and operation shows that both departments have opposite incentives. Development is interested in quick and frequent releases, but operations would prefer to avoid releases. Both sides are pursuing the same goal, namely to prove their own worth for the company. But that is exactly what regularly leads to conflicts, hostility and bad mood.

Blame game

As a rule, devs and ops meet under time pressure, for example when deploying a new release or when there is a problem (such as a system failure). The typical blame game then begins, in which both camps blame each other for the situation. Some examples:

  • The development department passes on a new release for deployment to the company, which simply does not succeed in making the software executable in the productive environment. When the company contacted the developers and described the errors that occurred, they blocked it: The software would run error-free in the development environment and it would therefore be clear that the error was in the company. As a result, both sides accuse each other of being to blame for the problem. There are crisis meetings and lots of nasty phone calls and emails between the departments and to the superiors. An investigation finally shows that the development and production environment differ in one important detail (e.g. the use of a component in clustering mode), which neither side was aware of beforehand.
  • The rush of users to the new website is so great that the response times get longer and longer and the site collapses completely a few hours later. From a business perspective, this is a disaster. But the first thing that the involved camps (development, operations and possibly other departments for database administration, quality assurance, etc.) do is to speculate heavily about the supposed cause: “That must clearly be a database problem!” Or “That is definitely them to blame for the new server! ”An objective analysis is not started until much later, but at this point the customers have already accepted that the new website is a disaster.
  • An annoying performance problem arises in the production system. The developers worked through several nights under great pressure and finally delivered a patch. However, the company is concerned that the patch will endanger the stability of the system because it includes changes to a critical component. Therefore, a precise quality control in a test environment is first required in order to check the solution in realistic test scenarios. Unfortunately, the required load cannot be adequately represented in the test environment. A lot of time goes by, and a month later the patch has still not been applied. The developers are disappointed because “it was obviously not in such a hurry”.

Those who have not yet experienced such situations can consider themselves lucky. Anyone who knows the Blame Game, on the other hand, knows that the time lost should have been better used to solve the problem. Assigning blame and being annoyed about it are still possible in retrospect.

The operation as a bottleneck

DevOps has added interpersonal components to the ideas of the agile movement.

Over time we got used to the blame game. However, since software development has been increasingly using agile methods, the situation has escalated. Methods such as Scrum rely on ongoing interaction between clients and developers as well as short release cycles, but the philosophy that goes with it usually does not continue in operation. In the end, the advantages of agile methods are thwarted if the last mile, specifically deployment and operation, is disregarded. Software releases are then created in short iterations, but the added value only becomes visible in the productive environment much later. The ever shorter release cycles reveal the operation increasingly as a bottleneck on the way of the software to the end user. Frequent releases also increase the potential for direct clashes and thus also the blame game between software development and operation. A movement is forming In the last few years a number of people decided independently of one another to do something about the conflict between development and operation. They gained experience, met and exchanged ideas, and eventually became what is now known as the DevOps movement. It is not surprising that these were almost exclusively employees in IT operations (and not developers). The often negative perception led to a desire for change, above all to the desire for a new self-confidence, away from being considered slow and away from slogans such as "A good day is if there is no disaster today". The parallels to the beginnings of the agile movement are obvious. In fact, even before the DevOps movement, there were efforts such as Agile Operations or Agile System Administration to increasingly use agile methods in IT operations. The key to the success of the DevOps movement was that it brought software development on board and thereby greatly increased the number of potential interested parties. DevOps has taken up the ideas of Agile Operations and Co., but supplemented them with interpersonal components. In the past two years, various goals have been formulated by DevOps. They can be divided into three areas: collaboration, automation and processes.

collaboration

Especially in the early phase of the DevOps movement, the interpersonal component was given a lot of emphasis. A statement by Patrick Debois, published on his blog following the first DevOpsDays, is indicative: "And remember it's all about putting the fun back into IT!"

For many representatives of the movement, mutual respect is the most urgent improvement in the interaction between development and operation, because it is a prerequisite for trust and good cooperation. Cultural building blocks of better cooperation are, for example, the self-commitment of those involved to goals, attentive listening, mutual further training and the establishment of common values. With a respectful and trusting cooperation, so the thought, the blame game can be avoided. The declared goal of the DevOps movement in interpersonal terms is to a certain extent to pull the lessons learned from the agile methods up one level and to establish them across departments.

automation

A central idea of ​​DevOps is the automation of processes that are otherwise carried out manually and are therefore not very transparent or are only poorly suited for quality control. Automation is made possible through the use of suitable tools. Many DevOps supporters refer to Infrastructure as Code as the most important part of automation. With this approach, all processes required to set up infrastructure or to carry out deployments are represented in source code. Many of the quality control solutions commonly used in software development can then also be used by the company. Possible benefits of the Infrastructure-as-Code approach are:

  • Central management of the source code using version control. • High transparency and avoidance of knowledge islands. • Automated testing of the configuration of servers and virtual machines.
  • Automated testing of deployments and the subsequent availability of participating systems and business-critical software functions.
  • Shared use of configuration and deployment rules through development and operations.

A central idea of ​​DevOps is the automation of processes.

Tools for configuration management, for example Puppet [3], Chef [4] (see also article by Martin Eigenbrodt, in Java Magazin 1.2012, page 53) or the older CFengine [5], are fundamental for realizing Infrastructure as Code. These tools offer domain-specific languages ​​(DSLs) to describe the desired end state of the system on an abstract, cross-platform level. Behind the scenes, these specifications are then carried out on the respective target systems using predefined images. A related group of tools focuses on automated deployment and the coordinated execution of actions on running, distributed servers. As a rule, these tools also offer DSLs to describe sequences of commands. Representatives of this category are Capistrano [6], ControlTier [7], Fabric [8], RunDeck [9] and Marionette Collective [10]. The usual suspects such as Mercurial or Git can be used to check the version of the formulated regulations (in the DSL used in each case).

A BDD tool such as Cucumber [11], which can be used together with Puppet or Chef, is ideal for automated testing of the source code. Extensions like Cucumber-Nagios [12] also make it possible to generate the output directly in the format of certain monitoring tools. A BDD approach is particularly recommended because it brings with it a culture of testing in which interesting use cases are tested and not just the mere availability of servers. With continuous integration servers such as Hudson [13] or Jenkins [14], these tests can also be carried out automatically. Tools for the fully automatic installation of operating systems or virtual machines should be of particular interest to developers. Newer representatives of this category are for example Cobbler [15] or Vagrant [16].

It is a declared goal of the DevOps movement to promote the automation of processes in IT operations and the sharing of tools between development and operations. For example, the developers can pass on the infrastructure configuration they use (e.g. as puppet manifests) to the company and coordinate it with them. Or development and operation develop the infrastructure code directly together and thus rule out incompatibilities between the development, test and production environments from the outset.

Processes

For the introduction of DevOps ideas in companies, it is essential to have processes. The DevOps movement still has a lot to do in this area and has yet to come up with concrete proposals. A cynical tweet from “DevOps Borat” [17] on the subject of DevOps processes: “To make error is human. To propagate error to all server in automatic way is #devops. ”Of course, the comment is not to be taken very seriously, but it clearly reflects a concern about blind actionism, in which the functions of development and operation are thrown together without considering possible risks .

Most DevOps proponents agree that it is neither necessary nor desirable to combine development and operations. Interdisciplinary experts who work at the interface between the two areas can of course still be helpful. However, “DevOps” is definitely not to be seen as a job or role designation. In general, when defining DevOps processes, one difficulty is that in many smaller companies the line between development and operation is not drawn as strictly as in large companies. It is therefore questionable whether processes and procedures that work well for small companies can also be used in large companies.Nevertheless, or precisely because of this, it is a declared goal of the DevOps movement to define suitable processes for the introduction and use of DevOps ideas.

Misunderstandings and criticism

The stated goals of DevOps are all understandable. However, the lack of focus of the DevOps movement on a clear, overarching goal has led to various misunderstandings and points of criticism. Let's consider a selection of them:

  • “DevOps is just an advertising name to advertise well-known things as new.” Almost without exception, the founders of the DevOps movement themselves belong to those people who have previously followed DevOps approaches. You know, of course, that many of the proposed practices and tools are not fundamentally new. Yet they have received tremendous attention to a problem that was previously not widely discussed in public. That alone is a great success. Incidentally, the initiators of the movement regularly emphasize that DevOps is not a money machine for them, and they even have to accept reproaches for not trying to make more capital out of DevOps [18].
  • “DevOps is a free ticket for developers to cause any damage to the productive system.” DevOps does not require the developers to have write access to the servers of the productive system or even their root password. Uniform deployment procedures and infrastructure as code can also be used while maintaining certain sensible restrictions.
  • “DevOps wants to replace development and operations with an elite of all-rounders.” Raising such an elite would be absurd. It is not without reason that specialization and the resulting separation of development and operation were introduced many years ago. Rather, DevOps seeks to improve collaboration and knowledge sharing between these areas.
  • “DevOps would like to abolish operations and let the developers do everything.” This thought, often formulated as a call for “NoOps”, is also absurd. The driving force behind DevOps are IT workers, and it is not their goal to get rid of them.
  • “With DevOps we have to learn new tools.” That may be, but why so negative? For those new to DevOps, it is an obvious decision to use tools to get started. It is precisely through automation that you can quickly gain added value and also create a common basis for cooperation between development and operation. Interesting in this context is a survey that Replay Solutions carried out in spring 2011, according to which tools are seen as very important for the success of DevOps. The respondents hope for improvements in defect tracking and version control in particular [19]. A survey by Puppet Labs from the summer of 2011 confirms this impression: Automation, e.g. B. with tools for configuration management, named as the greatest hoped-for improvement through DevOps [20]. Only then do more abstract goals follow, such as optimizing deployment processes or improving communication between the departments.
  • “Some people simply cannot be shown respect.” In fact, respect and trust do not come about at the push of a button, but have to be “acquired” through performance. You will always find people who do not perform well and therefore never get the trust of their team members. Nevertheless, the following applies in the context of DevOps: Development and operation must first understand what the other side is actually doing and what the everyday challenges look like before they can even remotely assess their performance. In the end, team members who are underperforming are of course a problem, but not a DevOps specific one.
  • “DevOps becomes superfluous with PaaS offers.” This is a criticism that has to be taken seriously, because the use of some cloud offers can significantly influence the specific tasks of the company. The reply of the DevOps advocates is that PaaS only creates a further abstraction layer or new interface for the classic functions of the operation (such as deployment, monitoring and ensuring quality requirements). So you still can't do without a business.

What is DevOps really about?

Overall, it can be said that DevOps wants to solve similar challenges as the agile methods for software development, only across departments. This of course makes it more difficult to find suitable solutions. Above all, however, it is becoming more difficult to establish DevOps approaches in practice. The answers to two fundamental questions are groundbreaking for DevOps:

  • What kind of incentive do development and operation have to invest a lot of time and effort in joint ventures when the day is already too short to do their own work? What kind of incentive is there for the company to use unknown tools for automation when the self-written shell scripts have been in use for years? Creating incentives for everyone involved is fundamental to establishing DevOps. Ultimately, however, these incentives can only be created through the objectives set by the management.
  • What kind of incentive does the management have to introduce DevOps approaches in their company? If we look at the question “What does DevOps bring us?” From the management's point of view, it quickly becomes clear that DevOps is only interesting if it provides (financial) added value.

 

Lately, therefore, a new perception of the problem addressed by DevOps has emerged. The goals of DevOps described at the beginning deal with symptoms, but ignore the core problem. The initial calls for more cooperation or better tools try to make everyday development and operations more pleasant.

We're not paid to have fun at work. For companies, it's the result that counts.

This approach helps to mobilize the masses and generate attention, but it does not necessarily convince the actual decision-makers to introduce DevOps in a company. It is therefore necessary to see the underlying problem as a business problem: How can maximum profit be generated in the shortest possible time? How can the original idea of ​​a software (the vision) be brought to the customer or end user quickly and yet stably so that it generates added value or income for the company? A little ironic about this development is that the name DevOps originally fitted very well, but currently obscures the view of the actual business problem.

If we take things a little more soberly, we find that we are not paid to have fun at work. For the company, what counts first and foremost is the result. If the result is correct, there is no business problem and no solution like DevOps is needed. Of course, it is ideal if the company's success can be increased using methods that employees also enjoy. However, the argument that development and operation are happy when you introduce a certain tool or eat pizza together does not work. The introduction of a measure such as “We are now doing DevOps!” Must first be necessary, for example because the current figures say so, and the success of the measure must also be quantifiable. The key word here is measurability. To do this, we need combined metrics for the joint performance of operations and development. You can supplement or replace the metrics used to date.

For comparison: no company introduces agility if it does not expect an increase in profit from it. Therefore, the path to success for DevOps also lies in the business case. And as with the agile methods, the business case can primarily be proven via success stories. It is therefore essential for the DevOps movement not only to think about techniques and tools, but primarily to report on the experiences of success through DevOps.

What can we expect with DevOps?

Regardless of whether the management specifies the introduction of DevOps or we try our own DevOps approaches, DevOps will mean a stronger focus on certain elements in our everyday work:

  • In software development, we will increasingly deal with operational aspects, e. B. the setting up of physical or virtual machines, the protection of systems or the meticulous planning and implementation of deployments. To this end, we will increasingly use tools for automation, together with employees from the company, from whom we will also learn. We will take more responsibility for our software and problems that arise in the production environment. We will support quick troubleshooting in emergencies, through logging and monitoring coordinated with the company, and perhaps even be included in alerting. We will make our software more robust, e.g. For example, by using feature flags with which new functions can be easily switched off from the outside if necessary (i.e. in the event of an error or problem). In the Java environment, the requirements for feature flags can be easily implemented using MBeans.
  • In IT operations, we will increasingly rely on automation, with infrastructure as code, version control and automated tests. We will learn from software development in this area. We will also use agile methods such as Kanban, possibly together with the developers. We will have to prepare for frequent releases and coordinate the process with the developers in this regard. Together with the developers, we will define suitable metrics in order to identify problematic situations on the productive system more quickly. In addition, we will be included in the planning of the software requirements at an early stage.

In addition, we can assume that the introduction of DevOps will involve cross-departmental metrics to measure success. We have to adapt to such common metrics, and thus common incentives for development and operation.

Where can I find more information about DevOps?

Most of the relevant information about DevOps can currently be found in blogs. Required Reading For DevOps- We Don't Get Paid To Have Fun At Work. For companies, it's the result that counts. Patrick Debois' blog is interested in the latest views on the DevOps movement and concrete examples of the use of DevOps tools [21]. Stephen Nelson-Smith runs a blog called “Agile Sysadmin”, on which, for example, a valuable article on the subject of “Kanban for Ops” can be found [22]. Damon Edwards 'or DTO Solutions' blog currently mainly deals with DevOps tools in the cloud environment, but also offers some groundbreaking articles from the early days [23]. Kief Morris runs a blog about Continuous Delivery, with many DevOps-oriented articles and thoughts on how to put it into practice [24]. The blog "Agile Web Development & Operations", written by Matthias Marschall and others, offers balanced and well thought-out content. The series of guest articles by various DevOps sizes should be particularly worth reading for beginners [25]. Other interesting blogs are "Agile Operations" [26] by Scott Wilson and "The agile Admin" [27] by Ernest Mueller. Finally, the “Planet DevOps” website is a central entry point with contributions from various authors [28]. German-language articles on DevOps are currently still a rarity. A recently published magazine article by Udo Pracht offers a good introduction to the subject [29].

Since DevOps is still in the discovery phase, no holistic book on this topic has yet been published (there is "DevOps" by Kevin Roebuck, but it has little to do with the DevOps movement and is not recommended). However, there are excellent books on related topics with a clear reference to DevOps. "Continuous Delivery" by Jez Humble and David Farley, which deals with configuration management, automated deployment and the processes required for this, is highly recommended. “Web Operations” by John Allspaw and Jesse Robbins is also recommended, which deals with the importance of suitable metrics, among other things. There are also books on various DevOps tools, for example "Test-Driven Infrastructure with Chef" by Stephen Nelson-Smith. Also worth mentioning is Michael Nygard's “Release It!”, Which is already a classic today. This book is particularly recommended for developers who want to increase their sensitivity to possible unexpected production problems of web and enterprise applications.

As far as conferences are concerned, the DevOpsDays [30] are probably the most important conference in the DevOps environment. The DevOps camp [31], which recently took place for the first time, should also be mentioned. At the well-known Velocity conferences [32], contributions on topics from the DevOps environment are also regularly made. There are also tool-specific conferences such as the Puppet Camp [33]. John Willis and Damon Edwards are offering a podcast on the subject under the title “DevOps Cafe” [34]. There is also a well-stocked weekly newsletter compiled by Gareth Rushgrove [35].

Also read the article "DevOps Light - DevOps in Regulated Environments" by Joachim Baumann.

Full article