2 suggestions and 1 recommendation for implementing DevOps

June 25, 2013 by Leave a comment

Before getting into the suggestions for how to implement DevOps it is necessary to answer the age old question, what is DevOps purpose?

“In an ideal state DevOps allows business features to be delivered quickly and continuously while maintaining 100% uptime.”

It always comes down to the bu$ine$$. All of us know that downtime costs real money and delaying features costs potential money. Many companies have even prioritized one or the other at different points in their maturity cycle. The goal of DevOps is to minimize both costs.

While automation is often a focus of DevOps, and I believe that a lazy admin is a good admin, I said nothing about automation in my definition. It wasn’t an oversight, automation is a means, not an end to DevOps goals. I also said nothing about developers or operations in this description since the roles are largely irrelevant if that goal is met. Here are 3 suggestions for bringing your organization into the light of DevOps.

Suggestion 1:
Lock your developers and operations teams into the same room and buy them enough beer to keep them happy with each other.

Ok, seems silly at first, but this is how some larger companies have done it. Each development team has 1 or 2 SRE members join and each literal DevOps team is focused on a particular business deliverable. This moves operations from a shared service to a team deliverable and gives all members a sense of ownership of the code as well as how it functions in production. From everything I have heard it has been a wildly successful brute force approach at several companies. Larger companies that have taken this approach seem to have a leg up on most of us since they can attract a level of talent most companies cannot and that they have the resources to expand each development team by an additional 1-2 people.

Suggestion 2:
Have your CEO look into the mirror and say “DevOps” three times.

While the ghost of DevOps won’t appear to slay your evil over-the-wall methodologies, getting top down support will put people on the path to figuring it out. A leader’s vision dedicated to moving in this direction can produce a successful DevOps strategy. I would be remiss to point out its flaws if done in isolation though. With a description as vague as my own, or any other DevOps description, it can be difficult to break down and measure progress on the path to DevOps. At larger organizations it can be an even greater challenge if some groups are happy with the status quo.

Suggestion 3:
What’s measured is managed!

There are 4 core DevOps metrics:

  • Mean Time To Detection (MTTD)
  • Mean Time To Resolution (MTTR)
  • Mean Time Between Incidents (MTBI)
  • Mean Time Between Releases (MTBR).

By measuring these for your team’s deliverables several conclusions can usually be drawn that promote the path to DevOps. For instance if your MTBI is high as a result of impacts during changes then you probably need to prioritize improvements to your release process or architecture to remove that downtime. Maintenance windows be damned, without 0 downtime deployments you will never be able to meet the goals of DevOps. If you aren’t able to decrease your MTBR then automation of deployments and testing will give a big boost. MTTR will be decreased when problems occur by providing accurate alerting and having well defined process, ownership and escalation. Keeping MTTD low ensures that monitoring is added as part of new features rather than after the fact.

Measuring the Mean Time application metrics will also identify hot spots in your application stack. Using these 4 metrics you can effectively score your applications and the risk they pose, not based on the potential impacts, but rather on the experience of your organization to deliver them.  This will remove barriers for applications that have a history of being successful in prod and ensure the right oversight is available when an application needs it.

When release processes need to be improved, code and infrastructure needs to be re-architected, monitoring needs to be refined and success is visibly measured it almost always leads to development and operations working together and DevOps becomes a byproduct of the business goals.

Filed under Thoughts

Jmeter JTL Reporting

June 6, 2013 by Leave a comment

When I first migrated a previous employer off Loadrunner and onto Jmeter several years ago 2 things happened quickly. In a big WIN! We jumped from 30->500 test cases (in <1 year) since we were able to provide unlicensed access to the system. In a big FAIL! we no longer had Loadrunner’s reports to use at meetings and show users after the fact. We quickly automated reporting that went through a number of iterations.

Now that I am in my new role I needed to rebuild a similar framework and reporting became even more important since it was for external customers rather than internal.

I realized others may need a single report from time to time and that the reporting can be a big hurdle in migrating technologies. The end result is JTL Report Creator.

This will take an JTL(XML output) process it into an aggregate report table with zoomable graphs using jqPlot for Latencies, Count, and Percent Failed. Since I am using my godaddy e-mail as a relay I am limited to 250 reports per day. This means all JTLs will queue and be processed every 5 minutes and then a link will be sent to your e-mail. Here is a simple example of the output.

This script requires that you save in the xml format AND that you have the following attributes
“save success”, s, Success flag (true/false)
“save elapsed time”, t, Elapsed time (milliseconds)
“save label”, lb, Label
“save timestamp”, ts, timeStamp (milliseconds since midnight Jan 1, 1970 UTC)

Only latencies of successful results are reported and averaged.
Please submit your file as a .jtl

Filed under Tools Tagged with , , , , , , ,

Mail Relays using python

June 6, 2013 by 1 Comment

I recently needed to send e-mail from my home computer in a python script. Unfortunately I use Comcast and they have decided that anyone that does this is a spammer so they have blocked all outbound communication over port 25. Big thanks to Lars for pointing me in the right direction!

In order to get around this both gmail and godaddy provide relay services.

Here is the solution:
Create the message

sender = 'ghoti@cultureofqualityengineering.com' #you will need to use your email of course
password = 'MyPassword'
receiver = #submitted email address

message = """From: Geoff <""" + sender + """>
To: To Person <""" + email + """>
Subject: Kapow!
Hey thanks for using this service!
and other interesting text
"""

Send the message

try:
  #session = smtplib.SMTP('smtp.gmail.com',587) #for gmail uncomment this
  #session = smtplib.SMTP_SSL('smtpout.secureserver.net',465) #for godaddy uncomment this line
  session.ehlo()
  #session.starttls() #uncomment for gmail
  #session.ehlo() #uncomment for gmail
  session.login(sender, password)
  session.sendmail(sender,receiver,message)
  session.quit()
except smtplib.SMTPException:
  print "Error: unable to send email"

Filed under Tools Tagged with , , , , , , , , ,