The DevOps Mindset by Rackspace
The DevOps Mindset by Rackspace
The DevOps Mindset by Rackspace
DevOps
Mindset:
Real-World Insights
from Tech Leaders
THE DEVOPS MINDSET REAL-WORLD INSIGHTS FROM TECH LEADERS 2
Certain facets have to be in place for DevOps to function, and the number
one factor is having a company culture in which it can thrive. To understand
the importance of culture, consider how DevOps can function within an
organization: Basically, it is both a social system and a technical system a
Socio-Technical System, Behr elaborates, and what DevOps does is bring
the social more in balance with the technical than it has been in the past.
The culture that can foster the DevOps mindset has to exist throughout a
company, not just within a single department, so people can collectively
contribute all their skills to solving problems. One of the key elements of
making DevOps successful, Behr emphasizes, is actually making sure that
it is part of an organizational plan. Organizations that want to be successful
with DevOps typically transition away from managing tasks to managing
If you want to transcend the status quo and reach new standards, you
must be willing to do what it takes to stay competitive. Flexibility is the
key, Behr says. You need that flexibility not just because of choice, but
because, now more than ever, the market is dictating it.
The key thing to be thinking of here is the mindset angle of DevOps: the
collaboration, the measurement, the sharing, says Behr. By balancing
the social and technical side of things, you can actually get much faster
feedback loops in your learning process and in your development and
operational processes. You can actually learn, advance and iterate much
quicker. But if you dont develop those sides equally, what you get is
automation without any thought or a lot of collaboration without any real
traction toward achieving your companys goals.
Why did you decide on DevOps? What was the catalyst to get you
started down the DevOps path?
HedgServ turns out they were scrambling to deal with whatever everyone else in the
organization threw at them. And in the process they were forced to learn
how to adapt in a world where they had little to no control.
Location:
New York/Ireland
When something broke, it was so easy to point it out after the fact, and
I was as critical as anyone, asking things like, "How could this happen?"
Company snapshot:
without understanding that it was unfair to ask this of the people being
HedgeServ is a global,
swamped by it.
independent fund
administrator.
Size of organization:
App. 700
"If senior One aspect of embracing DevOps is we learned that we had created the
system that caused this, and so we were then in a position to start to
management has correct it.
a desire to work How long did it take you to get from beginning the transition to the
an understanding I think the fundamental shift toward DevOps started when we got away
from focusing on individual team goals and elevated our conversation to
of the challenges organizational goals and let the teams drive toward them, probably a little
over a year ago. I suspect that could be called the beginning of our DevOps
of the people journey.
doing the work,
And now we have learned to communicate these goals in the form of
DevOps can intents, with leaders of cross-functional teams able to determine how to
achieve those intents. It has been amazing to watch this happen. We still
happen." have a long way to go. We are still experiencing frustration from people
wanting and trying to do good but not knowing how. When that goes away,
I would say we are doing DevOps successfully.
We are exploring ways in which we can measure our improvements, but I'm
not really satisfied with the measures I've seen and how they might apply
to us. My current observations of success are drawn from the behavior I see
in the people in our technology group, how their attitude is changing. For
example, the programmers have a blog where they describe how failures
occurred for all to learn from. We are just now wrapping up our new server
build, which was a combined effort between programmers and IT. While
that in itself is awesome, the even cooler thing is the fact that a whole new
avenue of conversation has opened up.
We recently had a case where one of our libraries was causing trouble
when we had multiple versions of our main application installed on the
same server. The programmer investigating reached out to the IT team
to ask about migrating this library into the base build instead of being
packaged with the app. That solution path would never have been explored
a year ago.
Did you run into roadblocks when you began your DevOps
implementation? If so, how did you navigate around them?
I would love to say that the journey has been smooth, but there are
roadblocks everywhere. You want to just point out to someone, "Hey,
look at thishere is a better way to work that will make your life more
rewarding" and they just jump at it, but it doesn't work that way.
I have been very fortunate to have a team of leaders who are willing to try
things with me and learn with me. Together we have worked with our teams
to ease people along the path to where we want to go.
Repeatability and scalability are our business. The capabilities that DevOps
gives us directly enable this by encouraging collaboration within the
organization to come up with better solutions.
The library example I gave above would have been solved by the
developers a year ago. It's not like the problem wouldn't have been solved
prior to our DevOps experiences, but it would have been solved within the
team silo and the solution would have been fragile.
I bet money that a solution developed the old way would have worked
for three years until one of the assumptions it made bit us. Getting more
robust solutions to all of the little problems is how DevOps helps us with
both scalability and reliability.
What DevOps myths (were too big, were not in Silicon Valley,
what we do isnt scalable) have you personally dispelled in your
transition to DevOps?
Perhaps the biggest one for us is that we are a Windows shop, and there
has traditionally been a bias toward Linux as the one true DevOps platform.
I can say with certainty that is not the case.
I would say it is okay to not know the answers. I feel one of the biggest
challenges in IT can sometimes be working with groups within and outside
your organization who expect certainty. Even if you don't know the path,
learn the goal and begin the journey.
By pushing communication from the start, everyone gets a better feel for
James others needs and how they do their jobs. Then they take those things into
Kenigsberg account while doing their own jobs working not just for themselves.
Annual revenue:
What kind of culture existed in your organization before you
"$83 million for year ended
transitioned to DevOps? Did it change? If so, how did you facilitate
12/31/2013."
that change?
"To achieve Is there a specific approach that will make the transition to DevOps
less painful?
true DevOps
The team that trains together stays together your teams should
collaboration, constantly cross-train across various departments. Your developers should
you need your know how QA departments work, and your IT operations group should
be aware of the needs of the developers. Employees in general must be
employees to aware of everyone else's needs so they can work in silos but still have the
transparency and communication necessary to be productive.
really think and
act as one, not What DevOps myths (were too big, were not in Silicon Valley,
what we do isnt scalable) have you personally dispelled in your
just be merged transition to DevOps?
Align your tech department with your business needs. We have areas
specifically aligned with the business (those responsible for learning tools
are in Learning Systems; those who run the backend are on Business
Systems).
Each area is fully loaded with a full stack of every role needed for a full
software development lifecycle. Because employees are aligned to the
business mission, it makes them not just people completing tasks, but turns
them into a nucleus of your company to achieve your mission. Its not about
the amount of code they write or tickets that they close; it becomes their
goal to achieve the mission of the company.
Was there a particular point in time that you started doing DevOps,
or has that always been part of the culture there?
No, it has not always been part of the culture. When I joined the company
a few years ago, we were 400 people. In the last two years, we have
grown to nearly 2,000, so the growth has been substantial. That growth
has been matched by growth of our infrastructure. So the original way the
infrastructure had been growing before I had come on at Riot, Ops was cut
off from developers and the rest of the organization, and there were a few
Jeff Hackert intermediary organizations that became primarily responsible for the build
of our software and a team that was responsible for deployment. They
Engineering were not organized together in any meaningful way, and they were not
Manager communicating.
Riot Games Was there a particular catalyst at Riot Games that got you started
down the DevOps path, or how did you come to do things in a more
collaborative way?
Location:
Santa Monica, CA (HQ)
When I got to Riot, there was a culture of certain personal heroics in
Company snapshot:
operations, but the cost was quite high and the resulting silos turned into
Riot Games is a game studio
some real challenges. The team that I started working with that really
that develops and produces
had a lot to do with this change was working with infrastructure; they
multiplayer online games,
were working specifically with Chef, but it was a small team of software
including League of Legends.
developers working to implement technology that both developers and the
operations team needed to participate, and there was no scenario in which
Size of organization: the silos could remain and we could reach any kind of definition of success.
App. 2,000 So this idea of building up or bringing DevOps into the organization was
just a natural extension of those activities.
KPIs tied to DevOps:
"We have KPIs for any How did you facilitate that change? How did you get people to come
endeavors, but we dont think out of their silos and be more collaborative and communicative?
about projects with that
moniker. We think of DevOps I will say that I dont know that, from my own side, I can claim responsibility
as a culture topic more than for having achieved those things. What I can say is that I definitely brought
as a class or designation or that conversation out into the open and forced real reflection around their
project." practices simply because I was standing in the middle between multiple
parties and had the opportunity to force that conversation.
"We had to engineers would not log onto a single box where there are a thousand
boxes that all shared a set of attributes. We would not log onto a single
flip the model, box in production and alter its configuration, no matter how beneficial that
change seemed to be. Instead, we would focus on cookbooks or recipes
from the idea of and the software expression of that node, or that group of nodes, as a way
and siloed Do you feel that you are doing DevOps? Would you describe the
environment, the culture, there as a DevOps shop?
responsibilities
to real, cross- Yeah, after nearly two years of very active conversations, we no longer have
what you would call a traditional Ops organization. By any sort of objective
functional measurement of what DevOps is from a culture perspective, we lie in that
model. But we do not refer to ourselves as doing or being DevOps.
collaboration."
The people who are doing DevOps often arent saying that theyre
doing it. So thats definitely in line with what weve heard from other
people.
What needed to happen for Riot to get where it needed to go, to be able
to manage at the scale were running at, which is pretty big, we would
want not to constrain artificially the creative intelligence of the people
working in that system. We want the purest expression of peoples creative
intelligence.
We want to honor and respect their abilities, not demean them by putting
artificial labels on practices they should define and hold. However, we do
That they have faith in their hiring practices, and that they have real
empathy and compassion for the people theyve hired. When you execute
on a really good hire, which should be all the time, every person you bring
in is bringing a set of experiences and a set of goals that they want to
achieve. So you dont want to stand in their way. What you want to do is to
guide or facilitate that creative output to solve the real problems that your
business is facing. If were talking about operations and infrastructure, the
idea of DevOps unlocks that, but it is not a set of proscriptions or
a recipe that will give you a defined best practices outcome.
Things like best practices will probably end up hurting you more than they
will help you. Your best practice is the collective wisdom of the very smart
people youve hired to solve your problems. And if what you have is a
siloed or broken environment, you want to increase the communication, not
impose new rules.
And do these teams work seamlessly with each other, or do they each
Bharat Krish have their defined responsibilities with no crossover?
Vice President, Theres a considerable amount of overlap. Its a startup culture, and a lot
Information of times people roll up their sleeves to get things done, so you might find
Technology my software developers who develop a product supporting the product
as well. If they dont have involvement in support, they may not know the
HBO context behind what they're buildingthey wouldnt know how to enhance
the product. So its intentional that I encourage the culture of overlap.
Latin America
Did that overlap exist before, or is this something youve
Location: encouraged? How did that culture come about?
Coral Gables, Florida
Its a cultural aspect that I continue to enrich all the time. In a large
Size of organization: organization, silos are built naturally. Part of my job is to break those silos
App. 800 on a constant basis and establish that overlap.
KPIs tied to DevOps: You talked about the developers being involved with the support
"None specifically
of the product. Does this lead to better end products and
tied to DevOps."
faster improvement and therefore an increase in your customer
satisfaction?
The nature of our business is that the products are evolving so fast, we see
it as building a quality product that meets the needs of the brand. In the
case of HBO, a premium entertainment brand, it needs to have a premium-
quality experience, whether its user experience or graphical representation
or content. Were consistently striving for a quality that will be measured on
how the audience lives the brand.
DOCUMENT DATE: AUGUST 2014
THE DEVOPS MINDSET REAL-WORLD INSIGHTS FROM TECH LEADERS 13
"In a large Do you feel that your developers, because you give them that
ownership, are able to live the brand and really care about it?
organization,
We want everybody to be successful, and thats a baseline, but mistakes are
silos are built encouraged. I always tell my team were not running an emergency room
naturally. Part here, so don't be afraid of making mistakes. It takes multiple iterations to
build something that has the quality of our brand written all over it, and it
of my job is to also takes multiple iterations to build something thats simple.
break those silos Its in the DNA of most of my team members that they are willing to take
that step and propose ideas and not have the fear of people thinking that
on a constant the ideas may be stupid or not encouraged. When I think of a DevOps
basis." culture, for me its building a foundation where development can happen in
a more agile fashion, so I focus on the foundation more than anything else.
Multiple things. Just a week ago, we did a workshop with all my managers
and part of that workshop was to reiterate the culture that we need to build
and maintain within the organization. I look at it as five things: Keep things
simple. Focus on managing people, supporting them, directing and training
them. Be humble. Respect everyone. Have integrity.
By creating those core values, the expectation is, we will have a lot more
people talking to each other about new ideas, not afraid to make mistakes
and being honest about the work. In this sense its going to reflect in a
great product, in tackling a lot of the competition in our industry or how
consumer behaviors change, being able to think out of the box and come
up with solutions.
Thats the culture were promoting, and its working. There are times we do
have things that we need to fix, and new employees bring a lot of baggage
with them that needs to be reset. Its a work in progress, but overall from
my experience working with multiple companies, Ive cracked the nutshell
on trying to be more innovative because of the cultural values that weve
implemented.
region. Latin America is very different from the U.S., which is very different
from Europe, which is very different from Asia.
As with the tech experts portrayed here, youll have your own reasons for
moving from a siloed IT department to an agile DevOps model. You might
recognize you need to achieve greater efficiency or want to reach new
levels of productivity. Whatever the motivation, your companys culture will
be the deciding factor in whether DevOps will be successful for you.
ACKNOWLEDGEMENTS Rackspace would like to thank all contributors for lending their
valuable time and insights to this project:
Kevin Behr
Chief Science Officer
Praxis Flow
Co-Author of The Phoenix Project
Jim Kimball
Chief Technology Officer
HedgeServ
James Kenigsberg
Chief Technology Officer
2U, Inc.
Jeff Hackert
Engineering Manager
Riot Games
Bharat Krish
VP, Information Technology
HBO Latin America