I was always (well, not always, once I'd proven myself) a senior admin/engineer, now I focus almost purely on AWS automation, and it bums me at out how many admins are completely happy with just being Next button admins who are absolutely AFRAID of touching a bash or zsh shell, or even a Cisco router without the UI.

I suppose it shouldn't bother me, it's done great for my career.

I would disagree with this point. If you're installing anything on a production box you really should know what that installer is going to do to your environment. Have you walked through the MS SQL Server installer wizard? You absolutely do need specialised knowledge if you're running that on a client's server.

SQL Server requires vastly less "specialized knowledge" than, say, Postgres. I would be surprised to find that most SQL Server installs come with someone with really in-depth knowledge.

The same applies to a lot of other things MS ships, like setting up AD. Sure, knowing what you're doing helps, but MS sorta built an empire on making a lot of that stuff pretty trivial to do. For another example, look at setting up IPSec. A cute little UI you click on, versus well, the process on Linux.

Or what accounts to run SQL's various services under and wonder why stuff doesn't work or their SQL box gets hijacked ("oh I'll just run this under the SYSTEM account").

The install wizard for SQL Server 2008R2 Standard Edition has around 20 pages (counting tabs on a page) of user selectable options and settings. You certainly do need to know what you're doing and understand the implications of each every one of those settings.

Sure if you're just doing a quicky install of the Dev Edition on your workstation you can afford to be a bit lax and click on through, but on a mission critical production box you need to RTFM or let a SQL admin who knows what they're doing perform the install.

I've been called in too many times to unf*ck a SQL server because a customer has declined one hour of installation support because their devs thought they knew better.... "oh yeah it's just clicky pointy install, why do I need help with that". Trust me GUI installers do not equate to "easy install" if you have no clue what the installer is asking you.

I'm a SQL (MS and MySQL) admin, amongst many other roles, for a webhoster with 19 years MS SQL Server experience (started on SQL 6.5 in '97), been there seen it all.

I've run SQL Server in life-critical production systems (911 call routing) and found it to be very straightforward. Just make sure the alarms are turned on and enabled for replication.

I've run it in HA clusters, both with a shared DAS and with the new shared-nothing system. A walk in the park. The only difficult thing I've personally run into with SQL Server was figuring out the Broker Service stuff and dealing with replication outside of a domain. And that was probably mostly networking configuration issues - the config wizards are pretty amazing.

I'm not implying that there is no specialized knowledge, and many SQL Server installs are probably suboptimal. But it's far from necessary to know much to go very far with SQL Server.

Compare to Postgres, where doing anything correctly requires mucking around with the config file, and, last time I used it, logging into the shell with a special user. And HA, well, good luck.

I may be the only one around though that doesn't find config files to be 'mucking' or a form of mysticism that all should kneel before the user who can wade through them. That also, is not 'specialized knowledge'.

I'm primarily a developer, but I'd love to understand networking/low-level OS concepts like these in greater detail. Every time I start with a tutorial, I just get overwhelmed with how much I don't know — is there a book you'd recommend for a developer getting into devops (advanced networking, low-level OS architecture, configuring a machine from scratch, etc.)?

For OS, I would recommend looking at Bach (on the design of UNIX) and Tannenbaum's books.

Microsoft revolutionized personal computers because they made things easy and cheap. Their stuff was meant to be used by the average joe, not a greybeard who'd survived the Unix Wars.

At the same time, IT is always seen as a "cost center" to be reduced and outsourced. Unix? Too expensive, replace them with Windows, and if there's a problem, re-image the system.

The sysadmins you describe still exist, but they're called SREs by Google or who knows what elsewhere. Today's "sysadmin" is an outsourced phonecenter guy in Bangalore.

If you as a business or employer want to continue to squeeze additional roles and knowledge into people, be prepared to pay for it.

Presuming this is USD, but even in the UK as GBP - "In my dreams" - and I'm a developer as well as a sysadmin for Windows (IIS/SQL) and Linux (Apache/MySQL), 25 years man and boy but have always kept on top of my game :)

Outside of London you'd be lucky to hit 45K GBP (~70K US$) in a permy role with that skillset, perhaps 60-65K GBP if contracting.

If you know of any place north of Chicago that pays 120K or more for a 20-year Unix/Linux engineer with C systems level programming talent, and won't burn you out in 2 years like the trading firms downtown, I'd be interested.

When I worked for a large Telco one of the reasons they employed me was to have an in house sysad for our Primes - we had the largest cluster in the UK running MR back in the early 80's

I rember one of my IBM mates at the time being horrified that we allowed to write all our own JCL

Its like the "Agile" push that companies thought they could get by buying a tool, and changing nothing else (release schedules, feature based release dates etc).

My favorite way of looking at it is simple - replace the word DevOps with empathy. That is the core of the idea, so lets try it - "Empathy Team", "Empathy Person", "Empathy Engineer".

DevOps is about dev and ops teams communicating and having empathy for the requirements of the other team.

Most posters here don't seem to get it. Most replies appear to be the idea that DevOps means you fire your Ops people and the Developers do Ops previous job and their own Development job as well.

It caused the dev team to implode (everybody but one person on the 8-man team left within a year). The systems' stability and overall architecture was utterly terrible, servers would fall over because they would log so many things that they'd run out of disk space ("what's logrotate?" "What's a slave database?"), no metrics, no definition of success for a service, etc.

I suppose you could pull this "DevOps" thing off if you really wanted to, but you have to hire folks who can program and actually like sysadmin work. Otherwise you end up hiring developers to do everything, who are terrible at sysadmin work, and who get burnt out because they never get to actually program.

DevOps is not get rid of the Ops team. DevOps is not get rid of the dev team. It is get Ops and Dev to talk and understand each other. Ops should not be completely clueless as to how the developers workflow works. That doesn't mean they are a developer. Development should not be completely clueless about what happens during deployment but that doesn't mean they're on call to handle OS issues.

DevOps is these two teams are working for the same company and same goal, so damn well talk to each other and act like it.

Hint: In a devops work flow, if you were large enough to need a separate ops team, you still have a separate ops team.

Previous job had a "Systems" team and a "Database" team. Both were just teams of obstructionists.

"Oh you need something today? Well, if you get a jira ticket made I could look into getting the requirements written up sometime in the next 2 weeks, to get the work assigned out next month."

Not saying that's the norm, but damn I hope I never experience that again.

My favorite was the marketing manager who thought an email, with a full size image attachment, and note saying 'this needs to go on the marketing site' was sufficient requirements.

Requirements and prioritization are not a bad thing. Excessive bureaucracy is.

  Both were just teams of obstructionists.
  "Oh you need something today?
    ... get a jira ticket made 
    ... work assigned out next month."

In my current job, I am the sole developer responsible for maintaining a certain set of our codebase. I have multiple internal customers, each of whom have separate (external) customer-facing needs. I have enough tickets created for three people working full time, so I am constantly having to triage which are the most important.

Part of the solution to this was to have a proposed plan for when to work on things -- I can then say, "Thank you for this well thought out ticket. I will be able to work on that in June." The most important thing is that my users' opinion about the criticality of things does not always match the overall picture, and I am always having to choose which tickets to postpone.

I wonder if one can draw parallels to the BYOD thing.

In both senses it seems to boil down to someone bypassing the IT admins because their manager wants to see results ASAP, and IT is going "sorry, budget and/or regulations says we can't at this time".

It seems to have stemmed from Ops teams saying "What can we do let dev teams build what they want, but in a way we can support it"

e.g. - a system needs an entry in the /etc/hosts file (a bad example I know, but a nice simple one for the basic use case)

old method: Dev files a Jira ticket, and the Ops team make it part of their golden images for production.

new method: Dev makes a change in the Config Management DB. Gets reviewed by a member of Ops, who either approves, or makes a suggestion to change it. Change gets tested against staging, then merged, and the system gets updated.

Ops know what the system needs, Dev knows how long it will take, and has an understanding of how it is deployed in real life.

Applications used to run on a fairly static number of instances that were maintained by an operations team. When more capacity was needed, operations would provision new machines. But now, especially with the increasing shift toward the cloud, infrastructure is seen more as something that's created programmatically, or even in response to events (load or otherwise) in the network. It's a powerful conceptual shift to think of infrastructure as something that isn't physical, but is more ephemeral. Just like a process on a server may die, a server on a cloud hosting platform may die. But you can engineer for failure at the infrastructure level the same way that you'd engineer for failure at the system level.

With this change, you can no longer rely on a fleet of sysadmins to manage everything for you. Instead, you rely on scripting everything..."Infrastructure as Code." You create recipes to create each and every part of your infrastructure and tools to orchestrate everything. Gone are the days where you actually ssh to boxes in a terminal...there are far too many boxes to make this practical. Instead, ssh becomes the protocol that your orchestration uses to communicate with its fleet of machines. Things are much more dynamic and you start to view your infrastructure more holistically rather than as a collection of individual machines.

Once you've started to think this way, you realize that neither the traditional development mindset or the traditional operations mindset work in isolation. You need to meld the two together so that operations understands better how to automate everything and development understands more about the environment in which their code will run.

The problem is in most orgs, the roles for people who fight fires in production tend to be demoted to a lowly "support" title, and paid/hired for accordingly, which is really not commensurate with the value they provide to the organization.

We were understaffed, underfunded, and underpaid.

Exactly, you have to fire the developers too.

DevOps is about not having separate dev and ops teams (or QA for that matter). It's about having one team that coordinates on implementing user stories and comes up with the right solutions. Some members of that team have certain skill sets - i.e. some are more ops focused and some are dev focused - but it's one team. It's about creating personal accountability; there's no more blindly assigning tickets to an offshore ops team and hoping it gets done right. If you're doing DevOps right, you developers and your operations people are on the same team, sit in the same area and attend the same scrums.

DevOps means that instead of saying "Ok, my work is done, but the deployment scripts need to be updated so I'll create a ticket with the ops team and they'll get it done next week", you can say "Ok, my work is done, but the deployment scripts need to be updated and I don't know how to do that, so I'll assign it to Mike." The feedback can go the other way too - if an implementation of a feature is too cumbersome to deploy, the ops guys are in the scrum with the dev guys and can work on a better design together, up front.

This scheme also works with diversity, quality, best practices, and change management.

A lot of us old sys admins are now calling ourselves devops because times are changing. To me devops isn't about taking a developer who likes ops to save some money. Its about taking ops out of their walled garden and into the development process. As someone in ops, I'd much rather be part of the development process as I'm the one supporting the code in production.

If its a choice of message queues.. I'd rather have a vote to the one I know about supporting.

It isn't really about cost savings as some companies I've been in I'm one of the highest paid engineers on the team. Its more about bringing ops into the process.

You still then have a friction point between "dev" and "ops" relating to hand off. Ops could go full PaaS, but more than likely you'll end up with a "cloud" infrastructure where your dev team will have to learn ops, or the traditional model where stuff gets thrown over the fence. Good luck deploying that service because there was no capacity planning and it'll take 6-8 weeks to get hardware from the vendor, regardless of whether ops can rack it and build it in under a day.

It's far better to have an org overlap so that ops is involved in dev planning to help address roadblocks, assist in capacity planning and management, as well as help reduce friction in tasks.

Before operations lived off tickets which just said what I did not how long it really took or if my estimates were off.

It may sound like that's Dev's job. But they're busy maintaining and detangling the codebase, and handling daily code deploys. Meanwhile Ops acts as a liaison with our hosting company, installs 3rd party software, performs some break/fix work in the environment, and does corporate IT work.

It used to seem crazy, and it probably looks crazy from the outside, but it's still working.

> It’s easier to train a developer to do sane operating procedures than to teach a sysadmin the developer mindset (testing, revision control, breaking down to logical units and similar).

Having done both jobs, I don't really think that's true, unless by 'sysadmin' one means the type of low-skill position which resets passwords. A system administrator is responsible for administering a system of hosts and software; his should be a more complex job than a developers. He should be responsible for test systems as well as for production systems; his scripts should live in source control. I'll grant that functional decomposition may not be as vitally important to his work as it is to a normal developers—but then, his scripts will inevitably grow to be pretty complex, so maybe it will be after all.

That doesn't mean that developers are stupid either. Sure, at BigEnterpriseCorp a developer is just a seat-warmer who implements directives like 'add a method to the Foo class which takes two arguments, adds them and returns the result,' but a real developer is a student (whether formally taught or informally self-taught) of computer science; he understands and can choose & implement both algorithms & data structures in order to solve a problem.

A good DevOps hire is going to have some skill in both areas, not as a weak sysadmin and a weak dev, but as a strong sysadmin and a strong dev. A good DevOps team is going to be more skilled (and expensive) than the equivalent dev and ops teams, but it will also be more productive and less error-prone.

If you're trying to build a devops team from Initech, Initron and Initrol low-skill employees, you're probably going to fail. If you're building a devops team to reduce costs, you're probably going to fail. But if you're building a devops team to increase profitability and competitiveness, you may succeed.

In the Windows world, a lot of the Sys Admins do not have those skills. It's changing, things like PowerShell and various others are helping, but a lot of Windows Admin are just not that comfortable away from the GUI. I think the comment you refer to is by someone with a Windows dev background. GUIs don't version...

The mainframe world folks tended to be very silo'd (and IBM engineers were never far to hold hands.)

DevOps, when implemented correctly can have enormous, order of magnitude increases in both productivity and quality, but it is not voodoo, despite this almost unbelievable claim. It involves empathy and cooperation and it requires this because in order to implement DevOps in a meaningful way, an organization has to vastly increase communication between "silos". It has to formalize, document and automate as many processes as possible.

DevOps is NOT easy to implement and involves a cultural shift and absolute management buy-in. It is best implemented in small projects, where clear wins can be demonstrated. Use this success to bootstrap the process to bigger projects.

I do agree with the OP that if you have a DevOps team, you are probably not doing DevOps. The whole point of DevOps is to get rid of silo mentality. No more us and them, but all us.

Implemented correctly, everyone is DevOps.

I've also had more or less the exact same experience with DevOps, though with (hopefully) a happier ending. I was on a team working for a client that desperately needed "DevOps-type" things. So I built them, and their infrastructure team ignored them.

Eventually we stopped working with that client, and I was moved to a different organization, where each development team was doing it's own thing, operations-wise, and had no interest in adopting a uniform approach, so my (short) efforts there were also for naught.

But then I had the opportunity to come back to the first client, as their DevOps Manager. Before I took the job, I talked to their current development organization about what their pain points were, and what actions they would be willing to take to alleviate them, and I built a strategy around that. A key part of this strategy was gradually eliminating the need for a DevOps team, as I firmly believe the best application of "DevOps" is as a joint project between Development and Operations to optimize shared processes. Once the friction is minimal, you don't need a team there anymore. So we have a developer embedded in our meetings, and have more or less full control of any infrastructure.

DevOps is not a department and it will not work if it is a department DevOps is a way of doing things to achieve maximum efficiency and everyone on development and operations must embrace it for it to work.

If either development or operations resist the change then either the resisting members should be eliminated or the switch over to the DevOps way aborted.

DevOps doesn't work without cooperation and effort from all team members.

In other words anything that can have a line of communication and still work can stay as it is but in order for DevOps to work development and operations must essentially become one with both developers and sysadmins working together on architecting a flow that works for both of them.

One office and one project is not really a requirement but it helps a lot in the beginning however if you can keep the cooperation a culture alive even working remotely could work.

The key is people who are always willing to learn and try new things and putting effort into change and staying on the same page with their teammates.

1) cost.

Ops are cheaper than devops, sometimes twice as much. Yes yes, flexibility. But thats a misnomer. If your dev teams are doing donkey sysadmin work, because they don't have the experience, or are up all hours fixing things, you're going to endup with a high staff turn over.

2) division of labour

As a company grows, they can employ more and more specialist staff. This gives them a competative edge as they are able to make nuanced decisions based on skill, or experience. A decent storage admin will be able to build a storage system that will scale relaibly, cheaply and with less downtime than someone who is not a specialist.

3) burn out.

A human is not designed to multitask 14 hours a day. Coding, testing, supporting, infrastructure procuring, for anything other than the smallest company is too much for a single role.

There are great things that come from a devop "culture"; greater personal autonomy. However like all things, its not one size fits all. There is a great deal of arrogance where people assume that a sysadmin never automate anything. or that they can't program.

They can and do, you know why? because they want to carry on searching ebay, scamming $someone boasting about network speed, or just plain getting shitfaced.

0: http://www.theregister.co.uk/2015/02/27/bofh_2015_episode_2/

The idea is that a good sysadmin is an idle sysadmin. (having offloaded tech support to the juniors)

unless they are playing with new stuff.

When I work with clients the inevitable question comes up: "What Principles should we implement then?" and the answers are usually more questions really. How long does it take developers to set up their working environments and get to coding? How is code tested and prepared for deployment? Are stakeholders feeling secure about deploying this new code to production users? Can we deploy effectively every time without deployment related issues? When something bad happens in production, what is the recovery procedure and how long does it take?

The answers to these questions and many others often show that there are a few issues. First, developers and operations aren't speaking to one another enough and aren't really working together. Second, there is very little trust in the current environment and that lack of trust is made up for by adding more human intervention. Third, there is very little automation and this causes the lack of trust but few people what to change their habits to fix it. And finally, there is such a great divid between engineering and operations that the pains felt by one group and not shared by the other.

DevOps has the principles needed to start to solve these problems. However, we don't need a department to implement those principles. We have enough departments.

As an employee of a large software focused corporation that does many of these things right, I had no idea how bad things can get for smaller, newer companies that haven't figured all of this out.

Dedicated QA teams should be avoided unless truly necessary. Otherwise, you have the same problem of developers throwing over under-tested code to a group of separate people that are smart but treated poorly.

A QA guy who loves to write automation gets the same promises... "oh you'll be programming mostly; manually test maybe once a month". But when they get into the door, it's the exact opposite.

Developers should all be passionate about QA (read as: the quality of their product), as well as how their product is actually deployed (read as: executed in the most important of environments--production!).

'Do not set out to create a DevOps team or employ people for that role'

My interpretation was the role simply doesn't exist. If your company has a team of developers and a team of people who keep the wheels turning - in an ideal world you already have a DevOps team as long as they collaborate. If you also employ smart people you will get 'developers that know how to do sysadmn' and 'sysadmins that know how to develop' - definitely this indeed should be embraced but not expected.

"Devops means giving a shit about your job enough to not pass the buck. Devops means giving a shit about your job enough to want to learn all the parts and not just your little world.

Developers need to understand infrastructure. Operations people need to understand code. People need to fucking work with each other and not just occupy space next to each other."

http://blog.lusis.org/blog/2013/06/04/devops-the-title-match...

I cannot disagree more. You can't have developers of your product knee deep in DevOps and be effective (at a certain size, smaller teams with less demanding operations needs can).

DevOps should have very clear demarcation of responsibility in an organization. What the OP describes is an blending of responsibilities and that is the root of his problems.

DevOps /is/ operations. It is just that simple. It is design and automation of support systems. Nothing more, nothing less.

As someone who works on the dev side, the ops side, and the mixture of the two I feel the OPs pain acutely and have had mixed experiences. People on the dev side need to understand and willing to learn about ops, it's not rocket science and you don't have to be knee deep in it to get the basics down. When ops becomes this black box and becomes a bottleneck is when you start to see dysfunction.

It is taking more of a gestalt systems approach to the entire support structure. This means more intelligent support programs and treating it like designing and delivering a software project (or, more accurately a series of subprojects) as opposed to a loose amalgam of scripts.

This also requires the product development team to be keenly aware of that support architecture and how their decisions factor into it especially in terms of testing, deployment, and module interactivity.

> Ops teams have been doing automation of support systems for a while, so what exactly does it mean to be DevOps?

DevOps has been around for a while -- way before there was a word for it. I think DevOps has come into its own, recently, though because platforms have become hugely more flexible while at the same time a consensus built on what it means to manage those systems. The difference, I think, is whether things are built to remain somewhat static or if change is designed for. Are development ideas like encapsulation and decoupling etc designed into the system? Smart sysadmins have been doing this stuff for a long time but now organizations are taking the approach into account all the way up and down the chain. And, yes, that means that product developers need to be fully aware of operational needs but, in my opinion, I don't think that means they build the operational tools themselves.

>your fancy 'DevOps' team just ends up being separated and isolated from the dev team you're back at square one

Demarcation of responsibility doesn't mean demarcation of information. The product development team will need to be intimately aware of what is happening in operations and vice versa. In a DevOps implementing environment it is unavoidable.

> the dev side need to understand and willing to learn about ops, it's not rocket science

These days it's getting pretty close. I don't think it is fair to product developers to say "Listen, we like the idea of what Docker can do for us so you are going to learn it inside and out." Somewhere you have to say "Hey, we probably want to implement Docker. Ops, dig in. You know how our product is currently structured and how operations currently works with that. Try to figure out what needs to change and what doesn't. X and Y from the dev team will be made available for deeper discussions and testing as needed. When the overall gist of what has to change is known, we'll all get together, ops and produce dev, to get everyone on the same page." It's a bleeding of information, not a full on 'shake it all up' scenario.

Neither side can be a black box anymore if the current state of software development and delivery is going to be taken advantage of. But neither can everyone know and do everything or context switch as needed.

Keep in mind: I am not a DevOps theorist -- maybe I have it completely wrong according to the people that came up with it but this is my practical understanding of the movement.

I disagree; although there's a huge value to consciously automating and designing support/deployment systems, there's even more value to developing products to be deployed ab initio.

You need automated production systems, and you also need developers who understand how their code is deployed and run. Really, you need smart, productive people, not interchangeable Java monkeys.

Design and automation of any systems is "development". It is just that simple. Nothing more, nothing less.

So, from your description "DevOps" is really development whose application domain is operations.

(Though I'm not sure I agree with your description -- I think that's part of DevOps but that DevOps is somewhat broader; in the same way that many approaches that fall under the general agile/lean umbrella have pushed erasing role-based team boundaries and, even within the lower level teams, role distinctions within the broader development realm -- e.g., between programmers, analysts, testers, etc. -- DevOps is about erasing, or at least blurring somewhat, the team and role boundaries between development and operations; this doesn't mean that people don't have distinct competencies, but it means that people with those competencies work together at lower organizational levels and in a more tightly integrated way focused on overall mission, rather than throwing things over the wall at each other.)

I think the author's story is more an admonition of blind foolishness and deceit than it is an indictment of the automation.

I say this because I've lived it...the last 3 (three!) companies I've contracted at were pushing dev code to live servers! Even after I setup svnserve and created the repos, no one was really interested in changing workflow. I choose svn because, lets face it, it has a much lower learning curve then git.

It wasn't until I absolutely refused to work on the live server that other the devs grudgingly started using SVN...I'm sure they are all back to FTPing to the live server now that I'm gone.

Without the proper culture from above, things just aren't going to change.

My job, just like it was when I was a sysadmin, is to keep things running smoothly. To optimize both sides of the coin and to take things off their plate that I myself can deal with that might be a blocker to them finishing tasks and to us getting features out or server migrations done.

Some devs know ops and some ops know dev, those guys, more often than not I just check in with and see that things are running smoothly. For the other folk I try to create systems and workflows that get blockers out of their way.

I try to be a useful and reliable tool to both sides as much as I can be.

Hopefully the way I've been doing this for the last two years since I moved into a more Devops related role has been helpful to everyone.

http://en.wiktionary.org/wiki/copacetic

I remember them fighting (mostly arguing) and the men going "Ey ey, lets keep things copacetic." (They might have said capasetti)

I always thought it was a pretty and a smart sounding word. I guess not.

Sounds good/smart to me! The wiktionary page mentions possible Italian-NY etymologies so you fit in there. Funnily enough I was going to try and be clever and guess you were a New Yorker [based on that] but noticed elsewhere you appear to be in TX.

Interesting, thanks for responding.

In theory that will eliminate the need for an additional tester and a second individual to do the change control.

But those short term gains expose the organization to the massive risks that the one and only one individual will make a mistake, with no second individual available to do a double check.

The reality is people make mistakes and by eliminating people from that double checking process may well save money in the short term, but in the long run it greatly adds to the of risk failure.

While there are short term gains, the risks to the organization can be massive:

http://www.bloomberg.com/bw/articles/2012-08-02/knight-shows...

The person deploying the software had no idea how it worked on an internal level.

The person who wrote the software had no idea how it was actually deployed in production.

They didn't have a quick rollback in place (this is the exact reason you have quick rollback ability)

The deploy was done by hand, with a user typing. That is probably the biggest risk you can have in a production system. It means that fully repeatable builds in test & pre production are impossible.

I am not saying "DevOps" would fix this, but a bit of empathy / comunications between the dev team and ops team might have averted it.

http://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stock...

My definition of devops: small teams totally own their project, iterating on design, implementation, deployment. I have no experience with one giant devops team for an entire company.

My experience is different: DevOps saves me many headaches by letting me easily deploy things for small clients. A larger company I work with uses Chef to manage their 40+ servers; the IT department doesn't touch these servers. Rather, 3 or 4 developers share in the work of writing cookbooks, adding instrumentation, deploying code and maintaining the servers. It would be handy to have a full-time guy as well (OP mentioned the amount of work needed to keep up with just your basic security patches) but we -- as developers -- love the access we have to our entire stack.

I don't think it came through in my post, but having a security expert (or any specialist) in the mix can be a huge help in making the product better - I know this from personal experience - you just have to make sure they're integrated, challenged, and respected in the group.

My take is that the code quality of whats allowed to be in production has gotten worse, that's it. It's an effect of an abundance of RAM and virtualization. The abundance of ram lets you just trash it and do things in a horribly inefficient way. Sure the extra memory can help with time memory trade off if you have something that doesnt need to scale a great deal, but processing things inline will still be faster. Developers have forgotten how to handle streams. Usually forcing a dev to learn a highly functional language that's immutable can fix these issues (it must involve a large dataset, where you know they will tax out the memory, and then code review to help).

I mention virtualization because a common pattern I've seen is to write poor code, and just put more boxes behind the load balancer when things start failing because the technical debt is so high it's an impossible mountain to climb. This obviously isn't the right approach and will leave your devops guys worn out chasing your everyday distributed systems problems that happen because it wasn't built right for distribution and the dev has never read the fallacies of distributed computing.

And while you're fixing this technical debt a team of competent engineers builds a competing product and trumps you hard (second mover advantage).

Maybe people paid a bit more attention when things cost 10x as much as they do now and every answer (both bad and good, and you can't tell the difference) wasn't a stackoverflow search away.....

Although in his post, he says:

"Another lesson learned: you can’t force cultural change. It has to start from the bottom up, and it needs breathing room to grow."

I would argue that cultural change has to also start from the top. You can't change the culture if the leaders of the company don't work with you on it.

The tendency of people at the bottom is to eventually start behaving like their managers as the managers project what is acceptable and not acceptable behavior. Trying to start change from the bottom up is a hard, hard thing to do, with a low probability of success if senior management isn't on board as well.

This is where the "breathing room" comes in. I think that fostering an environment where people are free to organize themselves without edicts and policy produces overall better processes (and ultimately better quality).

I think it's OK for management to lay a framework, provide a particular kind of training, but it's best to provide goals that forward the company, step back and let the people closest to the work decide the best way to achieve them.

I was part of a successful transition to devops from a thoroughly silo'd operation, where the boss basically gave the dev team the responsibility and authority to make it happen. He was also able to snag some of the traditional ops team and make them part of the new team. After a period of storming, the new team came up with some innovative ways to get the job done. Once the team realized that they were it in terms of getting it done, it got done. Quality and job satisfaction improved, cost was reduced.

I've also seen various unsuccessful transitions. In every case, the responsibility was divided into various silos, which sandbagged everything.

The memory bandwidth in one single person's brain is simply much higher than the bandwidth between people. To maximize performance you want to run your project on as few brains as possible.

In short - its not a "one solution fits all", and it is something that needs to be discussed with you SOX auditors, to find a working solution for your company

I used to be able to use "senior systems administrator" as a short-hand for those skill sets, but about 2010 that stopped working.

Now I need to use "DevOps" as an imperfect substitute in the title; otherwise only junior people apply, even though the rest of the ad goes into the same depth about skills and abilities.

What you want to look out for is the formation of "walls." People have an instinct to do this. People will form cliques -- which effectively are invisible barriers to understanding and honest collaboration.

In a 5 person startup, this is easy. You have one group. Most of the critical information flows are inside this boundary, and those which aren't, the group is already intensely aware of. (Investors, customers.)

However, when headcount has started to grow, and when you are forming a separate group to take on a task that requires intense collaboration, maybe you're doing the wrong thing? Maybe what you want is a separate role for people who get embedded in a variety of different groups? What if DevOps people belonged to DevOps, but also, and primarily, belonged with a particular group of developers? Then, they could have firsthand experience of intense collaboration, but also meet once a week with all the other DevOps role holders and communicate to standardize and avoid duplicated effort?

A generalization of this: Make sure you're building bridges and not compartments.

While I agree that sysadmins should be developers first it is increasingly less likely you'll find developers willing to become computer janitors and deal with the ugly realities of operations in, for example, larger companies. I wouldn't think that John Carmack ever liked having to figure out the installer scripts and keep them working across all the supported platforms through all the BS that Microsoft and different Linux distributions could throw at you, yet that's what a more "ideal" DevOps organization described by many model would have John Carmack do.

What should have happened though is that instead of hobbling the John Carmacks and dragging down their productivity, you should have fired the developers that refuse to cooperate with operations and those that keep actively putting off writing "boring" code or never documenting things that do matter for operations to figure out and learn through a failure that something broke. That kind of behavior is the developer equivalent of not being a team player at all. It's true that none of these things actually make you a better programmer in itself, but these things are what contribute to making you more PROFESSIONAL instead of a cowboy renegade programmer that doesn't care if nobody else can use his code because "clearly, if you can't figure it out quickly you don't deserve to use it" (not an exaggeration at all I've heard).

Having been in both development and operations in rather large (and small!) companies now, I'd far, far, far rather stay as a hermit-crab developer than deal with the constant dysfunctions of enterprise Faustian nightmares that are required for you to deal with as an operations engineer. You can generally ignore a lot of the noise in a super-siloed off R&D organization, for example, but you cannot ignore your pager going off at 2 am because a random server crashed causing someone to get very, very angry and demand immediate resolution when it is your JOB to keep those things going. Hardly going to write anything really meaningful with that ruining your work week pretty much every week.

Many of the lessons learned here aren't exactly universal, for example some of the better Devops cultures (e.g Google) do have separate dev & ops teams, they just have good collaboration.

> We had edicts to make things happen, under the assumption that if we built it, the developers would embrace it.

Followed by:

> What matters is good people working together without artificial boundaries.

Former will never produce latter. Always work with engineering before building anything that impacts their day to day.

The exact same thing I'm noticing is happening with the "automation" teams that are cropping up. It's like folks are discovering ways of getting around process, forgetting completely why the process exists in the first place.

Sure, there's this general idea that development and operations teams should work together, or of a programmer also having some experience adminning systems or vice versa. But this is as much a job description as "non-domestic car mechanic" is a job description. Generalized ideas that don't say anything but an indication of the background expected of the person.

"DevOps" or not makes no difference; all corporations face the same problems with multiple segregated teams and people doing whatever they want in order to get their jobs done, as well as a sort of tunnel vision that prevents one from properly supporting your teammates or a product. To this end, there is only one thing that will make a significant difference in the work done, no matter what you call your team or position.

The best engineer in the world becomes a janitor under the wrong leadership or teamwork.

If you're struggling to read it, try https://www.readability.com/shorten

https://wordpress.com/themes/sela/