Surviving the Unexpected

Are your work tasks interrupted by continuous “unforeseen events”?

In this post: discover strategies to deal with the unexpected and still make progress towards your goals. 🎯💪🏽

The Development teams and especially Operations not only work to satisfy the clientele for which our products are created, but also serve other teams within our organizations. Areas such as Customer Service, Professional Services, Sales and even Marketing depend on the implementations we carry out to optimize their respective responsibilities.

The act of serving, of creating a “something” of value for other people, is rewarding. But, what happens when the service provided for progress is compromised by the “unforeseen” from day to day in any type of organization? Without realizing it, these “emergencies” — fires that we must put out immediately🔥 — consume gigantic portions of our time and attention, resources that are not renewable.

To answer this question, let’s first look at the types of work that are generated on a daily basis. In the novel “The Phoenix Project”, these are classified as follows:

1. Business Projects

                                                              Types of work

This type of project aims to increase the economic income of companies, increase the number of people using the services (engagement) or reduce operating costs. In short, any initiative that has a financial impact.💸💲

Some tangible examples are: (pro-revenue) Release of new versions of products, marketing campaigns, research on technologies for the creation of new offers, (pro-cost reduction) removal of unused AWS EC2 instances or reduction in their size, or the use of DeepMind — AI from Google — to optimize the administration of Data Centers as that company did.

2. IT Projects (Internal)

IT Projects or Internal Projects do not directly affect customers, but they do affect teams or departments within organizations, who contribute to the creation of products and services that reach customers. These projects have the objective of optimizing the productivity of those equipment.🏃🏻‍

Creation of infrastructure for the Development and Quality teams, design and implementation of architectures to support products in the cloud, creation of CI/CD Pipelines and process automation for user administration are classic examples in this category.

3. Changes

Any type of modification to one of the two types of projects mentioned above. For example: server configuration update, modification to previously established architectures, changes to previously automated processes, etc.

4. Unplanned Work

Unforeseen events, emergencies, last-minute requests that they forgot to mention previously, “fires”, urgent patches to Production or whatever you call it. This is the kind of work that we didn’t imagine we would have to do when we started our workday (#innocents), but for some reason we must fix it immediately or the planet will stop rotating. 😫🚨

Unplanned Work — Unplanned Work — is the killer of productivity in any company and even in our personal life.

Humans have a poor record on predicting the future. However, we plan projects with ingenuity, forgetting the technological jungle where we live, in which the predators of our projects are those unforeseen events handled inadequately.

The Myth of Lack of Time

“I don’t have enough time”, “I would like to do what you ask me, but I have a lot to do.” If any of these thoughts ever crossed your mind, you’re not alone. We “always” have a lot to do in our job responsibilities. I know! But what is the cause of this circumstance?

I’m a firm believer in the saying: “No person is too busy, it is only a matter of priorities.” Although it may seem funny, many times we rush to remove the water that tries to sink our boat with buckets, instead of covering the hole through which it leaks in. ⛵️🕳⛳️

Living in constant alert and stress trying to put out “fires” does not mean being busy. It means not having clearly defined priorities.

But calm down! The goal is to protect our attention from what distracts it. We already have too much with email notifications, chats and the distractions of everyday life. If we want to progress in our projects, it is necessary to safeguard our concentration as much as possible. For this we have these strategies:

Strategies for Contingency Management


Strategy 1 — Cushion the unexpected

                                                              Strategy 1


If the unforeseen are not frequent in your daily responsibilities, it is possible to plan the backlog of things to do (ToDos) of a Sprint, in such a way that we reserve a cushion for when the unforeseen rear their head.

This strategy is suggested when there is a technological and operational maturity such that incidents are the exception to a quasi-steady state. (If you’re one of those teams, stop reading this article and please write your own to tell us how it feels! 😄 ).

…If you’re still reading, let’s look at the second strategy.


Strategy 2 — Tier III

With enough resources, create a dedicated team of people to deal with the unexpected — Level 3 or Tier III Support — so that they are the ones who minimize the impact on project execution.

Tier III team members have technical skills and in-depth product knowledge, so they are able to deal with eventualities while suggesting improvements to the Operations and Development teams.

If there are no fires to put out (great!), don’t worry. For this and the following strategy, it is suggested to have a backlog of Improvement and Innovation projects (Internal Projects) on which they can work. The Tier III team’s primary goal is incident resolution, and the secondary goal is improvement implementation and Proof of Concepts about innovative ideas. #kaizen + #innovation!


Strategy 3—Hybrid

I know what you’re thinking. There aren’t enough resources to make a Tier III team (yet). OK. That being the case, let’s use a hybrid strategy between 1 and 2.

In this case, within our current teams it is possible to assign shifts so that one (or several) person(s) is constantly available only to deal with the fires that need to be controlled. This sub-team acts as a shield to protect the other members who work on Projects and Changes.

From experience, I can suggest rotating the people on this team every two weeks (a Sprint in Agile Methodologies) to refresh perspectives on incidents and, at the same time, because a couple of weeks is a reasonable amount of time over which it is possible to make a retrospect incidents, learn from them, and suggest projects to mitigate them.

                           Left: Strategy 2 — Tier III. Right: Strategy 3 — Hybrid


Strategy 4— Forget all of the above

                                                                   Strategy 4


When none of the above strategies work and we still find ourselves in the conflict between constantly putting out fires and at the same time trying to progress on projects, then forget everything and focus only on solving the unforeseen.

Yeah! You read well. Give up any attempt to progress on projects. Why deceive yourself with false illusions?

Before applying a sling to an exposed fracture, it’s first necessary to stop the sacred 🤕. This is the same plan to follow in extreme project management situations.

The key in all these strategies lies in recording incidents in as much detail as possible, carrying out post-mortems to identify their causes and creating Internal Projects to resolve them proactively. For example:

  • If you realize that Hotfixes are frequently sent to Production to urgently patch products, then it means that it is necessary to launch an Internal Refactoring and Continuous Testing Project.
  • If you notice that it is constantly necessary to carry out a manual task to manage users, then it is time to create an Internal Project for the automation of said task.
  • If your infrastructure suffers unexpected crashes and you have to re-create it manually, then it is necessary to invest effort in a Project for the implementation of Infrastructure as Code and Monitoring.
  • If you continually receive Unplanned Work from a specific team, then it’s time to set limits with the leaders of that team.


Remember: working as a full-time firefighter is heroic, as long as your mission in life is to put out fires🔥👨🏻‍🚒.

The progress and innovation that the rest of us want to achieve will inevitably create fires, but when properly managed they become wicks for further innovation.

Keep on Learning…and firefighting smartly!


As always, leave me your feedback in a comment on this article. Was it helpful? What would you add or remove? Any other suggestions? You can also tweet me @jjruescas1 and put #DevOps at the end so I can find it.🙂

Leave a Reply

Contact us

All rights reserved. 2022. Desarrollado por Estudio AZA.