Contact ITInvolve

Archive for November, 2013

Knowledge isn’t free and neither are the donuts

Wednesday, November 20th, 2013

One of my former bosses is a Naval Reservist. Once a month, he goes away for the weekend to do his job for Uncle Sam. In the break room of whatever building he’s working at, there is a sign above a table that says – “Freedom isn’t free, Neither are the Donuts.” The idea is that if you take a donut, you’re supposed to drop some money in the jar so the tomorrow’s donuts can be purchased.

This got me thinking about other things that have costs associated with them, like experience and knowledge. So with a hat tip to the Navy, I submit that “Knowledge isn’t free, Neither are the donuts.”

There is a cost to developing and utilizing knowledge. There is the time it takes to develop the knowledge. For example, someone needs to write it down, make the video, etc. That takes time, which has a cost associated with it. An obvious cost is the knowledge developer’s hourly wage, but what about other costs? What else could that knowledge developer been doing instead of documenting the knowledge? Is their main job to develop knowledge or do they have other duties that have to be put on hold while they document?

We’ve gotten used to having much of the entire world’s knowledge in the palm of our hands via the Internet, Google and our smartphones. We can search for an answer to just about any question we might have, and at least 99% of the time, we can get a reasonable answer. Of course, depending on the question, I may have to spend time weeding through a whole bunch of entries to find it. I may have to ask the question a few different ways, or (heaven forbid) try to remember a little bit of Boolean logic to filter out unwanted results. In this sense, there is a very real cost with not just documenting but also accessing knowledge — my time.  What else could I be working on instead of trying to find this information?  How much time am I spending piecing together information, data or knowledge from other sources to figure out the truth?

Now, let me repeat this exercise, but with my work hat on. I vividly remember a few years ago that I was desperately searching for information to help me with a high priority project I was working on — upgrading the Point of Sales (POS) system. [The project goal was to deploy a newer version with better reporting capabilities for our logistics folks.] In this context, I was no longer going to my phone and googling something, instead I had to find and then go through tons of information about my company’s implementation and use of the POS system. No longer was everything nicely catalogued and key worded. No longer was everything easy to find. There were literally hundreds of places that I needed to go in order to find the data. There were build documents, CMDB entries, a bunch of Sharepoint sites, dozens of emails, file shares, several change management requests, some Wikis, and much more. The stuff was scattered everywhere and I had no idea if the three-year-old Visio diagram that I found was in any way representative of what was currently humming along in our data center 600 miles away.

So that meant I now had to seek out people – experts that could provide me with additional information or at least validate the information that I had earlier found. But first, whom did I need to talk to? The author of the Visio left the company 18 months prior. The VP of Retail Operations, who signed off on the Functional Requirements Spec, now worked for a competitor. Just trying to figure out the people who could maybe get me the information that I needed to do my job was a full time job unto itself. And, this wasn’t my only project, I had 15 other projects at the same time that I was involved with. Unfortunately, those other projects (and their value to the business) had to wait — even though people were waiting on for a number of tasks.

Fast forward a few weeks, and I’ve now got a bunch of information that I’m told through various meetings, phone conversations, and emails is current. I have a bunch of new hard copy documents on my desk, a picture of a white board diagram that a DBA spent an hour drawing for me, lots of emails, word documents, PDFs, you name it. I added that to the information I got from the vendor, information from three of our six CMDBs as well as some reports from three different monitoring tools. Now, I could finally start working on my tasks for the project.

I probably spent at least 30 hours trying to look for and validate this data and information. As an architect, my fully loaded hourly wage was around $75. So just assembling this information cost my company $2,250 not to mention the time that it cost from everyone I talked to which was easily several multiples of that. Plus, they all had other things to do that I was pulling them away from and there is a cost to that too.

Now the fun began. We didn’t have money for new hardware for this project so I had to make sure the new version of the POS software would run on what we had already. I went through all the information I had assembled. The build specs, in particular, were hilarious. Each was 60 pages long and there was so much boilerplate and filler in each one that it was mind numbing to read them all. Our CMDBs were filled with a bunch of useless (for my purposes) information that had been populated by our auto discovery tools, and I quickly started to drown in information. What I really needed to understand were just a couple of configuration settings to see if they were A) required, B) needed to be updated or C) completely irrelevant to what I was doing.

Aside from the Easter Egg hunt that I had to go through just to find the information I needed to do my job, I found the information itself was barely relevant. My company made IT people fill out all of this paperwork when they did something or built something, and most of these folks are IT majors not English majors. The people that came up with the document templates probably had no idea what kind of information would need to be captured, so what you think would be in one section is in another or the information didn’t fit the template so it had to truncated.

At this point, I didn’t have much confidence this upgrade will be successful. There were just too many unknown or little knowns, and it wasn’t from a lack of data. I actually had too much that it was hard to find the proverbial needle in the haystack I needed. What’s worse, what I had was mostly bad, or old or incorrect data. I couldn’t trust it and it didn’t provide the right context for me to effectively use it.

Unfortunately, this situation is very typical of many IT shops – the information and knowledge is not well documented and what is documented is a huge mess. IT information has a lifecycle of its own and the common methods of documenting, collecting, and accessing IT information are inadequate. IT Information is a living, breathing thing that must be nourished and maintained on an ongoing basis as part of daily work, because no one has time to do it after the fact. And even then, we’re really good at burying it in file shares, spreadsheets, emails and Sharepoint sites, so as soon as the information is created, it’s quickly forgotten about and withers and dies.  Then, you’re like me in this story, where all you have left is a corpse of information that might have ben useful once but probably doesn’t have much value today, because we move  so fast and what was true yesterday isn’t true today since that OS patch was put in place or the backup schedule was updated.

So what happened with the POS upgrade? Well, even with my lack of confidence, the change request for the upgrade was approved – because it had to be. There was no testing because we didn’t have a properly configured test environment, but I did ensure we had a full backup just in case something went wrong. We held our breath and we rolled it out. Amazingly it worked out okay, but to this day I feel like we just got lucky. Knowledge isn’t free, and this all to common status quo is no way to run an IT shop when your business depends on it.


IT Advice from Bill Nye “The Science Guy”

Tuesday, November 19th, 2013


I recently came across this quote and thought it very apropos to the situation in today’s complex IT organizations. Whether you are talking about server, storage, and network admins; developers; QA teams; security managers; and the many other experts in a typical IT organization, the fact is everyone in IT has specialized knowledge and a unique perspective on what they are responsible for.

It’s all too easy to get caught up in these individual perspectives and miss out on the big picture. But worse than that, because there are so many items associated with delivering a given application or service, and many of these items like a policy or unique setting an individual expert is not aware of, the best intended actions can often produce unexpectedly bad outcomes. Our own experiences and a quick Google search reveals that it is still all too common that outages are caused by “human error” and not an equipment failure or code issue.

George Spafford at Gartner has state the problem this way, “It is becoming impossible for any person or group [within IT] to completely understand how everything integrates together.” Because we don’t know what we don’t know, we can be lulled into a false sense of security as the bad outcomes all too clearly illustrate.

In response, a lot of IT organizations have tried to attack the problem with email, meetings and formalized change processes. This has helped many companies identify and minimize risks to a certain degree, but they have exchanged this benefit for a much slower change rate and over-involving too many personnel in change management.

A recently published metric from industry consulting firm Pink Elephant found that the average time from creation of a change request to its execution was 31 days! Whether the change is in response to a business need or is applying a patch or upgrade to make infrastructure better performing and resilient, I think we can all agree that a month is far too long. And a month is just the average! I am sure that complex changes with many change items greatly exceed a month in many IT organizations today. We need to do better as an industry – and that goes not just for practitioners but vendors and consultants too.

Here’s the second part of the problem with many current approaches. Because IT operations teams are rightly concerned about the instability that change represents, they pull far too many people into change planning meetings and change advisory board (CAB) meetings who don’t really need to be there or who could have just as easily provided their input offline. I can’t tell you how many times I have heard a change process owner complain about how they send emails out to change approvers and then have to hunt them down in person to get them to login to the change system and respond. And for their part, those approvers often complain that they get so many emails from the change system they can’t distinguish which are important and just end up ignoring them all.

So this brings me back to Bill Nye and his astute observation that we can learn something from everyone we meet. Let’s accept the fact that each of us doesn’t know everything we need to know to effectively manage today’s complex IT environments, despite the fact that we may indeed be experts in a particular area. It is only by capturing our collective knowledge and making it available to everyone that we can have a complete understanding of dependencies and risks. By using a modern approach like ITinvolve that allows IT knowledge workers to follow what they are responsible for or have an interest in, we can leverage the knowledge of others AND identify exactly who the right experts are and proactively engage them in a virtual collaboration to assess risk.

The result is that risks can be assessed more accurately but also more quickly, and without pulling people unnecessarily off of whatever else they are working on too. This assessment can then be provided proactively to the CAB and CAB members can approve or reject offline from meetings at a time of their convenience. If all CAB members approve, the change doesn’t even need to get discussed in the formal CAB meeting and can move straight to execution. This then enables IT to focus CAB meetings on the really important and high-risk changes that everyone hasn’t approved.

To get there, the first step is simply to recognize and appreciate that you can learn a lot from others by sharing what you know and having everyone do the same. We often here statements from our customers like “I’ve learned more about our environment using ITinvolve in the last three weeks than in the last five years I’ve worked here.” This is the reality, no matter how much of an expert we are, our knowledge of today’s complex IT landscape is limited. It’s only by working together and sharing what we know that we can deliver on our mission of helping IT become more agile while minimizing risk.

Matt Selheimer
VP of Marketing

Does your IT racecar enable agile business victories or are you stuck in the pits?

Monday, November 4th, 2013

In today’s business world, “fast eats slow.” And the IT department is now the primary driver (or inhibitor) for how fast a business can respond to the market and create new competitive advantages.

Is your business like a Formula 1 racecar, operating at top speed and agilely navigating the twists and turns of the track until you cross the finish line first? Or do you find that your racecar is frequently stuck in the pits with tire changes and other adjustments, or worse have you had a high profile blow out that dashed your company’s hopes of victory?

If you’re like most IT operations leaders I’ve spoken with in the last year:

  • Your environments have become more and more (not less) complex
  • You feel like you’re already operating at a frenetic pace
  • You worry that moving faster will lead to even less stable service delivery

If you’re an application development leader, you’ve probably gotten better and developing new releases faster by adopting agile principles or other related methodologies, but feel that operations simply can’t keep up and is holding you back. And if you’re a CIO, you are probably frustrated by the fact that development and operations aren’t working well and collaborating together to deliver on what your business needs.

I’m not going to suggest that you forklift how you develop software or run IT operations. That would be impractical and unreasonable. However, we’ve identified four areas that you can improve individually to help your business become more agile. And if you tackle all four of them, you will deliver a breakthrough in business agility.

These four areas exist at the boundaries between application development and IT operations. By focusing on improving collaboration and knowledge sharing in these four areas, you can foster a new hybrid DevOps function that will create new opportunities for victory in whatever industry you operate.

  1. Requirements Collaboration and Validation – As a project moves from business goals, to business requirements, to functional requirements, and to operational requirements the amount and frequency of handoffs between the business, application development, and IT operations is very large. At each handoff information can be lost or distorted. There’s actually a pretty well known and humorous cartoon about a tire swing that illustrates this problem.

What’s needed is a method to engage the right business, development and IT operations stakeholders to collaborate on and validate business, functional, and technical requirements. And this method should preserve those collaborations and their output so that others can view and access the information as they engage on project or support the release once deployed.

  1. Environment Synchronization – Have your ever found out right before a “go live” date that some of the operational requirements have changed? Or have you made changes in production that weren’t effectively communicated back to those responsible for pre-production QA and testing? With the pace IT operates today, this happens all the time in most IT organizations. The result is that launches are delayed because new requirements can’t be supported (e.g. I didn’t know we were going to need a UNIX environment for this project) or what’s tested in pre-production works fine only to fall over when deployed in production.

What’s needed is a method to synchronize configurations and operational requirements across the promote to production landscape, as well as the ability to communicate proactively any production changes that are relevant to development and testing efforts. Without this, we will continue to ship late and deliver unreliable applications that cause our business goals to be unmet and our business colleagues to continue to be frustrated by IT.

  1. Release Collaboration and Deployment – As you move closer to the release date, it’s critical to assess risks and timing. Unfortunately, risks assessments are often a time-consuming process and so they are cut short and only done superficially or they slow everything down causing the business value to be delayed or window of opportunity to be missed.

What’s needed is a method to engage business, development, and operations stakeholders to assess deployment risks, timing, and validate plans. And this method must incorporate what’s learned at each stage of the development lifecycle, and do so in a way that is inline with doing their daily jobs and not some big overhead administrative effort.

  1. Post-deployment Resolution and Collaboration – Once a new application or service is rolled out, there are likely to be some issues. All too often, we don’t get the right resources engaged to work the issue and that causes frustration on the part of the business, because the timing to resolve the issue elongates. What’s more, in an effort to get things fixed “quickly”, there is all hands on deck firefighting that pulls people off of new projects and far too many escalations of the same issues to senior resources without pushing down post-deployment learning to less senior staff to make use of. The hero culture so prevalent in IT, comes back to burn the organization because everyone gets caught up in this unplanned work instead of focusing on moving the business forward.

What’s needed is a method to identify and appropriate escalate issues to development or operations with just-in-time, in-context knowledge for rapid diagnosis, while also capturing experience to quickly solve ongoing issues without pulling valuable resources off of new assignments.

A solution like ITinvolve  that uniquely combines knowledge capture, analysis, and social collaboration for IT, enables you to do each of these things. And if you tackle all four areas, you will:

  • Accelerate your business’ response to market opportunities and competitive threats
  • Align business requirements, IT projects, development teams, and IT operations personnel to enable continuous delivery of new application releases
  • Minimize operational and compliance risks from new application releases (including those that have legacy integration requirements)

If you’re ready to change the status quo, we’re ready to help you.

Matt Selheimer
VP, Marketing