Observability Team Members

People and Skillset Considerations for Enterprise Observability and Monitoring

At the heart of an enterprise observability practice are the hardworking individuals who directly and indirectly support the numerous processes, monitoring technologies and activities. Sadly, most organizations are understaffed when it comes to observability- despite spending millions of dollars annually on tools licensing and subscriptions.

Various skills are required to drive an enterprise observability practice forward. These include technical, business and project management. Most organizations staff their observability and monitoring teams with product specialists who are aligned to their specific toolsets (e.g., Splunk or SolarWinds Administrators). While this is a necessary aspect of staffing a team, team members with an understanding of target architectures for monitoring are also needed to ensure that monitoring solutions are built around the environments to be observed and not the other way around. Understanding of critical technology functions and processes of target architectures and technology stacks are also required to avoid deployment of monitoring that fails to achieve minimal acceptable standards.

The concept of the observability architect - technologist with a monitoring background with understanding of infrastructure and applications has been growing for the last several years. It reflects the growing need of enterprises to have team members who can look beyond one observability capability and develop an end-to-end IT monitoring solution which traverses the various components of the runtime architecture including network, application, infrastructure, cloud and data. This is a critical skill which requires observability practitioners to wield knowledge of several monitoring capabilities, technology platforms and architectures.

Team of people sitting in front of a bank of computer screens displaying various information in an operations center

The Modern Observability Professional

In addition to architecture skills, data sciences and analytics skills are also becoming a requirement for modern observability teams. For years now, sophisticated observability platforms including those for log analytics, application performance management and AI Ops have required team members with skills in analytics, reporting, query language, API’s, data enrichment and consolidation to name but a few. With vendor platforms becoming more and more data analytics focused requiring coding skills to extract deeper metrics; data sciences is now a fundamental skills needed to attain return on investment for observability tools and provide maximum intelligence to the organization around business systems performance and customer satisfaction.

With so many skills and resource types needed to drive an effective observability program, where does an organization begin to build out these capabilities in a business-sustainable way? There are several methods that leading edge organizations employ which include a well-defined observability team organizational structure and roadmap of skills requirements aligned to monitoring team considerations. These considerations includes areas such as platform/ tools changes and enterprise evolution including cloud, application, data architecture which will require team members who understand how to best monitor these types of systems and vendor platforms.

Once roles, skills and resource planning have been set from a requirement standpoint, realistic budgeting and financial planning needs to occur with leadership to ensure that needed investment is available with a two to three year outlook. A clear understanding of what roles will be funded should drive a revision of the original resource plan to account for approved budgets and initiatives. Any observability programs that didn’t make the cut should be removed, delayed or scaled back. The number one pitfall for most enterprises is buying technology or establishing processes that have no resource or human capital behind them.

With the observability roadmap and funding in place, enterprises need to evaluate the best approach for staffing which includes internal sourcing and recruitment efforts through partnership with human resource teams as well as engaging with trusted consulting partners who have a focus and specialization in observability. A sound people strategy will include a sensible blend of full-time equivalents and consulting services partners who are bringing valuable skills and external experience. The blend is important as an overbalance of FTE’s or consulting partners will create problems in sustainability, knowledge and continuity.

With the team dynamic and blend of skills set, the journey is just beginning. Budget considerations need to contemplate for skills refresh and regular training to stay ahead of observability tools feature updates, new architectures and updated processes and/or techniques. Cultivating team members with current observability platform certifications or working with partners who can bring these updated skills sets is important to continue to leverage advanced capabilities which likely have relevance within the enterprise and are integral to driving return on investment and avoiding "staleness" of the solution within the enterprise.

Most importantly, observability-focused enterprises need to ensure a solid strategy to engage the larger user community across the organization including application, infrastructure, cloud, operations, testing, digital and other technology delivery groups. User focused training must be readily available with alignment with leadership and human resources to ensure the broadest user adoption possible. Without strong usage and penetration across these groups, most observability initiatives will not deliver acceptable return on investment and may create "investment fatigue."