What is Visualization?
Visualization is a human trait, it has been part of us since our ancestors painted on cave walls to map animal types, movement and the hunt. A more recent example of visualization as innovation was when Napoleon used them in the early 19th century wars. He didn’t need to put himself in danger to see everything at the top of a hill when he could visualize the battle from his tented HQ. Fast forward past our current time and we might see Minority Report interfaces come to life through the use of Xbox Kinect and holographic systems.
Across the technology industry today we are witnessing a visualization renaissance amongst the megatrends of Cloud Computing, Big Data and APIs. These technologies behind these systems are big, complex and constantly changing and can’t be managed by people staring at screens full of numbers.
Cloud pioneers such as Netflix openly talk about their struggles with daily work to separate signal from noise in their complex, changing systems? One of the winning answers to this is using visualization in your cloud operations.
Thanks to a new crop of visualization technologies and companies, the good news is you don’t need a be a PhD or have the resources of Google to enjoy these benefits, you don’t need to be a specialist in visualization, and you can start today.
Companies like Hyperglance are democratizing visualization for everyone else.
In this article, we look at:
- How IT Operations is a Cool Place to Work
- Cloud Pioneers Like Netflix Use Visualization
- High Performing IT Organizations use Visualization
- Using Visualization to Master Cloud Complexity
- How You Can Use Visualization in the Cloud Today
1. IT Operations is a Cool Place to Work
I remember when Operations was “contained” in the cellar of the datacenter where no daylight could reach. Today we can attend conferences like Operability.IO dedicated to Operations and its practitioners. These events are held in glamorous venues in cities like Berlin, London and New York with keynotes from seriously bright, talented and “seriously cool” speakers.
Attendees talk freely about using big data, visualization and empathy. Why? I see the evidence being an emerging culture, based on detailed research, to move towards the goal of becoming a high performing IT organization, as described by the 2015 State of DevOps report.
Much is said of the sharing economy but you can see free knowledge sharing in action at these events and online through blogs and social media. Operations thinkers and talkers are respected and listened to. One aspect of operations I’m particularly interested in, as it’s my sphere of expertise, is visualization in modern
Operations is cool and visualization is hot, and according to research like State of DevOps this is now recognized all the way up to the board room as something that can significantly impact business if done right.
2. Cloud Pioneers Like Netflix Use Visualization
Visualization is literally everywhere in the technology industry. We are surrounded by infographics, big data charts, and data-rich interactive maps.
Fact: People like pictures and stories, not impenetrable walls of text data.
Thought-leading software companies such as Netflix talk about operations a lot. They share their insights not just because they want to look clever or increase movie downloads.
Fact: Netflix publicly reveal insights into their operations because it attracts engineering and operations talent.
Cloud pioneers like Netflix prefer to show rather than tell, so they show the pictures they look at every day and the interfaces from their daily work. Here’s an example of a top-level visualization at Netflix:
Source: Netflix Tech Blog, Improving Netflix’s Operational Visibility with Real-Time Insight Tools
“Operational visibility with real-time insight enables us to deeply understand our operational systems, make product and service improvements, and find and fix problems quickly so that we can continue to innovate rapidly and delight our customers at every interaction. We are building a new set of tools and systems for operational visibility at Netflix with powerful insight capabilities.”
3. High Performing IT Organizations use Visualization
The 2015 State of DevOps Report, from Puppetlabs, is a must read for everyone in the IT business, from Operations staff up to the board. The authors are impeachable, the report methodology is solid and the insights unique and much higher quality than your usual industry surveys.
For the first time a report uncovers the metrics and practices that differentiate a high performing organization from the rest. One headline example from the report, and which company wouldn’t want this kind of performance?
High-performing IT organizations experience 60X fewer failures and recover from failure 168X faster than their lower-performing peers. They also deploy 30X more frequently with 200X shorter lead times.
In the report the authors identify visualization as an effective way to build quality into the system:
The more you build quality into the system — through automation, reducing batch sizes and shortening cycle times — and the more effectively you manage your team’s workload and visualize work queues, defects and bottlenecks, the more you increase throughput and stability.
The visualization aspect of DevOps is crucial to what author, Gene Kim, describes as The Second Way: Amplifying Feedback. The Second Way is a simple but powerful method of feeding back the system reality (in operations) back to the system creators (in development).
With visualization systems everyone can access a visual model of what’s happening in real time with data-rich interactive maps.
Source: Gene Kim, IT Revolution, The Three Ways: The Principles Underpinning DevOps
4. Using Visualization to Master Cloud Complexity
Amazon Web Services (AWS) is the public cloud company. Gartner’s Cloud VP, Lydia Leong, reports they are ten times the size of the rest of the market made up of fourteen other cloud providers.
It’s not just their size. The list of services that AWS offers is what sets them apart from all other cloud providers, the latest count being sixty-seven services across twelve categories.
What this means is that everyone using the AWS cloud has to use the AWS console and they are faced with a mountain of powerful features that looks like this:
Source: AWS Management Console
Now imagine lifting the lid of this console to expose all the configured, running systems underneath: the EC2 virtual machines connected to security groups and route tables and back to an on-premises datacenter via VPC and VPN. And that’s just EC2, one small part of AWS cloud infrastructure.
Trying to understand what is important from this noisy view is very difficult for new and experienced users alike. Pages of text look like noise to our brains. Humans need pictures, they need visualizations, especially if they need to all have the same picture like an operations team does.
Forcing a team of people find the same conclusions as to what’s operationally happening just from the AWS console is time-consuming and error-prone. People have different abilities in use the AWS console; they probably have different roles and therefore visibility into the console. Instead of looking at the same picture, they each bring their little piece and try to jigsaw it together.
5. How You Can Use Visualization in the Cloud
Using public cloud services means inevitable change for the IT organization. Mark Holt, CTO of The Trainline, has completed a full migration to AWS Cloud from their on-premises Oracle Exadata. His experience shows that many things change, not just the technology, and key to this is the whole team’s understand of the system using data and visualization. Each morning the first thing Mark does is to look at a visualization of application performance and user experience using NewRelic.
For both public and private cloud infrastructure, the best visualization option available today is Hyperglance which provides real-time, interactive, data-rich relationship graph models of AWS, Azure, OpenStack, VMware vSphere, Docker, Kubernetes, and SNMP networking – together – in your browser. If you’ve ever done problem-solving looking at the AWS Console, for example, then once you look at the Hyperglance interface you’ll feel how much more natural it is to use visualization.
Hyperglance doesn’t seek to replace the AWS, OpenStack or other consoles, its role is to complement it by making it easier for everyone to understand and assess the current state of resources and relationships. In fact, Hyperglance has direct links to the relevant source platforms’ consoles for easy access if needed. You can also directly connect to resources via SSH and the likes. Both analysis and troubleshooting is now much easier through a much more intuitive visual perspective.
Visualize Your Cloud Architecture
Trying to understand how your cloud is configured (let alone related), is becoming increasingly complex.
Within minutes, Hyperglance can show you your aggregated cloud inventory in an intuitive, interactive diagram:
- See all your AWS, Azure, & Kubernetes cloud architecture in one scalable view that updates continuously
- Overlay metadata onto your cloud diagram, including cost, to add important context
- Explore resource dependencies and links, and view/search detailed metadata
- Switch to see your cloud inventory & search results with one-click
- Export your diagrams, as often as you like, to VSDX (Visio), PNG, or CSV; automate exports using our RESTful API
- Hyperglance is self-hosted, deployed through the AWS & Azure Marketplaces, in Kubernetes, or installed on your own instance/VM