OCS & Air Traffic Control

OC Systems proudly supports the Federal Aviation Administration and the National Air Traffic Services with our Ada compiler and Aprobe products.

For more information on what the FAA’s En Route Automation Modernization (ERAM) project is all about, watch this video.

For a general overview of the FAA’s En Route Automation Modernization (ERAM), click here and for the latest update on ERAM click here

For a typical day in the UK NATS airspace, click here.

For more information on our products PowerAda or Aprobe, please visit our website.

Measure What You Need…

Projects collect lots of metrics that they do not need. All on this forum would agree that measurement is critical. But not all metrics are useful, and too many metrics can be confusing and obscure what’s important.

Furthermore, measuring takes time and space resources away from doing. As computers get faster, storage gets cheaper, metrics and logging frameworks come built-in and data analysis and display becomes more powerful, the temptation grows to collect everything, just in case you need it.

Here are some observations on why we collect too many metrics, and how we can avoid it.

1. IF YOUR JOB IS COLLECTING DATA, COLLECTING MORE DATA MAKES YOU LOOK MORE PRODUCTIVE

Collecting metrics is a means to an end, not an end unto itself. If you don’t get paid unless you find more numbers to squeeze from an application then your organization needs some adjustment.

Depending on your level in the organization, the jobs should be:

– Ask a question that a metric could answer

– Decide what metric answers a question

– Implement the collection of a requested metric

– Answer a question using the collected metric values

The end goal of metrics is either to identify a problem, or fix a problem.

2. SOMETIMES YOU CAN SEE “ANOMALIES” LOOKING AT OTHER METRICS YOU MIGHT NOT THINK RELEVANT

This is actually the most compelling argument for collecting a lot of metrics. But this should be done by choice, in a purposeful way, in a non-production but realistically-loaded environment, and the result should be analyzed by somebody with the time and qualifications to judge the value of these metrics. Just turning on all the metrics all the time and hoping the bug will jump out at you when you need it, is not an engineering approach.

3. IT’S EASIER TO BROWSE EXISTING METRICS THAN TO FIGURE OUT HOW TO ENABLE A NEW METRIC

It shouldn’t be, especially if it’s one of the many that you would have been collecting already. Good tools and infrastructure should make the mechanics easy, and their use is something your developers and operations people should know: How do I enable/disable specific metrics and adjust their collection frequency and persistence? Whether it’s one app-server’s JMX metrics or your external network bandwidth, somebody around there should know the points at which metrics are collected, how these are configured, and where the results go. If not, then that’s a problem to address.

When the person who knows is explicitly asked to look at the metrics being collected, chances are they’ll see some that are not used or useful. Or, they might see metrics or logging that are not enabled, but would have been useful in the past, and that’s even better. Either way: a requirement of your application’s implementation and documentation should be how to easily control metrics collection.

4. IT’S EASIER TO COLLECT ALL THE METRICS THAN TO FIGURE OUT WHICH ARE THE RIGHT FEW

How do you know which few metrics you need? Of course you don’t, always, in advance. This is the hardest problem and the biggest reason why we collect too much. There are two main approaches to identifying what to measure:

– negative or problem-focused

– positive or goal-focused

The negative approach might alternatively be called the House, MD(link is external) approach, where we do differential diagnosis to decide which tests to run on the patient. We build a diagnostic handbook for our application by listing problems, symptoms, metrics and value ranges which confirm the problem exists; and/or metrics and value ranges which exclude that problem.

This process has the added advantage of forcing us to identify potential problems, so our QA department can test for these in advance (see The AntifragileOrganization(link is external)). If testing or production shows additional problems, we add that problem, along with the metrics we used to identify and diagnose it, to our diagnostic handbook, and keep the collection of those useful metrics enabled, if possible.

The positive approach is the more familiar one: the SLA. Quantify what we want to achieve as metrics and measure that. We then use externally visible goals like the SLA to drive internal metrics, like measuring every operation comprising a transaction. Then measuring the resources used by the operations comprising a transaction. Then measuring the resources that compete with the resources that impact the operations that comprise a transaction … And this is the trap. Everything in the entire system contributes to your SLA, so it’s tempting to measure and report on everything.

However, considering both approaches together suggests a solution:

1. Measure what you want to achieve

Record user experience, transaction frequency, error rates, availability, system correctness. If you don’t measure that, you can’t know you have a problem. These metrics are generally those worth reporting to management and your team. (Metrics reporting follies are a topic worthy of a separate post, or book).

2. Measure what you need to know to solve the problems shown by point #1

Let diagnostic need drive the rest of your metrics, as well as your logging. When a metric proves useful, keep it enabled if it’s not costly (and if it is, see if you can get it another way for next time). But don’t bother producing reports about these metrics.

3. Disable all the metrics and logging that aren’t either (a) identifying problems or (b) helping you solve them

You’ll be amazed at how much lighter your load is.

APM is more than a software product…

I have been in the performance engineering business for about 30 years. I get involved early on during system development and move off as the systems move to maintenance. Then, I move onto the next one.

I sometimes get called in to troubleshoot a performance crisis. I have sometimes been compared to a locust moving onto the next greenfield and, at other times, compared to Kwai Chang Caine (of Kung Fu fame), wandering the earth sowing justice, peace and performance.

Whichever you choose, I have noticed some similarities between the various projects that I have worked on.

1. SPEED TO MEET THE NEED

There are two aspects to performance engineering:

– To speed up the system to meet the required response times

– To talk the customer into lowering their response time requirements

It continually amazes me how high level system people will start off with unnecessarily fast response requirements. One customer wanted sub-second browser response time, even though the user would stare at the resulting web page for many minutes. Really, it was ok if it took a couple of seconds to fetch the web page.

So don’t feel afraid to talk your customers into a lower performance criteria.

2. KEEP TRACK OF YOUR HISTORY

Performance requirements are always patchy. Yes, they get the big ones (user response time), but it is totally typical for the subcomponents to not have requirements on them, making it unclear where and how to make the end user response times. Complicating this, generally, all of the discussions leading to the system designs are lost. There was a big debate as to how large the system should be for how many users. Someone chose that 256 CPU server monster, but what was their thinking? Half way through the project, when the response requirements have been relaxed and the number of users has been increased, it sure would be nice to look though those emails and see what people were thinking so you can estimate what system changes need to be made to accommodate the business changes.

So during the early phases, have an archive account to which people can cc “important” emails. Later, these can be reviewed and all that wonderful history and knowledge will make the system change and maintenance a lot easier.

3. DEV VS. OPS

The development people do not really address the maintenance phase of the project. The development people are in a heated rush to meet performance goals and deadlines while not over running costs. The project managers are very focused on the deadline of “going live”.

Then the system goes live, everyone throws a party and then they realize that they have not planned well for this lesser period of effort: maintenance. What artifacts did the developers leave the maintainers? Did they leave a workable load test? Did they leave a well organized set of production diagnostics implemented as part of the application? Did they leave proper alert monitoring? Did they leave proper daily reporting?

Ideally, the last stage of development is to set up the system for long term, cost effective maintenance, not the first stage of maintenance. This task is properly done when the development team is still available, not after the brightest are gone to their next adventure. As much as possible the maintenance tasks (such as daily reports, regression tests and load tests) should be automated and that may require changes to the system, changes that the development team can do easily, but will be much more difficult for the whittled down maintenance team.

4. AN APM SOLUTION ITSELF IS NOT THE SOLUTION

Management often feels that buying a tool will solve a problem. “We have performance problems so we will buy an APM solution.” This belief often results in shelfware: management has underestimated the effort to implement the solution.

Yes, an APM solution is good. But you need to have enough labor to implement it and look at and understand the results. Plan on it up front. Don’t focus on the capital cost to your organization, focus on how it will integrate into your organization and what will be done on an ongoing basis.

5. HOW IMPORTANT IS PERFORMANCE?

How important is your project? Everyone feels that their project is important, but some are more so than others. Some systems really are not that performance intensive. So, you can save money by not worrying so much about performance; a high performing system is more expensive to monitor and maintain than one that requires less performance.

Make the decision early on as to how important performance is. If it is important, make sure that it is considered important by everyone. If it is not so important, embrace that too and make clear goals for performance. Even if performance is not important, clearly indicate the
goals. Even easy to reach goals should be articulated.

I have seen systems where the performance goal is to wait for users to complain, then address it. Sometimes that works fine.

Oliver Cole is President of OC Systems.  This blog was previously posted on APM Digest earlier this year.