It’s been nearly a month since the city-wide closure of Shanghai. But I have enough things to buy and distribute in my house that I have now evolved from the stage of worrying about running out of food to the stage of people running after the speed of food spoilage …

Recently, I have been investigating the architecture design of the monitoring and alarming system. When I was cooking at noon today, I suddenly had a brain hole: Why can’t the alarm system be displayed — whether it be a real-time alarm or a historical alarm — with a button behind each alarm called “One-click Google” .

To be honest, on the one hand, this idea is because some elements of the monitoring and alarm system design are considered every day, and on the other hand, it is mainly due to the fact that a friend has encountered several operation and maintenance problems in the past few days, and called at night to ask for solutions. In fact, it’s not that the problem is complicated, but because the operation and maintenance personnel have never encountered this problem, and then when they see some nonsensical errors, they may not know where to start for a while.

This situation is also normal, because sometimes the errors we see for the first time are generally the ones returned by the developer to the front end through the program, and the real error will not be directly displayed to the user. This is especially common on 2C systems. After all, on the one hand, non-professional users cannot understand it. If the exception is thrown directly to the user, the user will be confused; on the other hand, due to the purpose of network security, if a black hat appears Intentionally causing system exceptions through injection to make targeted attacks through error returns, then throwing exceptions directly is actually nothing.

However, since the users of the monitoring and alerting platform are all professional users, and they are all internal employees, the information of the monitoring and alerting platform should be structured and unstructured log information related to system resources, middleware, databases or containers. During the above collection, a huge data system is formed, which is displayed to the operation and maintenance personnel through the logical design of the DevOps platform developers, so that they can see the alarm and its detailed abnormal information at the first time.

So, in fact, if you see an abnormal error report on the front end, you can go to the monitoring and alarm platform to view the corresponding alarm information according to the time point and the system, and then deal with it, the incident can be resolved normally. The logic is simple and straightforward.

However, what if the operation and maintenance personnel still cannot use the “operation and maintenance personnel experience” to solve the problem when they see the alarm information?

At this time, operation and maintenance personnel generally do two things:

First: Open the search engine and search by the exception thrown to see if someone shares the solution;

Second: ask experienced or more senior personnel.

However, in general, due to human nature, these two schemes are generally operated serially. Because after all, if you can solve it yourself, you won’t usually bother people.

And when I raised this question, in fact, the pain point to solve is this weakness of human nature: maybe when you search on the search engine, what you search for is not the real error.

At this time, if the monitoring and alarm platform adds the function of “one-click Google”, then it is equivalent to the platform telling the operation and maintenance personnel: the error is this, follow the map to find out.

As for the idea of ​​this function, the simplest solution is to directly search for the information of this abnormal log when you click the button. Of course, the search source can be Google or an internal knowledge base.

A further evolution is that the most important keywords of the log can be learned through NLP.

A further evolution is the most important keyword that can correlate each system to find the root cause log.

When it comes to this, some people may find it funny: they are all used for NLP and root cause analysis, why not go directly to AIOps?

What to say, I always have a point of view: the use of AI in operation and maintenance can only be done well in simple self-healing processing and alarm-assisted decision-making. Whether the real event can be handled well, still needs people to the final decision.

Because operation and maintenance is not as fault-tolerant as AI applications in other fields, if there is a problem with face recognition, you can try it again. But the operation and maintenance itself is the only solution to return to normal. That is to say, the failure recovery success rate should be close to 100%. If this kind of work is given to AI, especially NLP is currently one of the slower development of several machine learning fields, and its accuracy has never been able to reach the fault tolerance level of operation and maintenance. The operation and maintenance work may be digging a hole for yourself.

Therefore, in this analysis and judgment idea, I think it is better to display this kind of auxiliary decision through a function that facilitates operation and maintenance personnel to find problems.

Of course, I am not completely negative about AIOps. After all, the accuracy rate is gradually achieved through the accumulation of computing power and samples. Perhaps the general accuracy of AIOps — not problem-specific accuracy — may someday be as fault-tolerant as operations work. At that time, it may really be a particularly happy day for the operation and maintenance personnel.

It could also be a day to look for other new jobs.