With the rapid development of mobile Internet technology, the data volume of operators increases exponentially. How to use massive data resources for value and commercial realization has become the key for operators to improve their core competitiveness and seize the market opportunity. ETL(Data warehouse technology), as a main technical means of data information conversion, is an important tool to extract, clean and transform the data of the business system and load it to the data center. With the growth of data volume and the change of business scenarios, the range of taking and calculation is constantly expanded, and the scheduling of the whole ETL process is increasingly complex, which not only brings the increase of operation and maintenance cost, but also makes it difficult to guarantee the quality of data production. Therefore, operators urgently need to seek new technical means.
Zte Soft Innovation combined with years of project operation service support experience, proposed to build a whole-process scheduling platform solution. The solution includes functions such as automatic progress prediction, abnormal monitoring and alarm, and data lineage analysis. It implements the scheduling capability of the whole process from data aggregation and data processing to data service, effectively helping operators improve data processing efficiency, and meeting their requirements for processing massive data and complex service scenarios.
FIG. 1 Architecture diagram of whole-process scheduling platform
FIG. 2 Function diagram of whole-process scheduling platform
The whole-process scheduling platform solution builds a three-layer stereo scheduling system:
Layer 1 end-to-end scheduling provides loose coupling, end-to-end scheduling, and monitoring functions from a service perspective. With data as the center, it organizes data, data dependency and production process tasks through data nodes to realize one-point management of scheduling information. By constructing a 360-degree scheduling operation view, the scheduling operation of the whole data center can be understood, and visual real-time monitoring can be provided.
The second layer of process-level scheduling controls dependencies between tasks through a process engine, allowing tasks to be executed in parallel or in serial. Tasks can be executed synchronously or asynchronously, shortening the entire process execution time and improving the running efficiency.
The third layer of task scheduling implements task execution policies and scheduling policies through officially released tasks to realize the periodic, temporary or real-time uninterrupted operation of task programs. Task scheduling Implements unified configuration and collaborative operation of multi-computing cluster tasks in a data center to optimize system resource allocation.
In 2017, ZTE Soft Chuang full scheduling platform in Hunan Telecom took the lead online. At present, we have undertaken 10000+ data processing tasks of Hunan Telecom Data center, and the problem verification rate has been increased by 50%. At the same time, the coverage of data quality is increased to 98%, and the training cycle is shortened from 1 month to 7 days, saving 60% of the labor cost. The platform has gradually become the connector of Hunan Telecom data center, supporting the internal and external data needs of Hunan Telecom with high quality, efficient, agile and secure means.
