Practical Application of Face Comparison and Heterogeneous Computing in the Intelligent Security System of HarmonyOS Next

This article aims to deeply explore the practical application of face comparison and heterogeneous computing technologies in building an intelligent security system based on the Huawei HarmonyOS Next system (up to API 12 as of now), and summarize it based on actual development experience. It mainly serves as a vehicle for technical sharing and communication. There may be mistakes and omissions. Colleagues are welcome to put forward valuable opinions and questions so that we can make progress together. This article is original content, and any form of reprint must indicate the source and the original author.

I. Requirements and Architecture Design of the Intelligent Security System

(1) Analysis of Functional Requirements

1. Requirements for Real-time Face Detection and Recognition In an intelligent security system, real-time face detection and recognition is one of the core functions. The system needs to be able to quickly and accurately detect faces in video streams in complex environments and compare them with the preset face database to identify the identities of people. For example, in public places such as airports and railway stations, it is necessary to monitor the flow of people in real-time and promptly detect suspicious individuals; in corporate office areas, access control can be realized to ensure that only authorized personnel can enter specific areas. This requires the face comparison algorithm to have high accuracy and low latency, and be able to process a large amount of face data in a short time.
2. Requirements for Efficiently Processing Data from Multiple Cameras With the expansion of the security monitoring scope, multiple cameras are usually deployed. Therefore, the system needs to have the ability to efficiently process data from multiple cameras and be able to receive and process video streams from multiple cameras simultaneously. This involves issues such as parallel processing of data, reasonable allocation of resources, and optimization of data transmission. For example, in the security system of a large shopping mall, there may be dozens of cameras covering different areas. The system must ensure that the video data of each camera can be processed in a timely manner without data backlog or excessive latency.

(2) Architecture Design Based on HarmonyOS Next

1. Hardware Selection To meet the high-performance requirements of the intelligent security system, hardware selection is crucial. Select devices that support heterogeneous computing, such as intelligent security cameras or edge computing devices equipped with both a CPU and an NPU. The CPU is responsible for handling the logical control of the system, data preprocessing, and some non-computation-intensive tasks, such as video decoding and data transmission control. The NPU focuses on accelerating computation-intensive tasks such as face comparison, and uses its powerful parallel computing capabilities to quickly process face feature extraction and comparison operations. For example, selecting an intelligent camera with a high-performance CPU and an NPU optimized for deep learning tasks can efficiently process the collected video data locally, reduce data transmission latency, and improve the real-time performance of the system.
2. Software Hierarchical Design
   • Data Collection Layer: It is responsible for collecting video data from multiple cameras. This layer needs to ensure the stable collection and transmission of data, adapt to different types of cameras, and handle possible network failures or hardware abnormalities. For example, using a reliable video stream transmission protocol, such as RTSP (Real-Time Streaming Protocol), to ensure that the video data can be transmitted to the processing layer accurately.
   • Processing Layer: It is the core part of the system and includes the implementation of face comparison and heterogeneous computing functions. In this layer, the Core Vision Kit is used for face feature extraction and comparison operations. At the same time, heterogeneous computing is implemented through the HiAI Foundation Kit, and different computing tasks are allocated to the CPU or NPU for execution according to the characteristics of the tasks. For example, the image preprocessing part of the face detection task is allocated to the CPU, while the face feature extraction and comparison tasks are allocated to the NPU to give full play to the advantages of the hardware.
   • Storage Layer: It is used to store the face database, system configuration information, log data, etc. When storing the face database, the security and efficient retrieval of data need to be considered. An appropriate database management system, such as MySQL or SQLite, should be adopted, and data encryption and backup should be carried out to ensure that the data is not leaked or lost. At the same time, the storage layer is also responsible for storing the processed result data, such as recognition records and alarm information, for subsequent query and analysis.

(3) Collaborative Work of Face Comparison and Heterogeneous Computing

In the system architecture, face comparison and heterogeneous computing technologies work closely together. After the data collection layer transmits the video stream to the processing layer, the CPU first performs preliminary video decoding and preprocessing, such as image cropping and normalization. Then, the Core Vision Kit is used to extract face features, and these feature extraction tasks will be allocated to the NPU for execution by the HiAI Foundation Kit to accelerate the calculation process. When performing face comparison, the extracted features are compared with the face database in the storage layer, and the comparison process is also carried out on the NPU to improve the comparison speed. Heterogeneous computing dynamically allocates computing resources according to the system load and task priorities to ensure that the face comparison task can be completed in a timely and accurate manner without affecting the execution of other system tasks. For example, when a large amount of video data is received from multiple cameras at the same time, heterogeneous computing will reasonably allocate the resources of the CPU and NPU, prioritize the face comparison task, and ensure the real-time and accuracy of the system.

II. Implementation of Core Functions and Technology Integration

(1) Implementation and Optimization of the Face Comparison Function

1. Implementation Process Using the Core Vision Kit The Core Vision Kit provides rich interfaces for the implementation of the face comparison function. First, through its face detection interface, the position and size of the face in the video frame are detected. Then, the face feature extraction interface is used to convert the detected face into a feature vector. The following is a simple code example (simplified version):

import { FaceDetector, FaceFeatureExtractor } from '@kit.CoreVisionKit';

// Initialize the face detection model
let faceDetector = new FaceDetector();
await faceDetector.init();

// Initialize the face feature extraction model
let faceFeatureExtractor = new FaceFeatureExtractor();
await faceFeatureExtractor.init();

// Read the video frame (assuming the video frame data has been obtained)
let frame = getVideoFrame();

// Detect the face
let faceResults = await faceDetector.detect(frame);
if (faceResults.length > 0) {
    // Extract the face features
    let faceFeature = await faceFeatureExtractor.extract(frame, faceResults[0].boundingBox);
    // Here, the extracted features can be stored or compared with the features in the database, etc.
}

In this example, the face detection and feature extraction models are first created and initialized, and then the face is detected from the video frame. If a face is detected, its features are extracted.

1. Technical Points of Performance Optimization and Code Examples To improve the performance of face comparison, optimization can be carried out in combination with the technical points in the document. For example, in the face detection stage, the image pyramid technique can be adopted to detect images at different resolutions, improving the detection rate of small-sized faces. In the feature extraction stage, the input image is normalized to make the feature extraction more stable and efficient. The following is a code snippet of face detection combined with the image pyramid technique (simplified version):

import { ImagePyramid, FaceDetector } from '@kit.CoreVisionKit';

// Create an image pyramid
let imagePyramid = new ImagePyramid(frame, { minSize: 30, maxSize: 300, scaleFactor: 1.2 });

// Traverse each level of the image pyramid for face detection
for (let i = 0; i < imagePyramid.getNumLevels(); i++) {
    let levelFrame = imagePyramid.getLevel(i);
    let faceResults = await faceDetector.detect(levelFrame);
    if (faceResults.length > 0) {
        // Process the detected face results, such as converting the coordinates to the original image coordinates, etc.
        for (let face of faceResults) {
            face.boundingBox = imagePyramid.convertToOriginalCoordinates(face.boundingBox, i);
        }
        break; // If a face is detected at a certain level, the loop can be ended early
    }
}

In this example, by creating an image pyramid, face detection is carried out on images at different resolutions, improving the accuracy and efficiency of detection.

(2) Application Implementation of Heterogeneous Computing in the Security System

1. Code Example of Task Allocation through the HiAI Foundation Kit The HiAI Foundation Kit provides powerful interfaces for implementing the allocation of heterogeneous computing tasks. The following is a simple example showing how to allocate a simple computing task (assuming it is matrix multiplication, used to simulate some calculations in face comparison) to the CPU and NPU for execution (simplified version):

import { HiaiEngine } from '@kit.HiAIFoundationKit';

// Define the matrix multiplication calculation function (assuming the matrix multiplication algorithm has been implemented here)
function matrixMultiplication(a: number[][], b: number[][]) {
    // The calculation logic is omitted, and the matrix multiplication algorithm needs to be implemented in practice
    return result;
}

// Create a CPU engine instance
let cpuEngine = new HiaiEngine('CPU');
// Create an NPU engine instance
let npuEngine = new HiaiEngine('NPU');

// Define two matrices
let matrixA = [[1, 2, 3], [4, 5, 6]];
let matrixB = [[7, 8], [9, 10], [11, 12]];

// Execute the matrix multiplication task on the CPU
let cpuResultPromise = cpuEngine.executeTask(() => {
    return matrixMultiplication(matrixA, matrixB);
});

// Execute the matrix multiplication task on the NPU (assuming the NPU supports matrix multiplication acceleration operations, this is just an example)
let npuResultPromise = npuEngine.executeTask(() => {
    return matrixMultiplication(matrixA, matrixB);
});

// Wait for the CPU task to complete and get the result
cpuResultPromise.then((result) => {
    console.log('CPU calculation result:', result);
});

// Wait for the NPU task to complete and get the result
npuResultPromise.then((result) => {
    console.log('NPU calculation result:', result);
});

In the actual intelligent security system, calculation-intensive tasks such as feature extraction and comparison in face comparison can be allocated to the NPU, while some data preprocessing and result processing tasks can be allocated to the CPU, thereby improving the overall processing efficiency of the system.

1. Resource Scheduling Strategies in Multitask Processing In a multitask processing scenario, such as performing face comparison on the video data from multiple cameras at the same time, a reasonable resource scheduling strategy is required. Scheduling can be carried out according to factors such as task priorities, the occupancy of computing resources, and the real-time requirements of tasks. For example, for face comparison tasks with high real-time requirements, NPU resources are preferentially allocated; when NPU resources are tight, some non-critical tasks (such as historical data statistics tasks) are temporarily suspended or transferred to the CPU for execution. At the same time, a task queue is used to manage tasks to ensure that tasks are executed in a reasonable order and avoid problems such as resource competition and deadlocks. For example, when a new video frame from a camera needs to be subjected to face comparison, the task is added to the task queue, and the tasks in the queue are executed in turn according to the resource situation and task priorities.

(3) Handling of Complex Scenarios

1. Methods for Handling Multiple Face Recognition In a multiple face recognition scenario, the system needs to accurately recognize multiple faces appearing in the video frame. First, all face positions and sizes are detected through the face detection algorithm, and then feature extraction and comparison are carried out for each face separately. To improve the efficiency of multiple face recognition, parallel computing can be adopted to process multiple faces simultaneously. For example, using the parallel computing capabilities of a multi-core CPU or NPU, the feature extraction and comparison tasks of each face are allocated to different computing cores for simultaneous execution. In the comparison result processing stage, the identity of each face is determined according to the comparison score and the preset threshold. If there are cases where the comparison scores of multiple faces are close to the threshold, secondary verification or manual intervention can be further adopted for confirmation to improve the accuracy of recognition.
2. Recognition Strategies in Case of Occlusion When a face is partially occluded, the difficulty of face comparison increases. For this situation, a recognition method based on local features can be adopted. First, the occluded area of the face is detected, and then the local features that are not occluded are focused on for comparison. For example, when the eyes or mouth are partially occluded, the features of the nose, cheeks, and other unoccluded areas are used for recognition. At the same time, multi-modal information, such as human body posture and behavior features, can be combined for auxiliary recognition. For example, if a person is wearing a mask, but their walking posture and behavior pattern are similar to those of a person in the database, it can be used as an auxiliary judgment basis to improve the accuracy of recognition. In addition, when training the face comparison model, face data in occluded situations should be added for training, so that the model can learn the feature change patterns in occluded situations and improve the robustness of the model in occluded scenarios.

III. Performance Evaluation and System Optimization

(1) Performance Evaluation Indicators and Methods

1. Evaluation of Face Comparison Accuracy Accuracy and recall rate are used as the main evaluation indicators. Accuracy represents the proportion of the number of correctly recognized faces to the total number of recognition times, and recall rate represents the proportion of actually existing faces that are correctly recognized. During the testing process, a large number of face images or video data containing different lighting, postures, expressions, and occlusion situations are collected to form a test set. These data are input into the intelligent security system for face comparison, and then compared with the real annotation results to calculate the accuracy and recall rate. For example, in a test set containing 1000 face images, the system correctly recognized 950 faces, and there were actually 1000 faces and the system correctly recognized 900 of them. Then the accuracy is 95%, and the recall rate is 90%.
2. Evaluation of Response Speed The time interval from when the camera collects the video frame to when the system outputs the face comparison result is measured as the evaluation indicator of the response speed. A high-precision timer is used to record the time, and the test is carried out under different system load conditions (such as processing the data from different numbers of cameras at the same time). For example, when processing the data from a single camera, record the average response time of the system; then gradually increase the number of cameras and observe the change in the response time. At the same time, analyze the components of the response time, including data collection time, data transmission time, calculation time (face comparison and heterogeneous computing), etc., and find out the bottleneck links that may affect the response speed.

(2) Analysis of Performance Bottlenecks and Optimization Strategies

1. Analysis and Optimization of Data Transmission Bottlenecks In an intelligent security system, data transmission may become one of the performance bottlenecks. For example, when the data transmission bandwidth between multiple cameras and the processing device is insufficient, it will lead to video data transmission latency, affecting the real-time performance of the system. Analyze factors such as the network topology, transmission protocol, and data traffic in the data transmission process to find out the possible problems. Optimization strategies include adopting network devices with higher bandwidth, optimizing the network configuration (such as setting reasonable IP addresses, subnet masks, gateways, etc.), and selecting more efficient transmission protocols (such as using the UDP protocol for video data transmission and combining it with a reliable transmission mechanism to ensure the integrity of the data). At the same time, the video data should be compressed to reduce the data transmission volume, but attention should be paid to the impact of the compression algorithm on the image quality and the subsequent face comparison performance, and an appropriate compression ratio and compression algorithm should be selected.
2. Problems of Unreasonable Allocation of Computing Resources and Optimization If the allocation of computing resources is unreasonable, it may lead to some tasks waiting for too long, reducing the overall performance of the system. For example, when multiple computation-intensive tasks compete for NPU resources at the same time, some tasks may queue up and wait, while the CPU resources are idle. By monitoring the resource usage of the system, such as the load of the CPU and NPU and the memory usage, analyze whether the allocation of computing resources is reasonable. Optimization strategies include dynamically adjusting the task allocation strategy, reasonably allocating the resources of the CPU and NPU according to the task priorities and resource requirements. For example, when the NPU load is too high, transfer some computing tasks with low real-time requirements to the CPU for execution; adopt task scheduling algorithms, such as First Come First Served, Shortest Job First, or Priority Scheduling algorithms, to optimize the execution order of tasks and improve resource utilization.

(3) Display of the Application Effect of the Optimized System and Experience Summary

1. Display of the Application Effect in the Actual Scene In an actual intelligent community security system, the optimized intelligent security system based on HarmonyOS Next for face comparison and heterogeneous computing has been deployed. The system is connected to multiple cameras in the community, covering key areas such as the community gate, corridors, and parking lots. During operation, the accuracy rate of face comparison reaches over 98%, and it can accurately identify community residents and outsiders. In terms of response speed, when processing the video data from 10 cameras at the same time, the average response time is controlled within 0.5 seconds, achieving real-time monitoring and early warning functions. For example, when a stranger enters the community, the system can identify it in a short time and issue an alarm to notify the community security personnel for handling, effectively improving the security of the community.
2. Experience Summary and Lessons Learned
   • Experience Summary: In the development process, making full use of the features of HarmonyOS Next and related technical tools is the key to improving the system performance. For example, reasonably using the Core Vision Kit and HiAI Foundation Kit can quickly realize the functions of face comparison and heterogeneous computing, and improve the system performance by optimizing the algorithm and resource allocation strategy. At the same time, in terms of data processing, pay attention to the quality of the data and the preprocessing work, such as diversifying the training data to improve the robustness of the face comparison model.