TDengine Achieves 10x Compression vs. Elasticsearch for Smart Vehicle Solution Provider

A smart vehicle solutions provider had developed a system for their tire products. As the scale of data within the system continued to grow, the overall business response speed and stability of the system became pain points for their internal and external customers. To improve performance with large-scale data, the project team responsible for the system decided to replace Elasticsearch with TDengine. This article provides a comprehensive analysis of how the system was transformed by the introduction of TDengine.

Project Background

Over the past two years, the system had been integrated with a smart rental system as well as the fleet management system of a major domestic logistics firm. With this expansion, Elasticsearch was no longer able to meet users’ data demands. The team began researching alternative data management platforms and in the end chose to deploy a specialized time-series database.

After conducting various evaluations, TDengine was ultimately selected as the time-series database for the smart tire system. The main reasons mentioned are listed as follows:

• Business Suitability: TDengine is designed and optimized for industrial use cases like IoT and connected cars.
• Operational Costs: TDengine Enterprise includes 24/7 technical support for operations and maintenance, significantly reducing operational costs.
• Data Compression: TDengine’s columnar storage architecture enables a high data compression ratio. Compared with Elasticsearch, real-world tests showed that TDengine achieved a compression ratio of over 1:10, cutting disk storage costs.
• High Read/Write Performance: TDengine has excellent write performance for time-series data. Tests with a three-node high-performance cloud disk setup achieved a write speed of 300,000 data points per second. Its read performance is also outstanding, with query speeds 10 times faster than Elasticsearch for the same volume of time-series data.
• Low Resource Consumption & High Availability: TDengine has low server resource usage and offers high availability and stability. It supports user-defined functions (UDFs) for algorithm integration and various built-in mathematical functions.
• Cold & Hot Data Separation: TDengine supports tiered storage, automatically migrating historical data to lower-cost storage media, further reducing enterprise data storage expenses.

System Architecture & Implementation Insights

The current project uses TDengine Enterprise 3.1.1.11 in a three-node cluster, with each node configured with a 16-core CPU, 64 GB of RAM, and 3 TB of high-performance cloud storage. Below is a simplified data flow architecture:

Edge gateways collect metrics such as tire temperature, pressure, and leakage status at set intervals (in seconds). After initial data integration and threshold comparison at the edge, the edge gateway sends data to the cloud gateway at three-minute intervals (or in seconds during alert conditions) via TCP and a proprietary protocol.

After passing through the cloud gateway, real-time streaming data undergoes decoding, real-time analysis, alert pushing, and data integration before being batch-written into TDengine. Data is presorted before ingestion to ensure sequential writes and reduce data fragmentation.

TDengine’s “one table per device” model was adopted for data storage, and the database for vehicle tire temperature and pressure data was configured as follows:

• Data retention period: 180 days
• File partitioning: 1-day intervals
• Data differentiation: Timestamp-based distinction for multiple tire positions on the same vehicle

Additionally, by leveraging TDengine’s built-in functions and user-defined functions, real-time stream processing was implemented for time-series algorithms (e.g., real-time temperature situational awareness) to improve warning efficiency, reduce false alarms, and enhance responsiveness.

Performance & Transformation Results

Since going live, the project has been running stably with a steady resource consumption rate. Compared with Elasticsearch, TDengine has demonstrated significant advantages in storing time-series data:

Disk Usage:

• Elasticsearch: Uses Lucene for JSON-based storage, leading to low compression and high disk usage.

• TDengine: Columnar storage ensures high compression, achieving a 10x reduction in disk usage compared to Elasticsearch.

Memory Usage (64 GB per node):

• Elasticsearch: Each of the 4 nodes used more than 54 GB of memory.

• TDengine: Each of the 3 nodes used only about 4 GB of memory.

Other advantages of TDengine include:

• High Read/Write Performance: Extremely fast sequential write speeds and highly efficient time-range queries.
• Complex Time-Series Computation: The UDF feature allows real-time stream computing for tire temperature trends and other complex time-series analytics without affecting overall system stability or latency.
• Data Subscription Services: Using TDengine’s data subscription feature, data is seamlessly transferred to a cloud compute platform for near real-time visualization and modeling. Additionally, third-party platforms can leverage TDengine’s data subscription feature for seamless data integration, facilitating data exchange with logistics companies, vehicle manufacturers, and public transportation providers.

Visualization & Future Prospects

In the field of vehicle tire temperature and pressure monitoring, the completeness, real-time performance, interactivity, and flexibility of displayed information are crucial. TDengine’s efficient query capabilities and easy-to-use SQL syntax enable seamless implementation of these requirements. TDengine has increased performance and reduced costs at this company, and they have also used its integrations to built a dashboard displaying various vehicle alert events, supporting fleet operators in making data-driven decisions.