The Apache SeaTunnel 2.3.8 version has officially been released! With this update, users can now enjoy the much-anticipated Docker image support, the Job-level logging feature, and other enhancements and optimizations. This article will detail the key updates in Apache SeaTunnel 2.3.8. We encourage more developers and users to join our open-source community.
- Download SeaTunnel 2.3.8: https://seatunnel.apache.org/download/
- Release Notes: https://github.com/apache/seatunnel/releases/tag/2.3.8
Key Updates
Job-Level Logging
In this update, we've optimized the logging functionality. In previous versions, logs from multiple tasks were written into a single file. When running multiple tasks simultaneously, the logs were intertwined, making troubleshooting difficult.
The new update supports two configurations for more efficient log querying.
The first option is adding a JobId to each log entry, allowing you to filter logs by JobId.
The second option is to split log files by JobId. Modifying the log configuration file allows you to create separate log files for each JobId.
New Docker Image
This update includes official Docker image support. The image comes with all connectors pre-installed, enabling users to run SeaTunnel more conveniently without downloading additional packages. This reduces deployment complexity and streamlines the process for users deploying SeaTunnel with Kubernetes.
For users requiring customization or additional development, the new version also offers a one-click command to build customized images:
Multi-Table Support for Flink/Spark Engines
In previous versions, the multi-table read and write feature was only supported on the Zeta engine. With this update, Spark and Flink engines now also support multi-table read and write operations.
Prometheus Integration for Cluster Monitoring
Previously, users needed to access API endpoints to retrieve cluster and task metrics. Now, users can export these metrics to Prometheus. Prometheus will regularly pull the status of SeaTunnel clusters and tasks, providing a visual interface to monitor the cluster and identify issues quickly.
Typesense Connector Support Added
Support for the Typesense connector has been added in this release.
Improvements and Optimizations
Embedding Transform Added
With the Embedding transform, SeaTunnel can embed machine learning models into the data transformation process. This allows raw fields to be converted into vector values and stored in the corresponding machine-learning databases. Currently, SeaTunnel supports models from vendors like Doubot, Qianfan, and OpenAI, with more model providers to be added in the future.
Kafka Support for Protobuf Read/Write
The Kafka connector now supports reading and writing Protobuf data formats. A Protobuf data type definition has been added under the Kafka connector for data reading and writing.
File Support for Reading Compressed Archives
Support for reading compressed file formats has been added, eliminating the need for manual extraction steps.
Finer-Grained Resource Loading Isolation
ClassLoader isolation has been improved from task group level to task level, avoiding dependency conflicts when using the same ClassLoader for Source/Sink.
Other optimizations include:
- Paimon stream read and dynamic bucket write support.
- SQL nested query support for querying Map structure fields.
- Iceberg-related optimizations.
- Kerberos authentication support.
- SaveMode adds IGNORE type.
- Optimization for reading across Redis versions.
- Adaptation for MySQL 8.1/8.2/8.3 versions.
- Support for TiDB CDC reads.
- Removal of system table restrictions in JDBC connectors.
- Event notification support for all connectors.
- Multi-table read support for the ES connector.
- Multi-table write support for the HBase connector.
Key Bug Fixes
- Fixed Hazelcast networking issues with non-TCP connections.
- Resolved resource isolation issues.
- Fixed Paimon dynamic bucket table and decimal precision loss issues.
- Fixed resource leakage at the end of Iceberg tasks.
For more detailed updates, please refer to the Release Notes: https://github.com/apache/seatunnel/releases/tag/2.3.8
Thanks to Our Contributors
Special thanks to @liunaijie for guidance and assistance with this release. Additionally, we’d like to thank the following community members for their efforts in making this release a success:
hailin0, hawk9821, cl0924, sunxiaojian, dailai, corgy-w, Hisoka-X, liunaijie, chl-wxp, zhangshenghang, ISADBA, loustler, chenqianwen, FuYouJ, xxsc0529, EricJoy2048, ZhangWeike2000, jw-itq, kevinjmh, Carl-Zhou-CN, FlechazoW, PeppaPage, liugddx, Cheun99, happyboy1024, CosmosNi, Anush008, BruceWong96, zqr10159, cloud456, Gxinge, xxsc0529, luzongzhu, jiamin13579, Zuhdan, yujian225.
Top comments (0)