DualTime: A Dual-Adapter Multimodal Language Model for Time Series Representation

DualTime is a multimodal language model that effectively leverages the complementary strengths of time series and textual data to improve time series representation, outperforming existing models in various settings and showcasing its potential in applications like medical diagnosis.

Abstract

The recent rapid development of language models (LMs) has attracted attention in the field of time series, including multimodal time series modeling. However, we note that current time series multimodal methods are biased, often assigning a primary role to one modality while the other assumes a secondary role. They overlook the mutual benefits and complementary of different modalities. For example, in seizure diagnosis, relying solely on textual clinical reports makes it difficult to pinpoint the area and type of the disease, while electroencephalograms (EEGs) alone cannot provide an accurate diagnosis without considering the symptoms. In this study, based on the complementary information mining of time series multimodal data, we propose DualTime, a Dual-adapter multimodal language model for Time series representation implementing temporal-primary and textual-primary modeling simultaneously. By injecting lightweight adaption tokens, the LM pipeline shared by dual adapters encourages embedding alignment and achieves efficient fine-tuning. Empirically, our method outperforms state-of-the-art models in both supervised and unsupervised settings, highlighting the complementary benefits of different modalities. In addition, we conduct few-shot label transfer experiments, which further verifies the transferability and expressiveness of our proposed DualTime.

Method

The authors of the paper proposed a novel methodology called DualTime, which is a dual-adapter multimodal language model for time series representation learning. This methodology involves the use of two adapters, one treating time series data as the primary modality and the other treating textual data as primary, to fully exploit the complementary information between the two modalities. The adapters share the parameters of a frozen pre-trained language model, which allows for efficient fine-tuning and alignment of the embedding spaces. The model introduces lightweight adaption tokens that are injected into the intermediate layers of the language model to achieve multimodal fusion, and it employs a zero-initialized gating strategy to preserve the pre-trained knowledge of the language model. The methodology is validated through extensive experiments, demonstrating its superior performance in both supervised and unsupervised learning tasks, as well as its transferability in few-shot learning scenarios.

Main Finding

The authors of the paper developed DualTime, a dual-adapter multimodal language model that significantly enhances time series representation by effectively leveraging the complementary information from both time series and textual data. Unlike previous models that favor one modality over the other, DualTime treats both modalities as primary, using lightweight adaption tokens for multimodal fusion within a shared pre-trained language model framework. This approach not only allows for efficient fine-tuning but also ensures better alignment of the embedding spaces. The model's superior performance in both supervised and unsupervised learning tasks, as well as its strong transferability in few-shot learning scenarios, demonstrates its potential for applications like medical diagnosis, where it can distinguish between different classes of time series data with high accuracy and efficiency.

Conclusion

The conclusion of the paper is that the authors have successfully proposed and validated a novel multimodal language model, DualTime, which effectively utilizes the complementary strengths of time series and textual data to improve time series representation learning. The model's dual-adapter design, which treats both modalities as primary, has been shown to outperform existing state-of-the-art models in various settings, including supervised and unsupervised learning tasks, as well as in few-shot label transfer experiments. This demonstrates the model's robustness, transferability, and discriminative power, making it a promising approach for applications such as medical diagnosis where the integration of multimodal data is crucial. The authors also suggest that future work could explore the impact of different language models on multimodal learning results, expressing confidence in the adaptability of their model design to various language model backbones.

Keywords

DualTime, multimodal language model, time series representation, dual-adapter, multimodal fusion, lightweight adaption tokens, pre-trained language model, fine-tuning efficiency, embedding space alignment, transferability, few-shot learning, medical diagnosis, ECG, EEG, clinical reports, seizure diagnosis, language models, LMs, time series modeling, multimodal data, complementary information, temporal-primary, textual-primary, multimodal fusion layers, gating strategy, supervised learning, unsupervised representation learning, contrastive learning, efficiency analysis, visualization analysis, UMAP, trainable parameters, training time, PTB-XL dataset, TUSZ dataset, ECG signals, EEG signals, clinical history, dataset statistics, data splitting, evaluation metrics, accuracy, precision, recall, F1 score, hyperparameter study, model efficiency, discriminative power.