Deep learning models have demonstrated their potential in learning effective molecular representations critical for drug property prediction and drug discovery. Despite significant advancements in leveraging multimodal drug molecule semantics, existing approaches often struggle with challenges such as low-quality data and structural complexity. Large language models (LLMs) excel in generating high-quality molecular representations due to their robust characterization capabilities. In this work, we introduce GICL, a cross-modal contrastive learning framework that integrates LLM-derived embeddings with molecular image representations. Specifically, LLMs extract feature representations from the SMILES strings of drug molecules, which are then contrasted with graphical representations of molecular images to achieve a holistic understanding of molecular features. Experimental results demonstrate that GICL achieves state-of-the-art performance on the ADMET task while offering interpretable insights into drug properties, thereby facilitating more efficient drug design and discovery.