Abstract—Multiple-Input Multiple-Output (MIMO) systems are fundamental to modern mobile communications, enabling beam steering, enhanced data capacity, and diversity gains in 5G networks. Space-Time Block Coding (STBC) improves the data rate of MIMO by creating orthogonal channels, yet this typically requires Channel State Information (CSI) obtained through pilot-based training. However, no existing algorithm can estimate the channel matrix for a general STBC system without modifying the pilot or pre‑coding structures. This limitation becomes particularly acute when Differential Chaos Shift Keying (DCSK)—valued for its robustness and inherent secrecy—is combined with MIMO, because its non‑periodic chaotic carrier is incompatible with conventional pilot‑based estimation. To address this issue, we propose an Adaptive Deep Neural Network-Autoencoder (ADNN-AE) for channel estimation in DCSK-based MIMO-STBC systems. In this framework, an autoencoder stage first denoises the chaotic signal, and a Deep Neural Network (DNN) then estimates the channel from the refined representation.
The hyperparameters—hidden‑neuron counts and training epochs of both stages—are automatically tuned by a Hybrid Artificial Gorilla Troops Optimizer-Anti-Coronavirus Optimizer (HAGT-ACO), eliminating manual trial‑and‑error. The proposed HAGT-ACO-ADNN-AE model was evaluated against ADNN‑AE baselines tuned with Sunflower Optimization (SFO), Deer Hunting Optimization Algorithm (DHOA), Artificial Gorilla Troops Optimization (AGTO) alone, and Anti-Coronavirus Optimization Algorithm (ACOA) alone, under 2 × 2 and 4 × 4 antenna configurations and multiple modulation schemes.
It achieves the lowest mean Bit Error Rate (BER) of 0.000184, yielding improvements of 30.6%, 44.2%, and 59.2% over SFO, DHOA, and ACOA-based baselines, respectively, and a 1.1% improvement over AGTO alone. Moreover, it exhibits the lowest standard deviation and the shortest execution time (5.17 s), confirming both the accuracy and efficiency of the proposed approach.  
 
Keywords—multiple input multiple output, space time block coding, differential chaos shift keying, adaptive deep neural network, optimization techniques