| The advent of the 5G and the evolution toward the Sixth Generation (6G) have
introduced ambitious requirements in terms of latency, throughput, and device density,
while also amplifying the importance of robust security. Balancing QoS and security in
such dynamic and heterogeneous environments presents a significant research challenge,
particularly as optimizing one often compromises the other.
This thesis addresses this challenge through the application of Deep Reinforcement
Learning (DRL) to optimize network parameters for both QoS and security simultaneously.
The study is presented in two parts, each corresponding to a peer-reviewed conference paper.
The first part explores the design of a DRL-based framework that dynamically adjusts
5G network parameters to strike an optimal balance between QoS and security using a custom
dataset called QoS 5G Sec. A comparative analysis of state-of-the-art DRL agents—Deep QNetwork (DQN), Proximal Policy Optimization (PPO), and Advantage Actor-Critic (A2C)—is
conducted with rigorous hyperparameter tuning. Results show that the DQN agent, optimized
using Optuna, offers the best performance in terms of stability, convergence, and adaptability.
The second part extends the investigation to 6G-relevant environments by addressing
the problem of intelligent handover in dense indoor WiGig networks. A novel reward function
is proposed to enhance learning in the presence of closely clustered channel gains. Multiple
DRL agents are evaluated across various user-density scenarios. The findings confirm that a
generalized DQN agent trained across all scenarios not only outperforms specialized models
but also demonstrates strong adaptability to unseen conditions.
Together, these contributions provide a unified approach to enhancing both QoS and
security in next-generation wireless networks, paving the way for scalable and intelligent
DRL-based network management.
Keywords: 5G, 6G, Quality of Service (QoS), Security, Deep Reinforcement Learning (DRL),
Intelligent Handover, WiGig Networks |