In order to achieve high data rates in future wireless packet switched cellular networks, aggressive frequency reuse is inevitable due to the scarcity of the radio resources. While intra-cell interference is mostly mitigated and can be ignored, inter-cell interference can severely degrade performances of end-users. Hence, Inter-Cell Interference Coordination is commonly identified as a key radio resource management mechanism to enhance system performance of 4G networks. This paper addresses the problem of ICIC in the downlink of Long Term Evolution (LTE) systems where the Resource Blocks (RB) selection process is inspired from the reinforcement learning theory targeted to address the adversarial Multi-Armed Bandit problem. We resort to the popular EXP3 algorithm whose goal is to steer autonomously the decision of each Base Station (BS) towards the least interfered RBs while ensuring reactivity to the possible changes that can occur in the common resource usage and radio channel quality. However, the EXP3 algorithm is computationally heavy as its strategy set grows exponentially with the number of needed RBs and the total amount of available RBs. Therefore, we propose an efficient adaptation of the EXP3 algorithm, deemed Q-EXP3, where the needed RBs are selected one by one requiring only polynomial time computation.