High precision modeling in industrial systems is difficult and costly. Model-free intelligent control methods, represented by reinforcement learning, have been applied in industrial systems broadly. The hard evaluated of production states and the low value density of processing data causes sparse rewards, which lead to an insufficient performance of reinforcement learning. To overcome the difficulty of reinforcement learning in sparse reward scenes, a reinforcement learning method with reward shaping and hybrid exploration is proposed. By perfecting the rewards distribution in the state space of environment, the reward shaping can make the state-value estimation of reinforcement learning more accurate. By improving the rewards distribution in time dimension, the hybrid exploration can make the iteration of reinforcement learning more efficient and more stable. Finally, the effectiveness of the proposed method is verified by simulations.