Rate-distortion and complexity joint optimization for fast motion estimation in H.264 video coding

H.264 video coding standard offers several coding modes including inter-prediction modes that use macroblock partitions with variable block sizes. Choosing a rate-distortion optimal mode among these possibilities contributes significantly to the superior coding efficiency of the H.264 encoder. Unfortunately, searching for optimal motion vectors of each possible subblock incurs a heavy computational cost. In this paper, in order to reduce the complexity of integer-pel motion estimation, we propose a rate-distortion and complexity joint optimization method that selects for each MB a subset of partitions to evaluate during motion estimation. This selection is based on simple measures of spatio-temporal activity within the MB. The procedure is optimized to minimize mode estimation error at a certain level of computational complexity. Simulation results show that the algorithm speeds up the motion estimation module by a factor of up to 20 with little loss in coding efficiency.