Terrain reconstruction aims at acquiring height maps by detecting corresponding feature points from two or more down-looking remote sensing images. This paper proposes a terrain reconstruction algorithm based on epipolar line rectification and dense matching. At first, it uses fundamental matrix to rectify stereo images to make their epipolar lines parallel and remove the disparity in vertical direction. Then, dense matching based on grid method is applied, which can provide sufficient matching points to estimate disparity of the rectified images. Finally, the heights of the matched points can be calculated according to the obtained disparity and flight parameters. Experiments show that our algorithm can generate precise and reliable height maps for well depicting terrain features.