Visual localization is an accurate and low-cost indoor localization solution. A bottleneck for visual localization is the computation efficiency of continuous image searching and matching. In this paper, an indoor visual localization method is proposed to realize continuous and accurate indoor localization based on image matching. This method uses smartphones to collect multi-sensor data, including video frames and inertial readings. To improve the computation efficiency of the proposed visual localization method, a spatial model is developed to optimize the spatial organization of geo-tagged images in a dataset. Several spatial constraint-based image searching strategies are also designed to further reduce the computation time. Based on the spatial model and spatial constraint-based strategies, a visual localization algorithm is proposed. The experimental results show that the localization errors of the image querying, continuous offline localization and online localization of this method are approximately 0.4 m, 0.7 m and 0.9 m, respectively. This method can achieve an accuracy of 1.3 m, even under a random camera opening condition. The average computation time (i.e.. the average time needed to provide a location estimation result) is approximately 0.59 s. The results indicate that the proposed method can realize efficient and continuous indoor localization with high localization accuracy.