The availability of vast amounts of location-based data from social media platforms such as Twitter has enabled us to look deeply into the dynamics of human movement. In particular, geo-tagged tweets provide us with extensive, fine-grained data on people’s movements both in space and in time. The aim of this paper is to leverage a large collection of geo-tagged tweets and the street networks of two major metropolitan areas—London and Tokyo—to (1) show the heterogeneity of human urban movements, and (2) explore the underlying mechanism that determines the heterogeneity of human mobility patterns. The study relies on two topological representations based on city hotspots and natural streets to capture the spatial heterogeneity of urban space and the human activities contained therein. For the two target cities, hundreds of thousands of tweet locations and road segments were processed to generate city hotspots and natural streets. User movement trajectories and city hotspots were then used to build a hotspot network capable of quantitatively characterizing the heterogeneous movement patterns of people within the cities. To emulate observed movement patterns, the study conducts a two-level agent-based simulation that includes random walks through the hotspot networks and movements in the street networks using each of three distance types—metric, angular and combined. Comparisons of the simulated and observed movement flows at the segment and street levels show that the heterogeneity of human urban movements at the collective level are mainly shaped by the scaling structure of the urban space.