Special Articles on XR—Initiatives to Form an XR Ecosystem—
“Virtual Ginza” Based on an Urban Spatial Dataset

Cloud Rendering MEC Mobile Spatial Statistics

Daizo Ikeda
Strategic Business Development Division, NTT Corporation

Masahiro Yuguchi and Takafumi Yamazoe
Service Innovation Department

Abstract
As part of the Lifestyle Co-Creation Lab’s efforts to create a “Wellbeing Society,” NTT DOCOMO reproduced the Ginza district of Tokyo in a virtual space under the name “Virtual Ginza,” where it is verifying the value a 3D space offers mainly for regional revitalization through tour experiences and gamification. At docomo Open House’22, we built a Virtual Ginza in NTT DOCOMO’s Open Innovation Cloud (OIC) as a cloud-gaming system and implemented a system configuration that transmits images of the virtual space from the avatar’s viewpoint operated by the user to their terminals using high-speed, high-capacity, low-latency 5G SA communications. This configuration enables an immersive and exhilarating gamification experience in the vast Ginza area regardless of the processing performance of the user’s terminal.

01. Introduction

• As part of the Lifestyle Co-Creation Lab’s efforts to create, ...

  Open

  As part of the Lifestyle Co-Creation Lab’s efforts to create a “Well-being Society,”^*1 NTT DOCOMO reproduced the Tokyo district of Ginza in a virtual space (hereafter referred to as “Virtual Ginza”) and is verifying the value a 3D space offers mainly for regional revitalization.

  Recently, the virtual space for communication is called the metaverse^*2, which consists of 3D spatial data, objects including 3D avatars and geographic objects, projections of sensor data, executions of simulations, and interactions between 3D avatars and objects, etc. In Japan, the Ministry of Land, Infrastructure, Transport and Tourism is leading the development of a nationwide 3D city model^*3 dataset as “Project PLATEAU” and providing it as open data for promoting the development of digital twins^*4 (in which the real world is projected onto the virtual world) and developing use cases [1]. This urban spatial dataset, called a 3D city model, is generated in accordance with the international standard data format City Generalized Markup Language (CityGML)^*5 [2] by adding 3D information from aerial images to basic data about buildings as a planar map developed for urban-planning purposes. In addition to shape information about urban space such as polygon data, geographic objects such as buildings and streets can be created and defined in units of objects, which makes it possible to preserve attribute data such as the name, type, and year of construction of the object. Therefore, simulation, which is essential for digital twinning, and interaction through 3D avatars can be efficiently implemented on a 3D city model.

  The urban space is defined in terms of the concept called the Level Of Detail (LOD)^*6 in the 3D city model, and a dataset representing the roof shapes of buildings, called LOD2, can be used for the 23 wards of the Tokyo metropolitan area (as of March 2022).

  To reproduce Ginza virtually, a 3D city model corresponding to a 1-km-square block (including Harumi Avenue and Ginza Avenue), which run through the center of Ginza, was used (Figure 1). Since Virtual Ginza provides an experience of gamification^*7, the 3D city model in LOD2 was changed to pastel colors, and detailed models were created and placed as some of the buildings around the Ginza 4-Chome intersection (Figure 2). As a communication space between 3D avatars on a vast field in Ginza, a multiplayer system is constructed with Cloud Direct^*8 using a 5G Stand Alone (SA)^*9 mobile network, which enables high-speed, large-capacity, and low-latency communications. Moreover, the application of cloud-rendering^*10 technology eliminates the need to install an application on the terminal side and enables real-time communications via a Web browser. By utilizing the 3D city model and Mobile Spatial Statistics^*11[3], it is possible to generate a group of Non-Player Characters (NPCs)^*12, thus reproducing the bustling atmosphere of Ginza. In this article, the overview and design of Virtual Ginza are provided and a verification system utilizing 5G—which was exhibited at docomo Open House’22— is described.

  

  Figure 1  Extent of Ginza reproduced by CG

  

  Figure 2  A streetscape of Ginza reproduced by CG

  1. Wellbeing Society: NTT DOCOMO R&D aims to create a society in which each individual shines and all possibilities grow while staying close to others.
  2. Metaverse: 3D virtual space built on a computer network and the services using it.
  3. 3D city model: 3D geospatial data that reproduces real-world cities in the virtual world (cyberspace). It is being developed by the Ministry of Land, Infrastructure, Transport and Tourism as part of Project PLATEAU.
  4. Digital twin: A technology that uses various data to reproduce places, objects, and events that exist in the real world as if they were twins in a digital space.
  5. CityGML: A data format for 3D city models developed as an international standard by the Open Geospatial Consortium (OGC), an international standardization organization in the field of geospatial information.
  6. LOD: Level of detail of modeling in 3D CG.
  7. Gamification: Application of game design and rules to fields other than games.
  8. Cloud Direct: A service that provides closed communications separated from the Internet on DOCOMO’s 5G lines.
  9. 5G SA: A 5G service based on the SA model in which dedicated 5G equipment and 5G base stations are used, in contrast to NSAs (see *23) that use 4G equipment.
  10. Cloud rendering: Remotely using CG images drawn (rendered) on a server on a network.
  11. Mobile Spatial Statistics: Population statistics generated from mobile network operational data, which can be generated 24 hours a day, 365 days a year across Japan.
  12. NPC: A game term referring to a game character that is not controlled by the user.

02. Overview of Virtual Ginza

• As a cosmopolitan district full of history, culture, and ...

  Open

  As a cosmopolitan district full of history, culture, and modern attractions such as sightseeing, entertainment, and shopping, Ginza is one of the areas of Tokyo that has been greatly affected by the recent COVID-19 pandemic. By reproducing Ginza in a virtual space by using a 3D city model, we developed Virtual Ginza to revitalize the Ginza area in the “new normal”^*13 era through various experiences, and we started a demonstration experiment of Virtual Ginza in March 2021.

  Virtual Ginza offers a 3D avatar-based experience of touring and competitive gaming, which were created through gamification. Its aim is to improve the attractiveness of Ginza, which still retains its historic streetscape, by (1) creating detailed models of spots such as the birthplace of Ginza and the Kabukiza Theater, which are symbols of Ginza’s history and culture, and (2) by introducing a gamification experience through which visitors collect silver coins, which gave the area its name by being once made in Ginza, while touring historical and cultural spots. To express the lively aspect of the Ginza district, Mobile Spatial Statistics were utilized to arrange crowds as NPCs according to the number of visitors to Ginza as indicated by human flow data in those statistics. Mobile Spatial Statistics can generate nationwide population statistics 24 hours a day, 365 days a year. Those statistics can then be used to reproduce the bustle of Ginza on the basis of the population of Ginza before the COVID-19 pandemic and to include NPCs that follow the population in real time.

  The 3D avatar tour experience allows visitors to visit historical and cultural spots as well as several indoor spaces, purchase digital items, and change avatar costumes. At the NTT DOCOMO Store, one of the indoor spaces, visitors can purchase digital items that are representative of Ginza (such as goods related to Kabuki), and those purchases will improve the running speed and other abilities of the 3D avatars. Visitors can also visit other representative Ginza spots such as the Kabukiza Theater, POLA GINZA, and NISSAN CROSSING, and those visits will increase awareness and appeal of various brands, create opportunities for purchases, and lead to customers visiting actual stores.

  Through such gamification, Virtual Ginza has the effect of promoting goods related to Ginza. For example, several ad trucks equipped with large displays are placed on streets running through Ginza, and the volume of sound from the ad truck changes according to the distance from the 3D avatar to the ad truck and the viewpoint of the avatar. As a result, digital signage^*14 is implemented in a virtual space (Figure 3). The 3D acoustic effect thereby created causes the user to notice ads with images that are out of the user’s sight while experiencing a sense of presence by taking advantage of the 3D space.

  

  Figure 3  Ad trucks set up in the field

  As for the competitive game, the users play the game “parkour,” a dynamic sport that involves running, climbing, and jumping without the use of tools, in pairs of friends in manner that gives the users the experience of running through the streets of Ginza in pursuit of a goal (Figure 4). Playing parkour in Ginza provides users with an unusual and exhilarating experience. The game also creates a space in which users can interact with each other through the district of Ginza. That interaction is generated by interweaving collaborative experiences, such as catapulting a pair of users into the air when they jump on a launch ramp simultaneously or reaching a goal in alignment, as well as real-time communication such as voice chat.

  

  Figure 4  Appearance of multiplayers in a competitive game

  1. New normal: A state in which a new common sense has irreversibly taken hold as a result of changes in the social environment and circumstances.
  2. Digital signage: Advertising media using digital technology. Using displays or projectors to change advertising content in response to time or location, this technology is gaining attention as an alternative to conventional advertising media such as posters, etc.

03. Implementation of Urban Spatial Expression

• As for Virtual Ginza, meta-information about 3D city models and buildings, ...

  Open

  As for Virtual Ginza, meta-information about 3D city models and buildings, statistical information such as Mobile Spatial Statistics, item data for enabling gamification in 3D space, NPCs, and event scripts are combined to create an urban spatial dataset, which is used for expression of the urban space on a game engine^*15. In recent years, game engines have begun to be applied to a wide range of fields such as architectural visualization and video production as well as the development of games and Virtual Reality (VR) content. As for this demonstration experiment, we used Unreal Engine 4, a game engine capable of advanced graphic expression, to represent vast urban spaces and enable multiplayer play—both in the cloud and in local network environments—and verified the touring experience and gamification in a multiplayer environment.

  3.1 3D City Model

  The 3D-city-model dataset from the Ministry of Land, Infrastructure, Transport and Tourism’s Project PLATEAU (hereafter referred to as “Plateau dataset”) was used for representing Virtual Ginza in 3D CG. The Plateau dataset is a series of open-data releases of 3D CG data and meta-information about buildings in various cities, and 56 city datasets are available as of March 2022. However, since the Plateau dataset is generated in 3D CG from satellite-photo information, the resolution of texture^*16 is an issue when the first-person view from the ground is used; therefore, NTT DOCOMO created its own 3D CG data for some buildings. To verify an event using urban space, we also created data on the interior of some buildings (Figures 5, 6).

  

  Figure 5  Exterior and interior views of a NTT DOCOMO shop

  

  Figure 6  Inside the stadium (virtual building) that can be removed from the Ginza area

  3.2 Visualization of People-flow Data through Mobile Spatial Statistics

  With Mobile Spatial Statistics, data on people flows throughout Japan are generated as population statistics per area defined as, for example, a 500-m mesh^*17. In Virtual Ginza, the flow of people is represented as the number of NPCs passing by; however, as the volume of NPCs for displaying increases, the processing load for drawing them increases. Accordingly, it is necessary to reduce the volume by drawing only the range of the field of view from the user’s perspective; however, the size of the mesh of the people-flow data and the range of the drawing in the 3D space corresponding to the user’s field of view do not match.

  Given the above-described mismatch, the displayed number of passers-by passing through a zone covered by a 125-m mesh from the user’s perspective was calculated as N [person-hours: number of people passing through the city (population flow)] × t [seconds: time it takes an NPC to traverse the mesh in the range of the field of view from the user’s perspective (transit time)]. For example, if N per 125-m mesh per hour (3,600 s) is 2,000 and t of an NPC passing through a 125-m mesh is 90 s (equivalent to 5 km/h), the number of people who should be simultaneously drawn is 2,000/3,600 × 90 = 50 people. In addition to moving passersby, stationary crowds (lightweight models that do not compute movement) can be randomly placed on the sidewalk in the field of view from user’s viewpoint, etc., according to the user’s preference; as a result, the appropriate number of people can be drawn without adding processing load (Figures 7, 8).

  

  Figure 7  Variation in passerby NPCs

  

  Figure 8  NPC Passerby NPCs and stationary NPCs

  3.3 Verification of Effect of Touring Experience and Gamification

  In a demonstration experiment of Virtual Ginza, as part of the use case development measures [4] in the FY2020 Plateau Project of the Ministry of Land, Infrastructure, Transport and Tourism, a total of 40 subjects were tested to verify the effectiveness of gamification in improving the attractiveness of the area. We evaluated the user experience through questionnaires, play logs, and interviews, the results of which found that gamification, such as allowing users to acquire digital items and play parkour, improves the value of the experience of touring in a virtual space and creates opportunities to visit various places. The result also suggested that the experience of touring around Ginza in a virtual space may arouse interest in visiting the actual Ginza.

  1. Game engine: A framework for implementing various forms of games. In recent years, game engines have been increasingly used for multiple platforms and in different industries.
  2. Texture: An image applied to the surface of an object to add the sense of hand feeling to 3D data.
  3. Mesh: A grid of land mapped out in terms of latitude and longitude.

04. Cloud Gaming with docomo OIC

• At docomo Open House’22 held in January 2022, we demonstrated Virtual Ginza ...

  Open

  At docomo Open House’22 held in January 2022, we demonstrated Virtual Ginza by using the cloud gaming^*18 system utilizing the docomo Open Innovation Cloud (docomo OIC), which is a Multi-access Edge Computing (MEC)^*19 environment directly connected to 5G lines, to demonstrate the low latency and stability of 5G SA in a real operational environment.

  4.1 Creation of a Seamless “Open-world Experience” on a Web Browser

  Content in open world games^*20, such as Virtual Ginza, that provide the experience of seamlessly moving through a vast field rendered using 3D CGs, are generally designed to enable users to experience precise level design^*21, dynamic loading of various data, and other features in a simple and easy manner. Note that, the use of a high-specification PC equipped with Graphics Processing Units (GPUs)^*22 (called a “gaming PC”) is generally recommended for the open world game experience.

  Particularly in mobile environments, 3D CG processing power and memory usage are limited, and providing an experience in which the user seamlessly moves across a vast field places a heavy burden on content production. Moreover, depending on the design of the content, long data-loading time may be required at the start of use of the content, creating a barrier to easy user access to the content. In the case of browser-based systems (i.e., a Web application), design and development must take into account not only data loading but also the startup time.

  In this demonstration experiment, we aimed to provide an environment in which users can experience real-time communications through 3D avatars on a vast field in Ginza while reducing barriers to user, such as restrictions on usable terminals and installing applications. We achieved this aim by using a cloud-gaming system using cloud-rendering technology to enable real-time communications via a Web browser.

  With cloud gaming, various types of processing are performed on the server side, and the user terminal that operates the content performs only input and display, so high processing power is not required on the terminal side. If stable communication quality and low latency between the terminal and the server environment (which uses a high-performance GPU) can be ensured, rich 3D spatial representation can be achieved independently of the performance of the terminal using the content.

  4.2 Utilization of docomo OIC Environment and 5G SA

  For docomo Open House’22, we built Virtual Ginza as a verification system rather than a commercial service, and we provided 5G SA as a limited corporate solution in some locations (in a closed, Cloud Direct connection environment). Accordingly, in environments limiting the equipment that can be set up, such as exhibitions, it is necessary to combine a general wired-Internet environment for connecting to external environments in which 5G SA is not available, multiplayer cloud gaming in a docomo OIC environment, and voice chat. We therefore implemented a system that combines different layers of communications, namely, communications between the Internet and the docomo OIC, communications between the docomo OIC and 5G terminals (Cloud Direct connection with 5G SA), and communications between the Internet and 5G terminals (a communication-path bypass connecting Cloud Direct and the Internet) (Figure 9). These configurations made it possible to stably operate multiple terminals at the same location for extended periods of time at docomo Open House’22.

  

  Figure 9  System configuration at docomo Open House’22

  Regarding cloud gaming, we compared response times for LTE, 5G Non-Stand Alone (NSA)^*23, which is the current common 5G environment, and 5G SA, and the difference in latency was about 5 to 10 ms in each case, which confirms the low latency of 5G SA. The user did not notice any particular delay in the operation of cloud gaming for any of these communications systems.

  Since cloud gaming assumes the use of costly, high-performance GPUs and requires 3-GB transmission per hour, its cost is an issue in both terms of operation and utilization (mainly the amount of transmission due to video transmission). When we compare running applications on a 5G closed network connection, as shown by the red dashed lines in Fig. 9, and running applications on local PCs, it becomes clear that using a 5G closed-network connection facilitates optimization of large amounts of data in architectural visualization and security design in multiplayer games, and its effectiveness in realizing short-term and limited concepts and operating it at one-time events was confirmed. We believe that the above-described configuration can be considered when taking a trade-off between the amount of work involved in content.

  1. Cloud gaming: A system that enables games to be played by remotely controlling cloud-rendered video.
  2. MEC: A mechanism of installing servers or storage within a carrier network at locations near users.
  3. Open world game: A type of computer game that provides the experience of seamlessly moving through a vast field composed of 3D CGs.
  4. Level design: Originally a term for designing difficulty levels in computer games, it has become a game-engine term that includes the design of processing and behavior in the CG space and the level of detail in CG representation.
  5. GPU: A processor that excels in parallel computing. It is ideal for deep-learning processes that require parallel computation.
  6. NSA: A 5G wireless access network designed for use with enhanced LTE (eLTE).

05. Conclusion

• In this article, Virtual Ginza was described from the viewpoints of ...

  Open

  In this article, Virtual Ginza was described from the viewpoints of regional revitalization through gamification using a vast 3D-CG field for multiplayers and constructing a system as a use case of 5G SA at docomo Open House’22. Since we could confirm the value proposition provided by Virtual Ginza through experimental verification, we will continue to collaborate with partner companies and develop use cases in regions other than Ginza to contribute to the creation of a “wellbeing society” in the future. The challenges facing system design include cost efficiency in cloud gaming, and we aim to address such challenges not by simply reducing data volume but by utilizing the 5G environment, such as optimizing processing by MEC, to implement a design that can be more easily tried and tested and operated.

• REFERENCES

  Open

  1. [1] Ministry of Land, Infrastructure, Transport and Tourism: “PLATEAU by MLIT” (in Japanese).
     https://www.mlit.go.jp/plateau/
  2. [2] I. Okajima et al.: “Supporting Growth in Society and Industry Using Statistical Data from Mobile Terminal Networks—Overview of Mobile Spatial Statistics—,” NTT DOCOMO Technical Journal, Vol. 14, No. 3, pp. 4-9, Jan. 2013.
     https://www.docomo.ne.jp/english/binary/pdf/corporate/technology/rd/technical_journal/bn/vol14_3/vol14_3_004en.pdf (PDF format:1,019KB)
  3. [3] Open Geospatial Consortium: “CityGML,” May 2022.
     https://www.ogc.org/standards/citygml
  4. [4] Ministry of Land, Infrastructure, Transport and Tourism: “Examples of Services Utilizing 3D City Models” (in Japanese).
     https://www.mlit.go.jp/plateau/new-service/

VOL.24 NO.2