Special Articles on AI Services—Document and Image Processing Technologies—
Edge and Cloud Distributed Processing, and Object Detection AI Technology of Dashcam Video Data

Automotive Camera Image Recognition AI Edge and cloud distributed processing

Takuya Kitade†1 and Fumihiko Kato†2
Service Innovation Department

Yukihiro Namba
X-Tech Development Department

Hideyuki Mitani, Tatsunari Kondo and Ayaka Kitamura
NTT Communications Corporation, Platform Service Division, 5G & IoT Services

†1 Currently, X-Tech Development Department
†2 Currently, R&D Strategy Department

Abstract
In recent years, there has been a remarkable spread of dashcams in Japan, as these devices have been fitted to many automobiles, both large and small. While the main use of a dashcam is to record videos during accidents, some models have been developed that can upload data during normal driving to the cloud. However, as it is difficult to manually search for desired videos from such a large amount of data, the utilization of dashcam data has become an issue. With this issue in mind, NTT DOCOMO and NTT Communications have developed a system to detect construction work that may cause damage to underground infrastructure such as gas and water pipes, with a view to utilizing dashcam data. In this system, for effective use of communications bandwidth, some objects related to construction work are detected by the dashcam (edge terminal), and only the video data in which the objects are detected is uploaded to the cloud. The cloud then detects multiple construction-related objects, and queries whether the video really is a construction site. This makes it possible to collect and analyze a wide range of dashcam video while minimizing the amount of communications for video data upload.

01. Introduction

  • Dashcams, which are video cameras installed in vehicles such as cars, ...

    Open

    Dashcams, which are video cameras installed in vehicles such as cars, are being installed in many vehicles, large and small, as tools for recording videos during accidents, because they can constantly record videos while driving and automatically save videos under specific conditions such as sudden acceleration or braking. In addition, the use of dashcams with communications functions is increasing, and it is now possible to check captured data on the cloud.

    However, issues remain regarding utilization of the vast amount of video data that has been captured. It is not realistic from both performance and cost perspectives to upload all video data to the cloud and centrally manage it for effective data utilization.

    To address this issue, NTT Communications is developing an AI Roadworks Detection Solution (tentative name) that utilizes image recognition AI*1 technology developed by NTT DOCOMO and Mobiscan®, a system that enables distributed management of video data captured by dashcams on dashcams and the cloud and uploading of only necessary data, by using dashcams equipped with communications functions and image recognition AI.

    This article describes the mechanism and features of Mobiscan and the AI Roadworks Detection Solution.

    1. Image recognition AI: A form of AI that takes images as its input and uses them to generate results by making judgments, performing estimates, and so on.
  • 02. Mechanism of Mobiscan

  • Mobiscan is a distributed video management platform service provided ...

    Open

    Mobiscan is a distributed video management platform service provided by NTT Communications that stores only metadata*2 such as location information and capture time related to the video data from a dashcam installed in a car or other vehicle in the cloud in real time and uploads the video data as needed. As shown in Figure 1, this service enables the use of video data collected by dashcams onboard vehicles operated by mobility partners by storing the data in Mobiscan, analyzing and processing the stored video data, and providing the data to the data utilizing partner. Here, “mobility partners” refer to partner companies such as bus and taxi operators and transportation companies that operate vehicles constantly and are capable of collecting a wide range of video data by installing dashcams. Data utilizing partners refer to partner companies that utilize the data stored in Mobiscan.

    Mobility partners can provide data to Mobiscan simply by installing dedicated dashcams in the vehicles with which they conduct their existing business, thereby creating a new source of revenue. Meanwhile, data utilizing partners can reduce costs by utilizing Mobiscan data to perform some of the tasks that used to be performed visually by visiting target sites, such as inspections and patrol work.

    Figure 1 Mobiscan service
    1. Metadata: Information related to content data, not content data itself. For example, the camera position (latitude and longitude) for an image file.
  • 03. Features of Mobiscan

  • Mobiscan can sequentially collect video data from multiple vehicles ...

    Open

    Mobiscan can sequentially collect video data from multiple vehicles on the road, but if all data are stored in Mobiscan, communications, data processing, and storage costs will increase. For these reasons, the distributed processing and video sharing mechanisms described below are used to achieve both operational cost efficiency and video comprehensiveness.

    3.1 Distributed Processing with Edge AI and Cloud AI

    In Mobiscan, the edge AI*3 installed in the dashcam selects and uploads to the cloud only video data requested by the data utilizing partner, thereby reducing communications costs and data storage costs. To ensure that the video data required by the data utilizing partner is covered, edge AI reacts even when only a portion of the specific object to be detected is included by using image recognition AI and saves and uploads to the cloud several seconds of video before and after the object is captured, just as a typical dashcam does (Figure 2).

    Meanwhile, the cloud AI running on the cloud that makes up Mobiscan performs detailed image recognition on all data filtered by the edge AI. This process makes it possible to extract only the events that the data utilizing partner is looking for.

    Figure 2 Cloud AI and edge AI linkage

    3.2 Distributed Data Management

    As mentioned above, Mobiscan enables upload of only required video data to the cloud using the edge AI installed in the dashcam. The system is also equipped with a function to upload specific video data stored on the dashcam so that the existing video data can be utilized even when the data utilization method, i.e., the recognition target, is changed. The cloud maintains a list of video data stored in the dashcam, and by requesting video data from the dashcam, the dashcam can upload video data necessary for additional analysis to the cloud. For example, if the metadata for a location specified by the user on a map indicates that the data is stored on the dashcam, the dashcam uploads the data to the cloud according to the request, and if the data has not yet been captured, it is reserved to be uploaded to the cloud the next time it is captured by the dashcam.

    1. Edge AI: In this article, edge AI processing refers to AI processing performed on devices.
  • 04. Case Study of Mobiscan Application “AI Roadworks Detection Solution (Tentative Name)”

  • Using company vehicles, infrastructure providers such as gas and ...

    Open

    Using company vehicles, infrastructure providers such as gas and electricity companies patrol the roads where their facilities are buried to ensure that no roadworks are taking place that may damage buried facilities by unintentional underground cutting. This entails costs such as the personnel costs of patrolling operators, the fuel costs required to run the vehicles, and vehicle maintenance costs, as well as the difficulty of recruiting the operators.

    Therefore, we developed the “AI Roadworks Detection Solution (tentative name)” using Mobiscan. Using this solution to analyze videos on the road makes it possible to replace the work of driving on and visually checking the actual road.

    An overview of the AI Roadworks Detection Solution (tentative name) using Mobiscan is shown in Figure 3. The system performs a two-stage AI judgement on the video collected by the mobility partner’s vehicle, thereby suppressing unnecessary data communications while ensuring the accuracy of the AI.

    Specifically, when a construction cone, an element of a construction project, is detected by the edge AI during data collection, it uploads to the cloud the video of the five seconds before and after the point of detection. From the uploaded video, the cloud AI uses image recognition AI to detect multiple elements that make up a construction, such as construction signs and cone bars, which are then analyzed and scored over multiple frames. Only those videos that reach a certain score are provided to the data utilizing partner along with location information. Personal information such as faces and license plate numbers included in the provided videos are masked to ensure privacy.

    The image recognition AI used for image analysis in the cloud AI was developed as “NTT DOCOMO’s contribution to the creation of a sustainable society by promoting the Digital Transformation (DX)*4 of social infrastructure maintenance through the use of AI and big data.”

    The data utilizing partner can check videos of the construction site with a single click on a map on the User Interface (UI) screen, making it possible for patrols to grasp construction work on a map while reducing actual patrols using company vehicles.

    Figure 3 AI Roadworks Detection Solution (tentative name) using Mobiscan
    1. DX: The use of IT technology to revolutionize services and business models, promote business, and change the lives of people for the better in diverse ways.
  • 05. Conclusion

  • This article has provided an overview of Mobiscan, a distributed video ...

    Open

    This article has provided an overview of Mobiscan, a distributed video management platform service that efficiently stores and analyzes video data acquired from dashcams, and has described the “AI Roadworks Detection Solution (tentative name),” an example application of Mobiscan. Although this case study involved the use of Mobiscan for infrastructure maintenance management, we hope to contribute to the further utilization of the video data from dashcams by advancing AI technology for video analysis in Mobiscan as on-road video data is expected to be utilized in various fields, such as disaster prevention and urban planning.

VOL.26 NO.1

Go to top of the page