Classifying Apparatus, Classifying Method, and Non-Transitory Computer-Readable Storage Medium

The present disclosure generally relates to a classifying apparatus, a classifying method, and a non-transitory computer-readable storage medium.

There are techniques to use a vibration sensor to monitor an object, such as a road. PTL1 discloses a technique to use a distributed acoustic sensing (DAS) system as a vibration sensor to obtain a waterfall data that indicates an amplitude of vibration sensed by the vibration sensor for each one of multiple locations and for each one of multiple points in time.

PTL1: Japanese Unexamined Patent Application Publication No.2021-121917

PTL1 does not teach a case where some points of the vibration sensor are not placed along the object to be monitored. An objective of this disclosure is to provide a novel technique to handle data obtained from a vibration sensor that is installed to monitor an object.

The present disclosure provides a classifying apparatus comprising at least one memory that is configured to store instructions and at least one processor.

The at least one processor is configured to execute the instructions to: acquire a waterfall data that indicates amplitude of vibration for each point in time and for each sensing point in a vibration sensor that is placed along a target object; perform semantic segmentation on the waterfall data to generate a class data that indicates a normal class or an abnormal class for each element of the waterfall data, the normal class being assigned to the element whose sensing point is predicted to be a monitoring point, the abnormal class being assigned to the element whose sensing point is predicted to be a non-monitoring point, the monitoring point being the sensing point that is placed along the target object, the non-monitoring point being the sensing point that is not placed along the target object; and classify the sensing points into the monitoring point and the non-monitoring point based on the class data.

The present disclosure further provides a classifying method performed by a computer.

The classifying method comprises: acquiring a waterfall data that indicates amplitude of vibration for each point in time and for each sensing point in a vibration sensor that is placed along a target object; performing semantic segmentation on the waterfall data to generate a class data that indicates a normal class or an abnormal class for each element of the waterfall data, the normal class being assigned to the element whose sensing point is predicted to be a monitoring point, the abnormal class being assigned to the element whose sensing point is predicted to be a non-monitoring point, the monitoring point being the sensing point that is placed along the target object, the non-monitoring point being the sensing point that is not placed along the target object; and classifying the sensing points into the monitoring point and the non-monitoring point based on the class data.

The present disclosure further provides a non-transitory computer readable storage medium storing a program.

The program causes a compute to execute: acquiring a waterfall data that indicates amplitude of vibration for each point in time and for each sensing point in a vibration sensor that is placed along a target object; performing semantic segmentation on the waterfall data to generate a class data that indicates a normal class or an abnormal class for each element of the waterfall data, the normal class being assigned to the element whose sensing point is predicted to be a monitoring point, the abnormal class being assigned to the element whose sensing point is predicted to be a non-monitoring point, the monitoring point being the sensing point that is placed along the target object, the non-monitoring point being the sensing point that is not placed along the target object; and classifying the sensing points into the monitoring point and the non-monitoring point based on the class data.

According to the present disclosure, a novel technique to handle data obtained from a vibration sensor that is installed to monitor an object is provided.

Example embodiments according to the present disclosure will be described hereinafter with reference to the drawings. The same numeral signs are assigned to the same elements throughout the drawings, and redundant explanations are omitted as necessary. In addition, predetermined information (e.g., a predetermined value or a predetermined threshold) is stored in advance in a storage device to which a computer using that information has access unless otherwise described.

1 FIG. 1 FIG. 2000 2000 2000 2000 illustrates an overview of a classifying apparatusof the first example embodiment. It is noted that the overview illustrated byshows an example of operations of the classifying apparatusto make it easy to understand the classifying apparatus, and does not limit or narrow the scope of possible operations of the classifying apparatus.

2000 10 30 30 10 20 20 30 30 The classifying apparatusis configured to handle a waterfall datathat indicates an amplitude of vibration sensed by a vibration sensorfor each one of two or more points in the vibration sensorand for each one of two or more points in time. In some embodiments, the waterfall datamay be a time series of two or more sensing data. The sensing datais generated by the vibration sensor, and indicates an amplitude of vibration sensed at each one of two or more points of the vibration sensorat a point in time.

30 30 40 The vibration sensoris installed (placed) along an object to be monitored, such as a road. Hereinafter, the object to be monitored by the vibration sensoris called “target object”.

30 30 40 An example of the vibration sensoris a DAS system that includes a DAS device and an optical fiber cable. In the case where the DAS system is employed as the vibration sensor, the optical fiber cable is placed along the target objectand is attached to the DAS device. The DAS device is configured to transmit a laser pulse through the optical fiber cable and to receive the reflection of the transmitted laser pulse.

40 40 40 20 30 Since vibration occurred at a certain point of the target objectaffects the laser pulse that is traveling at that point of the target objectat that time, the DAS device can measure an amplitude of the vibration occurred at that point of the target objectby analyzing the reflection of the transmitted laser pulse. Thus, the DAS device can generate the sensing datathat indicates the amplitude of vibration that is sensed at each one of two or more points of the optical fiber cable. Hereinafter, a point of the vibration sensor(e.g., the optical fiber cable) at which the amplitude of vibration is sensed is called “sensing point”.

10 30 10 30 In some implementations, the waterfall datamay be formed as a matrix data denoted by W. The row of the matrix data W may represent point in time whereas the column thereof represents sensing point of the vibration sensor. In this case, an element at the i-th row and the j-th column of the waterfall data(i.e., W[i][j]) represents the amplitude of vibration sensed at the j-th sensing point of the vibration sensorat the i-th point in time.

10 10 20 10 10 In addition, the sequence of the elements in the i-th row of the waterfall data(i.e., {W[i][0], W[i][1], . . . , W[i][M]} where M represents the total number of the sensing points indicated by the waterfall data) represents the sensing datathat is generated at the i-th point in time. On the other hand, the sequence of the elements in the j-th column of the waterfall data(i.e., {W[0][j], W[1][j], . . . , W[N][j]} where N represents the total number of the points in time indicated by the waterfall data) represents a time series of the amplitudes of vibration that are sensed at the j-th sensing point.

10 30 10 10 The waterfall dataformed as a matrix data may be handled as an image data, called “waterfall image” hereinafter. In this case, W[i][j] corresponds to a value of the pixel (i,j) of the waterfall image. Suppose that the amplitudes of vibration sensed by the vibration sensorare quantized and normalized to a rage of 0 to 255. In this case, the waterfall datacan be formed as a grayscale image. It is noted, however, that the waterfall datais not necessarily handled as an image data.

30 40 40 30 32 32 40 40 40 32 1 FIG. The vibration sensormay include one or more sensing points that are not placed along the target object(in other words, not suitably located to monitor the vibration of the target object). For example, the vibration sensormay have some extra segmentas depicted by. The amplitude of vibration sensed at the sensing points in those extra segmentsdo not accurately indicate the amplitude of vibration of the target object. Hereinafter, the sensing point that is placed along the target objectis called “monitoring point”, whereas the sensing point that is not placed along the target object(e.g., the sensing point included in the extra segment) is called “non-monitoring point”.

2000 10 30 2000 10 10 10 Taking the existence of the non-monitoring points into consideration, the classifying apparatusis configured to use the waterfall datato detect monitoring points of the vibration sensor. Specifically, the classifying apparatusacquires the waterfall data, and performs semantic segmentation on the waterfall data. Semantic segmentation is a technique to analyze a collection of two or more data to classify them into two or more classes. Through the semantic segmentation, one of two or more classes (types, in other words) is assigned to each element of the waterfall data.

10 30 10 30 The class may include NORMAL and ABNORMAL. NORMAL is assigned to the element of the waterfall datathat is predicted to indicate the amplitude of vibration that is sensed at a monitoring point of the vibration sensor. On the other hand, ABNORMAL is assigned to the element of the waterfall datathat is predicted to indicate the amplitude of vibration that is sensed at a non-monitoring point of the vibration sensor.

10 2000 10 10 10 As a result of the semantic segmentation on the waterfall data, the classifying apparatusgenerates a class data that indicates the class for each element of the waterfall data. The class data may be formed in a manner similar to the waterfall data. Suppose that the waterfall datais formed as a N×M matrix denoted by W. In this case, the class data may be formed as a N×M matrix denoted by C wherein C[i][j] indicates the class assigned to W[i][j].

2000 2000 10 2000 10 2000 10 Based on the class data, the classifying apparatusdetermines which sensing points are the monitoring points; in other words, the classifying apparatusclassifies the sensing points into the monitoring point and the non-monitoring point based on the class data. When the waterfall datais a matrix data whose column represents sensing point, it can be rephrased that the classifying apparatusdetermines which columns of the waterfall datashow amplitudes of vibrations sensed at the monitoring points based on the class data; in other words, the classifying apparatusclassifies the columns of the waterfall datainto the column corresponding to the monitoring points and the column corresponding to the non-monitoring point based on the class data. Some other examples of classifying apparatus which determines monitoring and non-monitoring points or portions of monitoring and non-monitoring points could be analytical techniques based on statistical measures of the waterfall data.

2000 10 30 10 10 According to the classifying apparatusof the first example embodiment, the elements of the waterfall dataare classified into the normal class and the abnormal class through semantic segmentation, and the sensing points of the vibration sensorare classified into the monitoring point and the non-monitoring point based on the result of the classification of the elements of the waterfall data. This is a novel way of handling the waterfall data, which is acquired from a vibration sensor that monitors an object.

2000 40 10 10 As explained in detail later, the classification of the sensing points into the monitoring point and the non-monitoring point is useful in various manners. For example, the classifying apparatuscan remove the influence of the non-monitoring points, with which the amplitude of vibrations of the target objectcannot be measured accurately, from the waterfall databy removing the regions of the non-monitoring points from the waterfall data.

2000 Hereinafter, more detailed explanation of the classifying apparatuswill be described.

2 FIG. 2000 2000 2020 2040 2060 2020 10 2040 10 10 2060 30 is a block diagram illustrating an example of the functional configuration of the classifying apparatusof the first example embodiment. The classifying apparatusincludes an acquiring unit, an segmenting unit, and classifying unit. The acquiring unitacquires the waterfall data. The segmenting unitperforms semantic segmentation on the waterfall datato generate the class data that indicates one of two or more classes for each element of the waterfall data. The classifying unitclassifies the sensing points of the vibration sensorinto the monitoring point and the non-monitoring point based on class data.

2000 2000 The classifying apparatusmay be realized by one or more computers. Each of the one or more computers may be a special-purpose computer manufactured for implementing the classifying apparatus, or may be a general-purpose computer like a personal computer (PC), a server machine, or a mobile device.

2000 2000 2000 2 FIG. The classifying apparatusmay be realized by installing an application in the computer. The application is implemented with a program that causes the computer to function as the classifying apparatus. In other words, the program is an implementation of the functional units of the classifying apparatusthat are exemplified by.

3 FIG. 3 FIG. 1000 2000 1000 1020 1040 1060 1080 1100 1120 is a block diagram illustrating an example of the hardware configuration of a computerrealizing the classifying apparatusof the first example embodiment. In, the computerincludes a bus, a processor, a memory, a storage device, an input/output (I/O) interface, and a network interface.

1020 1040 1060 1080 1100 1120 1040 1060 1080 1100 1000 1120 1000 The busis a data transmission channel in order for the processor, the memory, the storage device, and the I/O interface, and the network interfaceto mutually transmit and receive data. The processoris a processer, such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), or FPGA (Field-Programmable Gate Array). The memoryis a primary memory component, such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The storage deviceis a secondary memory component, such as a hard disk, an SSD (Solid State Drive), or a memory card. The I/O interfaceis an interface between the computerand peripheral devices, such as a keyboard, mouse, or display device. The network interfaceis an interface between the computerand a network. The network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

1000 2000 3 FIG. The hardware configuration of the computeris not restricted to that shown in. For example, as mentioned-above, the classifying apparatusmay be realized as a combination of multiple computers. In this case, those computers may be connected with each other through the network.

4 FIG. 2000 2020 10 102 2040 10 104 2060 106 is a flowchart illustrating an example flow of processes performed by the classifying apparatusof the first example embodiment. The acquiring unitacquires the waterfall data(S). The segmenting unitperforms semantic segmentation on the waterfall datato generate the class data (S). The classifying unitclassifies the sensing points into the monitoring point and the non-monitoring point (S).

10 102 Acquisition of Waterfall data: S

2020 10 102 10 20 2020 20 20 10 2020 20 10 The acquiring unitacquires the waterfall data(S). As mentioned above, the waterfall datarepresents a time series of the sensing data. In some embodiments, the acquiring unitmay acquire two or more sensing dataof different points in time from each other, thereby acquiring a time series of those sensing dataas the waterfall data. In other words, the acquiring unitconverting the acquired two or more sensing datainto the waterfall data.

20 30 20 2000 2020 20 30 20 2000 2020 20 30 20 2020 20 There are various ways to acquire the sensing data. In some embodiments, the vibration sensorputs the sensing datainto a storage device to which the classifying apparatushas access. In this case, the acquiring unitmay access to this storage device to acquire the sensing data. In other embodiments, the vibration sensorsends the sensing datato the classifying apparatus. In this case, the acquiring unitmay receive the sensing datasent by the vibration sensor, thereby acquiring the sensing data. It is noted that the acquiring unitmay acquire two or more sensing dataone by one or simultaneously.

20 10 2020 10 The conversion of two or more sensing datainto the waterfall datamay be performed by another computer in advance. In this case, the acquiring unitmay acquire the waterfall dataat once.

2040 10 104 2000 10 The segmenting unitperforms semantic segmentation on the waterfall data(S). As mentioned above, the classifying apparatusmay handle two classes, called NORMAL and ABNORMAL. In this case, one of these two classes is assigned to each element of the waterfall dataas a result of the semantic segmentation.

10 10 10 10 10 10 There are various ways to perform semantic segmentation on the waterfall data. In some embodiments, a machine learning-based model, called “segmenting model”, is used to perform the semantic segmentation on the waterfall data. The segmenting model may be configured to take the waterfall dataas input, analyze the waterfall datato determine the class of each element, and output the class data. The analysis of the waterfall datamay include: extracting features from the waterfall data; and upsampling the extracted features to the same size as the input data to generate the class data.

The segmenting model may be implemented as one of various types of machine learning-based model, such as a neural network. A few examples of neural network suitable for implementing the segmenting model are U-net, region based convolution neural network (R-CNN), Fast R-CNN, and Faster R-CNN.

2000 2000 The segmenting model is trained in advance of an operating phase (testing phase, in other words) of the classifying apparatus. Hereinafter, a computer that performs the training of the segmenting model is called “training apparatus”. The training apparatus may be the classifying apparatusor may be another apparatus.

To train the segmenting model, the training apparatus uses a training dataset that includes two or more training data. The training data may be formed as a combination of a training input data and a ground-truth data. The training input data represents the waterfall data whereas the ground-truth data represents the class data corresponding to the training input data. Specifically, the ground truth data is the class data each of whose element indicates the class that should be assigned to the corresponding element of the training input data.

It is noted that there are various well-known techniques to train machine learning-based models using the training dataset, and any one of those techniques can be applied to the training apparatus to train the segmenting model. For example, the training apparatus inputs the training input data into the segmenting model and obtain the class data from the segmenting model. Then, the training apparatus updates trainable parameters of the segmenting model (e.g., weights of edges and biases of a neural network) based on a loss that represents a degree of difference between the ground truth data and the class data that is output from the segmenting model. The training apparatus trains the segmenting model by repeatedly updating the segmenting model with multiple training data in the training data set.

10 2040 10 2040 2040 10 It is noted that the size of data that the segmenting model can handle at once may be less than that of the waterfall data. In this case, the segmenting unitdivides the waterfall datainto two or more partial data, called “patches”, whose sizes are the same as the size of the input of the segmenting model. Then, the segmenting unitinputs the patches into the segmenting model, thereby obtaining the class data for each patch. The segmenting unitcan obtain the class data of a whole of the waterfall databy concatenating the class data of each patch.

2040 10 30 10 30 30 30 In some embodiments, the segmenting unitmay further take one or more measurement conditions into consideration to perform the semantic segmentation on the waterfall data. For example, the measurement condition may include a period of time during which the vibration sensorperforms the measurement to generate the waterfall data. In another example, the measurement condition may include one or more weather conditions, such as a weather type (e.g., sunny, cloudy, or rainy), a temperature, or a humidity during the measurement is performed by the vibration sensor. In another example, the measurement condition may include parameters related to the vibration sensor, such as sensitivity of the vibration sensor.

10 When one or more measurement conditions are used for the semantic segmentation, the segmenting model may be configured to further take the one or more measurement conditions as input. In addition, the segmenting model may be further configured to extract features from each one of the waterfall dataand the measurement conditions, compute combined features of those extract features, and upsample the combined features to the same size as the input data to generate the class data.

The segmenting model is required to be trained not only with the waterfall data but also with the measurement conditions. Thus, the training input data further includes the measurement conditions as well as the waterfall data so that the training apparatus can trains the segmenting model with the measurement conditions.

2020 2020 2000 2000 2020 In order to use the measurement conditions for the semantic segmentation, the acquiring unitacquires the measurement conditions. For example, the acquiring unitmay acquire the measurement conditions from a storage device in which the measurement conditions are stored in advance and to which the classifying apparatushas access. In another example, the measurement conditions may be sent from another computer to the classifying apparatus, and the acquiring unitmay receive those measurement conditions.

2060 106 10 The classifying unitclassifies the sensing points into the monitoring point and the non-monitoring point based on the class data (S). Conceptually, the sensing point is more likely to be the monitoring point as more elements of the waterfall datacorresponding to that sensing point are classified into the NORMAL class.

10 10 2060 10 2060 10 Since the waterfall dataincludes two or more elements for each sensing point, it is possible that both the NORMAL class and the ABNORMAL class are assigned to the elements of the waterfall datacorresponding to the same sensing point as each other. To handle this situation, the classifying unitmay determine, for each sensing point, whether or not the sensing point is the monitoring point based on the number of the elements of the waterfall datacorresponding to the sensing point to which the NORMAL class is assigned. Specifically, for each sensing point, the classifying unitmay use the class data to determine the number of the elements of the waterfall datacorresponding to the sensing point to which the NORMAL class is assigned, and determines whether the determined number is larger than or equal to a predefined threshold.

2060 2060 When the determined number is larger than or equal to the threshold, the classifying unitdetermines that the sensing point is the monitoring point. On the other hand, when the determined number is less than the threshold, the classifying unitdetermines that the sensing point is the non-monitoring point.

10 10 2060 Suppose that the waterfall datais a N×M matrix data whose column represents the sensing point and whose row represents point in time. In addition, the threshold mentioned above is set to be T. In this case, for each column j (1<=j<=M) of the waterfall data, the classifying unitdetermines the number of the elements in the column j (denoted by C[j]) to which the NORMAL class is assigned. If C[j]>=T (i.e., the NORMAL class is assigned to T or more elements in the column j), the sensing point corresponding to the column j is determined to be the monitoring point. On the other hand, if C[j]<T (i.e., the NORMAL class is assigned to less than T elements in the column j), the sensing point corresponding to the column j is determined to be the non-monitoring point.

10 10 2060 In other embodiments, instead of the number of the elements in the column j of the waterfall datato which the NORMAL class is assigned, the percentage of the elements in the column j of the waterfall data(denoted by P[j]) to which the NORMAL class is assigned may be used to detect the monitoring point. P[j] can be defined as “P[j]=C[j]/N”. In this case, the classifying unitmay compare P [j] with a predetermined threshold for each column j to determine whether the sensing point corresponding to the column j is the monitoring point or the non-monitoring point.

2060 30 40 30 40 In other embodiment, the classifying unitmay takes the total number of the monitoring points in the vibration sensorinto consideration. When the length of the target objectis known in advance, it is possible to predict the total number of the monitoring points in the vibration sensor, which is denoted by Ns. Suppose that the length of the target objectis L[m] and the interval between the monitoring points are defined to be a[m] in advance. In this case, the total number of the monitoring points can be predicted as Ns=L/a.

2060 10 Taking the total number of the monitoring points into consideration, the classifying unitmay sort the sensing points in the descending order of the likelihood of being the monitoring point, and determine the 1st to (L/a)-th sensing point as the monitoring point. The rest of the sensing points are determined to be the non-monitoring point. The likelihood of the sensing point being the monitoring point may be represented the number of the elements of the waterfall datacorresponding to that sensing point to which the NORMAL class is assigned; e.g., represented by C[j] in the case exemplified above.

2060 The classifying unitmay generate information, called “sensing point information”, that indicates the result of the classification of the sensing points. Specifically, the sensing point information may indicate two lists called “monitoring point list” and “non-monitoring point list”. The monitoring point list indicates the identifiers of the sensing points that are classified as the monitoring point. On the other hand, the non-monitoring point list indicates the identifies of the sensing points that are classified as the non-monitoring point.

2000 10 2000 10 40 The sensing point information may be used in various manners. For example, the classifying apparatusmay use the sensing point information to remove the influence of the non-monitoring points from the waterfall data. Specifically, the classifying apparatusmay remove the elements of the waterfall datacorresponding to the non-monitoring points, thereby obtaining a time-series data that represents the amplitude of vibration for each monitoring point: in other words, the amplitude of vibration for each point of the target object. Hereinafter, this time-series data is called “monitoring data”.

5 FIG. 5 FIG. 10 10 2000 102 106 illustrates a way to generate the monitoring data based on the waterfall data. In this example, the waterfall datais a matrix data whose column represents sensing point and whose row represent point in time. The classifying apparatusperforms the steps Sto Sto classify the sensing points into the monitoring point and the non-monitoring point. In, the columns of the non-monitoring points are filled with a diagonal stripe pattern.

2000 10 50 The classifying apparatusremoves the columns of the non-monitoring points from the waterfall data, and concatenates the columns that are not removed into a single matrix data. This matrix data is handled as the monitoring data.

2000 40 2000 40 40 40 50 2000 50 40 The classifying apparatuscan use the sensing point information to localize one or more locations of the target object. Specifically, the classifying apparatuscan determine the interval of the monitoring points by dividing the length of the target objectby the number of the monitoring points. When the interval of the monitoring points is determined to be I[m], the location of the target objectcorresponding to the k-th monitoring point can be determined to be at I*k[m] from the start point of the target object. By applying the result of the localization to the monitoring data, the classifying apparatuscan modify the monitoring dataso as to indicate the time series of the amplitude of vibration for each location of the target objectthat corresponds to the monitoring point.

10 10 2000 2000 The sensing point information may be used not only for the current waterfall databut also for the waterfall dataobtained in the future. It enables the classifying apparatusto avoid frequently performing the classification of the sensing points, thereby reducing computer resources used by the classifying apparatus.

6 FIG. 2000 is a flowchart illustrating an example flow of processes performed by the classifying apparatusthat uses the sensing point information in the future processes.

2000 10 30 202 204 The classifying apparatusacquires the waterfall datathat is generated by the vibration sensor(S), and determines whether or not the sensing point information is stored in a storage device (S).

204 2000 10 206 208 2000 210 2000 50 10 212 When it is determined that the sensing point information is not stored in the storage device (S: NO), the classifying apparatusperforms semantic segmentation on the waterfall data(S) and classifies the sensing points (S). Then, the classifying apparatusgenerates the sensing point information and saves it in the storage device (S). Based on the sensing point information, the classifying apparatusgenerates the monitoring datafrom the waterfall data(S).

204 204 2000 214 50 10 212 When it is determined in the step Sthat the sensing point information is stored (S: YES), the classifying apparatusacquires the sensing point information from the storage device (S), and generates the monitoring datafrom the waterfall databased on the sensing point information (S).

2000 2000 50 2000 206 210 It is noted that an expiration period may be set to the sensing point information in order to re-generate the sensing point information some time. In this case, the classifying apparatusalso determines whether or not the sensing point information in the storage device is valid based on its expiration period. Then, only when the valid sensing point information is stored in the storage device, the classifying apparatususe that sensing point information to generate the monitoring data. Otherwise, the classifying apparatusperforms the steps Sto Sto generate a new sensing point information.

2000 50 The classifying apparatusmay be configured to output one or more pieces of information, generally called “output information”, that are related to the result of the classification of the sensing points. The output information may include the sensing point information, the monitoring data, or both.

2000 It is noted that there are various ways to output the output information. In some implementations, the output information may be put into a storage device, displayed on a display device, or sent to another computer such as a PC or smart phone of the user of the classifying apparatus.

7 FIG. 7 FIG. 2000 2000 2000 2000 illustrates an overview of the classifying apparatusof the second example embodiment. It is noted that the overview illustrated byshows an example of operations of the classifying apparatusof the second example embodiment to make it easy to understand the classifying apparatusof the second example embodiment, and does not limit or narrow the scope of possible operations of the classifying apparatusof the second example embodiment.

70 40 40 40 70 10 10 In this example embodiment, it is assumed that there are one or more moving objectson the target objectthat cause the target objectto vibrate. For example, when the target objectis a road, the moving objectsmay be vehicles (e.g., cars or mortar cycles) which run on the road. In addition, it is assumed that the waterfall datais formed as a matrix data whose column represents sensing points whereas whose row represents points in time, or vice versa. Unless otherwise stated, the column of the waterfall datarepresents sensing points whereas the row thereof represents points in time.

2000 70 10 2060 2060 2060 70 Under the assumption mentioned above, the classifying apparatusdetects a trajectory (e.g., a time series of locations) for one or more moving objectsfrom the waterfall data, and uses the detected trajectory to modify the sensing point information (i.e., the result of the classification of the sensing points that is performed by the classifying unit). In other words, some sensing points that are classified by the classifying unitas the monitoring point may be re-classified as the non-monitoring points, some sensing points that are classified by the classifying unitas the non-monitoring point may be re- classified as the monitoring points, or both. Hereinafter, a trajectory of the moving objectis called “object trajectory”.

70 10 It is considered that the closer a sensing point is to the location of the moving object, the larger the amplitude of the vibration that is sensed at that sensing point. Thus, the object trajectory can be detected based on the amplitude of the vibration indicated by the waterfall data.

8 FIG. 10 10 60 60 80 60 illustrates the object trajectory detected from the waterfall data. In this example, the waterfall datais formed as a waterfall imagewhose X axis represents sensing points and whose Y axis represents points in time. The waterfall imageis illustrated as a grayscale image whose pixel has a larger value as the amplitude of the vibration corresponding to the pixel is larger. For the convenience of illustration, the darker color is depicted with denser and larger dots. The object trajectoryis depicted with white lines thar are superimposed on the waterfall image.

8 FIG. 8 FIG. 80 80 10 As illustrated by, the object trajectorymay be non-continuous (cut off, in other words) due to the existence of the non-monitoring points as illustrated by. If the sensing points are not correctly classified into the monitoring point and the non-monitoring point, the object trajectorybecomes non-continuous when the regions corresponding to the non-monitoring points are removed from the waterfall data.

9 FIG. 9 FIG. 80 60 90 80 2000 90 60 50 illustrates the object trajectoryin a case where the sensing points are not correctly classified. The waterfall imageincludes an abnormal sectionthat is a region of one or more continuous non-monitoring sections. In the case illustrated by, the object trajectorybecomes non-continuous when the classifying apparatusremoves the abnormal sectionfrom the waterfall imageto generate the monitoring data.

80 10 80 80 50 2000 90 60 50 10 FIG. 10 FIG. On the other hand, if the sensing points are correctly classified into the monitoring point and the non-monitoring point, the object trajectorybecomes continuous when the regions corresponding to the non-monitoring points are removed from the waterfall data.illustrates the object trajectoryin a case where the sensing points are correctly classified. In the case illustrated by, the object trajectorybecomes continuous in the monitoring datawhen the classifying apparatusremoves the abnormal sectionfrom the waterfall imageto generate the monitoring data.

2000 80 10 90 80 90 80 90 10 Taking the fact mentioned above into consideration, the classifying apparatusof the second example embodiments corrects the sensing point information based on the object trajectorydetected from the waterfall data. Specifically, the start point, the end point, or both of the abnormal sectionis corrected so as to make the object trajectoryalmost continuous before and after the abnormal section(in other words, so that the object trajectorybecomes almost continuous when the abnormal sectionis removed from the waterfall data).

2000 80 70 40 According to the classifying apparatusof the second example embodiment, the sensing point information is corrected based on the object trajectory, which is the trajectory of the moving objectthat moves on the target object. As a result, errors in the sensing point information due to misclassification can be reduce, thereby making the sensing point information more accurate.

2000 Hereinafter, more detailed explanation of the classifying apparatuswill be described.

11 FIG. 2000 2000 2000 2020 2040 2060 2000 2080 2100 2080 80 10 2100 80 is a block diagram illustrating an example of the functional configuration of the classifying apparatusof the second example embodiment. Same as the classifying apparatusof the first example embodiment, the classifying apparatusof the second example includes the acquiring unit, the segmenting unit, and the classifying unit. In addition, the classifying apparatusof the second example embodiment further includes a detecting unitand a correcting unit. The detecting unitdetects one or more object trajectoryfrom the waterfall data. The correcting unitcorrects the sensing point information based on the object trajectory.

2000 2000 2000 1000 1080 2000 3 FIG. The classifying apparatusof the second example embodiment may be implemented in a similar manner to the manner by which the classifying apparatusof the first example embodiment is realized. For example, the classifying apparatusof the second example embodiment is realized by the computerthat is illustrated by. However, the storage deviceof the second example embodiment includes the program that implements the functions of the classifying apparatusof the second example embodiment.

12 FIG. 2000 2000 102 106 106 2080 80 10 302 80 304 is a flowchart illustrating an example flow of processes performed by the classifying apparatusof the second example embodiment. The classifying apparatusof the second example embodiment may perform the steps Sto Sin the same manner as that of the first example embodiment. After performing the step S, the detecting unitdetects the object trajectoryfrom the waterfall data(S), and corrects the sensing point information based on the object trajectory(S).

2080 80 10 302 70 2080 2080 20 10 70 20 70 80 70 The detecting unitdetects one or more object trajectoriesfrom the waterfall data(S). There are well-known ways to detect a trajectory of a moving objectfrom a time series data that indicates the amplitude of vibration for two or more locations, and one of those ways can be applied to the detecting unit. For example, the detecting unitmay detect, for each sensing datain the waterfall data, one or more locations (i.e., sensing points) each of which a moving objectis predicted to be located at. Specifically, there may be one or more maximum points in the sensing data, and the sensing points corresponding to the maximum points are predicted to be the locations of the moving objects. Then, the object trajectoryis detected by connecting the detected locations of the moving objectsover time.

2100 2060 304 2000 90 10 90 2100 90 80 90 The correcting unitcorrects the sensing point information that is generated by the classifying unit(S). To do so, the classifying apparatusdetermines one or more abnormal sectionsfrom the waterfall databased on the sensing point information. Then, for each abnormal section, the correcting unitcorrects the start point, the end point, or both of the abnormal sectionso as to make the object trajectoryalmost continuous during the abnormal section, thereby correcting the sensing point information.

80 90 2100 80 90 90 90 When there is a single object trajectorythat crosses an abnormal section, the correcting unitmay determine a target width Wt based on the object trajectory, and change the width of the abnormal sectioninto Wt, thereby correcting the sensing point information. Changing the width of the abnormal sectionincludes re-classifying the sensing points around the borders of the abnormal section.

90 90 2100 Suppose that the start point and the end point of the abnormal sectionare the sensing points Ss and Se, respectively. In addition, the target width Wt of the abnormal sectionis six smaller than the current width of thereof. In this case, the correcting unitmay shift the start point and the end point by +3 and −3, respectively. To do so, the sensing point Ss, Ss+1, Se+2, Se, Se−1, and Se−2 are re-classified into the monitoring point.

80 10 FIG. The target width Wt of the abnormal section can be determined, but not limited to, by finding the end points of the object trajectoryusing techniques such as line detection, kink detection or vehicle tracking algorithms. The width Wt denotes the length, or distance, of the non-monitoring section. Wt, when estimated correctly, makes the object trajectory continuous as shown inafter removing this abnormal section. If the object trajectory is not continuous, the width Wt is estimated again by correcting the end point coordinates of the vehicle trajectories.

80 90 2100 90 80 90 90 90 80 90 2100 90 80 90 When there are two or more object trajectoriesthat cross the same abnormal sectionas each other, the correcting unitmay determine the target width Wt of the abnormal sectionbased on the object trajectoriesthat cross the abnormal section, and may shift the start point and the end point of the abnormal sectionby the same distance as each other so as to change the width of the abnormal sectioninto Wt. Specifically, for each object trajectorycrossing the same abnormal sectionas each other, the correcting unitdetermines a candidate width Wc of the abnormal section, and computes a statistical value (e.g., an average value) of the candidate widths Wc as Wt. The candidate width Wc corresponding to an object trajectory OT1 may be determined in the same way as the way to determine the target width Wt in the case where there is no object trajectoryother than OT1 that crosses the abnormal section, which is explained above.

90 2100 80 2100 2100 When determining the target width Wt of the abnormal section, the correcting unitmay exclude one or more outliers (called “outlier trajectory”) from the object trajectoriesfor which the candidate width Wc are computed. Suppose that there are four object trajectories OT1, OT2, OT3, and OT4 that cross the abnormal section A1. In addition, the object trajectory OT2 is determined to be an outliner trajectory. In this case, the correcting unitcomputes the candidate width Wc1, Wc3, and Wc4 for OT1, OT3, and OT4, respectively. Since OT2 is determined to be an outlier, the candidate width Wc is not computed for OT2. Then, the correcting unitcomputes a statistical value SV1 of Wc1, Wc2, and Wc3 as the target width Wt of the abnormal section A1, and modifies the start point and the end point of A1 included in the sensing point information so that the width of A1 is changed into Wt(=SV1).

80 2100 80 2100 80 80 2100 80 80 2100 80 To determine whether or not the object trajectoryis an outlier trajectory, the correcting unitmay compute a degree of irregularity of the object trajectory. The correcting unitdetermines whether or not the degree of irregularity of the object trajectoryis less than a predefined threshold. When it is determined that the degree of irregularity of the object trajectoryis less than the predefined threshold, the correcting unitcomputes the candidate width of the abnormal section based on that object trajectory. On the other hand, when it is determined that the degree of irregularity of the object trajectoryis not less than the predefined threshold, the correcting unitdoes not compute the candidate width of the abnormal section based on that object trajectory.

80 80 80 80 2100 80 In some embodiments, a degree of linearity (proportionality, in other words) of the object trajectorycan be used to represent the degree of irregularity of the object trajectory. Specifically, the degree of irregularity of the object trajectoryis determined to be higher as the degree of linearity of the object trajectoryis lower. There are well-known ways to measure a degree of linearity of a curve, and one of those ways can be applied to the correcting unitto compute the degree of linearity of the object trajectory.

80 In other embodiments, the degree of irregularity may be measured based on direction, change in speed, overall travel behavior, or two or more thereof that are computed using the object trajectory. Specifically by tracking the object trajectory for measures as mentioned before for travelling behaviors such as low speeds, over-speeding or sudden change in speeds. The irregular trajectories show measure of these behaviors as outliers compared to the neighboring vehicle trajectories in this measuring section.

2000 50 10 50 80 10 In some embodiments, the classifying apparatususes the corrected sensing point information to generate the monitoring datafrom the waterfall data. Then, the monitoring dataand the object trajectoriesdetected from the waterfall datacan be used in a traffic flow monitoring application. The traffic flow monitoring application may compute traffic flow properties, such as vehicle speeds and vehicle count. These properties can be used to monitor traffic flow.

The program can be stored and provided to a computer using any type of non-transitory computer readable media. Non-transitory computer readable media include any type of tangible storage media. Examples of non-transitory computer readable media include magnetic storage media (such as floppy disks, magnetic tapes, hard disk drives, etc.), optical magnetic storage media (e.g., magneto-optical disks), CD-ROM (compact disc read only memory), CD-R (compact disc recordable), CD-R/W (compact disc rewritable), and semiconductor memories (such as mask ROM, PROM (programmable ROM), EPROM (erasable PROM), flash ROM, RAM (random access memory), etc.). The program may be provided to a computer using any type of transitory computer readable media. Examples of transitory computer readable media include electric signals, optical signals, and electromagnetic waves. Transitory computer readable media can provide the program to a computer via a wired communication line (e.g., electric wires, and optical fibers) or a wireless communication line.

Although the present disclosure is explained above with reference to example embodiments, the present disclosure is not limited to the above-described example embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the invention.

The whole or part of the example embodiments disclosed above can be described as, but not limited to, the following supplementary notes.

at least one memory that is configured to store instructions; and at least one processor that is configured to execute the instructions to: acquire a waterfall data that indicates amplitude of vibration for each point in time and for each sensing point in a vibration sensor that is placed along a target object; perform semantic segmentation on the waterfall data to generate a class data that indicates a normal class or an abnormal class for each element of the waterfall data, the normal class being assigned to the element whose sensing point is predicted to be a monitoring point, the abnormal class being assigned to the element whose sensing point is predicted to be a non-monitoring point, the monitoring point being the sensing point that is placed along the target object, the non-monitoring point being the sensing point that is not placed along the target object; and classify the sensing points into the monitoring point and the non-monitoring point based on the class data. A classifying apparatus comprising:

computing the number of the elements of the waterfall data that correspond to the sensing point and to which the normal class are assigned; determining whether the sensing point is the monitoring point or the non-monitoring point based on the computed number. wherein the classifying the sensing points includes performing, for each sensing point: The classifying apparatus according to supplementary note 1,

generate sensing point information that indicates, for each sensing point, whether the sensing point is the monitoring point or the non-monitoring point; detect one or more trajectories of moving objects from the waterfall data, the moving object being an object that moves on the target object; and correct the sensing point information based on the detected trajectories. wherein the at least one processor is further configured to: The classifying apparatus according to supplementary note 1 or 2,

for each one of trajectories that cross the abnormal section, determining a candidate width of the abnormal section based on the trajectory; computing a statistical value of the computed candidate widths as a target width of the abnormal section; and modifying a width of the abnormal section into the target width. wherein the correcting the sensing point information includes performing, for each one of abnormal sections that are regions of one or more consecutive non- monitoring points in the waterfall data: The classifying apparatus according to supplementary note 3,

computing a degree of irregularity of the trajectory; and computing the candidate width of the abnormal section based on the trajectory when the degree of irregularity of the trajectory is less than a predefined threshold. wherein determining the candidate width of the abnormal section for the trajectory including: The classifying apparatus according to supplementary note 4,

wherein the degree of irregularity of the trajectory is determined based on a degree of linearity of the trajectory. The classifying apparatus according to supplementary note 5,

acquiring a waterfall data that indicates amplitude of vibration for each point in time and for each sensing point in a vibration sensor that is placed along a target object; performing semantic segmentation on the waterfall data to generate a class data that indicates a normal class or an abnormal class for each element of the waterfall data, the normal class being assigned to the element whose sensing point is predicted to be a monitoring point, the abnormal class being assigned to the element whose sensing point is predicted to be a non-monitoring point, the monitoring point being the sensing point that is placed along the target object, the non-monitoring point being the sensing point that is not placed along the target object; and classifying the sensing points into the monitoring point and the non- monitoring point based on the class data. A classifying method that is computed by a computer, comprising:

computing the number of the elements of the waterfall data that correspond to the sensing point and to which the normal class are assigned; determining whether the sensing point is the monitoring point or the non-monitoring point based on the computed number. wherein the classifying the sensing points includes performing, for each sensing point: The classifying method according to supplementary note 7,

generating sensing point information that indicates, for each sensing point, whether the sensing point is the monitoring point or the non-monitoring point; detecting one or more trajectories of moving objects from the waterfall data, the moving object being an object that moves on the target object; and correcting the sensing point information based on the detected trajectories. The classifying method according to supplementary note 7 or 8, further comprising:

for each one of trajectories that cross the abnormal section, determining a candidate width of the abnormal section based on the trajectory; computing a statistical value of the computed candidate widths as a target width of the abnormal section; and modifying a width of the abnormal section into the target width. wherein the correcting the sensing point information includes performing, for each one of abnormal sections that are regions of one or more consecutive non-monitoring points in the waterfall data: The classifying method according to supplementary note 9,

computing a degree of irregularity of the trajectory; and computing the candidate width of the abnormal section based on the trajectory when the degree of irregularity of the trajectory is less than a predefined threshold. wherein determining the candidate width of the abnormal section for the trajectory including: The classifying method according to supplementary note 10,

wherein the degree of irregularity of the trajectory is determined based on a degree of linearity of the trajectory. The classifying method according to supplementary note 11,

acquiring a waterfall data that indicates amplitude of vibration for each point in time and for each sensing point in a vibration sensor that is placed along a target object; performing semantic segmentation on the waterfall data to generate a class data that indicates a normal class or an abnormal class for each element of the waterfall data, the normal class being assigned to the element whose sensing point is predicted to be a monitoring point, the abnormal class being assigned to the element whose sensing point is predicted to be a non-monitoring point, the monitoring point being the sensing point that is placed along the target object, the non-monitoring point being the sensing point that is not placed along the target object; and classifying the sensing points into the monitoring point and the non-monitoring point based on the class data. A non-transitory computer-readable storage medium storing a program that causes a computer to execute:

computing the number of the elements of the waterfall data that correspond to the sensing point and to which the normal class are assigned; determining whether the sensing point is the monitoring point or the non-monitoring point based on the computed number. wherein the classifying the sensing points includes performing, for each sensing point: The storage medium according to supplementary note 13,

generating sensing point information that indicates, for each sensing point, whether the sensing point is the monitoring point or the non-monitoring point; detecting one or more trajectories of moving objects from the waterfall data, the moving object being an object that moves on the target object; and correcting the sensing point information based on the detected trajectories. wherein the program causes the computer to further execute: The storage medium according to supplementary note 13 or 14,

for each one of trajectories that cross the abnormal section, determining a candidate width of the abnormal section based on the trajectory; computing a statistical value of the computed candidate widths as a target width of the abnormal section; and modifying a width of the abnormal section into the target width. wherein the correcting the sensing point information includes performing, for each one of abnormal sections that are regions of one or more consecutive non-monitoring points in the waterfall data: The storage medium according to supplementary note 15,

computing a degree of irregularity of the trajectory; and computing the candidate width of the abnormal section based on the trajectory when the degree of irregularity of the trajectory is less than a predefined threshold. wherein determining the candidate width of the abnormal section for the trajectory including: The storage medium according to supplementary note 16,

wherein the degree of irregularity of the trajectory is determined based on a degree of linearity of the trajectory. The storage medium according to supplementary note 17,

10 waterfall data 20 sensing data 30 vibration sensor 40 target object 50 monitoring data 60 waterfall image 70 moving object 80 object trajectory 90 abnormal section 1000 computer 1020 bus 1040 processor 1060 memory 1080 storage device 1100 input/output interface 1120 network interface 2000 classifying apparatus 2020 acquiring unit 2040 segmenting 2060 classifying unit 2080 detecting unit 2100 correcting unit