Systems and Methods for Detecting Traffic Light Violations

Traffic laws may govern and regulate vehicles on roadways. Traffic laws may define rules involving observing speed limits, observing traffic lights, following traffic signs, yielding to special vehicles (e.g., school buses and emergency vehicles), etc. Individuals that violate traffic laws may be subjected to fines or other types of punishment.

The following detailed description of example implementations refers to the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements.

A camera may be installed on a dashboard of a vehicle and capture video of the road when the vehicle is driving on the road. The vehicle may be a connected vehicle. The camera may be an onboard camera, mounted on the dashboard, that records a scene associated with the vehicle. The camera may start capturing video after the vehicle is turned on and may stop capturing video after the vehicle is turned off. When the camera detects an event (e.g., hard acceleration or harsh cornering), a video recording from the captured video may be created. The video recording may include video of the road and surrounding areas associated with the event. For example, the video recording may include objects near the vehicle, such as stop signs, traffic lights, and/or surrounding vehicles. The camera may send the video recording to a server. The server may review the video recording, and based on the video recording, the server may classify the event using a video detection algorithm. For example, the server may classify the event as being related to an unsafe driving behavior. Alternatively, the classification of the event may be performed by a local computing device associated with the vehicle. A notification may be sent to a driver of the vehicle and/or a supervisor. Depending on the video recording and the classification of the event, the driver may be coached in safter driving habits.

In one example, the event may be a red traffic light violation. The red traffic light violation may occur when the vehicle crosses an intersection after a traffic light controlling the intersection has turned red. In other words, the red traffic light violation may occur when the vehicle enters the intersection (e.g., passes a stop bar) after the traffic light has turned red. A detection of the red traffic light violation may involve determining a state of the traffic light in the scene, determining which traffic light is relevant to the vehicle, and/or determining whether the vehicle is crossing the relevant traffic light when the state of the traffic light is red.

However, detecting red traffic light violations in video recordings obtained from connected vehicles may be difficult, and in some cases, red traffic light violations may be mistakenly detected or not detected altogether. Traffic light relevance may be one cause of misdetections for red traffic light violations or non-detections of traffic light violations. In a scene captured from the intersection in which the traffic light regulates the passage of vehicles, dozens of traffic lights may be turned on at the same time and with different colors. For example, some traffic lights may be red, other traffic lights may be yellow, and still other traffic lights may be green. A determination as to which traffic light regulates a passage for a given vehicle (and corresponding driver) may be challenging. Such a traffic light may be considered to be a relevant traffic light. In the scene, zero, one, or more traffic lights may be considered to be relevant traffic lights. In some cases, especially with multiple lanes, determining which traffic light is relevant to a particular vehicle may be difficult.

As an example, in a scene, the only relevant traffic light out of multiple traffic lights may be a traffic light in the middle of the scene, as a traffic light in a left portion of the scene may be regulating just a left turn and a traffic light in a right portion of the scene may be regulating a crossing for pedestrians. In another example, in a scene, one traffic light may be considered to be relevant when the vehicle is performing a right turn. In this case, the relevant traffic light may not be aligned with a center of the scene, so knowledge of a context of the vehicle's movement (e.g., performing the right turn) may be needed to determine which traffic light is relevant. In these examples, detecting which traffic light is relevant to the vehicle may be challenging.

As an example, in a scene, a traffic light on a highway ramp may turn green just for a few seconds, which may indicate that a single vehicle is able to pass. When the vehicle is actually passing the traffic light, a state of the traffic light may already be red again. As another example, in a scene, a traffic light may be red, but the vehicle may lawfully make a right turn when the traffic light is red. In this example, such an event should not be classified as a traffic light violation, even though the traffic light is red when the vehicle makes the right turn. As yet another example, a traffic light may flash red or yellow (depending on different meanings), and generally a vehicle may be permitted to cross an intersection in such a scenario, which may happen more frequently at night. In these examples, the vehicle may legally cross a relevant traffic light while the traffic light's state is red, which may mistakenly trigger a traffic light violation to be detected.

In some implementations, an anti-causal algorithm, using speed and video, may combine deep learning models with heuristics to accurately detect traffic light violations in videos obtained from connected vehicles. The detection of traffic light violations may be without the usage of high-definition satellite map information. The detection of traffic light violations may be part of a fleet management application. The detection of traffic light violations may be relatively accurate, even in the presence of multiple traffic lights and even when the vehicle performs a special case (e.g., making a right turn on a red light). The detection of traffic light violations may be used to identify coachable events, so that drivers in a commercial fleet may improve their behavior and increase overall safety.

In some implementations, by combining speed information, video information, deep learning models, and/or heuristics, video obtained from a connected vehicle may be analyzed to accurately detect traffic light violations. Video captured by a camera onboard the vehicle may be analyzed to determine whether the vehicle unlawfully crosses an intersection when the traffic light is red. The anti-causal algorithm may be able to account for corner cases, such as traffic lights on highway ramps that are green for only a short period of time, lawfully permitted right turns by vehicles even when the traffic light is red, and/or traffic lights that flash red or yellow during which the vehicle is permitted to cross the intersection. Such corner cases may be considered and may not unnecessarily result in false traffic light violation predictions. As a result, an ability to accurately predict whether traffic light violations occur may be achieved, thereby improving an overall system performance.

is a diagram of an example 100 associated with detecting traffic light violations. As shown in, example 100 includes a camera, a vehicle, a server, a user device, and sensor(s). The cameramay be onboard the vehicle. For example, the cameramay be installed on a dashboard of the vehicle.

The cameramay capture a video recording of a scene surrounding the vehicle. The cameramay be able to record video of the scene that is in front of the vehicle. For example, the cameramay be able to record objects and pedestrians that are in front of the vehicle. The cameramay be continuously recording the scene when the vehicle is turned on.

One or more sensorsmay capture sensor information. The one or more sensorsmay include a gyroscope. The gyroscope may be integrated with the camera, or the gyroscope may be external to the camera. In this example, the sensor information may include rotation information, orientation information, and/or angular velocity information associated with the vehicle. The one or more sensorsmay include a global positioning system (GPS). In this example, the sensor information may include speed information associated with the vehicle.

The camera(or another computing device associated with the vehicle) may transmit the video recording and the sensor information to the server, where the servermay detect a traffic light violation based on the video recording and the sensor information. Alternatively, the video recording and the sensor information may be processed locally by the computing device associated with the vehicle. In this example, a traffic light violation detection may be performed locally at the vehicle. The servermay obtain the video recording and the sensor information, where the video recording may be of the scene captured by the cameraonboard the vehicle, and the sensor information may be associated with the vehicle(e.g., orientation information and/or speed information).

The servermay perform an object detection (OD) that indicates a presence of a traffic light in a frame of the video recording. The object detection is described in greater detail in. The object detection may not necessarily detect which traffic light of multiple traffic lights is relevant in the frame. Rather, the object detection may be used to determine whether one or more traffic lights are present in the frame, and which color(s) are associated with the traffic lights. A red traffic light may indicate that the vehicleis to stop at an intersection, a yellow traffic light may indicate that the vehicleis to slow down and stop at the intersection if possible, and a green traffic light may indicate that the vehicleis allowed to pass the intersection without stopping.

The servermay perform, based on the sensor information, a turn detection (RT) that indicates whether the vehicleis performing a turn during the frame. The turn detection is described in greater detail in. For example, the turn detection may be used to detect whether the vehicleis performing a right turn on the intersection. A right turn detection may be useful because, in some cases, the vehiclemay be permitted to make the right turn even when the traffic light is red. In other words, such an action may not constitute a traffic signal violation. The servermay determine whether the vehicleis making the right turn based on gyroscope information.

The servermay perform, based on the sensor information, a speed detection(S) that indicates a speed associated with the vehicleduring the frame. The speed detection is described in greater detail in. The vehiclemay include a GPS that tracks a speed of the vehiclein real-time. The servermay correlate a timestamp associated with the frame and a timestamp associated with the speed of the vehicle, such that the servermay be able to determine, when the frame was taken, the speed of the vehicleat that time.

The servermay determine, based on an image classifier, a red light probability (P) that the frame contains at least one relevant red traffic light for the vehicle. The red light probability is described in greater detail in. The image classifier may be used to categorize road images captured by the camera. The image classifier may be fed with additional inputs other than an image, such as the results of on an object detector with multiple attributes, and the multiple attributes may include a first attribute for traffic light relevance and a second attribute for traffic light state. The image classifier may be trained using a training set, where the training set may include a plurality of historical images. As a result, the servermay determine, for the frame, a probability that a particular red light is relevant to the vehicle.

The servermay calculate a violation score based on the object detection, the turn detection, and the red light probability with respect to the frame, and based on the speed detection. The violation score may account for, based on a grace period, a traffic light that turns green for a limited time period that allows only a single vehicle to pass and then turns red. The violation score may account for the vehicle making a lawfully permitted right turn when the traffic light is red. The frame may be one of multiple frames, and the violation score may account for the traffic light flashing red or flashing yellow based on the multiple frames. The violation score may be a numerical value, and the violation score may fall within a defined range (e.g., 0 to 1, or 0 to 100). The violation score may be computed using different components, which may be related to the object detection, the turn detection, the red light probability, and/or the speed detection.

In some implementations, the servermay determine, based on the image classifier, a not relevant probability that the frame contains no traffic light or that the frame contains one or more traffic lights that are not relevant to the vehicle, and/or a green light probability that the frame contains at least one relevant green traffic light for the vehicle.

The servermay determine whether the vehicle is associated with a traffic light violation based on the violation score in relation to a threshold. The server may compare the violation score to the threshold, and depending on whether the violation score satisfies the threshold, the servermay predict that the traffic light violation has occurred. The traffic light violation may involve the vehicle driving past the intersection when the traffic light is red, and no exception exists that lawfully permits the vehicle to cross the intersection when the traffic light is red. A detection of the traffic light violation may be based on video information, speed information, and heuristics, and the detection of the traffic light violation may be without a use of satellite map information. In other words, the detection of the traffic light violation may not be based on satellite map information.

In some implementations, as part of a traffic light violation detector, multiple time series may be combined together in a single violation score. A time series may be given by: R(t)=OD(t)*RT(t)*P(t), where t denotes a given frame, OD is associated with object detection, RT is associated with right turn detection, and Pis associated with a probability of a relevant red traffic light. When a value of R(t) is close to one, the vehiclemay be in the presence of a relevant red traffic light since the object detection mask is not zero, Pis close to one, and the vehicleis not turning right since RT(t) is not zero.

In some implementations, an aggregation of the time series Pover a rolling window of length T may be represented by:

where Pis associated with a probability of no relevant traffic light. Ideally, right after the vehiclepasses through the intersection, a value may change from zero to close to one (e.g., a probability of any traffic light state dropped to zero), meaning that no traffic light is detected in an upcoming time window of length T. Before passing the intersection, one of the probabilities for other colors (e.g., green, yellow, or red) may be close to one. When R(t)*NR(t) is close to one for some value t′, then at time t′ the vehiclemay have run a red light (which may not necessarily mean a traffic light violation). The vehiclemay have stopped late at the intersection and the traffic light may no longer be visible, or a camera installation may cause a camera to be pointed downward, and in proximity of the intersection, so that the traffic light may quickly disappear from the camera field of view. In these cases, no traffic light violation may actually occur, but a false positive may be triggered. To account for such cases, the speed of the vehicle may be considered. When the speed is below a given threshold, the traffic light violation may not be considered to be possible, so an actual score may be given by: violation_scoreR=R(t)*NR(t)*S(t), where S is associated with the speed detection.

In some implementations, to make the violation score more robust, filtering may be applied to a speed mask. The filtering may involve calculating a mean speed over a rolling window. If the traffic light violation happens at a very low speed, a detection of the traffic light violation may not be possible. A GPS speed may already have some uncertainty and considering zero as a limit value without any tolerance may not be feasible.

In some implementations, a time series Pmay be considered for a highway ramp scenario (e.g., a traffic light flashes between green and red), where Pis associated with a probability of a relevant green traffic light. When a relevant traffic light state transitions from red to green (or green to red), a grace period may be used for the next few seconds. The grace period may serve to artificially set a score to zero. For detecting a transition from red to green (or green to red), the time series Pmay be aggregated and a value of its product may be monitored with R (t), such that:

where for a red-green transition to occur, R(t)*GR(t)˜1 should be satisfied.

In some implementations, one scenario that may be accounted for is flashing traffic lights. In this scenario, a state transition from red to not relevant may be periodic. In NR(t), an average of a not relevant probability may be taken over a next T seconds. When a flashing period is less than T (e.g., 0.5*T), NR will be equal to 0.5, and hence this particular event may be ranked in a lower position then an actual red light violation. The traffic light violation may be predicated when a score is above a threshold/confidence score, where the threshold/confidence score may be determined by monitoring a performance of an algorithm in a dataset. An additional minimum filter may be applied to the score, which may avoid a bad prediction of a relevant state, even when just happening for one frame, leading to a false positive. Further, an anti-causal window may be used, such that the score cannot be computed in real time, but rather with a delay that is at least equal to a length of a time window.

In some implementations, the servermay determine, based on the image classifier, a yellow light probability that the frame contains at least one relevant yellow traffic light for the vehicle. The servermay determine, based on the image classifier, the not relevant probability that the frame contains no traffic light or that the frame contains one or more traffic lights that are not relevant to the vehicle. The servermay perform, based on the sensor information, the speed detection that indicates the speed associated with the vehicleduring the frame. The servermay determine a yellow stop score that indicates a severity of a yellow light violation. The yellow stop score may be based on the speed and a duration of a detected yellow relevant traffic light. The servermay calculate the violation score based on the object detection, the turn detection, the yellow light probability, the not relevant probability, the speed detection, and the yellow stop score.

In some implementations, the vehiclemay run a yellow light. Passing through a traffic light regulated stop with a yellow light turned on may not necessarily be a traffic light violation. The vehicleshould stop at the traffic light when possible, and the vehicleis permitted to not stop at the traffic light when not possible. Given that Y(t)=OD(t)*RT (t)*P(t), when Y(t)*NR(t) is close to one for some value t′, a yellow light violation may have happened at t′. In this example, Pis associated with a probability of a relevant yellow traffic light. A violation score may be represented by: violation_score=Y(t)*NR(t)*S(t)*YellowStopScore, where YellowStopScore may capture a severity of a yellow light violation. The severity of the yellow light violation may leverage a speed of the vehicleand a duration of a detected yellow relevant traffic light. In a specific example, YellowStopScore=Duration(t)*60/min (S(t), 60). A higher duration of the yellow traffic light may result in a higher YellowStopScore (e.g., a yellow light that is seen for a longer period of time gives a driver of the vehiclemore opportunity to stop the vehiclein time). On the other hand, a duration may be divided by a term that considers a minimum of the speed and a threshold that is set to 60 km/h. In this way, when the vehicleis traveling 60 km/h or more, this term may be set to one. When the vehicleis traveling at a slower speed (e.g., a speed less than 60 km/h), this term may increase and raise an overall YellowStopScore. For a vehiclethat is traveling relatively fast, stopping at a yellow light may be less feasible, but for a vehiclethat is traveling relatively slow, stopping at the yellow light may be more feasible.

As shown by reference number, the servermay transmit, to the user device, a notification that indicates whether the vehicleis associated with the traffic light violation. The user devicemay be associated with the driver of the vehicle. In this example, the notification may indicate a recommendation for the driver of the vehicleto improve a driving behavior and increase safety in response to the traffic light violation. The user devicemay be associated with a supervisor of the driver. In this example, the notification may indicate that the driver was involved in the traffic light violation.

As indicated above,is provided as an example. Other examples may differ from what is described with regard to. The number and arrangement of devices shown inare provided as an example. In practice, there may be additional devices, fewer devices, different devices, or differently arranged devices than those shown in. Furthermore, two or more devices shown inmay be implemented within a single device, or a single device shown inmay be implemented as multiple, distributed devices. Additionally, or alternatively, a set of devices (e.g., one or more devices) shown inmay perform one or more functions described as being performed by another set of devices shown in.

is a diagram of an example 200 associated with object detection.

In some implementations, object detection may be used for road scene analysis. Object detection may be used to determine, at each frame of a video, a location, a size, and/or a shape of each traffic light. Object detection may not provide information regarding traffic light relevance and state, but may be used to check the presence of traffic lights in a given video. An absence of a traffic light in the video, as determined using object detection, may result in no detection of a traffic light violation because a presence of a traffic light is critical for detecting the traffic light violation.

In some implementations, results of object detection may be used to create a mask to select time windows that may be of interest for traffic light violation detection in the given video. A time series given by a maximum area of the traffic lights detected in all frames of the video may be analyzed. When no traffic light is present, a value of the time series may be set to zero. Intervals of interest may be an interval in which a value of the time series is above a certain threshold. A resulting mask may be a time series that is set to zero when the value is below the threshold and set to one elsewhere. The resulting mask may be known as a square wave.

As shown in, for object detection, when a vehicle approaches an intersection, a maximum traffic light area in an image of a scene may increase from zero to a certain value, which may be due to the movement of the vehicle towards a traffic light. When the vehicle is stopped at the intersection, the maximum traffic light area may be constant for a period of time. When the vehicle passes the intersection, the traffic light may no longer be detected, so the maximum traffic light area may drop to zero. An interval of interest may start as the vehicle approaches the intersection (e.g., once a certain threshold is satisfied) and the area of interest may end when the vehicle passes the intersection. A mask (OD) may be created that corresponds to the interval of interest. The resulting mask may be set to one during the interval of interest, and the resulting mask may be set to zero at other times.

As indicated above,is provided as an example. Other examples may differ from what is described with regard to.

is a diagram of an example 300 associated with turn detection.

In some implementations, turn detection may be used for detecting a right turn of a vehicle. When the vehicle is permitted to turn right at an intersection at a red light, a traffic light violation should not be predicted. The vehicle may initially stop at the red light, and then after a period of time (e.g., two seconds), the vehicle may be lawfully permitted to make the right turn while the traffic light is still red. The vehicle may have to initially stop at the red light based on a traffic sign that instructs a driver of the vehicle to stop. For the turn detection, a z-axis component of an angular velocity measured by a gyroscope sensor in the camera may be used. When a value of the z-axis component is a negative value, the vehicle may be turning right. A mask may be created, where the mask may start from an angular velocity checking whenever the value is below a given negative threshold. The mask may be made narrower by adding a fixed buffer of X seconds before and after the zero seconds, where X is a positive integer. An estimate may become more robust when, instead of an instantaneous value, a mean filter is applied on the angular velocity.

As shown in, for turn detection, the z-axis component of the angular velocity may be tracked over a period of time. A value of the z-axis component being below a given negative threshold may indicate that the vehicle is making the right turn. A mask (RT) may be created, starting from a positive value of the z-axis component, to check whenever the value falls below the given negative threshold, which may indicate that the vehicle is making the right turn. The mask may be set to one when the value of the z-axis component is above the given negative threshold. The mask may be set to zero when the value of the value of the z-axis component reaches the given negative threshold and a right turn is detected.

As indicated above,is provided as an example. Other examples may differ from what is described with regard to.

is a diagram of an example 400 associated with speed detection.

As shown in, for speed detection, a detection of a traffic light violation may depend on a speed of a vehicle. The vehicle may have to move at a given speed in order to run a red light. When the vehicle is stopped, a traffic light violation may not be possible. The speed of the vehicle may be measured using a GPS associated with the vehicle. A mask(S) may be created to track the speed of the vehicle over a period of time. When the speed is below a certain threshold, the mask may be set to zero. When the speed is above the certain threshold, the mask may be set to one.

As indicated above,is provided as an example. Other examples may differ from what is described with regard to.

is a diagram of an example 500 associated with traffic light state classification.

In some implementations, for traffic light state classification, an image classifier may be used to categorize road images into different classes. An image in which no traffic light is present, or an image that has at least one traffic light but none of the traffic lights are relevant for a vehicle that is capturing the image, may be associated with a not relevant class. An image in which at least one relevant green traffic light is present for the vehicle may be associated with a relevant green class. An image in which at least one relevant yellow traffic light is present for the vehicle may be associated with a relevant yellow class. An image in which at least one relevant red traffic light is present for the vehicle may be associated with a relevant red class. The image classifier may be run for each frame in a video, which may produce a time series containing probabilities for each state (e.g., not relevant, relevant green, relevant yellow, or relevant red). A complementary value to a sum of the probabilities of each of the colors (e.g., green, yellow, and red) may represent a probably that no relevant traffic light is present for the vehicle. Each one of the time series may be represented by P, P, P, and P, where Pis associated with a probability of no relevant traffic light, Pis associated with a probability of a relevant green traffic light, Pis associated with a probability of a relevant yellow traffic light, and Pis associated with a probability of a relevant red traffic light. In some cases, the image classifier may not be used, but rather an object detector with multiple attributes (e.g., one attribute for traffic light relevance and one attribute for state) may be used instead. When a probability of a given state drops to zero and a dominant probability becomes not relevant, the vehicle may have just passed an intersection.

As shown in, a time series of images may be captured by a camera onboard a vehicle. Initially, the image classifier may determine that one or more frames indicate that no relevant traffic light is likely, which may be based on a state probability. After a certain point of time, the image classifier may determine that one or more frames indicate that a green traffic light is likely, which may be based on the state probability. The image classifier may then determine that one or more frames indicate that a yellow light is likely, which may be based on the state probability. The image classifier may then determine that one or more frames indicate that a red light is likely, which may be based on the state probability. A traffic light may change from green, to yellow, and to red as the vehicle approaches an intersection. After a certain point of time at which the red light is likely, the state probability may switch to zero, at which point no relevant traffic light is likely. The state probability may switch to zero as a result of the vehicle passing the intersection (e.g., the red light is no longer present in the frame). When the vehicle passes the red light, a potential traffic light violation may occur. For example, a traffic light violation may occur when the vehicle passes the intersection and the traffic light is still red. Further, as shown in, a time series of Pmay indicate a value close to zero when no relevant traffic light is likely, when the green light is likely, and when the yellow light is likely. When the red light becomes likely, Pmay increase to one (indicating that a presence of at least one relevant red traffic light is very likely).

As indicated above,is provided as an example. Other examples may differ from what is described with regard to.

is a diagram of an example environmentin which systems and/or methods described herein may be implemented. As shown in, environmentmay include a camera, a vehicle, a server, a user device, and a network. Devices of environmentmay interconnect via wired connections, wireless connections, or a combination of wired and wireless connections.