System and methods for automatically detecting double parking violations

This application claims the benefit of U.S. Provisional Patent Application No. 63/507,183 filed on Jun. 9, 2023, the content of which is incorporated herein by reference in its entirety.

This disclosure relates generally to the field of computer-based parking violation detection and, more specifically, to systems and methods for automatically detecting double parking violations.

Non-public vehicles double parking is a significant transportation problem for municipalities, counties, and other government entities. Double parking is often defined in municipal codes as a vehicle that is illegally parked on a lane of a roadway next to a permitted parking lane or next to a parked vehicle in the permitted parking lane. The permitted parking lane is often a lane or portion of a roadway that is closest to a road edge (e.g., a right road edge in the United States). Double-parked vehicles can cause traffic congestion, accidents, and discourage usage of specialty lanes blocked by such double-parked vehicles.

For example, a vehicle double-parked in a no-parking lane can disrupt schedules of municipal fleet vehicle and frustrate those that depend on such municipal fleet vehicles. Similarly, vehicles double-parked in a bike lane can force bicyclists to ride on the road, making their rides more dangerous and discouraging the use of bicycles as a safe and reliable mode of transportation. Moreover, double-parked vehicles can disrupt crucial municipal services such as street sweeping, waste collection, and firefighting operations.

Traditional photo-based traffic enforcement technology and approaches are often unsuited for today's fast-paced environment. For example, photo-based traffic enforcement systems often rely heavily on human reviewers to review and validate evidence packages containing images or videos captured by one or more stationary cameras. This requires large amounts of human effort and makes the process slow, inefficient, and costly. In particular, traffic enforcement systems that rely on human reviewers are often not scalable, require more time to complete the validation procedure, and do not learn from their past mistakes. Furthermore, these photo-based traffic enforcement systems often fail to take into account certain contextual factors or features that may provide clues as to whether a captured event is or is not a potential parking violation.

Therefore, an improved computer-based parking violation detection system is needed that can undertake certain evidentiary reviews automatically without relying on human reviewers and can take into account certain automatically detected contextual factors or features that may aid the system in determining whether a double parking violation has indeed occurred. Such a solution should be accurate, scalable, and cost-effective to deploy and operate.

Disclosed herein are methods, apparatus, and systems for automatically detecting double parking violations. For example, one aspect of the disclosure concerns a method for automatically detecting double parking violations comprising determining, using one or more processors of an edge device, a location of a road edge of a roadway from one or more video frames of a video captured by one or more video image sensors of the edge device. The method can further comprise determining a layout of one or more lanes of the roadway, including at least one no-parking lane, based on the road edge determined by the edge device. The method can also comprise bounding the no-parking lane using a lane bounding polygon, bounding a vehicle detected from the one or more video frames using a vehicle bounding polygon, and detecting a potential double parking violation based in part on an overlap of at least part of the vehicle bounding polygon with at least part of the lane bounding polygon.

The method can also comprise determining whether the vehicle is moving or static when captured by the video and detecting the potential double parking violation only in response to the vehicle being determined to be static when captured by the video (i.e., only if the vehicle is determined to be static or not moving when captured by the video).

In some embodiments, the step of determining the road edge can further comprise fitting a line representing the road edge to a plurality of road edge points using a random sample consensus algorithm (RANSAC).

In some embodiments, the line fitted to the plurality of road edge points can be parameterized by a slope and an intercept. Each of the slope and the intercept can be calculated using a sliding window or moving average algorithm such that the slope is an average slope value and the intercept is an average intercept value calculated from one or more video frames captured prior in time.

In some embodiments, the plurality of road edge points can be determined by selecting a subset of points, but not all points, along a mask or heatmap representing the road edge.

In some embodiments, the mask or heatmap can be outputted by a lane segmentation deep learning model running on the edge device.

In some embodiments, the method can also comprise passing the one or more video frames to one or more deep learning models (e.g., an object detection deep learning model, a lane segmentation deep learning model, or a combination thereof) running on the edge device or the server to determine a context surrounding the double parking violation. The context surrounding the double parking violation can be used by the edge device or a server communicatively coupled to the edge device to detect the double parking violation. For example, the context surrounding the double parking violation can be used by the edge device or the server to validate or confirm whether a static vehicle is actually parked or is only temporarily static and is in the process of moving.

In certain embodiments, the one or more deep learning models can comprise an object detection deep learning model, a lane segmentation deep learning model, or a combination thereof. At least one of the deep learning models can be configured to output a multiclass classification concerning a feature associated with the context.

For example, the feature can be brake light status of the vehicle. Also, for example, the feature can be a traffic condition surrounding the vehicle. Moreover, the feature can also be a roadway intersection status.

In some embodiments, the one or more video frames can be captured by an event camera of the edge device. At least one of the video frames can be passed to a license plate recognition deep learning model running on the edge device to automatically recognize a license plate of the vehicle.

In some embodiments, the one or more video frames can be captured by an event camera of the edge device coupled to a carrier vehicle while the carrier vehicle is in motion.

In some embodiments, the method of determining whether the vehicle is moving or static can comprise determining GPS coordinates of vehicle bounding polygons across multiple event video frames, transforming the GPS coordinates into a local Cartesian coordinate system such that the GPS coordinates are transformed coordinates, and determining whether the vehicle is static or moving based on a standard deviation of the transformed coordinates in both a longitudinal direction and a latitudinal direction and a cross correlation of the transformed coordinates along the longitudinal direction and the latitudinal direction.

A longitudinal axis of the local Cartesian coordinate system can be in a direction of travel of a carrier vehicle carrying the edge device. A latitudinal axis of the local Cartesian coordinate system can be in a lateral direction perpendicular and can be perpendicular to the longitudinal axis.

Another aspect of the disclosure concerns a device for automatically detecting a double parking violation. The device can comprise one or more video image sensors configured to capture a video of a vehicle and a roadway including a road edge of the roadway. The device can also comprise one or more processors programmed to determine a location of the road edge of the roadway from one or more video frames of the video captured by the one or more video image sensors, determine a layout of one or more lanes of the roadway, including a no-parking lane, based on the road edge determined by the device, bound the no-parking lane using a lane bounding polygon; bound a vehicle detected from the one or more video frames using a vehicle bounding polygon, and detect a potential double parking violation based in part on an overlap of at least part of the vehicle bounding polygon with at least part of the lane bounding polygon.

Yet another aspect of the disclosure concerns one or more non-transitory computer-readable media comprising instructions stored thereon, that when executed by one or more processors, cause the one or more processors to perform operations comprising determining a location of a road edge of a roadway from one or more video frames of a video captured by one or more video image sensors of an edge device, determining a layout of one or more lanes, including a no-parking lane, of the roadway based on the road edge, bounding the no-parking lane using a lane bounding polygon, bounding a vehicle detected from the one or more video frames using a vehicle bounding polygon, and detecting a potential double parking violation based in part on an overlap of at least part of the vehicle bounding polygon with at least part of the lane bounding polygon.

1 FIG.A 100 100 102 104 106 illustrates one embodiment of a systemthat can be used to automatically detect double parking violations. The systemcan comprise a plurality of edge devicescommunicatively coupled to or in wireless communication with a serverin a cloud computing environment.

104 104 104 The servercan comprise or refer to one or more virtual servers or virtualized computing resources. For example, the servercan refer to a virtual server or cloud server hosted and delivered by a cloud computing platform (e.g., Amazon Web Services®, Microsoft Azure®, or Google Cloud®). In other embodiments, the servercan refer to one or more stand-alone servers such as a rack-mounted server, a blade server, a mainframe, a dedicated desktop or laptop computer, one or more processors or processor cores therein, or a combination thereof.

102 104 104 102 The edge devicescan communicate with the serverover one or more networks. In some embodiments, the networks can refer to one or more wide area networks (WANs) such as the Internet or other smaller WANs, wireless local area networks (WLANs), local area networks (LANs), wireless personal area networks (WPANs), system-area networks (SANs), metropolitan area networks (MANs), campus area networks (CANs), enterprise private networks (EPNs), virtual private networks (VPNs), multi-hop networks, or a combination thereof. The serverand the plurality of edge devicescan connect to the network using any number of wired connections (e.g., Ethernet, fiber optic cables, etc.), wireless connections established using a wireless communication protocol or standard such as a 3G wireless communication standard, a 4G wireless communication standard, a 5G wireless communication standard, a long-term evolution (LTE) wireless communication standard, a Bluetooth™ (IEEE 802.15.1) or Bluetooth™ Lower Energy (BLE) short-range communication protocol, a wireless fidelity (WiFi) (IEEE 802.11) communication protocol, an ultra-wideband (UWB) (IEEE 802.15.3) communication protocol, a ZigBee™ (IEEE 802.15.4) communication protocol, or a combination thereof.

102 104 104 108 108 108 108 The edge devicescan transmit data and files to the serverand receive data and files from the servervia secure connections. The secure connectionscan be real-time bidirectional connections secured using one or more encryption protocols such as a secure sockets layer (SSL) protocol, a transport layer security (TLS) protocol, or a combination thereof. Additionally, data or packets transmitted over the secure connectioncan be encrypted using a Secure Hash Algorithm (SHA) or another suitable encryption algorithm. Data or packets transmitted over the secure connectioncan also be encrypted using an Advanced Encryption Standard (AES) cipher.

104 102 107 106 107 107 107 104 107 107 The servercan store data and files received from the edge devicesin one or more databasesin the cloud computing environment. In some embodiments, the databasecan be a relational database. In further embodiments, the databasecan be a column-oriented or key-value database. In certain embodiments, the databasecan be stored in a server memory or storage unit of the server. In other embodiments, the databasecan be distributed among multiple storage nodes. In some embodiments, the databasecan be an events database.

102 110 110 1 FIG.C As will be discussed in more detail in the following sections, each of the edge devicescan be carried by or installed in a carrier vehicle(seefor examples of different types of carrier vehicles).

102 110 110 For example, the edge device, or components thereof, can be secured or otherwise coupled to an interior of the carrier vehicleimmediately behind the windshield of the carrier vehicle.

1 FIG.A 102 112 114 116 118 120 As shown in, each of the edge devicescan comprise a control unit, an event camera, a license plate recognition (LPR) camera, a communication and positioning unit, and a vehicle bus connector.

114 116 110 114 116 110 In some embodiments, the event cameraand the LPR cameracan be coupled to at least one of a ceiling and headliner of the carrier vehiclewith the event cameraand the LPR camerafacing the windshield of the carrier vehicle.

102 110 102 110 102 110 In other embodiments, the edge device, or components thereof, can be secured or otherwise coupled to at least one of a windshield, window, dashboard, and deck of the carrier vehicle. Also, for example, the edge devicecan be secured or otherwise coupled to at least one of a handlebar and handrail of a micro-mobility vehicle serving as the carrier vehicle. Alternatively, the edge devicecan be secured or otherwise coupled to a mount or body of an unmanned aerial vehicle (UAV) or drone serving as the carrier vehicle.

114 122 110 114 124 124 112 1 5 5 5 FIGS.B,A,B, andC The event cameracan capture videos of vehicles (including a potentially offending vehicle, see, e.g.,) parked near the carrier vehicle. The videos captured by the event cameracan be referred to as event videos. Each of the event videos can be made up of a plurality of event video frames. The event video framescan be processed and analyzed by the control unitin real-time or near real-time to determine whether any of the vehicles have committed a double parking violation.

112 306 114 124 112 124 112 102 124 134 122 134 122 122 122 122 122 124 3 FIG. 3 FIG. For example, one or more processors of the control unitcan be programmed to apply a plurality of functions from a computer vision library(see, e.g.,) to the videos captured by the event camerato read the event video frames. The one or more processors of the control unitcan then pass at least some of the event video framesto a plurality of deep learning models (see, e.g.,) running on the control unitof the edge device. The deep learning models can automatically identify objects from the event video framesand classify such objects (e.g., a car, a truck, a bus, etc.). In some embodiments, the deep learning models can also automatically identify a set of vehicle attributesof a potentially offending vehicle(also referred to as a target vehicle) involved in a double parking violation. The set of vehicle attributescan include a color of the potentially offending vehicle, a make and model of the potentially offending vehicle, and a vehicle type of the potentially offending vehicle(for example, if the potentially offending vehicleis a personal vehicle or a municipal vehicle such as a fire truck, ambulanee, parking enforcement vehicle, police car, etc.). The potentially offending vehiclecan be detected along with other vehicles in the event video frame(s).

116 122 110 116 126 126 112 128 129 122 114 116 The LPR cameracan capture videos of license plates of the vehicles (including the potentially offending vehicle) parked near the carrier vehicle. The videos captured by the LPR cameracan be referred to as license plate videos. Each of the license plate videos can be made up of a plurality of license plate video frames. The license plate video framescan be analyzed by the control unitin real-time or near real-time to extract alphanumeric strings representing license plate numbersof license platesof the potentially offending vehicles. The event cameraand the LPR camerawill be discussed in more detail in later sections.

118 118 The communication and positioning unitcan comprise at least one of a cellular communication module, a WiFi communication module, a Bluetooth® communication module, and a high-precision automotive-grade positioning unit. The communication and positioning unitcan also comprise a multi-band global navigation satellite system (GNSS) receiver configured to concurrently receive signals from a GPS satellite navigation system, a GLONASS satellite navigation system, a Galileo navigation system, and a BeiDou satellite navigation system.

118 102 118 112 130 122 112 130 122 112 130 122 The communication and positioning unitcan provide positioning data that can allow the edge deviceto determine its own location at a centimeter-level accuracy. The communication and positioning unitcan also provide positioning data that can be used by the control unitto determine a locationof a potentially offending vehicle. For example, the control unitcan use positioning data concerning its own location to substitute for the locationof the potentially offending vehicle. The control unitcan also use positioning data concerning its own location to estimate or approximate the locationof the potentially offending vehicle.

102 120 120 102 110 102 102 110 120 102 110 120 120 102 130 122 The edge devicecan also comprise a vehicle bus connector. The vehicle bus connectorcan allow the edge deviceto obtain certain data from the carrier vehiclecarrying the edge device. For example, the edge devicecan obtain wheel odometry data from a wheel odometer of the carrier vehiclevia the vehicle bus connector. Also, for example, the edge devicecan obtain a current speed of the carrier vehiclevia the vehicle bus connector. As a more specific example, the vehicle bus connectorcan be a J1939 connector. The edge devicecan take into account the wheel odometry data to determine the locationof the potentially offending vehicle.

102 132 122 130 302 102 102 102 132 132 132 The edge devicecan also record or generate at least a plurality of timestampsmarking the time when the potentially offending vehiclewas detected at a location. For example, the localization and mapping engineof the edge devicecan mark the time using a global positioning system (GPS) timestamp, a Network Time Protocol (NTP) timestamp, a local timestamp based on a local clock running on the edge device, or a combination thereof. The edge devicecan record the timestampsfrom multiple sources to ensure that such timestampsare synchronized with one another in order to maintain the accuracy of such timestamps.

102 102 104 136 136 124 126 As will be discussed in more detail in later sections, if an edge devicedetects that a double parking violation has occurred, the edge devicecan transmit data, information, videos, and other files to the serverin the form of an evidence package. The evidence packagecan comprise the event video framesand the license plate video frames.

136 127 124 102 127 127 In some embodiments, the evidence packagecan also comprise one or more classification resultsobtained by feeding the event video framesinto one or more deep learning models running on the edge device. The classification resultscan be associated with a context surrounding the potential double parking violation. The classification resultswill be discussed in more detail in later sections.

102 122 The deep learning models running on the edge devicecan make predictions or classifications (e.g., multi-class classifications) concerning the context-related features. In some embodiments, such predictions or classifications can be in the form of confidence scores or numerical values. In these and other embodiments, such predictions or classification can ultimately be used to obtain a qualitative or binary classification (e.g., was the potentially offending vehiclestatic or moving).

136 128 129 102 126 130 122 102 110 132 112 134 122 124 The evidence packagecan also comprise at least one license plate numberof a license platerecognized by the edge deviceusing the license plate video framesas inputs, a locationof the potentially offending vehicledetermined by the edge device, the speed of the carrier vehiclewhen the double parking violation was detected, any timestampsrecorded by the control unit, and vehicle attributesof the potentially offending vehiclecaptured by the event video frames.

1 FIG.A 104 138 also illustrates that the servercan transmit certain data and files to a third-party computing device/resource or client device. For example, the third-party computing device can be a server or computing resource of a third-party double parking violation processor. As a more specific example, the third-party computing device can be a server or computing resource of a government vehicle registration department. In other examples, the third-party computing device can be a server or computing resource of a sub-contractor responsible for processing double parking violations for a municipality or other government entity.

138 138 138 The client devicecan refer to a portable or non-portable computing device. For example, the client devicecan refer to a desktop computer or a laptop computer. In other embodiments, the client devicecan refer to a tablet computer or smartphone.

104 332 138 3 FIG. The servercan also generate or render a number of graphical user interfaces (GUIs)(see, e.g.,) that can be displayed through a web portal or mobile app run on the client device.

332 332 The GUIscan also provide data or information concerning times/dates of double parking violations and locations of the double parking violations. The GUIscan also provide a video player configured to play back video evidence of the double parking violation.

332 102 332 332 In another embodiment, at least one of the GUIscan comprise a live map showing real-time locations of all edge devices, double parking violations, and violation hot-spots. In yet another embodiment, at least one of the GUIscan provide a live event feed of all flagged events or double parking violations and the validation status of such double parking violations. The GUIsand the web portal or app will be discussed in more detail in later sections.

104 102 102 The servercan also determine that a double parking violation has occurred based in part on comparing data and videos received from the edge deviceand other edge devices.

1 FIG.B 1 FIG.A 1 FIG.B 100 122 140 illustrates an example scenario where the systemofcan be utilized to detect a double parking violation. As shown in, a potentially offending vehiclecan be double parked or otherwise stopped in a no-parking lane.

140 In some embodiments, the no-parking lanecan be a lane next to or adjacent to (laterally offset from) a permitted parking lane. The permitted parking lane can be a lane or portion of a roadway that is closest to a road edge (e.g., a right road edge in the United States) or curb.

140 In other embodiments, the no-parking lanecan be a bus lane when the bus lane is outside of a bus lane enforcement period, a bike lane, a no-parking or no-stopping zone, or a combination thereof.

110 102 110 122 140 110 122 110 140 110 122 1 FIG.C 1 FIG.A A carrier vehicle(see also,) having an edge device(see also) mounted or installed within the carrier vehiclecan drive by (i.e., next to or in a lane lateral to) or behind the potentially offending vehicleparked or otherwise stopped in the no-parking lane. For example, the carrier vehiclecan be driving in a lane blocked by the potentially offending vehicle. Alternatively, the carrier vehiclecan be driving in a lane adjacent to or at least two lanes lateral to the no-parking lane. The carrier vehiclecan encounter the potentially offending vehiclewhile traversing its daily route (e.g., bus route, garbage collection route, etc.).

102 122 140 701 114 116 701 700 140 12 114 116 114 116 7 7 FIGS.A andC The edge devicecan capture videos of the potentially offending vehicle, at least part of the no-parking lane, and a road edge(see) using the event camera(and, in some instances, the LPR camera). The road edgeis the edge of a roadwaythat encompasses the no-parking lane. In one embodiment, the videos can be in the MPEG-4 Partor MP4 file format. In some embodiments, the videos can refer to multiple videos captured by the event camera, the LPR camera, or a combination thereof. In other embodiments, the videos can refer to one compiled video comprising multiple videos captured by the event camera, the LPR camera, or a combination thereof.

112 102 130 122 118 112 130 122 110 120 The control unitof the edge devicecan then determine a locationof the potentially offending vehicleusing, in part, a positioning data (e.g., GPS data) obtained from the communication and positioning unit. The control unitcan also determine the locationof the potentially offending vehicleusing, in part, inertial measurement data obtained from an IMU and wheel odometry data obtained from a wheel odometer of the carrier vehiclevia the vehicle bus connector.

112 124 126 112 122 140 One or more processors of the control unitcan also be programmed to automatically identify objects from the videos by applying a plurality of functions from a computer vision library to the videos to, among other things, read video frames from the videos and pass at least some of the video frames (e.g., the event video framesand the license plate video frames) to a plurality of deep learning models (e.g., one or more neural networks) running on the control unit. For example, the potentially offending vehicleand the no-parking lanecan be identified as part of this detection step.

112 124 126 112 134 122 134 122 122 122 122 In some embodiments, the one or more processors of the control unitcan also pass at least some of the video frames (e.g., the event video frames, the license plate video frames, or a combination thereof) to one or more deep learning models running on the control unitto identify a set of vehicle attributesof the potentially offending vehicle. The set of vehicle attributescan include a color of the potentially offending vehicle, a make and model of the potentially offending vehicleand a vehicle type of the potentially offending vehicle(e.g., whether the potentially offending vehicleis a personal vehicle or a public service vehicle such as a fire truck, ambulanee, parking enforcement vehicle, police car, etc.).

112 126 116 112 128 129 122 As a more specific example, the control unitcan pass the license plate video framescaptured by the LPR camerato a license plate recognition engine (e.g., a license plate recognition deep learning model) running on the control unitto recognize an alphanumeric string representing a license plate numberof the license plateof the potentially offending vehicle.

112 102 136 126 126 130 122 132 134 128 122 104 The control unitof the edge devicecan also wirelessly transmit an evidence packagecomprising at least some of the event video framesand the license plate video frames, the locationof the potentially offending vehicle, one or more timestamps, the recognized vehicle attributes, and the extracted license plate numberof the potentially offending vehicleto the server.

136 127 124 126 102 127 127 The evidence packagecan also comprise one or more classification resultsobtained by feeding the event video frames(and, in some instances, the license plate video frames) into one or more of the deep learning models running on the edge device. The classification resultscan be associated with a context surrounding the potential double parking violation. The classification resultswill be discussed in more detail in later sections.

102 110 110 112 102 136 122 104 Each edge devicecan be configured to continuously take videos of its surrounding environment (i.e., an environment outside of the carrier vehicle) as the carrier vehicletraverses its usual carrier route. In these embodiments, the one or more processors of the control unitof each edge devicecan periodically transmit evidence packagescomprising video frames from such videos and data/information concerning the potentially offending vehiclescaptured in the videos to the server.

104 127 104 102 102 110 The servercan confirm or validate that a double parking violation has indeed occurred based in part on the classification results. Moreover, the servercan confirm or validate that a double parking violation has indeed occurred based in part on comparing data and videos received from multiple edge devices(where each edge deviceis mounted or otherwise coupled to a different carrier vehicle).

1 FIG.C 110 110 illustrates that, in some embodiments, the carrier vehiclecan be a municipal fleet vehicle. For example, the carrier vehiclecan be a transit vehicle such as a municipal bus, train, or light-rail vehicle, a school bus, a street sweeper, a sanitation vehicle (e.g., a garbage truck or recycling truck), a traffic or parking enforcement vehicle, or a law enforcement vehicle (e.g., a police car or highway patrol car), a tram or light-rail train.

110 110 In other embodiments, the carrier vehiclecan be a semi-autonomous vehicle such as a vehicle operating in one or more self-driving modes with a human operator in the vehicle. In further embodiments, the carrier vehiclecan be an autonomous vehicle or self-driving vehicle.

110 In certain embodiments, the carrier vehiclecan be a private vehicle or vehicle not associated with a municipality or government entity.

102 102 In alternative embodiments, the edge devicecan be carried by or otherwise coupled to a micro-mobility vehicle (e.g., an electric scooter). In other embodiments contemplated by this disclosure, the edge devicecan be carried by or otherwise coupled to an unmanned aerial vehicle (UAV) or drone.

2 FIG.A 1 FIG.C 102 100 102 102 102 102 110 illustrates one embodiment of an edge deviceof the system. The edge devicecan be any of the edge devices disclosed herein. For purposes of this disclosure, any references to the edge devicecan also be interpreted as a reference to a specific component, processor, module, chip, or circuitry within the edge device. The edge devicecan be configured for placement behind a windshield of a carrier vehicle(e.g., a fleet vehicle, see).

2 FIG.A 102 112 114 112 116 112 102 118 120 114 116 112 As shown in, the edge devicecan comprise a control unit, an event cameracommunicatively coupled to the control unit, and one or more license plate recognition (LPR) camera camerascommunicatively coupled to the control unit. The edge devicecan further comprise a communication and positioning unitand a vehicle bus connector. The event cameraand the LPR cameracan be connected or communicatively coupled to the control unitvia high-speed camera interfaces such as a Mobile Industry Processor Interface (MIPI) camera serial interface.

112 114 116 112 112 114 116 The control unitcan comprise a plurality of processors, memory and storage units, and inertial measurement units (IMUs). The event cameraand the LPR cameracan be coupled to the control unitvia high-speed buses, communication cables or wires, and/or other types of wired or wireless interfaces. The components within each of the control unit, the event camera, or the LPR cameracan also be connected to one another via high-speed buses, communication cables or wires, and/or other types of wired or wireless interfaces.

112 The processors of the control unitcan include one or more central processing units (CPUs), graphics processing units (GPUs), Application-Specific Integrated Circuits (ASICs), field-programmable gate arrays (FPGAs), tensor processing units (TPUs), or a combination thereof. The processors can execute software stored in the memory and storage units to execute the methods or instructions described herein.

112 21 For example, the processors can refer to one or more GPUs and CPUs of a processor module configured to perform operations or undertake calculations. As a more specific example, the processors can perform operations or undertake calculations at a terascale. In some embodiments, the processors of the control unitcan be configured to perform operations atteraflops (TFLOPS).

112 114 116 The processors of the control unitcan be configured to run multiple deep learning models or neural networks in parallel and process data received from the event camera, the LPR camera, or a combination thereof. More specifically, the processor module can be a Jetson Xavier NX™ module developed by NVIDIA Corporation. The processors can comprise at least one GPU having a plurality of processing cores (e.g., between 300 and 400 processing cores) and tensor cores, at least one CPU (e.g., at least one 64-bit CPU having multiple processing cores), and a deep learning accelerator (DLA) or other specially designed circuitry optimized for deep learning algorithms (e.g., an NVDLA™ engine developed by NVIDIA Corporation).

102 In some embodiments, at least part of the GPU's processing power can be utilized for object detection and license plate recognition. In these embodiments, at least part of the DLA's processing power can be utilized for object detection and lane line detection. Moreover, at least part of the CPU's processing power can be used for lane line detection and simultaneous localization and mapping. The CPU's processing power can also be used to run other functions and maintain the operation of the edge device.

The memory and storage units can comprise volatile memory and non-volatile memory or storage. For example, the memory and storage units can comprise flash memory or storage such as one or more solid-state drives, dynamic random access memory (DRAM) or synchronous dynamic random access memory (SDRAM) such as low-power double data rate (LPDDR) SDRAM, and embedded multi-media controller (eMMC) storage. For example, the memory and storage units can comprise a 512 gigabyte (GB) SSD, an 8 GB 128-bit LPDDR4x memory, and 16 GB eMMC 5.1 storage device. The memory and storage units can store software, firmware, data (including video and image data), tables, logs, databases, or a combination thereof.

Each of the IMUs can comprise a 3-axis accelerometer and a 3-axis gyroscope. For example, the 3-axis accelerometer can be a 3-axis microelectromechanical system (MEMS) accelerometer and a 3-axis MEMS gyroscope. As a more specific example, the IMUs can be a low-power 6-axis IMU provided by Bosch Sensortec GmbH.

102 102 For purposes of this disclosure, any references to the edge devicecan also be interpreted as a reference to a specific component, processor, module, chip, or circuitry within a component of the edge device.

118 The communication and positioning unitcan comprise at least one of a cellular communication module, a WiFi communication module, a Bluetooth® communication module, and a high-precision automotive-grade positioning unit.

For example, the cellular communication module can support communications over a 5G network or a 4G network (e.g., a 4G long-term evolution (LTE) network) with automatic fallback to 3G networks. The cellular communication module can comprise a number of embedded SIM cards or embedded universal integrated circuit cards (eUICCs) allowing the device operator to change cellular service providers over-the-air without needing to physically change the embedded SIM cards. As a more specific example, the cellular communication module can be a 4G LTE Cat-12 cellular module.

112 110 112 The WiFi communication module can allow the control unitto communicate over a WiFi network such as a WiFi network provided by a carrier vehicle, a municipality, a business, or a combination thereof. The WiFi communication module can allow the control unitto communicate over one or more WiFi (IEEE 802.11) communication protocols such as the 802.11n, 802.11ac, or 802.11ax protocol.

112 The Bluetooth® module can allow the control unitto communicate with other control units on other carrier vehicles over a Bluetooth® communication protocol (e.g., Bluetooth® basic rate/enhanced data rate (BR/EDR), a Bluetooth® low energy (BLE) communication protocol, or a combination thereof). The Bluetooth® module can support a Bluetooth® v4.2 standard or a Bluetooth v5.0 standard. In some embodiments, the wireless communication modules can comprise a combined WiFi and Bluetooth® module.

118 118 118 118 The communication and positioning unitcan comprise a multi-band global navigation satellite system (GNSS) receiver configured to concurrently receive signals from a GPS satellite navigation system, a GLONASS satellite navigation system, a Galileo navigation system, and a BeiDou satellite navigation system. For example, the communication and positioning unitcan comprise a multi-band GNSS receiver configured to concurrently receive signals from at least two satellite navigation systems including the GPS satellite navigation system, the GLONASS satellite navigation system, the Galileo navigation system, and the BeiDou satellite navigation system. In other embodiments, the communication and positioning unitcan be configured to receive signals from all four of the aforementioned satellite navigation systems or three out of the four satellite navigation systems. For example, the communication and positioning unitcan comprise a ZED-F9K dead reckoning module provided by u-blox holding AG.

118 102 118 112 102 130 122 112 130 122 112 130 122 1 FIG.B The communication and positioning unitcan provide positioning data that can allow the edge deviceto determine its own location at a centimeter-level accuracy. The communication and positioning unitcan also provide positioning data that can be used by the control unitof the edge deviceto determine the locationof the potentially offending vehicle(see). For example, the control unitcan use positioning data concerning its own location to substitute for the locationof the potentially offending vehicle. The control unitcan also use positioning data concerning its own location to estimate or approximate the locationof the potentially offending vehicle.

2 FIG.A 102 120 112 120 112 110 102 120 112 122 also illustrates that the edge devicecan comprise a vehicle bus connectorcoupled to the control unit. The vehicle bus connectorcan allow the control unitto obtain wheel odometry data from a wheel odometer of a carrier vehiclecarrying the edge device. For example, the vehicle bus connectorcan be a J1939 connector. The control unitcan take into account the wheel odometry data to determine the location of the potentially offending vehicle.

102 102 102 110 102 The edge devicecan also comprise a power management integrated circuit (PMIC). The PMIC can be used to manage power from a power source. In some embodiments, the components of the edge devicecan be powered by a portable power source such as a battery. In other embodiments, one or more components of the edge devicecan be powered via a physical connection (e.g., a power cord) to a power outlet or direct-current (DC) auxiliary power outlet (e.g., 12V/24V) of a carrier vehiclecarrying the edge device.

114 200 202 204 202 206 202 The event cameracan comprise an event camera image sensorcontained within an event camera housing, an event camera mountcoupled to the event camera housing, and an event camera skirtcoupled to and protruding outwardly from a front face or front side of the event camera housing.

202 204 202 204 202 204 114 110 204 202 114 110 114 110 110 204 202 The event camera housingcan be made of a metallic material (e.g., aluminum), a polymeric material, or a combination thereof. The event camera mountcan be coupled to the lateral sides of the event camera housing. The event camera mountcan comprise a mount rack or mount plate positioned vertically above the event camera housing. The mount rack or mount plate of the event camera mountcan allow the event camerato be mounted or otherwise coupled to a ceiling and/or headliner of the carrier vehicle. The event camera mountcan allow the event camera housingto be mounted in such a way that a camera lens of the event camerafaces the windshield of the carrier vehicleor is positioned substantially parallel with the windshield. This can allow the event camerato take videos of an environment outside of the carrier vehicleincluding vehicles parked near the carrier vehicle. The event camera mountcan also allow an installer to adjust a pitch/tilt and/or swivel/yaw of the event camera housingto account for a tilt or curvature of the windshield.

206 110 200 110 206 200 200 206 206 202 206 114 206 110 The event camera skirtcan block or reduce light emanating from an interior of the carrier vehicleto prevent such light from interfering with the videos captured by the event camera image sensor. For example, when the carrier vehicleis a municipal bus, the interior of the municipal bus is often lit by artificial lights (e.g., fluorescent lights, LED lights, etc.) to ensure passenger safety. The event camera skirtcan block or reduce the amount of artificial light that reaches the event camera image sensorto prevent this light from degrading the videos captured by the event camera image sensor. The event camera skirtcan be designed to have a tapered or narrowed end and a wide flared end. The tapered end of the event camera skirtcan be coupled to a front portion or front face/side of the event camera housing. The event camera skirtcan also comprise a skirt distal edge defining the wide flared end. In some embodiments, the event cameracan be mounted or otherwise coupled in such a way that the skirt distal edge of the event camera skirtis separated from the windshield of the carrier vehicleby a separation distance. In some embodiments, the separation distance can be between about 1.0 cm and 10.0 cm.

206 206 206 In some embodiments, the event camera skirtcan be made of a dark-colored non-transparent polymeric material. In certain embodiments, the event camera skirtcan be made of a non-reflective material. As a more specific example, the event camera skirtcan be made of a dark-colored thermoplastic elastomer such as thermoplastic polyurethane (TPU).

200 200 200 200 The event camera image sensorcan be configured to capture video at a frame rate of between 15 frame per second and up to 60 frames per second (FPS). For example, the event camera image sensorcan be a high-dynamic range (HDR) image sensor. The event camera image sensorcan capture video images at a minimum resolution of 1920×1080 (or 2 megapixels). As a more specific example, the event camera image sensorcan comprise one or more CMOS image sensors provided by OMNIVISION Technologies, Inc.

114 110 110 110 112 124 124 112 140 124 As previously discussed, the event cameracan capture videos of an environment outside of the carrier vehicle, including any vehicles parked near the carrier vehicle, as the carrier vehicletraverses its usual carrier route. The control unitcan be programmed to apply a plurality of functions from a computer vision library to the videos to read event video framesfrom the videos and pass the event video framesto a plurality of deep learning models (e.g., neural networks) running on the control unitto automatically identify objects (e.g., cars, trucks, buses, etc.) and roadways (e.g., a roadway encompassing the no-parking lane) from the event video framesin order to determine whether a double parking violation has occurred.

2 FIG.A 102 116 116 208 210 212 210 214 210 As shown in, the edge devicecan also comprise an LPR camera. The LPR cameracan comprise at least two LPR image sensorscontained within an LPR camera housing, an LPR camera mount, coupled to the LPR camera housing, and an LPR camera skirtcoupled to and protruding outwardly from a front face or front side of the LPR camera housing.

210 212 210 212 210 212 116 110 212 210 The LPR camera housingcan be made of a metallic material (e.g., aluminum), a polymeric material, or a combination thereof. The LPR camera mountcan be coupled to the lateral sides of the LPR camera housing. The LPR camera mountcan comprise a mount rack or mount plate positioned vertically above the LPR camera housing. The mount rack or mount plate of the LPR camera mountcan allow the LPR camerato be mounted or otherwise coupled to a ceiling and/or headliner of the carrier vehicle. The LPR camera mountcan also allow an installer to adjust a pitch/tilt and/or swivel/yaw of the LPR camera housingto account for a tilt or curvature of the windshield.

212 210 116 110 116 110 The LPR camera mountcan allow the LPR camera housingto be mounted in such a way that the LPR camerafaces the windshield of the carrier vehicleat an angle. This can allow the LPR camerato capture videos of license plates of vehicles directly in front of or on one side (e.g., a right side or left side) of the carrier vehicle.

116 216 218 216 216 The LPR cameracan comprise a daytime image sensorand a nighttime image sensor. The daytime image sensorcan be configured to capture images or videos in the daytime or when sunlight is present. Moreover, the daytime image sensorcan be an image sensor configured to capture images or videos in the visible spectrum.

218 The nighttime image sensorcan be an infrared (IR) or near-infrared (NIR) image sensor configured to capture images or videos in low-light conditions or at nighttime.

216 216 In certain embodiments, the daytime image sensorcan comprise a CMOS image sensor manufactured or distributed by OmniVision Technologies, Inc. For example, the daytime image sensorcan be the OmniVision OV2311 CMOS image sensor configured to capture videos between 15 FPS and 60 FPS.

218 The nighttime image sensorcan comprise an IR or NIR image sensor manufactured or distributed by OmniVision Technologies, Inc.

116 116 In other embodiments not shown in the figures, the LPR cameracan comprise one image sensor with both daytime and nighttime capture capabilities. For example, the LPR cameracan comprise one RGB-IR image sensor.

220 220 2 FIG.A The LPR camera can also comprise a plurality of IR or NIR light-emitting diodes (LEDs)configured to emit IR or NIR light to illuminate an event scene in low-light or nighttime conditions. In some embodiments, the IR/NIR LEDscan be arranged as an IR/NIR light array (see).

220 110 220 218 220 218 The IR LEDscan emit light in the infrared or near-infrared (NIR) range (e.g., about 800 nm to about 1400 nm) and act as an IR or NIR spotlight to illuminate a nighttime environment or low-light environment immediately outside of the carrier vehicle. In some embodiments, the IR LEDscan be arranged as a circle or in a pattern surrounding or partially surrounding the nighttime image sensor. In other embodiments, the IR LEDscan be arranged in a rectangular pattern, an oval pattern, and/or a triangular pattern around the nighttime image sensor.

116 218 220 220 218 220 218 In additional embodiments, the LPR cameracan comprise a nighttime image sensor(e.g., an IR or NIR image sensor) positioned in between two IR LEDs. In these embodiments, one IR LEDcan be positioned on one lateral side of the nighttime image sensorand the other IR LEDcan be positioned on the other lateral side of the nighttime image sensor.

116 220 116 In certain embodiments, the LPR cameracan comprise between 3 and 12 IR LEDS. In other embodiments, the LPR cameracan comprise between 12 and 20 IR LEDs.

220 220 In some embodiments, the IR LEDscan be covered by an IR bandpass filter. The IR bandpass filter can allow only radiation in the IR range or NIR range (between about 780 nm to about 1500 nm) to pass while blocking light in the visible spectrum (between about 380 nm to about 700 nm). In some embodiments, the IR bandpass filter can be an optical-grade polymer-based filter or a piece of high-quality polished glass. For example, the IR bandpass filter can be made of an acrylic material (optical-grade acrylic) such as an infrared transmitting acrylic sheet. As a more specific example, the IR bandpass filter can be a piece of poly(methyl methacrylate) (PMMA) (e.g., Plexiglass™) that covers the IR LEDs.

214 214 214 214 In some embodiments, the LPR camera skirtcan be made of a dark-colored non-transparent polymeric material. In certain embodiments, the LPR camera skirtcan be made of a polymeric material. For example, the LPR camera skirtcan be made of a non-reflective material. As a more specific example, the LPR camera skirtcan be made of a dark-colored thermoplastic elastomer such as thermoplastic polyurethane (TPU).

2 FIG.A 116 214 116 110 218 Althoughillustrates an embodiment of the LPR camerawith only one LPR camera skirt, it is contemplated by this disclosure that the LPR cameracan comprise an outer LPR camera skirt and an inner LPR camera skirt. The inner LPR camera skirt can block IR light reflected by the windshield of the carrier vehiclethat can interfere with the videos captured by the nighttime image sensor.

214 214 116 110 214 110 116 The LPR camera skirtcan comprise a first skirt lateral side, a second skirt lateral side, a skirt upper side, and a skirt lower side. The first skirt lateral side can have a first skirt lateral side length. The second skirt lateral side can have a second skirt lateral side length. In some embodiments, the first skirt lateral side length can be greater than the second skirt lateral side length such that the first skirt lateral side protrudes out further than the second skirt lateral side. In these and other embodiments, any of the first skirt lateral side length or the second skirt lateral side length can vary along a width of the first skirt lateral side or along a width of the second skirt lateral side, respectively. However, in all such embodiments, a maximum length or height of the first skirt lateral side is greater than a maximum length or height of the second skirt lateral side. In further embodiments, a minimum length or height of the first skirt lateral side is greater than a minimum length or height of the second skirt lateral side. The skirt upper side can have a skirt upper side length or a skirt upper side height. The skirt lower side can have a skirt lower side length or a skirt lower side height. In some embodiments, the skirt lower side length or skirt lower side height can be greater than the skirt upper side length or the skirt upper side height such that the skirt lower side protrudes out further than the skirt upper side. The unique design of the LPR camera skirtcan allow the LPR camerato be positioned at an angle with respect to a windshield of the carrier vehiclebut still allow the LPR camera skirtto block light emanating from an interior of the carrier vehicleor block light from interfering with the image sensors of the LPR camera.

116 110 110 112 126 126 112 128 126 112 126 112 112 The LPR cameracan capture videos of license plates of vehicles parked near the carrier vehicleas the carrier vehicletraverses its usual carrier route. The control unitcan be programmed to apply a plurality of functions from a computer vision library to the videos to read license plate video framesfrom the videos and pass the license plate video framesto a license plate recognition deep learning model running on the control unitto automatically extract license plate numbersfrom such license plate video frames. For example, the control unitcan pass the license plate video framesto the license plate recognition deep learning model running on the control unitto extract license plate numbers of all vehicles detected by an object detection deep learning model running on the control unit.

112 124 102 102 3 FIG. The control unitcan also pass the event video framesto a plurality of deep learning models running on the edge device(see). The deep learning models can be neural networks running on the edge devicethat determine a context surrounding the double parking violation.

112 112 136 124 126 127 104 112 128 129 122 136 1 FIG.A If the control unitdetermines that a double parking violation has occurred, the control unitcan generate an evidence packagecomprising at least some of the event video frames, the license plate video frames, the classification results, and data/information concerning the double parking violation for transmission to the server(see). The control unitcan include the automatically recognized license plate numberof the license plateof the potentially offending vehicleinvolved in the double parking violation in the evidence package.

3 FIG. 104 136 104 124 126 104 104 102 As will be discussed in more detail with respect to, once the serverhas received the evidence package, the one or more processors of the servercan be programmed to pass the event video framesand the license plate video framesto a plurality of deep learning models running on the server. The deep learning models (e.g., neural networks) running on the servercan then output its own predictions or classifications to validate the evidence received from the edge device(s).

124 104 104 136 102 104 136 104 For example, such predictions or classifications can concern context-related features automatically extracted from the event video framesby the deep learning models running on the server. The servercan then use these classification results to automatically validate or reject the evidence packagereceived from the edge device. Moreover, the servercan also use these classification results to determine whether the evidence packageshould be recommended for further review by a human reviewer or another round of automated review by the serveror another computing device.

2 FIG.B 104 100 104 104 104 illustrates one embodiment of the serverof the system. As previously discussed, the servercan comprise or refer to one or more virtual servers or virtualized computing resources. For example, the servercan refer to a virtual server or cloud server hosted and delivered by a cloud computing platform (e.g., Amazon Web Services®, Microsoft Azure®, or Google Cloud®). In other embodiments, the servercan refer to one or more physical servers or dedicated computing resources or nodes such as a rack-mounted server, a blade server, a mainframe, a dedicated desktop or laptop computer, one or more processors or processors cores therein, or a combination thereof.

104 104 For purposes of the present disclosure, any references to the servercan also be interpreted as a reference to a specific component, processor, module, chip, or circuitry within the server.

104 222 224 226 222 224 226 For example, the servercan comprise one or more server processors, server memory and storage units, and a server communication interface. The server processorscan be coupled to the server memory and storage unitsand the server communication interfacethrough high-speed buses or interfaces.

222 222 224 222 222 The one or more server processorscan comprise one or more CPUs, GPUs, ASICS, FPGAs, TPUs, or a combination thereof. The one or more server processorscan execute software stored in the server memory and storage unitsto execute the methods or instructions described herein. The one or more server processorscan be embedded processors, processor cores, microprocessors, logic circuits, hardware FSMs, DSPs, or a combination thereof. As a more specific example, at least one of the server processorscan be a 64-bit processor.

224 224 224 224 The server memory and storage unitscan store software, data (including video or image data), tables, logs, databases, or a combination thereof. The server memory and storage unitscan comprise an internal memory and/or an external memory, such as a memory residing on a storage node or a storage server. The server memory and storage unitscan be a volatile memory or a non-volatile memory. For example, the server memory and storage unitscan comprise nonvolatile storage such as NVRAM, Flash memory, solid-state drives, hard disk drives, and volatile storage such as SRAM, DRAM, or SDRAM.

226 226 226 104 102 226 104 226 The server communication interfacecan refer to one or more wired and/or wireless communication interfaces or modules. For example, the server communication interfacecan be a network interface card. The server communication interfacecan comprise or refer to at least one of a WiFi communication module, a cellular communication module (e.g., a 4G or 5G cellular communication module), and a Bluetooth®/BLE or other type of short-range communication module. The servercan connect to or communicatively couple with each of the edge devicesvia the server communication interface. The servercan transmit or receive packets of data using the server communication interface.

2 FIG.C 102 102 114 116 102 118 102 illustrates an alternative embodiment of the edge devicewhere the edge deviceis a personal communication device such as a smartphone or tablet computer. In this embodiment, the event cameraand the LPR cameraof the edge devicecan be the built-in cameras or image sensors of the smartphone or tablet computer. Moreover, references to the one or more processors, the memory and storage units, the communication and positioning unit, and the IMUs of the edge devicecan refer to the same or similar components within the smartphone or tablet computer.

102 104 108 110 110 Also, in this embodiment, the smartphone or tablet computer serving as the edge devicecan also wirelessly communicate or be communicatively coupled to the servervia the secure connection. The smartphone or tablet computer can also be positioned near a windshield or window of a carrier vehiclevia a phone or tablet holder coupled to the ceiling/headliner, windshield, window, console, and/or dashboard of the carrier vehicle.

3 FIG. 102 104 102 300 302 304 313 illustrates certain modules and engines of one embodiment of an edge deviceand the server. In some embodiments, the edge devicecan comprise at least an event detection engine, a localization and mapping engine, and a license plate recognition engine, and a vehicle movement classifier.

102 Software instructions run on the edge device, including any of the engines and modules disclosed herein, can be written in the Java® programming language, C++ programming language, the Python® programming language, the Golang™ programming language, or a combination thereof.

102 102 114 116 102 2 FIG.A 2 FIG.A As previously discussed, the edge devicecan continuously capture videos of an external environment surrounding the edge device. For example, the event camera(see) and the LPR camera(see) of the edge devicecan capture everything that is within a field of view of the cameras.

114 124 116 126 In some embodiments, the event cameracan capture videos comprising a plurality of event video framesand the LPR cameracan capture videos comprising a plurality of license plate video frames.

114 126 116 124 In alternative embodiments, the event cameracan also capture videos of license plates that can be used as license plate video frames. Moreover, the LPR cameracan capture videos of a double parking violation event that can be used as event video frames.

102 124 126 102 114 116 114 116 300 The edge devicecan retrieve or grab the event video frames, the license plate video frames, or a combination thereof from a shared camera memory. The shared camera memory can be an onboard memory (e.g., non-volatile memory) of the edge devicefor storing video frames captured by the event camera, the LPR camera, or a combination thereof. Since the event cameraand the LPR cameraare capturing videos at approximately 15 to 60 video frames per second (fps), the video frames are stored in the shared camera memory prior to being analyzed by the event detection engine. In some embodiments, the video frames can be grabbed using a video frame grab function such as the GStreamer tool.

300 306 300 102 The event detection enginecan call a plurality of functions from a computer vision libraryto enhance one or more video frames by resizing, cropping, or rotating the one or more video frames. For example, the event detection enginecan crop and resize the one or more video frames to optimize the one or more video frames for analysis by one or more deep learning models or neural networks running on the edge device.

300 102 300 102 For example, the event detection enginecan crop and resize at least one of the video frames to produce a cropped and resized video frame that meets certain size parameters associated with the deep learning models running on the edge device. Also, for example, the event detection enginecan crop and resize the one or more video frames such that the aspect ratio of the one or more video frames meets parameters associated with the deep learning models running on the edge device.

306 306 In some embodiments, the computer vision librarycan be the OpenCV® library maintained and operated by the Open Source Vision Foundation. In other embodiments, the computer vision librarycan be or comprise functions from the TensorFlow® software library, the SimpleCV® library, or a combination thereof.

300 124 308 102 124 308 300 308 124 300 The event detection enginecan pass or feed at least some of the event video framesto an object detection deep learning model(e.g., a neural network trained for object detection) running on the edge device. By passing and feeding the event video framesto the object detection deep learning model, the event detection enginecan obtain as outputs from the object detection deep learning modelpredictions, scores, or probabilities concerning the objects detected from the event video frames. For example, the event detection enginecan obtain as outputs a confidence score for each of the object classes detected.

308 308 308 308 308 In some embodiments, the object detection deep learning modelcan be configured or trained such that only certain vehicle-related objects are supported by the object detection deep learning model. For example, the object detection deep learning modelcan be configured or trained such that the object classes supported only include cars, trucks, buses, etc. Also, for example, the object detection deep learning modelcan be configured or trained such that the object classes supported also include bicycles, scooters, and other types of wheeled mobility vehicles. In some embodiments, the object detection deep learning modelcan be configured or trained such that the object classes supported also comprise non-vehicle classes such as pedestrians, landmarks, street signs, fire hydrants, bus stops, and building façades.

308 308 102 In some embodiments, the object detection deep learning modelcan be configured to detect more than 100 (e.g., between 100 and 200) objects per video frame. Although the object detection deep learning modelcan be configured to accommodate numerous object classes, one advantage of limiting the number of object classes is to reduce the computational load on the processors of the edge device, shorten the training time of the neural network, and make the neural network more efficient.

308 308 308 The object detection deep learning modelcan comprise a plurality of convolutional layers and connected layers trained for object detection (and, in particular, vehicle detection). In one embodiment, the object detection deep learning modelcan be a convolutional neural network trained for object detection. For example, the object detection deep learning modelcan be a variation of the Single Shot Detection (SSD) model with a MobileNet backbone as the feature extractor.

308 In other embodiments, the object detection deep learning modelcan be the You Only Look Once Lite (YOLO Lite) object detection model.

308 308 134 134 300 In some embodiments, the object detection deep learning modelcan also identify or predict certain attributes of the detected objects. For example, the object detection deep learning modelcan identify or predict a set of attributes of an object identified as a vehicle (also referred to as vehicle attributes) such as the color of the vehicle, the make and model of the vehicle, and the vehicle type (e.g., whether the vehicle is a personal vehicle or a public service vehicle). The vehicle attributescan be used by the event detection engineto make an initial determination as to whether the vehicle shown in the video frames is subject to a municipality's double parking violation rules or policies.

308 102 102 110 308 102 308 The object detection deep learning modelcan be trained, at least in part, from video frames of videos captured by the edge deviceor other edge devicesdeployed in the same municipality or coupled to other carrier vehiclesin the same carrier fleet. The object detection deep learning modelcan be trained, at least in part, from video frames of videos captured by the edge deviceor other edge devices at an earlier point in time. Moreover, the object detection deep learning modelcan be trained, at least in part, from video frames from one or more open-sourced training sets or datasets.

3 FIG. 5 FIG.C 102 304 304 128 122 304 126 116 102 310 102 310 128 122 129 304 124 310 128 122 124 As shown in, the edge devicecan also comprise a license plate recognition engine. The license plate recognition enginecan be configured to recognize license plate numbersof potentially offending vehicles(see, also,) in the video frames. For example, the license plate recognition enginecan pass license plate video framescaptured by the dedicated LPR cameraof the edge deviceto a license plate recognition (LPR) deep learning modelrunning on the edge device. The LPR deep learning modelcan be specifically trained to recognize license plate numbersof vehicles (e.g., the potentially offending vehicle) from video frames or images containing license platesof such vehicles. Alternatively, or additionally, the license plate recognition enginecan also pass event video framesto the LPR deep learning modelto recognize license plate numbersof vehicles (e.g., the potentially offending vehicle) from such event video frames.

310 310 In some embodiments, the LPR deep learning modelcan be a neural network trained for license plate recognition. In certain embodiments, the LPR deep learning modelcan be a modified version of the OpenALPR™ license plate recognition model.

304 In other embodiments, the license plate recognition enginecan also undertake automated license plate recognition using a text-adapted vision transformer.

310 102 304 128 129 By feeding video frames or images into the LPR deep learning modelor the text-adapted vision transformer, the edge devicecan obtain as an output from the license plate recognition engine, a prediction in the form of an alphanumeric string representing the license plate numberof the license plate.

304 310 102 310 126 In some embodiments, the license plate recognition engineor the LPR deep learning modelrunning on the edge devicecan generate or output a confidence score associated with a prediction confidence representing the confidence or certainty of its own recognition result (i.e., indicative of or represent the confidence or certainty in the license plate recognized by the LPR deep learning modelfrom the license plate video frames).

512 136 104 136 128 310 5 FIG.C The plate recognition confidence score (see, e.g., confidence scorein) can be a number between 0 and 1.00. As previously discussed, the plate recognition confidence score can be included as part of an evidence packagetransmitted to the server. The evidence packagecan comprise the plate recognition confidence score along with the license plate numberpredicted by the LPR deep learning model.

102 302 303 302 130 122 102 118 102 102 110 102 302 102 130 122 2 FIG.A As previously discussed, the edge devicecan also comprise a localization and mapping enginecomprising a map layer. The localization and mapping enginecan calculate or otherwise estimate the locationof the potentially offending vehiclebased in part on the present location of the edge deviceobtained from at least one of the communication and positioning unit(see, e.g.,) of the edge device, inertial measurement data obtained from the IMUs of the edge device, and wheel odometry data obtained from the wheel odometer of the carrier vehiclecarrying the edge device. For example, the localization and mapping enginecan use the present location of the edge deviceto represent the locationof the potentially offending vehicle.

302 130 122 122 102 302 122 102 122 302 130 122 122 In other embodiments, the localization and mapping enginecan estimate the locationof the potentially offending vehicleby calculating a distance separating the potentially offending vehiclefrom the edge deviceand adding such a separation distance to its own present location. As a more specific example, the localization and mapping enginecan calculate the distance separating the potentially offending vehiclefrom the edge deviceusing video frames containing the license plate of the potentially offending vehicleand a computer vision algorithm (e.g., an image depth analysis algorithm) designed for distance calculation. In additional embodiments, the localization and mapping enginecan determine the locationof the potentially offending vehicleby recognizing an object or landmark (e.g., a bus stop sign) with a known geolocation associated with the object or landmark near the potentially offending vehicle.

303 102 303 104 104 303 303 303 The map layercan comprise one or more semantic maps or semantic annotated maps. The edge devicecan receive updates to the map layerfrom the serveror receive new semantic maps or semantic annotated maps from the server. The map layercan also comprise data and information concerning the widths of all lanes of roadways in a municipality. For example, the known or predetermined width of each of the lanes can be encoded or embedded in the map layer. The known or predetermined width of each of the lanes can be obtained by performing surveys or measurements of such lanes in the field or obtained from one or more publicly-available map databases or municipal/governmental databases. Such lane width data can then be associated with the relevant streets/roadways, areas/regions, or coordinates in the map layer.

303 303 303 The map layercan further comprise data or information concerning a total number of lanes of certain municipal roadways and the direction-of-travel of such lanes. Such data or information can also be obtained by performing surveys or measurements of such lanes in the field or obtained from one or more publicly-available map databases or municipal/governmental databases. Such data or information can be encoded or embedded in the map layerand then associated with the relevant streets/roadways, areas/regions, or coordinates in the map layer.

102 132 122 130 302 102 102 132 132 132 The edge devicecan also record or generate at least a plurality of timestampsmarking the time when the potentially offending vehiclewas detected at the location. For example, the localization and mapping enginecan mark the time using a global positioning system (GPS) timestamp, a Network Time Protocol (NTP) timestamp, a local timestamp based on a local clock running on the edge device, or a combination thereof. The edge devicecan record the timestampsfrom multiple sources to ensure that such timestampsare synchronized with one another in order to maintain the accuracy of such timestamps.

300 124 312 102 312 706 700 124 706 701 706 705 705 703 701 7 7 FIGS.A-C 7 FIG.B 7 7 FIGS.A andC In some embodiments, the event detection enginecan also pass the event video framesto a lane segmentation deep learning modelrunning on the edge device. As will be discussed in more detail in relation to, one prediction head of the lane segmentation deep learning modelcan generate or output a maskor heatmap (see. e.g.,) representing an edge or boundary of a roadwayshown in the event video frames. The maskor heatmap can be considered an initial rough approximation of a road edge(see). The pixels of the maskor heatmap can be reduced through various culling procedures until only a few road edge pointsremain. These road edge pointscan be further culled using an iterative non-deterministic outlier detection algorithm to yield a linerepresenting the road edge.

In some embodiments, the iterative non-deterministic outlier detection algorithm can be a random sample consensus (RANSAC) algorithm that randomly selects a subset of points to be inliers, attempts to fit a line (e.g., a linear regression model) to the subset of points, discards outlier points, and repeats the process until additional outlier points are identified and discarded.

300 700 140 703 701 707 300 707 303 7 7 FIGS.A andC The event detection enginecan then determine a layout of one or more lanes of a roadway, including the no-parking lane, using the fitted lineto represent the road edge(see). The layout of the lanescan be determined by the event detection engineusing a known or predetermined width of each of the lanesencoded or embedded in the map layer.

300 140 703 701 707 303 140 707 700 For example, the event detection enginecan determine the location of a no-parking lanebased on the fitted linerepresenting the road edge, the known or predetermined width of the lanes(obtained from the map layer), and the location or position of the no-parking lanerelative to the other lanesof the roadway.

140 312 140 708 312 708 7 7 FIGS.A andC Once the location of the no-parking laneis determined, the lane segmentation deep learning modelcan then bound the no-parking laneusing a lane-of-interest (LOI) polygon(see). The lane segmentation deep learning modelcan also output image coordinates associated with the LOI polygon.

312 102 312 In some embodiments, the lane segmentation deep learning modelrunning on the edge devicecan be a neural network or convolutional neural network trained for lane detection and segmentation. For example, the lane segmentation deep learning modelcan be a multi-headed convolutional neural network comprising a residual neural network (e.g., a ResNet such as a ResNet34) backbone with a standard mask prediction.

312 312 124 102 In certain embodiments, the lane segmentation deep learning modelcan be trained using a dataset designed specifically for lane detection and segmentation. In other embodiments, the lane segmentation deep learning modelcan also be trained using event video framesobtained from other deployed edge devices.

308 122 124 500 500 308 500 5 5 7 FIGS.A,D, andA As will be discussed in more detail in the following sections, the object detection deep learning modelcan bound a potentially offending vehicledetected within an event video framewith a vehicle bounding polygon(see). In some embodiments, the vehicle bounding polygoncan be a vehicle bounding box. The object detection deep learning modelcan also output image coordinates associated with the vehicle bounding polygon.

500 516 312 500 708 300 500 708 7 FIG.A The image coordinates associated with the vehicle bounding polygoncan be compared with the image coordinates associated with the lane bounding polygonsoutputted by the lane segmentation deep learning model. The image coordinates associated with the vehicle bounding polygoncan be compared with the image coordinates associated with the LOI polygon(see). The event detection enginecan detect a potential double parking violation based in part on an amount of overlap of at least part of the vehicle bounding polygonand the LOI polygon.

102 102 104 136 136 124 126 127 127 102 136 310 310 136 308 308 If the edge devicedetects that a potential double parking violation has occurred, the edge devicecan transmit data, videos, and other files to the serverin the form of an evidence package. As previously discussed, the evidence packagecan comprise the event video frames, the license plate video frames, and one or more classification resultsrelated to a context surrounding the double parking violation. The one or more classification resultscan be confidence scores or probabilities outputted by the one or more deep learning models running on the edge device. The evidence packagecan also comprise a confidence score outputted by the LPR deep learning modelconcerning a license plate automatically recognized by the LPR deep learning model. The evidence packagecan also comprise confidence scores outputted by the object detection deep learning modelconcerning vehicles detected by the object detection deep learning model.

136 128 102 126 130 122 102 110 132 112 134 122 124 The evidence packagecan also comprise at least one license plate numberrecognized by the edge deviceusing the license plate video framesas inputs, a locationof the potentially offending vehicleestimated or otherwise calculated by the edge device, the speed of the carrier vehiclewhen the double parking violation was detected, any timestampsrecorded by the control unit, and vehicle attributesof the potentially offending vehiclecaptured by the event video frames.

3 FIG. 102 313 313 122 114 140 As shown in, the edge devicecan also comprise a vehicle movement classifier. The vehicle movement classifiercan be configured to determine whether the potentially offending vehicleis static or moving when captured by the event camera. It is important to differentiate between a vehicle that is moving and a vehicle that is static because a moving vehicle detected within a no-parking lanecannot be assessed a double parking violation.

102 122 313 124 122 124 500 9 14 FIGS.- In some embodiments, the edge devicecan determine that a double parking violation has occurred only if the potentially offending vehicleis determined to be static. As will be discussed in more detail in relation to, the vehicle movement classifiercan receive as inputs a number of event video frameswhere the potentially offending vehicleappearing in such event video frameshas been bounded by a vehicle bounding polygon.

313 500 124 500 The vehicle movement classifier(or vehicle movement classification module) can track the vehicle bounding polygonsacross multiple event video frames. The vehicle bounding polygonscan be connected across multiple frames using a tracking algorithm such as a mixed integer linear programming (MILP) algorithm.

313 118 102 124 313 500 500 124 The vehicle movement classifiercan first associate GPS coordinates (obtained from the positioning unitof the edge device) with timestamps of each of the event video frames. The vehicle movement classifiercan then determine the GPS coordinates of the vehicle bounding polygonsusing a homography localization algorithm. The GPS coordinates of the vehicle bounding polygonsacross multiple event video framescan then be transformed or converted into a local Cartesian coordinate system.

110 102 The longitudinal axis of the local Cartesian coordinate system can be in a direction of travel of the carrier vehiclecarrying the edge device. The latitudinal axis of the local Cartesian coordinate system can be in a lateral direction perpendicular to the longitudinal axis.

9 14 FIGS.- 313 122 As will be discuss in more detail in relation to, the vehicle movement classifiercan determine that the potentially offending vehicleis static or moving based on a standard deviation of the transformed GPS coordinates in both the longitudinal direction and the latitudinal direction and a cross-correlation of the transformed GPS coordinates along the longitudinal direction and the latitudinal direction.

15 FIG. 308 312 102 122 140 313 122 122 122 140 140 As will be discussed in more detail in relation to, the object detection deep learning modeland the lane segmentation deep learning modelrunning on the edge devicecan also be used to output confidence scores or classification results concerning a context surrounding the potential double parking violation that can assist in determining whether the potentially offending vehicleis actually parked in the no-parking laneor is only temporarily stopped. As previously discussed, the vehicle movement classifiercan determine whether the potentially offending vehicleis static or moving. However, the potentially offending vehiclecan be static without being actually parked. For example, these include instances where the potentially offending vehicleis stuck in traffic due to other vehicles in the no-parking laneor is stopped temporarily in the no-parking laneat an intersection (e.g., is about to make a turn at the intersection).

308 312 1500 1500 1502 122 1504 122 1506 15 FIG. The object detection deep learning model, the lane segmentation deep learning model, or a combination thereof can be configured to output predictions or classification results concerning featuresassociated with the context surrounding the potential double parking violation. Such featurescan comprise a brake light statusof the potentially offending vehicle, a traffic conditionsurrounding the potentially offending vehicle, and a roadway intersection status(see).

308 1502 122 1504 122 312 1506 The object detection deep learning modelcan be trained to classify or make predictions concerning the brake light statusof the potentially offending vehicleand the traffic conditionsurrounding the potentially offending vehicle. In these and other embodiments, the lane segmentation deep learning modelcan be trained to classify or make predictions concerning the roadway intersection status.

308 1508 1508 1508 308 122 122 140 122 122 140 1512 1512 127 136 104 In certain embodiments, the objection detection deep learning modelcan comprise a plurality of prediction headsor detectors. The prediction headsor detectors can be multi-class detectors that can be configured to undertake a multi-class prediction. One of the prediction headsof the object detection deep learning modelcan be trained to distinguish between brake lights that are on, brake lights that are off, and brake lights that are flashing. For example, if the brake lights of a potentially offending vehicleare off, it is more likely that the potentially offending vehiclestopped in the no-parking laneis parked rather than temporarily stopped. On other hand, if the brake lights of the potentially offending vehicleare on or are flashing, it is more likely that the potentially offending vehiclestopped in the no-parking laneis only temporarily stopped. The prediction head can also generate a set of brake light confidence scoresassociated with the predictions or classifications. The brake light confidence scorescan be included as part of a set of classification resultsincluded in the evidence packagetransmitted to the serveror to a third-party evidence processor.

15 FIG. 1508 308 122 122 140 140 122 Also, as shown in, another one of the prediction headsof the object detection deep learning modelcan be trained to detect a traffic condition surrounding the potentially offending vehicleby determining whether one or more other vehicles are immediately in front of the potentially offending vehiclein the no-parking laneor whether the no-parking laneis otherwise clear with no other vehicles impeding the movement of the potentially offending vehicle.

122 122 122 140 122 For example, if one or more other vehicles are immediately in front of the potentially offending vehicle, the presence of such vehicles can indicate that the movement of the potentially offending vehicleis impeded or blocked (possibly due to a traffic jam, an incident, or a traffic light being red, etc.). On other hand, if no vehicles are detected immediately in front of the potentially offending vehicleand the no-parking laneis otherwise clear, this can indicate that the potentially offending vehicleis double-parked.

1514 1514 127 136 104 The prediction head can also generate a set of traffic condition confidence scoresassociated with the predictions or classifications. The traffic condition confidence scorescan be included as part of a set of classification resultsincluded in the evidence packagetransmitted to the serveror to a third-party evidence processor.

312 124 122 140 122 140 6 FIG. 15 FIG. One of the prediction heads of the lane segmentation deep learning model(see, e.g.,and) can be trained to detect whether an intersection is present in at least one of the event video framescapturing the potential double-parking violation. For example, if an intersection is detected, it is more likely that the potentially offending vehiclestopped in the no-parking lanein order to make a turn at the intersection. On other hand, if no intersection is detected, it is more likely that the potentially offending vehiclestopped in the no-parking laneis double parked.

1516 1516 127 136 104 The prediction head can also generate a set of intersection detection confidence scoresassociated with the predictions or classifications. The intersection detection confidence scorescan be included as part of a set of classification resultsincluded in the evidence packagetransmitted to the serveror to a third-party evidence processor.

1512 1514 1516 122 140 1512 1514 1516 136 In some embodiments, the brake light confidence scores, the traffic condition confidence scores, the intersection detection confidence scores, or a combination thereof can be used directly to determine whether the potentially offending vehiclestopped in the no-parking lane(i.e., the static vehicle) is only temporarily stopped or is actually double-parked. For example, one or more thresholds can be set and if at least one of the brake light confidence scores, the traffic condition confidence scores, or the intersection detection confidence scoresdo not meet one of the threshold or exceeds one of the thresholds, the evidence packagecan be rejected.

1512 1514 1516 104 104 136 1512 1514 1516 In other embodiments, the brake light confidence scores, the traffic condition confidence scores, the intersection detection confidence scores, or a combination thereof can also be provided as inputs to a decision tree algorithm running on the serveras part of an evidence evaluation procedure. The servercan automatically approve or reject the evidence packagebased on a final score calculated by the decision tree algorithm. The brake light confidence scores, the traffic condition confidence scores, and the intersection detection confidence scorescan be factored into the calculation of the final score.

3 FIG. 104 314 303 316 318 308 312 313 As shown in, the servercan comprise at least a knowledge enginecomprising an instance of the map layer, an events database, and an evidence validation modulecomprising instances of the object detection deep learning model, the lane segmentation deep learning model, and the vehicle movement classifier.

104 102 124 308 312 104 The servercan double-check the detection made by the edge deviceby feeding or passing at least some of the same event video framesto instances of the objective detection deep learning modeland the lane segmentation deep learning modelrunning on the server.

3 FIG. 318 104 314 316 314 316 104 104 Althoughillustrates the evidence validation moduleas being on the same serveras the knowledge engineand the events database, it is contemplated by this disclosure and it should be understood by one of ordinary skill in the art that at least one of the knowledge engineand the events databasecan be run on another server or another computing device communicatively coupled to the serveror otherwise accessible to the server.

104 3 FIG. Software instructions run on the server, including any of the engines and modules disclosed herein and depicted in, can be written in the Ruby® programming language (e.g., using the Ruby on Rails® web application framework), Python® programming language, or a combination thereof.

314 102 314 102 314 102 314 102 The knowledge enginecan be configured to construct a virtual 3D environment representing the real-world environment captured by the cameras of the edge devices. The knowledge enginecan be configured to construct three-dimensional (3D) semantic annotated maps from videos and data received from the edge devices. The knowledge enginecan continuously update such maps based on new videos or data received from the edge devices. For example, the knowledge enginecan use inverse perspective mapping to construct the 3D semantic annotated maps from two-dimensional (2D) video image data obtained from the edge devices.

102 118 102 110 The semantic annotated maps can be built on top of existing standard definition maps and can be built on top of geometric maps constructed from sensor data and salient points obtained from the edge devices. For example, the sensor data can comprise positioning data from the communication and positioning unitsand IMUs of the edge devicesand wheel odometry data from the carrier vehicles.

314 314 102 The geometric maps can be stored in the knowledge enginealong with the semantic annotated maps. The knowledge enginecan also obtain data or information from one or more government mapping databases or government GIS maps to construct or further fine-tune the semantic annotated maps. In this manner, the semantic annotated maps can be a fusion of mapping data and semantic labels obtained from multiple sources including, but not limited to, the plurality of edge devices, municipal mapping databases, or other government mapping databases, and third-party private mapping databases. The semantic annotated maps can be set apart from traditional standard definition maps or government GIS maps in that the semantic annotated maps are: (i) three-dimensional, (ii) accurate to within a few centimeters rather than a few meters, and (iii) annotated with semantic and geolocation information concerning objects within the maps. For example, objects such as lane lines, lane dividers, crosswalks, traffic lights, no parking signs or other types of street signs, fire hydrants, parking meters, curbs, trees or other types of plants, or a combination thereof are identified in the semantic annotated maps and their geolocations and any rules or regulations concerning such objects are also stored as part of the semantic annotated maps. As a more specific example, all no-parking lanes within a municipality and their enforcement periods can be stored as part of a semantic annotated map of the municipality.

104 102 110 102 102 102 314 102 102 104 102 The semantic annotated maps can be updated periodically or continuously as the serverreceives new mapping data, positioning data, and/or semantic labels from the various edge devices. For example, a bus serving as a carrier vehiclehaving an edge deviceinstalled within the bus can drive along the same bus route multiple times a day. Each time the bus travels down a specific roadway or passes by a specific landmark (e.g., building or street sign), the edge deviceon the bus can take video(s) of the environment surrounding the roadway or landmark. The videos can first be processed locally on the edge device(using the computer vision tools and deep learning models previously discussed) and the outputs from such detection can be transmitted to the knowledge engineand compared against data already included as part of the semantic annotated maps. If such labels and data match or substantially match what is already included as part of the semantic annotated maps, the detection of this roadway or landmark can be corroborated and remain unchanged. If, however, the labels and data do not match what is already included as part of the semantic annotated maps, the roadway or landmark can be updated or replaced in the semantic annotated maps. An update or replacement can be undertaken if a confidence level or confidence score of the new objects detected is higher than the confidence level or confidence score of objects previously detected by the same edge deviceor another edge device. This map updating procedure or maintenance procedure can be repeated as the serverreceives more data or information from additional edge devices.

3 FIG. 104 303 104 315 102 303 102 104 315 315 102 140 102 315 As shown in, the servercan store the semantic annotated maps as part of a map layer. The servercan transmit or deploy revised or updated instances of the semantic annotated mapsto the edge devices(to be stored as part of the map layerof each of the edge devices). For example, the servercan transmit or deploy revised or updated semantic annotated mapsperiodically or when an update has been made to the existing semantic annotated maps. The updated semantic annotated mapscan be used by the edge deviceto determine the locations of no-parking lanesto ensure accurate detection. Ensuring that the edge deviceshave access to updated semantic annotated mapsreduces the likelihood of false positive detections.

104 136 316 316 124 126 136 102 In some embodiments, the servercan store event data or files included as part of the evidence packagesin the events database. For example, the events databasecan store event video framesand license plate video framesreceived as part of the evidence packagesreceived from the edge devices.

318 136 136 The evidence validation modulecan analyze the contents of an evidence packageand can make a decision concerning whether the evidence packagesis automatically approved, is automatically rejected, or requires further review.

104 136 316 136 316 136 318 The servercan store the contents of the evidence packagein the events databaseeven when the evidence packagehas been automatically rejected or has been subject to further review. In certain embodiments, the events databasecan store the contents of all evidence packagesthat have been evaluated by the evidence validation module.

318 136 318 136 In some embodiments, the evidence validation modulecan undertake an initial review of the evidence packageautomatically without relying on human reviewers. In these embodiments, the evidence validation modulecan undertake the initial review of the evidence packageby taking into account certain automatically detected context-related features surrounding a potential double parking violation to determine whether the double parking violation has indeed occurred.

136 124 126 102 122 136 127 The evidence packagecan comprise, among other things, one or more event video framesand license plate video framescaptured by the camera(s) of the edge deviceshowing a potentially offending vehicleinvolved in a double parking violation. The evidence packagecan also comprise one or more classification resultsassociated with the context-related features.

104 102 122 318 124 136 313 104 In some embodiments, the servercan also double-check or validate the detection made by the edge deviceconcerning whether the potentially offending vehiclewas static or moving. The evidence validation modulecan feed the event video framesfrom the evidence packageinto the vehicle movement classifierrunning on the server.

104 332 330 138 138 138 138 The servercan also render one or more graphical user interfaces (GUIs)that can be accessed or displayed through a web portal or mobile applicationrun on a client device. The client devicecan refer to a portable or non-portable computing device. For example, the client devicecan refer to a desktop computer or a laptop computer. In other embodiments, the client devicecan refer to a tablet computer or smartphone.

104 136 104 332 In some embodiments, one of the GUIs can provide information concerning the context-related features used by the serverto validate the evidence packagesreceived by the server. The GUIscan also provide data or information concerning times/dates of double parking violations and locations of the double parking violations.

332 332 124 126 At least one of the GUIscan provide a video player configured to play back video evidence of the double parking violation. For example, at least one of the GUIscan play back videos comprising the event video frames, the license plate video frames, or a combination thereof.

332 102 332 In another embodiment, at least one of the GUIscan comprise a live map showing real-time locations of all edge devices, double parking violations, and violation hot-spots. In yet another embodiment, at least one of the GUIscan provide a live event feed of all flagged events or double parking violations and the validation status of such double parking violations.

138 136 In some embodiments, the client devicecan be used by a human reviewer to review the evidence packagesmarked or otherwise tagged for further review.

4 FIG. 400 402 300 102 illustrates that part of a methodfor detecting a double parking violation can be undertaken by a plurality of workersof the event detection engineof an edge device.

402 402 The workerscan be software programs or modules dedicated to performing a specific set of tasks or operations. Each workercan be a software program or module dedicated to executing the tasks or operations within a docker container.

4 FIG. 402 402 402 102 As shown in, the output from one worker(e.g., the first workerA) can be transmitted to another worker (e.g., the third workerC) running on the same edge device. For example, the output or results (e.g., the inferences or predictions) provided by one worker can be transmitted to another worker using an inter-process communication protocol such as the user datagram protocol (UDP).

300 102 402 402 402 300 402 300 402 402 4 FIG. In some embodiments, the event detection engineof each of the edge devicescan comprise at least a first workerA, a second workerB, and a third workerC. Althoughillustrates the event detection enginecomprising three workers, it is contemplated by this disclosure that the event detection enginecan comprise four or more workersor two workers.

4 FIG. 402 402 124 404 404 102 114 114 404 402 402 As shown in, both the first workerA and the second workerB can retrieve or grab video frames (e.g., event video frames) from a shared camera memory. The shared camera memorycan be an onboard memory (e.g., non-volatile memory) of the edge devicefor storing videos captured by the event camera. Since the event cameracan capture approximately 15 to 60 video frames per second, the video frames can be stored in the shared camera memoryprior to being analyzed by the first workerA or the second workerB. In some embodiments, the video frames can be grabbed using a video frame grab function such as the GStreamer tool.

402 500 402 707 707 516 140 708 5 5 FIGS.A andD 5 FIG.D 7 FIG.A As will be discussed in more detail in the following sections, the objective of the first workerA can be to detect objects of certain object classes (e.g., cars, trucks, buses, etc.) within a video frame and bound each of the objects with a vehicle bounding polygon(see, e.g.,). The objective of the second workerB can be to detect one or more laneswithin the same video frame and bound the lanesin lane bounding polygons(see, e.g.,) including bounding a no-parking lanein an LOI polygon(see).

402 800 500 708 402 402 8 8 FIGS.A andB The objective of the third workerC can be to detect whether a double parking violation has occurred by calculating a lane occupancy score(see, e.g.,) using outputs (e.g., the vehicle bounding polygonand the LOI polygon) produced and received from the first workerA and the second workerB.

4 FIG. 402 124 404 406 402 124 102 402 308 102 illustrates that the first workerA can crop and resize an event video frameretrieved from the shared camera memoryin operation. The first workerA can crop and resize the event video frameto optimize the video frame for analysis by one or more deep learning models or convolutional neural networks running on the edge device. For example, the first workerA can crop and resize the video frame to optimize the video frame for the object detection deep learning modelrunning on the edge device.

402 308 402 308 In one embodiment, the first workerA can crop and resize the video frame to meet certain size parameters associated with the object detection deep learning model. For example, the first workerA can crop and resize the video frame such that the aspect ratio of the video frame meets certain parameters associated with the object detection deep learning model.

114 300 402 700 707 As a more specific example, the video frames captured by the event cameracan have an aspect ratio of 1920×1080. When the event detection engineis configured to determine double parking violations, the first workerA can be programmed to crop the video frames such that vehicles and roadwayswith lanesare retained but other objects or landmarks (e.g., sidewalks, pedestrians, building façades) are cropped out.

308 402 308 When the object detection deep learning modelis a variation of the Single Shot Detection (SSD) model with a MobileNet backbone as the feature extractor, the first workerA can crop and resize the video frames such that the aspect ratio of the video frames meets certain parameters associated with the object detection deep learning model.

400 122 122 500 408 402 308 502 504 506 500 5 FIG.A The methodcan also comprise detecting a potentially offending vehiclefrom the video frame and bounding the potentially offending vehicleshown in the video frame with a vehicle bounding polygonin operation. The first workerA can be programmed to pass the video frame to the object detection deep learning modelto obtain an object class, an object detection confidence score, and a set of image coordinatesfor the vehicle bounding polygon(see, e.g.,).

308 308 308 502 308 502 308 502 In some embodiments, the object detection deep learning modelcan be configured such that only certain vehicle-related objects are supported by the object detection deep learning model. For example, the object detection deep learning modelcan be configured such that the object classessupported only consist of cars, trucks, and buses. In other embodiments, the object detection deep learning modelcan be configured such that the object classessupported also include bicycles, scooters, and other types of wheeled mobility vehicles. In other embodiments, the object detection deep learning modelcan be configured such that the object classessupported also comprise non-vehicles classes such as pedestrians, landmarks, street signs, fire hydrants, bus stops, and building façades.

308 308 502 502 102 In certain embodiments, the object detection deep learning modelcan be designed to detect up to 60 objects per video frame. Although the object detection deep learning modelcan be designed to accommodate numerous object classes, one advantage of limiting the number of object classesis to reduce the computational load on the processors of the edge deviceand make the neural network more efficient.

308 308 In some embodiments, the object detection deep learning modelcan be a convolutional neural network comprising a plurality of convolutional layers and connected layers trained for object detection (and, in particular, vehicle detection). In one embodiment, the object detection deep learning modelcan be a variation of the Single Shot Detection (SSD) model with a MobileNet backbone as the feature extractor.

308 308 308 134 In other embodiments, the object detection deep learning modelcan be the You Only Look Once Lite (YOLO Lite) object detection model. In some embodiments, the first object detection deep learning modelcan also identify certain attributes of the detected objects. For example, the object detection deep learning modelcan identify a set of vehicle attributesof an object identified as a car such as the color of the car, the make and model of the car, and the car type (e.g., whether the vehicle is a personal vehicle or a public service vehicle).

308 102 102 110 308 102 308 The object detection deep learning modelcan be trained, at least in part, from video frames of videos captured by the edge deviceor other edge devicesdeployed in the same municipality or coupled to other carrier vehiclesin the same carrier fleet. The object detection deep learning modelcan be trained, at least in part, from video frames of videos captured by the edge deviceor other edge devices at an earlier point in time. Moreover, the object detection deep learning modelcan be trained, at least in part, from video frames from one or more open-sourced training sets or datasets.

402 504 308 504 402 500 504 As previously discussed, the first workerA can obtain an object detection confidence scorefrom the object detection deep learning model. The object detection confidence scorecan be between 0 and 1.0. The first workerA can be programmed to not apply a vehicle bounding polygonto a vehicle if the object detection confidence scoreof the detection is below a preset confidence threshold. For example, the confidence threshold can be set at between 0.65 and 0.90 (e.g., at 0.70). The confidence threshold can be adjusted based on an environmental condition (e.g., a lighting condition), a location, a time-of-day, a day-of-the-week, or a combination thereof.

402 506 500 506 500 506 500 500 506 500 500 As previously discussed, the first workerA can also obtain a set of image coordinatesfor the vehicle bounding polygon. The image coordinatescan be coordinates of corners of the vehicle bounding polygon. For example, the image coordinatesfor the vehicle bounding polygoncan be x- and y-coordinates for an upper left corner and a lower right corner of the vehicle bounding polygon. In other embodiments, the image coordinatesfor the vehicle bounding polygoncan be x- and y-coordinates of all four corners or the upper right corner and the lower left corner of the vehicle bounding polygon.

500 500 In some embodiments, the vehicle bounding polygoncan bound the entire two-dimensional (2D) image of the vehicle captured in the video frame. In other embodiments, the vehicle bounding polygoncan bound at least part of the 2D image of the vehicle captured in the video frame such as a majority of the pixels making up the 2D image of the vehicle.

400 402 308 402 410 402 308 506 500 502 402 308 402 5 FIG.A The methodcan further comprise transmitting the outputs produced by the first workerA and/or the object detection deep learning modelto a third workerC in operation. In some embodiments, the outputs produced by the first workerA and/or the object detection deep learning modelcan comprise the image coordinatesof the vehicle bounding polygonand the object classof the object detected (see, e.g.,). The outputs produced by the first workerA and/or the object detection deep learning modelcan be packaged into UDP packets and transmitted using UDP sockets to the third workerC.

402 308 402 In other embodiments, the outputs produced by the first workerA and/or the object detection deep learning modelcan be transmitted to the third workerC using another network communication protocol such as a remote procedure call (RPC) communication protocol.

4 FIG. 402 124 404 412 124 402 124 402 illustrates that the second workerB can crop and resize an event video frameretrieved from the shared camera memoryin operation. In some embodiments, the event video frameretrieved by the second workerB can be the same as the event video frameretrieved by the first workerA.

124 402 402 124 402 In other embodiments, the event video frameretrieved by the second workerB can be a different video frame from the video frame retrieved by the first workerA. For example, the event video framecan be captured at a different point in time than the video frame retrieved by the first workerA (e.g., several seconds or milliseconds before or after). In all such embodiments, one or more vehicles and lanes should be visible in the video frame.

402 124 102 402 124 312 The second workerB can crop and resize the event video frameto optimize the video frame for analysis by one or more deep learning models or convolutional neural networks running on the edge device. For example, the second workerB can crop and resize the event video frameto optimize the video frame for the lane segmentation deep learning model.

402 312 402 124 312 In one embodiment, the second workerB can crop and resize the video frame to meet certain parameters associated with the lane segmentation deep learning model. For example, the second workerB can crop and resize the event video framesuch that the aspect ratio of the video frame meets certain parameters associated with the lane segmentation deep learning model.

124 114 402 124 As a more specific example, the event video framescaptured by the event cameracan have an aspect ratio of 1920×1080. The second workerB can be programmed to crop the event video framessuch that vehicles and lanes are retained but other objects or landmarks (e.g., sidewalks, pedestrians, building façades) are cropped out.

402 The second workerB can crop and resize the video frames such that the aspect ratio of the video frames is about 448×256.

400 505 505 707 114 102 505 114 114 505 102 5 FIG.A When cropping the video frame, the methodcan further comprise an additional step of determining whether a vanishing point(see, e.g.,) is present within the video frame. The vanishing pointcan be one point or region in the video frame where distal or terminal ends of the lanesshown in the video frame converge into the point or region. Alternatively, the event cameraon the edge devicecan be physically adjusted (for example, as part of an initial calibration routine) until the vanishing pointis shown in the video frames captured by the event camera. Adjusting the cropping parameters or the event camerauntil a vanishing pointis detected in the video frame can be part of a calibration procedure that is run before deploying the edge devicesin the field.

505 707 402 505 707 The vanishing pointcan be used to approximate the sizes of lanesdetected by the second workerB. For example, the vanishing pointcan be used to detect when one or more of the laneswithin a video frame are obstructed by an object (e.g., a bus, car, truck, or another type of vehicle).

400 312 707 414 312 707 516 312 5 7 7 FIGS.C,A, andC The methodcan also comprise passing the processed video frame (i.e., the cropped, resized, and smoothed video frame) to the lane segmentation deep learning modelto detect and bound lanescaptured in the video frame in operation. The lane segmentation deep learning modelcan bound the lanesin a plurality of lane bounding polygons(see, e.g.,). The lane segmentation deep learning modelcan be a convolutional neural network trained specifically for lane detection and segmentation.

312 600 312 6 FIG. In some embodiments, the lane segmentation deep learning modelcan be a multi-headed convolutional neural network comprising a plurality of prediction heads(see, e.g.,). For example, the lane segmentation deep learning modelcan be a multi-headed convolutional neural network comprising a residual neural network (e.g., a ResNet) backbone with a standard mask prediction.

600 312 707 312 140 140 312 516 140 516 In some embodiments, each of the headsof the lane segmentation deep learning modelcan be configured to detect a specific type of lane and/or lane marking(s). At least one of the lanesdetected by the lane segmentation deep learning modelcan be a no-parking lane. The no-parking lanecan be identified by the lane segmentation deep learning modeland a lane bounding polygoncan be used to bound the no-parking lane. Lane bounding polygonswill be discussed in more detail in later sections.

400 402 312 402 416 402 312 516 708 402 312 402 7 FIG.A 4 FIG. The methodcan further comprise transmitting the outputs produced by the second workerB and/or the lane segmentation deep learning modelto a third workerC in operation. In some embodiments, the outputs produced by the second workerB and/or the lane segmentation deep learning modelcan be coordinates of the lane bounding polygonsincluding coordinates of a LOI polygon(see, e.g.,). As shown in, the outputs produced by the second workerB and/or the lane segmentation deep learning modelcan be packaged into UDP packets and transmitted using UDP sockets to the third workerC.

402 312 402 In other embodiments, the outputs produced by the second workerB and/or the lane segmentation deep learning modelcan be transmitted to the third workerC using another network communication protocol such as an RPC communication protocol.

4 FIG. 402 402 402 418 402 As shown in, the third workerC can receive the outputs/results produced by the first workerA and the second workerB in operation. The third workerC can receive the outputs/results as UDP packets received over UDP sockets.

402 312 502 506 500 402 312 707 516 707 708 The outputs or results received from the first workerA can be in the form of predictions or detections made by the object detection deep learning modelof the objects captured in the video frame that fit a supported object class(e.g., car, truck, or bus) and the image coordinatesof the vehicle bounding polygonsbounding such objects. The outputs or results received from the second workerB can be in the form of predictions made by the lane segmentation deep learning modelof the lanescaptured in the video frame and the coordinates of the lane bounding polygonsbounding such lanesincluding the coordinates of at least one LOI polygon.

400 402 402 420 The methodcan further comprise validating the payloads of UDP packets received from the first workerA and the second workerB in operation. The payloads can be validated or checked using a payload verification procedure such as a payload checksum verification algorithm. This is to ensure the packets received containing the predictions were not corrupted during transmission.

400 402 402 402 422 The methodcan also comprise the third workerC synchronizing the payloads or messages received from the first workerA and the second workerB in operation. Synchronizing the payloads or messages can comprise checks or verifications on the predictions or data contained in such payloads or messages such that any comparison or further processing of such predictions or data is only performed if the predictions or data concern objects or lanes in the same video frame (i.e., the predictions or coordinates calculated are not generated from different video frames captured at significantly different points in time).

400 500 516 708 424 402 402 500 516 The methodcan further comprise translating the coordinates of the vehicle bounding polygonand the coordinates of the lane bounding polygons(including the coordinates of the LOI polygon) into a uniform coordinate domain in operation. Since the same video frame was cropped and resized differently by the first workerA (e.g., cropped and resized to an aspect ratio of 500×500 from an original aspect ratio of 1920×1080) and the second workerB (e.g., cropped and resized to an aspect ratio of 752×160 from an original aspect ratio of 1920×1080) to suit the needs of their respective convolutional neural networks, the pixel coordinates of pixels used to represent the vehicle bounding polygonand the lane bounding polygonsmust be translated into a shared coordinate domain or back to the coordinate domain of the original video frame (before the video frame was cropped or resized). This is to ensure that any subsequent comparison of the relative positions of boxes and polygons are done in one uniform coordinate domain.

400 800 500 708 426 800 800 8 8 FIGS.A andB The methodcan also comprise calculating a lane occupancy score(see, e.g.,) based in part on the translated coordinates of the vehicle bounding polygonand the LOI polygonin operation. In some embodiments, the lane occupancy scorecan be a number between 0 and 1. The lane occupancy scorecan be calculated using one or more heuristics.

402 800 708 402 500 708 708 800 500 800 For example, the third workerC can calculate the lane occupancy scoreusing a lane occupancy heuristic. The lane occupancy heuristic can comprise the steps of masking or filling in an area within the LOI polygonwith certain pixels. The third workerC can then determine a pixel intensity value associated with each pixel within at least part of the vehicle bounding polygon. The pixel intensity value can range between 0 and 1 with 1 being a high degree of likelihood that the pixel is located within the LOI polygonand with 0 being a high degree of likelihood that the pixel is not located within the LOI polygon. The lane occupancy scorecan be calculated by taking an average of the pixel intensity values of all pixels within at least part of the vehicle bounding polygon. Calculating the lane occupancy scorewill be discussed in more detail in later sections.

400 800 402 136 800 428 The methodcan further comprise detecting that a double parking violation has occurred when the lane occupancy scoreexceeds a predetermined threshold value. The third workerC can then generate an evidence packagewhen the lane occupancy scoreexceeds a predetermined threshold value in operation.

136 124 114 118 102 110 132 124 134 122 128 122 136 402 102 104 138 In some embodiments, the evidence packagecan comprise the event video frameor other video frames captured by the event camera, the positioning data obtained by the communication and positioning unitof the edge device, the speed of the carrier vehiclewhen the double parking violation was detected, certain timestampsdocumenting when the event video framewas captured, a set of vehicle attributesconcerning the potentially offending vehicle, and an alphanumeric string representing the recognized license plate numberof the potentially offending vehicle. The evidence packagecan be prepared by the third workerC or another worker on the edge deviceto be sent to the serveror a third-party computing device/resource or client device.

5 FIG.A 3 FIG. 124 122 500 124 300 114 102 300 124 114 308 102 illustrates an example of an event video frameshowing a potentially offending vehiclebounded by a vehicle bounding polygon. The event video framecan be one of the video frames grabbed or otherwise retrieved by the event detection enginefrom the videos captured by the event cameraof the edge device. As previously discussed, the event detection enginecan periodically or continuously pass event video framesfrom the videos captured by the event camerato an object detection deep learning modelrunning on the edge device(see).

5 FIG.A 308 122 500 300 308 502 504 506 500 As shown in, the object detection deep learning modelcan bound the potentially offending vehiclein the vehicle bounding polygon. The event detection enginecan obtain as outputs from the object detection deep learning model, predictions concerning the objects detected within the video frame including at least an object class, an object detection confidence scorerelated to the object detected, and a set of image coordinatesfor the vehicle bounding polygon.

504 112 102 504 The object detection confidence scorecan be between 0 and 1.0. In some embodiments, the control unitof the edge devicecan abide by the results of the detection only if the object detection confidence scoreis above a preset confidence threshold. For example, the confidence threshold can be set at between 0.65 and 0.90 (e.g., at 0.70).

300 506 500 506 500 506 500 506 500 The event detection enginecan also obtain a set of image coordinatesfor the vehicle bounding polygon. The image coordinatescan be coordinates of corners of the vehicle bounding polygon. For example, the image coordinatescan be x- and y-coordinates for an upper left corner and a lower right corner of the vehicle bounding polygon. In other embodiments, the image coordinatescan be x- and y-coordinates of all four corners or the upper right corner and the lower left corner of the vehicle bounding polygon.

500 122 124 500 122 124 122 In some embodiments, the vehicle bounding polygoncan bound the entire two-dimensional (2D) image of the potentially offending vehiclecaptured in the event video frame. In other embodiments, the vehicle bounding polygoncan bound at least part of the 2D image of the potentially offending vehiclecaptured in the event video framesuch as a majority of the pixels making up the 2D image of the potentially offending vehicle.

300 308 134 122 134 300 140 The event detection enginecan also obtain as an output from the object detection deep learning modelpredictions concerning a set of vehicle attributessuch as a color, make and model, and vehicle type of the potentially offending vehicleshown in the video frames. The vehicle attributescan be used by the event detection engineto make an initial determination as to whether the vehicle shown in the video frames is subject to the double parking violation policy (e.g., whether the vehicle is allowed to park or otherwise stop in a no-parking lane).

5 FIG.A 1 FIG.A 5 FIG.A 5 FIG.A 100 122 110 122 140 110 102 140 116 102 129 122 116 110 illustrates an example scenario where the systemofcan be utilized to detect a double parking violation of a potentially offending vehicleparked in the same lane occupied by the carrier vehicle. As shown in, a potentially offending vehiclecan be double parked or otherwise stopped in a no-parking lanethat is also the lane-of-travel (or “ego lane”) of a carrier vehiclecarrying the edge device. As shown in, the no-parking lanecan be a lane next to or adjacent to a permitted parking lane. In this instance, the LPR cameraof the edge devicecannot be used to capture the license plateof the potentially offending vehicledue to the positioning of the LPR camera(which, in this case, may be pointed to the right side of the carrier vehicle).

122 124 129 116 102 304 114 114 124 114 310 102 When a potentially offending vehicleis detected in the event video framebut a license plateis not captured by the LPR camera, the edge device(e.g., the license plate recognition engine) can trigger the event camerato operate as an LPR camera. When the event camerais triggered to act as the LPR camera (at least temporarily), the event video framescaptured by the event cameracan be passed to the LPR deep learning modelsrunning on the edge device.

110 129 122 126 114 124 310 For example, when a carrier vehicle(e.g., a municipal bus) is stopped and the license plateof the potentially offending vehicleis not detected in the license plate video frames, the event cameracan be triggered to operate (at least temporarily) as the LPR camera where the event video framesare passed to the LPR deep learning modelsfor automatic license plate recognition.

5 FIG.B 116 114 110 102 122 116 122 illustrates a flowchart showing the logic of switching from the LPR camerato the event camerafor automated license plate recognition when a carrier vehiclecarrying the edge deviceis approaching a potentially offending vehicledirectly from behind or at an insufficient angle such that the LPR camerais not in a position to capture the license plate of the potentially offending vehicle.

5 FIG.B 122 129 124 116 124 129 122 310 102 128 129 129 124 124 114 310 102 For example,illustrates that once the potentially offending vehicleis detected or identified, a query can be made as to whether a license plateappears in any of the license plate video framescaptured by the LPR camera. If the answer to this query is yes, such license plate video framescontaining the license plateof the potentially offending vehiclecan be passed to an LPR deep learning modelrunning on the edge deviceto automatically recognize the license plate numberof the license plate. Alternatively, if the answer to this query is no and no license plateis detected in the license plate video frames, the event video framescaptured by the event cameracan be used for automated license plate recognition by being passed to the LPR deep learning modelrunning on the edge device.

5 FIG.C 3 FIG. 126 129 122 510 126 304 116 102 304 126 116 310 102 illustrates an example of a license plate video frameshowing a license plateof the potentially offending vehiclebounded by a license plate bounding polygon. The license plate video framecan be one of the video frames grabbed or otherwise retrieved by the license plate recognition enginefrom the videos captured by the LPR cameraof the edge device. As previously discussed, the license plate recognition enginecan periodically or continuously pass license plate video framesfrom the videos captured by the LPR camerato an LPR deep learning model(see) running on the edge device.

310 126 310 112 102 310 128 122 128 112 310 512 The LPR deep learning modelcan be specifically trained to recognize license plate numbers from video frames or images. By feeding the license plate video frameto the LPR deep learning model, the control unitof the edge devicecan obtain as an output from the LPR deep learning model, a prediction concerning the license plate numberof the potentially offending vehicle. The prediction can be in the form of an alphanumeric string representing the license plate number. The control unitcan also obtain as an output from the LPR deep learning modelan LPR confidence scoreconcerning the recognition.

512 112 102 512 The LPR confidence scorecan be between 0 and 1.0. In some embodiments, the control unitof the edge devicecan abide by the results of the recognition only if the LPR confidence scoreis above a preset confidence threshold. For example, the confidence threshold can be set at between 0.65 and 0.90 (e.g., at 0.70).

5 FIG.D 3 FIG. 5 FIG.A 124 122 500 516 124 114 102 300 102 124 308 312 102 308 122 500 112 102 308 502 504 506 500 illustrates another example of an event video frameshowing a potentially offending vehiclebounded by a vehicle bounding polygonand a lane bounded by a lane bounding polygon. The event video framecan be one of the video frames grabbed or otherwise retrieved from the videos captured by the event cameraof the edge device. The event detection engineof the edge devicecan periodically or continuously pass event video framesto the object detection deep learning modeland the lane segmentation deep learning modelrunning on the edge device(see). As discussed above in relation to, the object detection deep learning modelcan bound the potentially offending vehiclein the vehicle bounding polygonand the control unitof the edge devicecan obtain as outputs from the object detection deep learning model, predictions concerning the object class, the object detection confidence score, and a set of image coordinatesfor the vehicle bounding polygon.

300 124 312 707 124 300 707 140 140 The event detection enginecan also pass or feed event video framesto the lane segmentation deep learning modelto detect one or more lanesshown in the event video frames. Moreover, the event detection enginecan also recognize that one of the lanesdetected is a no-parking lane. In some embodiments, the no-parking lanecan be a lane next to or adjacent to a permitted parking lane.

5 FIG.D 312 140 516 312 518 516 As shown in, the lane segmentation deep learning modelcan bound the no-parking lanein a lane bounding polygon. The lane segmentation deep learning modelcan also output image coordinatesassociated with the lane bounding polygon.

516 516 In some embodiments, the lane bounding polygoncan be a quadrilateral. More specifically, the lane bounding polygoncan be shaped substantially as a trapezoid.

300 122 140 500 122 516 140 506 500 518 516 500 516 300 122 140 500 516 The event detection enginecan determine that the potentially offending vehicleis parked in the no-parking lanebased on the amount of overlap between the vehicle bounding polygonbounding the potentially offending vehicleand the lane bounding polygonbounding the no-parking lane. For example, the image coordinatesassociated with the vehicle bounding polygoncan be compared with the image coordinatesassociated with the lane bounding polygonto determine an amount of overlap between the vehicle bounding polygonand the polygon. As a more specific example, the event detection enginecan calculate a lane occupancy score to determine whether the potentially offending vehicleis parked in the no-parking lane. A higher lane occupancy score can be equated with a higher degree of overlap between the vehicle bounding polygonand the lane bounding polygon.

5 FIG.D 7 FIG.C 5 5 FIGS.A andD 516 516 500 510 516 102 104 107 500 510 516 Althoughillustrates only one instance of a lane bounding polygon, it is contemplated by this disclosure that multiple lanes can be bound by lane bounding polygonsin the same video frame (see, e.g.,). Moreover, althoughillustrate a visual representation of the vehicle bounding polygon, the license plate bounding polygon, and the lane bounding polygon, it should be understood by one of ordinary skill in the art that the image coordinates of such bounding boxes and polygons and can be used as inputs only by the edge deviceor the serveror stored in the databasewithout the actual vehicle bounding polygon, license plate bounding polygon, or lane bounding polygonbeing visualized.

6 FIG. 6 FIG. 312 312 312 illustrates a schematic representation of one embodiment of the lane segmentation deep learning model. As shown in, the lane segmentation deep learning modelcan be a multi-headed neural network trained for lane detection and segmentation. For example, the lane segmentation deep learning modelcan be a multi-headed convolutional neural network.

6 FIG. 312 600 600 600 600 600 600 600 600 600 600 602 As shown in, the lane segmentation deep learning modelcan comprise a plurality of prediction headsoperating on top of several shared layers. For example, the prediction headscan comprise a first headA, a second headB, a third headC, and a fourth headD. The first headA, the second headB, the third headC, and the fourth headD can share a common stack of network layers including at least a convolutional backbone(e.g., a feature extractor).

602 124 402 602 The convolutional backbonecan be configured to receive as inputs event video framesthat have been cropped and re-sized by pre-processing operations undertaken by the second workerB. The convolutional backbonecan then pool certain raw pixel data and sub-sample certain raw pixel regions of the video frames to reduce the size of the data to be handled by the subsequent layers of the network.

602 600 The convolutional backbonecan extract certain essential or relevant image features from the pooled image data and feed the essential image features extracted to the plurality of prediction heads.

600 600 600 600 600 The prediction heads, including the first headA, the second headB, the third headC, and the fourth headD, can then make their own predictions or detections concerning different types of lanes captured by the video frames.

312 600 402 600 By designing the lane segmentation deep learning modelin this manner (i.e., multiple prediction headssharing the same underlying layers), the second workerB can ensure that the predictions made by the various prediction headsare not affected by any differences in the way the image data is processed by the underlying layers.

600 312 600 600 300 308 312 600 Although reference is made in this disclosure to four prediction heads, it is contemplated by this disclosure that the lane segmentation deep learning modelcan comprise five or more prediction headswith at least some of the headsdetecting different types of lanes. Moreover, it is contemplated by this disclosure that the event detection enginecan be configured such that the object detection workflow of the object detection deep learning modelis integrated with the lane segmentation deep learning modelsuch that the object detection steps are conducted by an additional headof a singular neural network.

600 312 110 In some embodiments, the first headA of the lane segmentation deep learning modelcan be trained to detect a lane-of-travel. The lane-of-travel can also be referred to as an “ego lane” and is the lane currently occupied by the carrier vehicle.

600 600 102 The lane-of-travel can be detected using a position of the lane relative to adjacent lanes and the rest of the video frame. The first headA can be trained using a dataset designed specifically for lane detection and segmentation. In other embodiments, the first headA can also be trained using video frames obtained from deployed edge devices.

600 312 704 704 7 FIG.A In these and other embodiments, the second headB of the lane segmentation deep learning modelcan be trained to detect lane markings(see, e.g.,). For example, the lane markingscan comprise lane lines, text markings, markings indicating a crosswalk, markings indicating turn lanes, dividing line markings, or a combination thereof.

600 312 140 140 140 600 140 7 FIG.A 7 7 FIGS.A-C In some embodiments, the third headC of the lane segmentation deep learning modelcan be trained to detect the no-parking lane(see, e.g.,). In some embodiments, the no-parking lanecan be a lane next or adjacent to a permitted parking lane. In other embodiments, the no-parking lanecan be a bus lane when the bus lane is outside of a bus lane enforcement period, a bike lane, a fire lane, or a no-stopping zone. The third headC can detect the no-parking lanebased on an automated lane detection algorithm. The lane detection algorithm can be disclosed in relation to.

600 102 600 The third headC can be trained using video frames obtained from deployed edge devices. In other embodiments, the third headC can also be trained using training data (e.g., video frames) obtained from a dataset.

600 312 702 140 702 140 600 702 140 600 102 600 7 FIG.C The fourth headD of the lane segmentation deep learning modelcan be trained to detect one or more adjacent or peripheral lanes(see, e.g.,) after the no-parking laneis detected. In some embodiments, the adjacent or peripheral lanescan be lanes immediately adjacent to the no-parking laneor the lane-of-travel, or lanes further adjoining the immediately adjacent lanes. In certain embodiments, the fourth headD can detect the adjacent or peripheral lanesbased on a determined position of the no-parking laneand/or the lane-of-travel. The fourth headD can be trained using video frames obtained from deployed edge devices. In other embodiments, the fourth headD can also be trained using training data (e.g., video frames) obtained from a dataset.

600 600 600 600 600 600 In some embodiments, the training data (e.g., video frames) used to train the prediction heads(any of the first headA, the second headB, the third headC, or the fourth headD) can be annotated using semantic segmentation. For example, the same video frame can be labeled with multiple labels (e.g., annotations indicating a no-parking lane, a lane-of-travel, adjacent/peripheral lanes, crosswalks, etc.) such that the video frame can be used to train multiple or all of the prediction heads.

7 FIG.A 7 FIG.A 7 FIG.A 124 140 516 122 500 705 703 701 700 701 700 700 700 701 700 705 703 705 703 illustrates an event video frameshowing a no-parking lanebounded by a lane bounding polygonand vehicles, including a potentially offending vehicle, bounded by vehicle bounding polygons.also includes a visualization of a plurality of road edge pointsthat can be used to fit a linerepresenting a road edgeof a roadway. For purposes of this disclosure, the road edgecan be an edge or line formed by a roadwaywhere the roadwaymeets a curb or sidewalk directly adjacent to or abutting the roadway. For example, the road edgecan follow a gutter line or a line that serves as the interface between the gutter of the roadwayand the curb/sidewalk. Although the road edge pointsand the lineare shown in, it should be understood by one of ordinary skill in the art that the method(s) disclosed herein can be made operable without displaying or visualizing the road edge pointsor the line.

140 140 701 700 124 701 One technical problem faced by the applicant when it comes to detection of double parking violations is how to determine the location of a no-parking lane. This can be made more difficult by the fact that, oftentimes, the no-parking laneis offset from a curb or sidewalk by one or more parking lanes or loading zones where parking or stopping is permitted. One technical solution discovered and developed by the applicant is to first determine the location of a road edgeof the roadwayfrom video frames (e.g., event video frames) capturing a potential double parking violation and then to determine the location of the other lanes, including the location of a no-stopping or no-parking parking lane, in relation to the road edgeusing data or information concerning the widths of municipal roadway lanes (which are often standardized or can be easily measured/surveyed).

701 124 102 312 102 600 312 706 700 124 124 6 FIG. 7 FIG.B One method for determining the location of the road edgecan comprise the step of feeding one or more event video framescaptured by the edge deviceto a lane segmentation deep learning modelrunning on the edge device(see, e.g.,). One prediction headof the lane segmentation deep learning model(e.g., a semantic segmentation head) can generate or output a maskor heatmap (see, e.g.,) representing an edge or boundary of a roadwayshown in the event video frame(s). The mask or heatmap can be obtained by performing pixel-level segmentation of the one or more event video frames.

7 FIG.B 7 FIG.B 706 600 312 706 700 124 illustrates an example maskor heatmap that can be generated or outputted by a prediction headof the lane segmentation deep learning model. As shown in, the maskor heatmap can represent a right edge or boundary of the roadwayshown in the event video frame.

312 706 100 312 706 100 In some embodiments, the lane segmentation deep learning modelcan be programmed to select the maskor heatmap on the right or right-hand side when the systemor methods disclosed herein are deployed in a country (e.g., the United States) or region where the flow of traffic is on the right-hand side. In other embodiments, the lane segmentation deep learning modelcan be programmed to select a maskor heatmap on the left or left-hand side when the systemor methods disclosed herein are deployed in a country (e.g., the United Kingdom) or region where the flow of traffic is on the left-hand side.

706 701 706 The maskor heatmap can be considered an initial rough approximation of the road edge. The next step in the method can comprise discarding or ignoring all pixels of the maskor heatmap that do not exceed a preset mask value threshold (e.g., mask value>0.25). At this point, each of the remaining pixels can be represented by a tuple comprising the pixel's x-coordinate, y-coordinate, and mask value.

706 706 706 701 Since the number of pixels remaining, even after the thresholding step above, can still be quite high and contain numerous redundant pixels (especially along the horizontal direction), the number of pixels remaining can be further culled through a slicing technique. The slicing technique can comprise slicing the maskor heatmap horizontally and vertically with a step of N and M pixels, respectively. The maskor heatmap can then be analyzed along each slice by finding a center of weight of each slice. This is done by using mask values of the pixels as weighting factors of each coordinate (the x-coordinate or the y-coordinate, depending on the horizontal or vertical slice). For example, each of the horizontal slices can produce a single pair comprising a “weighted” x-coordinate and a y-coordinate, which is defined by the step of the slicing. In certain embodiments, a mean value of the mask scores along a given slice can be calculated and assigned to such a coordinate pair as a confidence score of the point. This way, the maskor heatmap which began as a rough approximation of the road edgecan be reduced to a K number of points where the max value for K is determined by the following: K=(video_frame_width//M)+ (video_frame_height//N), where the double slash (//) is a division operator that produces a result that is rounded down to the nearest whole number.

705 705 703 701 In some embodiments, the resulting K number of points can be considered the road edge points. The road edge pointscan be further culled using an iterative non-deterministic outlier detection algorithm to yield a linerepresenting the road edge.

In some embodiments, the iterative non-deterministic outlier detection algorithm can be a random sample consensus (RANSAC) algorithm that randomly selects a subset of points to be inliers, attempts to fit a line (e.g., a linear regression model) to the subset of points, discards outlier points, and repeats the process until additional outlier points are identified and discarded.

705 In other embodiments, the road edge pointscan also be further culled using another outlier detection algorithm such as an M-estimator sample consensus (MSAC) algorithm, a maximum likelihood estimator sample consensus (MLESAC) algorithm, or a least-median of squares algorithm.

703 705 703 124 703 705 701 In certain embodiments, the linefitted to the plurality of road edge pointscan be parameterized by a slope and an intercept. Each of the slope and the intercept of the linecan be calculated using a sliding window or moving average algorithm such that the slope is an average slope value and the intercept is an average intercept value calculated from several event video framesprior in time. This additional step can ensure that the linefitted to the road edge pointsis robust and accurately represents the road edge.

700 140 703 701 At this point, the method can further comprise determining a layout of one or more lanes of the roadway, including the no-parking lane, using the fitted lineto represent the road edge.

7 FIG.C 703 705 703 701 140 703 705 illustrates a linefitted to a plurality of road edge pointsusing an iterative non-deterministic outlier detection algorithm (e.g., RANSAC) and four lanes positioned to the left of the linerepresenting the road edge. The layout of the four lanes, including the location of the no-parking lane, can be determined based on the linefitted to the plurality of road edge points.

707 140 702 102 707 303 102 707 303 In some embodiments, the layout of the lanes(e.g., the no-parking lane, the peripheral lanes, the lane-of-travel, etc.) can be determined by the edge deviceusing a known or predetermined width of each of the lanesencoded or embedded in a map layerstored on each of the edge devices. The known or predetermined width of each of the lanescan be obtained by performing surveys or measurements of such lanes in the field or obtained from one or more publicly-available map databases or municipal/governmental databases. Such lane width data can then be associated with the relevant streets/roadways, areas/regions, or coordinates in the map layer.

303 140 707 700 303 707 700 707 707 303 303 The map layercan also comprise data or information concerning the position of the no-parking lanerelative to the other lanesof the roadway. The map layercan further comprise data or information concerning a total number of lanesof the roadwayand the direction-of-travel of such lanes. Such data or information can also be obtained by performing surveys or measurements of such lanesin the field or obtained from one or more publicly-available map databases or municipal/governmental databases. Such data or information can be encoded or embedded in the map layerand then associated with the relevant streets/roadways, areas/regions, or coordinates in the map layer.

303 302 302 102 303 102 303 104 104 The map layercan be stored as part of the localization and mapping engineor accessible by the localization and mapping enginerunning on each of the edge devices. For example, the map layercan comprise one or more semantic maps or semantic annotated maps. The edge devicecan receive updates to the map layerfrom the serveror receive new semantic maps or semantic annotated maps from the server.

102 130 118 303 707 707 140 707 701 707 102 707 707 140 707 707 303 110 102 700 The edge devicecan use its own current location(e.g., obtained from the communication and positioning unit) to query the map layerfor the known or predetermined width of the lanes, the total number of lanes, the location or position of the one or more no-parking lanesrelative to the other lanesor the road edge, and the direction-of-travel for such lanesfor that particular location. In some embodiments, the edge devicecan pull map data (including the known or predetermined width of the lanes, the total number of lanes, the location or position of the one or more no-parking lanesrelative to the other lanes, and the direction-of-travel of such lanes) from the map layeras the carrier vehiclecarrying the edge devicedrives along different roadwaysor drives along its daily route.

102 707 700 703 701 303 102 140 703 701 707 140 701 707 The edge devicecan determine the layout of the lanesof the roadwayusing the lineto represent the road edge(e.g., the right road edge) and using certain data or information retrieved or pulled from the map layer. For example, the edge devicecan determine the location of one or more no-parking lanesbased on the linerepresenting the road edge, the known or predetermined width of the lanes, and the location or position of the one or more no-parking lanesrelative to the road edgeor relative to the other lanes.

124 707 700 701 114 102 707 701 122 140 Since the event video framecaptures the lanesof the roadway(and the road edge) from a perspective of the event camera, the edge devicecan also transform the layout of the lanes(and the road edge) using a perspective transformation algorithm (i.e., homography) to provide more insights concerning the location of the potentially offending vehiclerelative to the no-parking lane.

7 FIG.C 140 140 140 140 also illustrates that the no-parking lanecan be a lane that serves as a bus lane during a bus lane enforcement period (e.g., between the hours of 6 am and 10 pm on weekdays). When outside of this bus lane enforcement period (e.g., between the hours of 10 μm and 6 am on the weekdays and on weekends), this lane can serve as a no-parking lane. For example, vehicles that stop or park in this no-parking lanecan be assessed a bus lane violation during the bus lane enforcement period and vehicles that stop or park in this no-parking laneoutside of the bus lane enforcement period (i.e., during all other times) can be assessed a double-parking violation.

303 302 300 102 130 102 132 In some embodiments, the various enforcement time periods can be stored as part of the map layerof the localization and mapping enginesuch that the event detection engineof the edge devicecan determine the applicable traffic rule based on the current locationof the edge deviceand the timestampsrecorded.

300 The event detection enginecan dynamically switch detection algorithms (e.g., double parking violation detection algorithms, bus lane violation detection algorithms, etc.) based on the applicable enforcement rules at the time.

8 8 FIGS.A andB 7 7 FIGS.A andC 800 800 506 500 518 708 506 500 708 illustrate one embodiment of a method of calculating a lane occupancy score. In this embodiment, the lane occupancy scorecan be calculated based in part on the translated image coordinatesof the vehicle bounding polygonand the translated image coordinatesof the LOI polygon(see). As previously discussed, the translated image coordinatesof the vehicle bounding polygonand the LOI polygoncan be based on the same uniform coordinate domain (for example, a coordinate domain of the video frame originally captured).

8 8 FIGS.A andB 500 500 802 802 500 500 802 500 As shown in, an upper portion of the vehicle bounding polygoncan be discarded or left unused such that only a lower portion of the vehicle bounding polygon(also referred to as a lower bounding polygon) remains. In some embodiments, the lower bounding polygoncan be a truncated version of the vehicle bounding polygonincluding only the bottom 5% to 30% (e.g., 15%) of the vehicle bounding polygon. For example, the lower bounding polygoncan be the bottom 15% of the vehicle bounding polygon.

802 500 500 802 500 802 804 122 124 500 802 As a more specific example, the lower bounding polygoncan be substantially rectangular with a height dimension equal to between 5% to 30% of the height dimension of the vehicle bounding polygonbut with the same width dimension as the vehicle bounding polygon. As another example, the lower bounding polygoncan be substantially rectangular with an area equivalent to between 5% to 30% of the total area of the vehicle bounding polygon. In all such examples, the lower bounding polygoncan encompass the tiresof the potentially offending vehiclecaptured in the event video frame. Moreover, it should be understood by one of ordinary skill in the art that although the word “box” is sometimes used to refer to the vehicle bounding polygonand the lower bounding polygon, the height and width dimensions of such bounding “boxes” do not need to be equal.

800 708 708 708 402 308 402 312 The method of calculating the lane occupancy scorecan also comprise masking the LOI polygonsuch that the entire area within the LOI polygonis filled with pixels. For example, the pixels used to fill the area encompassed by the LOI polygoncan be pixels of a certain color or intensity. In some embodiments, the color or intensity of the pixels can represent or correspond to a confidence level or confidence score of a detection undertaken by the first workerA (from the object detection deep learning model), the second workerB (from the lane segmentation deep learning model), or a combination thereof.

802 312 708 802 708 802 708 The method can further comprise determining a pixel intensity value associated with each pixel within the lower bounding polygon. The pixel intensity value can be a decimal number between 0 and 1. In some embodiments, the pixel intensity value corresponds to a confidence score or confidence level provided by the lane segmentation deep learning modelthat the pixel is part of the LOI polygon. Pixels within the lower bounding polygonthat are located within a region that overlaps with the LOI polygoncan have a pixel intensity value closer to 1. Pixels within the lower bounding polygonthat are located within a region that does not overlap with the LOI polygoncan have a pixel intensity value closer to 0. All other pixels including pixels in a border region between overlapping and non-overlapping regions can have a pixel intensity value in between 0 and 1.

8 FIG.A 122 140 708 708 708 802 500 For example, as shown in, a potentially offending vehiclecan be parked in a no-parking lanethat has been bounded by an LOI polygon. The LOI polygonhas been masked by filling in the area encompassed by the LOI polygonwith pixels. A lower bounding polygonrepresenting a lower portion of the vehicle bounding polygonhas been overlaid on the masked LOI polygon to represent the overlap between the two bounded regions.

8 FIG.A 8 FIG.A 8 FIG.A 8 FIG.A 802 806 806 806 806 1 806 806 2 806 402 806 402 806 402 illustrates three pixels within the lower bounding polygonincluding a first pixelA, a second pixelB, and a third pixelC. Based on the scenario shown in, the first pixelA is within an overlap region (shown as Ain), the second pixelB is located on a border of the overlap region, and the third pixelC is located in a non-overlapping region (shown as Ain). In this case, the first pixelA can have a pixel intensity value of about 0.99 (for example, as provided by the second workerB), the second pixelB can have a pixel intensity value of about 0.65 (as provided by the second workerB), and the third pixelC can have a pixel intensity value of about 0.09 (also provided by the second workerB).

8 FIG.B 8 FIG.B 8 FIG.B 8 FIG.B 122 140 708 122 140 808 808 808 808 1 808 808 2 808 402 808 402 808 402 illustrates an alternative scenario where a potentially offending vehicleis parked in a lane adjacent to a no-parking lanethat has been bounded by an LOI polygon. In this scenario, the potentially offending vehicleis not actually in the no-parking lane. Three pixels are also shown inincluding a first pixelA, a second pixelB, and a third pixelC. The first pixelA is within a non-overlapping region (shown as Ain), the second pixelB is located on a border of the non-overlapping region, and the third pixelC is located in an overlap region (shown as Ain). In this case, the first pixelA can have a pixel intensity value of about 0.09 (for example, as provided by the second workerB), the second pixelB can have a pixel intensity value of about 0.25 (as provided by the second workerB), and the third pixelC can have a pixel intensity value of about 0.79 (also provided by the second workerB).

800 800 802 800 708 802 With these pixel intensity values determined, a lane occupancy scorecan be calculated. The lane occupancy scorecan be calculated by taking an average of the pixel intensity values of all pixels within each of the lower bounding polygons. The lane occupancy scorecan also be considered the mean mask intensity value of the portion of the LOI polygonwithin the lower bounding polygon.

800 For example, the lane occupancy scorecan be calculated using Formula I below:

802 708 140 402 312 where n is the number of pixels within the lower portion of the vehicle bounding polygon (or lower bounding polygon) and where the Pixel Intensity Value; is a confidence level or confidence score associated with each of the pixels within the LOI polygonrelating to a likelihood that the pixel is depicting part of a no-parking lane such as a no-parking lane. The pixel intensity values can be provided by the second workerB using the lane segmentation deep learning model.

800 The method can further comprise detecting a double parking violation when the lane occupancy scoreexceeds a predetermined threshold value.

8 8 FIGS.A andB 8 FIG.A 8 FIG.B 8 FIG.A 8 FIG.B 800 122 800 122 800 402 300 800 136 104 138 402 Going back to the scenarios shown in, the lane occupancy scoreof the potentially offending vehicleshown incan be calculated as approximately 0.89 while the lane occupancy scoreof the potentially offending vehicleshown incan be calculated as approximately 0.19. In both cases, the predetermined threshold value for the lane occupancy scorecan be set at 0.75. With respect to the scenario shown in, the third workerC of the event detection enginecan calculate the lane occupancy scoreand determine that a double parking violation has occurred and can begin to generate an evidence packageto be sent to the serveror a third-party computing device/client device. With respect to the scenario shown in, the third workerC can determine that a double parking violation has not occurred.

313 102 313 104 500 122 9 FIG. In some embodiments, a vehicle movement classifierrunning on the edge deviceor another instance of the vehicle movement classifierrunning on the servercan track vehicle bounding polygonsacross multiple video frames to determine whether the potentially offending vehicleis moving or static (see, e.g.,).

9 FIG. 124 122 124 500 124 313 102 500 124 114 102 500 illustrates example event video frameswhere a potentially offending vehicleappearing in such event video framesis bounded by a vehicle bounding polygonin each of the event video frames. The vehicle movement classifieror vehicle movement classification module of the edge devicecan track the vehicle bounding polygonsacross multiple event video framescaptured by the event cameraof the edge device. The vehicle bounding polygonscan be connected across multiple frames using a tracking algorithm such as a mixed integer linear programming (MILP) algorithm.

In other embodiments, the tracking algorithm can be the Hungarian Algorithm, an Intersection-over-Union (IOU) algorithm, a centroid distance algorithm, or a tracking-by-detection algorithm.

500 122 114 114 The vehicle bounding polygonscan be tracked as part of a method for determining whether the potentially offending vehicleis moving or static. A “moving vehicle” is a vehicle that is determined to be moving when captured in an event video by the event camera. A “static vehicle” is a vehicle that is determined to be not moving (e.g., the vehicle is parked or otherwise stopped temporarily) when captured in the event video by the event camera.

140 It is important to differentiate between a vehicle that is moving and a vehicle that is static because a moving vehicle detected within a no-parking lanecannot be assessed a double parking violation.

122 118 102 124 122 500 124 124 114 110 124 114 122 500 124 110 118 102 110 114 102 124 500 110 102 500 122 110 102 110 102 500 2 FIG.A 10 FIG.A The method of determining whether the potentially offending vehicleis moving or static can comprise the step of first associating GPS coordinates (obtained from the positioning unitof the edge device, see) with timestamps of each of the event video frames. The GPS coordinates of the potentially offending vehiclebounded by the vehicle bounding polygoncan then be computed or determined from the event video framesusing a homography localization algorithm (seefor a visualization of the GPS coordinates overlaid on a map graphic). For example, a transformation matrix can be calculated that transforms 2D points in the event video frame(or the image plane of the event camera) to road or GPS coordinates with respect to the carrier vehicle. The transformation matrix can be calculated by measuring GPS coordinates and associating such GPS coordinates with 2D points in the event video frame(or the image plane of the event camera). An optimization process can then be used to derive the transformation matrix. The GPS coordinates of the potentially offending vehiclebounded by the vehicle bounding polygoncan then be computed or determined from the event video framesusing the derived transformation matrix and using the GPS coordinates of the carrier vehicle(which can be obtained using the positioning unitof the edge devicemounted to the carrier vehicle). Since the event cameraof the edge deviceis also the origin point of the event video frame, the location of the vehicle bounding polygonin the image plane can be transformed into GPS coordinates based on the GPS location of the carrier vehicle(or edge device). As a more specific example, if the vehicle bounding polygonbounding the potentially offending vehicleis 5 meters in front and 3 meters to the right of the carrier vehicle(or edge device), the GPS coordinates of the carrier vehicle(or edge device) can be adjusted by this amount to obtain the GPS coordinates of the vehicle bounding polygon.

500 124 122 The GPS coordinates of the vehicle bounding polygonsacross multiple event video framescan then be tracked and a cluster or set of GPS coordinates tracking the potentially offending vehiclecan be transformed or converted into a local Cartesian coordinate system {L}.

10 FIG.A 10 FIG.B 1000 122 1002 1000 122 1000 1004 illustrates the tracked GPS coordinatesof the potentially offending vehiclevisualized on a map graphic. Each dot on the map graphic represents a computed location (in the form of GPS coordinates) of the potentially offending vehicleper frame. As previously discussed, the GPS coordinatescan then be transformed or converted into a local Cartesian coordinate system(see).

10 FIG.B 10 FIG.A 1004 1000 1006 1004 illustrates an example of a local Cartesian coordinate systemwhere the GPS coordinatesofare transformed or converted and then plotted as transformed coordinatesin the local Cartesian coordinate system.

1004 1004 110 1000 1004 The local Cartesian coordinate systemcan be centered at an average GPS coordinate (which can be considered a naïve mean at such a small local scale), the Z-axis of the coordinate systemcan be pointed in a vertical direction, the Y-axis can be aligned with the heading or direction of travel of the carrier vehicle, and the X-axis can be in a lateral direction (according to the right-hand rule). The purpose of converting or normalizing the GPS coordinatesusing the local Cartesian coordinate systemis to better align the event trajectory data across different events and to eliminate the effects that different street directions have on how the data is used.

11 11 FIGS.A andB 11 11 FIGS.A andB illustrate normalized trajectories of a moving vehicle and a static vehicle, respectively, visualized on separate map graphics. As shown in, the trajectory of the moving vehicle is noticeably different from the trajectory of the static vehicle. The trajectories can be visualized using a GPS visualizer such as a keyhole markup language (KML) visualizer.

12 FIG. 110 102 122 122 110 is a histogram illustrating the number of times that a carrier vehiclecarrying an edge devicetraveled a particular distance when detecting a static vehicle (e.g., a potentially offending vehiclethat is static) versus a moving vehicle (e.g., a potentially offending vehiclethat is moving). As can be seen from the histogram, an initial observation can be made that carrier vehiclesalmost never moved more than 35 meters (>35 meters) when detecting a static vehicle. As such, a threshold of about 35 meters can be used to establish a naïve classifier when it comes to classifying a static vehicle versus a moving vehicle.

13 13 FIGS.A-C 1006 1000 1004 1006 1006 are pairwise 2D scatter plots showing three features selected to differentiate between moving vehicles and static vehicles using the transformed coordinates(the GPS coordinatesthat were transformed/normalized into the local Cartesian coordinate system). The three features include: (1) a standard deviation (SD) of the latitudinal coordinates (hereinafter “std_latitudinal”) where the latitudinal coordinates are the X values of the transformed coordinates; (2) SD of the longitudinal coordinates (hereinafter “std_longitudinal”) where the longitudinal coordinates are the Y values of the transformed coordinates; and a (3) correlation of the latitudinal and longitudinal SDs (hereinafter “cross_corr_std”) representing how correlated are the coordinates along each of the two directions (X and Y).

13 13 FIGS.A-C 122 The scatter plots ofillustrate that moving vehicles and static vehicles exhibit noticeably different movement patterns. As such, the three features (std_latitudinal, std_longitudinal, and cross_corr_std) can be provided as inputs to a logistic regression model or another type of binary classification model to predict whether a potentially offending vehicleis static or moving. Other models that can be used comprise support vector machines (SVMs), decision trees, convolutional neural networks (CNNs), transformers, or Hidden Markov models (HMMs).

122 1000 122 1004 In some embodiments, the whole GPS trajectory could be fed to the model. This makes the problem a sequence classification problem with a variable length input sequence. The entire trajectory could be fed to a Long Short-Term Memory (LSTM) network or another type of sequence classification network to predict whether a potentially offending vehicleis static or moving using the computed GPS coordinatesof the potentially offending vehiclethat have been transformed or converted into the local Cartesian coordinate system.

102 313 104 318 In some embodiments, the logistic regression model can be running on the edge device(for example, as part of the vehicle movement classifier). In other embodiments, the logistic regression model can be running on the serveras part of the evidence validation module.

1400 14 FIG. The logistic regression model can receive as inputs the calculated values of the std_latitudinal feature, the std_longitudinal feature, and the cross_corr_std feature. The logistic regression model can output a probability estimate or prediction score(see) that can be used to make a prediction concerning whether the potentially offending vehicle is moving or static.

14 FIG. 1400 1400 is a histogram of prediction scoresfor 190 ground truth annotated events involving detection of static and moving vehicles. As depicted in the histogram, the prediction scores(the “is_moving_score”) were mostly concentrated on two ends of the 0 to 1 interval. As such, a threshold of 0.5 was selected as the threshold for prediction.

With the threshold being 0.5, the 190 events were classified with a precision of 0.986, a recall of 0.897, and an f1_score of 0.940 based on the ground truths of such events.

124 In summary, by providing the three features of std_latitudinal, std_longitudinal, and cross_corr_std as inputs to a logistic regression model, 70 out of the 78 cases with a moving vehicle was correctly identified and only 1 out of 112 cases with a static vehicle was misidentified. This single false positive detection involved a very long vehicle (a bus) with a detection polygon that was positioned near an edge of the event video frame. Such cases can be mitigated by excluding certain bus classes or using a different type of detection polygon. Moreover, 7 out of the 78 cases of moving vehicles were misidentified with 5 of them due to the small location range caused by short trajectories (very small movement distances).

122 122 122 140 140 As will be discussed in more detail in the following sections, even if a potentially offending vehicleis determined to be static, there are certain extenuating circumstances that would prevent the potentially offending vehiclefrom being considered double-parked. For example, these include instances where the potentially offending vehicleis stuck in traffic due to other vehicles in the no-parking laneor is stopped temporarily in the no-parking laneat an intersection.

122 One technical problem when it comes to detection of double parking violations is determining whether the potentially offending vehicleis static or moving since a moving vehicle cannot be assessed a double parking violation. One technical solution discovered and developed by the applicant is to determine GPS coordinates of vehicle bounding polygons across multiple event video frames, converting or transforming these GPS coordinates into a local Cartesian coordinate system (such that the GPS coordinates are transformed coordinates), and then determining whether the vehicle is static or moving by based on a standard deviation of the transformed coordinates in both a longitudinal and a latitudinal direction and a cross correlation of the transformed coordinates. For example, the standard deviation of the transformed coordinates in both the longitudinal and latitudinal direction and the cross correlation of the transformed coordinates can be fed as inputs to a logistic regression model to obtain an output in the form of a prediction score concerning whether the vehicle is moving.

15 FIG. 308 312 102 104 122 140 122 122 122 140 140 is a schematic diagram illustrating that the object detection deep learning modeland the lane segmentation deep learning model, running on at least one of the edge deviceand the server, can be used to output predictions or classification results concerning a context surrounding the potential double parking violation that can assist in determining whether the potentially offending vehicleis actually parked in the no-parking laneor is only temporarily stopped. As previously discussed, the potentially offending vehiclecan be determined as being static as opposed to moving. However, there are instances where the potentially offending vehiclecan be static without being actually parked. For example, these include instances where the potentially offending vehicleis stuck in traffic due to other vehicles in the no-parking laneor is stopped temporarily in the no-parking laneat an intersection (e.g., is about to make a turn at the intersection). Thus, being able to gain more insights into the context surrounding the potential double parking violation can cut down on the likelihood of a false positive detection.

308 312 1500 1500 1502 122 1504 122 1506 The object detection deep learning model, the lane segmentation deep learning model, or a combination thereof can be configured to output predictions or classification results concerning featuresassociated with the context surrounding the potential double parking violation. Such featurescan comprise a brake light statusof the potentially offending vehicle, a traffic conditionsurrounding the potentially offending vehicle, and a roadway intersection status.

308 1502 122 1504 122 312 1506 In some embodiments, the object detection deep learning modelcan be trained to classify or make predictions concerning the brake light statusof the potentially offending vehicleand the traffic conditionsurrounding the potentially offending vehicle. In these and other embodiments, the lane segmentation deep learning modelcan be trained to classify or make predictions concerning the roadway intersection status.

308 1508 1508 In certain embodiments, the objection detection deep learning modelcan comprise a plurality of prediction headsor detectors. The prediction headsor detectors can be multi-class detectors that can be configured to undertake a multi-class prediction.

15 FIG. 1508 308 1508 122 122 140 122 122 140 As shown in, one of the prediction headsof the object detection deep learning model(e.g., prediction headA) can be trained to distinguish between brake lights that are on, brake lights that are off, and brake lights that are flashing. For example, if the brake lights of a potentially offending vehicleare off, it is more likely that the potentially offending vehiclestopped in the no-parking laneis parked rather than temporarily stopped. On other hand, if the brake lights of the potentially offending vehicleare on or are flashing, it is more likely that the potentially offending vehiclestopped in the no-parking laneis only temporarily stopped.

308 124 102 1508 1508 1512 1512 136 104 The object detection deep learning modelcan receive as inputs event video framescaptured by the edge device. The prediction headA can classify the input video frames into one of the following classes: (1) brake lights that are on (lights_on); (2) brake lights that are off (lights_off); and (3) brake lights that are flashing (lights_flashing). The prediction headA can also generate a set of brake light confidence scoresassociated with the predictions or classifications. The brake light confidence scorescan be included as part of a set of classification results included in the evidence packagetransmitted to the serveror to a third-party evidence processor.

15 FIG. 1508 308 1508 122 122 140 140 122 Also, as shown in, another one of the prediction headsof the object detection deep learning model(e.g., prediction headB) can be trained to detect a traffic condition surrounding the potentially offending vehicleby determining whether one or more other vehicles are immediately in front of the potentially offending vehiclein the no-parking laneor whether the no-parking laneis otherwise clear with no other vehicles impeding the movement of the potentially offending vehicle.

122 122 122 140 122 For example, if one or more other vehicles are immediately in front of the potentially offending vehicle, the presence of such vehicles can indicate that the movement of the potentially offending vehicleis impeded or blocked (possibly due to a traffic jam, an incident, or a traffic light being red, etc.). On other hand, if no vehicles are detected immediately in front of the potentially offending vehicleand the no-parking laneis otherwise clear, this can indicate that the potentially offending vehicleis double-parked.

308 124 102 1508 1508 1514 1514 136 104 1508 122 122 The object detection deep learning modelcan receive as inputs event video framescaptured by the edge device. The prediction headB can classify the input video frames into one of the following classes: (1) car(s) in front (car_in_front), (2) no car(s) in front (lane_clear), or (3) traffic accident ahead (traff_accident). The prediction headB can also generate a set of traffic condition confidence scoresassociated with the predictions or classifications. The traffic condition confidence scorescan be included as part of a set of classification results included in the evidence packagetransmitted to the serveror to a third-party evidence processor. In other embodiments, one of the prediction headscan also detect whether the offending vehicleis an emergency vehicle by classifying the vehicle type of the offending vehicle.

15 FIG. 600 312 600 124 122 140 122 140 As shown in, one of the prediction headsof the lane segmentation deep learning model(e.g., prediction headA) can be trained to detect whether an intersection is present in at least one of the event video framescapturing the potential double-parking violation. For example, if an intersection is detected, it is more likely that the potentially offending vehiclestopped in the no-parking lanein order to make a turn at the intersection. On other hand, if no intersection is detected, it is more likely that the potentially offending vehiclestopped in the no-parking laneis double parked.

312 124 102 600 600 1516 1516 136 104 1516 312 124 122 140 The lane segmentation deep learning modelcan receive as inputs event video framescaptured by the edge device. The prediction headA can classify the input video frames into one of the following classes: (1) traffic light detected (light_detected), (2) traffic light not detected (no_light), (3) stop sign detected (sign_detected), or (4) stop sign not detected (no_sign). The prediction headA can also generate a set of intersection detection confidence scoresassociated with the predictions or classifications. The intersection detection confidence scorescan be included as part of a set of classification results included in the evidence packagetransmitted to the serveror to a third-party evidence processor. The classification results and/or the intersection detection confidence scorescan be used by the lane segmentation deep learning modelto determine whether an intersection was detected in at least one of the event video framescapturing the potential double-parking violation. As previously discussed, if an intersection was detected, it is more likely that the potentially offending vehiclewas stopped in the no-parking lanein order to make a turn at the intersection.

1512 1514 1516 122 140 1512 1514 1516 136 In some embodiments, the brake light confidence scores, the traffic condition confidence scores, the intersection detection confidence scores, or a combination thereof can be used directly to determine whether the potentially offending vehiclestopped in the no-parking lane(i.e., the static vehicle) is only temporarily stopped or is actually double-parked. For example, one or more thresholds can be set and if at least one of the brake light confidence scores, the traffic condition confidence scores, or the intersection detection confidence scoresdo not meet one of the threshold or exceeds one of the thresholds, the evidence packagecan be rejected.

1512 1514 1516 104 104 136 1512 1514 1516 In other embodiments, the brake light confidence scores, the traffic condition confidence scores, the intersection detection confidence scores, or a combination thereof can also be provided as inputs to a decision tree algorithm running on the serveras part of an evidence evaluation procedure. The servercan automatically approve or reject the evidence packagebased on a final score calculated by the decision tree algorithm. The brake light confidence scores, the traffic condition confidence scores, and the intersection detection confidence scorescan be factored into the calculation of the final score.

308 312 1518 124 102 124 316 308 312 124 316 124 136 104 138 The objection detection deep learning modeland the lane segmentation deep learning modelcan be trained using training datacomprising event video framespreviously captured by the edge devicesor event video framesstored in an events database. The objection detection deep learning modeland the lane segmentation deep learning modelcan be continuously trained in order to improve the accuracy and efficacy of such models. The event video framesretrieved from the events databasecan be event video frameswhere the evidence packagescontaining such video frames were previously validated by the server, a computing deviceof a third-party processor, a human reviewer, or a combination thereof.

A number of embodiments have been described. Nevertheless, it will be understood by one of ordinary skill in the art that various changes and modifications can be made to this disclosure without departing from the spirit and scope of the embodiments. Elements of systems, devices, apparatus, and methods shown with any embodiment are exemplary for the specific embodiment and can be used in combination or otherwise on other embodiments within this disclosure. For example, the steps of any methods depicted in the figures or described in this disclosure do not require the particular order or sequential order shown or described to achieve the desired results. In addition, other steps operations may be provided, or steps or operations may be eliminated or omitted from the described methods or processes to achieve the desired results. Moreover, any components or parts of any apparatus or systems described in this disclosure or depicted in the figures may be removed, eliminated, or omitted to achieve the desired results. In addition, certain components or parts of the systems, devices, or apparatus shown or described herein have been omitted for the sake of succinctness and clarity.

Accordingly, other embodiments are within the scope of the following claims and the specification and/or drawings may be regarded in an illustrative rather than a restrictive sense.

Each of the individual variations or embodiments described and illustrated herein has discrete components and features which may be readily separated from or combined with the features of any of the other variations or embodiments. Modifications may be made to adapt a particular situation, material, composition of matter, process, process act(s) or step(s) to the objective(s), spirit, or scope of the present invention.

Methods recited herein may be carried out in any order of the recited events that is logically possible, as well as the recited order of events. Moreover, additional steps or operations may be provided or steps or operations may be eliminated to achieve the desired result.

Furthermore, where a range of values is provided, every intervening value between the upper and lower limit of that range and any other stated or intervening value in that stated range is encompassed within the invention. Also, any optional feature of the inventive variations described may be set forth and claimed independently, or in combination with any one or more of the features described herein. For example, a description of a range from 1 to 5 should be considered to have disclosed subranges such as from 1 to 3, from 1 to 4, from 2 to 4, from 2 to 5, from 3 to 5, etc. as well as individual numbers within that range, for example 1.5, 2.5, etc. and any whole or partial increments therebetween.

All existing subject matter mentioned herein (e.g., publications, patents, patent applications) is incorporated by reference herein in its entirety except insofar as the subject matter may conflict with that of the present invention (in which case what is present herein shall prevail). The referenced items are provided solely for their disclosure prior to the filing date of the present application. Nothing herein is to be construed as an admission that the present invention is not entitled to antedate such material by virtue of prior invention.

Reference to a singular item includes the possibility that there are plural of the same items present. More specifically, as used herein and in the appended claims, the singular forms “a,” “an,” “said” and “the” include plural referents unless the context clearly dictates otherwise. It is further noted that the claims may be drafted to exclude any optional element. As such, this statement is intended to serve as antecedent basis for use of such exclusive terminology as “solely,” “only” and the like in connection with the recitation of claim elements, or use of a “negative” limitation. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

Reference to the phrase “at least one of”, when such phrase modifies a plurality of items or components (or an enumerated list of items or components) means any combination of one or more of those items or components. For example, the phrase “at least one of A, B, and C” means: (i) A; (ii) B; (iii) C; (iv) A, B, and C; (v) A and B; (vi) B and C; or (vii) A and C.

In understanding the scope of the present disclosure, the term “comprising” and its derivatives, as used herein, are intended to be open-ended terms that specify the presence of the stated features, elements, components, groups, integers, and/or steps, but do not exclude the presence of other unstated features, elements, components, groups, integers and/or steps. The foregoing also applies to words having similar meanings such as the terms, “including”, “having” and their derivatives. Also, the terms “part,” “section,” “portion,” “member” “element,” or “component” when used in the singular can have the dual meaning of a single part or a plurality of parts. As used herein, the following directional terms “forward, rearward, above, downward, vertical, horizontal, below, transverse, laterally, and vertically” as well as any other similar directional terms refer to those positions of a device or piece of equipment or those directions of the device or piece of equipment being translated or moved.

Finally, terms of degree such as “substantially”, “about” and “approximately” as used herein mean the specified value or the specified value and a reasonable amount of deviation from the specified value (e.g., a deviation of up to +0.1%, +1%, +5%, or +10%, as such variations are appropriate) such that the end result is not significantly or materially changed. For example, “about 1.0 cm” can be interpreted to mean “1.0 cm” or between “0.9 cm and 1.1 cm.” When terms of degree such as “about” or “approximately” are used to refer to numbers or values that are part of a range, the term can be used to modify both the minimum and maximum numbers or values.

The term “engine” or “module” as used herein can refer to software, firmware, hardware, or a combination thereof. In the case of a software implementation, for instance, these may represent program code that performs specified tasks when executed on a processor (e.g., CPU, GPU, or processor cores therein). The program code can be stored in one or more computer-readable memory or storage devices. Any references to a function, task, or operation performed by an “engine” or “module” can also refer to one or more processors of a device or server programmed to execute such program code to perform the function, task, or operation.

It will be understood by one of ordinary skill in the art that the various methods disclosed herein may be embodied in a non-transitory readable medium, machine-readable medium, and/or a machine accessible medium comprising instructions compatible, readable, and/or executable by a processor or server processor of a machine, device, or computing device. The structures and modules in the figures may be shown as distinct and communicating with only a few specific structures and not others. The structures may be merged with each other, may perform overlapping functions, and may communicate with other structures not shown to be connected in the figures. Accordingly, the specification and/or drawings may be regarded in an illustrative rather than a restrictive sense.

This disclosure is not intended to be limited to the scope of the particular forms set forth, but is intended to cover alternatives, modifications, and equivalents of the variations or embodiments described herein. Further, the scope of the disclosure fully encompasses other variations or embodiments that may become obvious to those skilled in the art in view of this disclosure.