Appliance and Camera Based User Interface

The present disclosure is related generally to appliances and methods for operating an appliance.

Appliances generally include user control interfaces electrically coupled to an appliance controller and fixed into surface of the appliance, such as a cabinet or door. User control interfaces generally include wires that route through the appliance to the user control interface, such as to provide energy for operation and to transmit and receive control signals. User control interfaces generally include buttons, knobs, switches, haptic feedback devices, touchscreens, or other electrical or mechanical user input interfaces. The user input interfaces are communicatively coupled to the appliance controller to transmit and receive signals for operating components at the appliance.

Additionally, user control interfaces may require insulation and seals to protect the user control interface and associated electrical couplings from various environmental conditions, such as moisture, heat, or cold. Accordingly, appliances may require space, volume, and interfaces for accommodating insulation, seals, and protective structures at the user control interface, which may add cost and complexity to the appliance, and limit potential places at which the user control interface may be positioned.

Additionally, or alternatively, electrical couplings and electrical or mechanical user input interfaces may be prone to degradation and deterioration through repeated usage, which may limit appliance functionality or require costly repair or replacement.

As such, there is a need for improved user control interfaces that address one or more of the aforementioned issues.

Aspects and advantages of the invention will be set forth in part in the following description, or may be apparent from the description, or may be learned through practice of the invention.

An aspect of the present disclosure is directed to an appliance computing device including a processor operably coupled to an imaging device. The processor is configured to execute instructions that cause operations. The operations include generating a virtual user interface relative to a surface within a field of view of the imaging device. The virtual user interface includes a digital marker corresponding to an appliance control function. The operations further include generating a control signal based on a physical user interaction relative to the digital marker.

An aspect of the present disclosure is directed to a method for operating an appliance. The method includes generating a virtual user interface corresponding to a surface of the appliance within a field of view of an imaging device operably coupled to an appliance computing device, the virtual user interface including a digital marker corresponding to an appliance control function; generating a control signal based on a physical user interaction relative to the digital marker; and transmitting the control signal to articulate an operational component of the appliance, wherein articulation of the operational component corresponds to the appliance control function.

An aspect of the present disclosure is directed to an appliance. The appliance includes a surface, an operational component configured to perform a control function, an imaging device configured with a field of view toward the surface, and a computing device including a processor operably coupled to the imaging device. The processor is configured to execute operations, the operations including generating a virtual user interface corresponding to the surface within the field of view of the imaging device, the virtual user interface including a digital marker corresponding to the control function; generating a control signal based on a physical user interaction relative to the digital marker; and transmitting the control signal to perform the control function at the operational component.

These and other features, aspects and advantages of the present invention will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

Repeat use of reference characters in the present specification and drawings is intended to represent the same or analogous features or elements of the present invention.

Reference now will be made in detail to embodiments of the present invention, one or more examples of which are illustrated in the drawings. The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any implementation described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other implementations. Moreover, each example is provided by way of explanation, rather than limitation of, the technology. In fact, it will be apparent to those skilled in the art that modifications and variations can be made in the present technology without departing from the scope or spirit of the claimed technology. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure covers such modifications and variations as come within the scope of the appended claims and their equivalents. The detailed description uses numerical and letter designations to refer to features in the drawings. Like or similar designations in the drawings and description have been used to refer to like or similar parts of the invention.

As used herein, the terms “first”, “second”, and “third” may be used interchangeably to distinguish one component from another and are not intended to signify location or importance of the individual components. The singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise. The terms “coupled,” “fixed,” “attached to,” and the like refer to both direct coupling, fixing, or attaching, as well as indirect coupling, fixing, or attaching through one or more intermediate components or features, unless otherwise specified herein. As used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of features is not necessarily limited only to those features but may include other features not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, “or” refers to an inclusive-or and not to an exclusive-or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).

Terms of approximation, such as “about,” “generally,” “approximately,” or “substantially,” include values within ten percent greater or less than the stated value. When used in the context of an angle or direction, such terms include within ten degrees greater or less than the stated angle or direction. For example, “generally vertical” includes directions within ten degrees of vertical in any direction, e.g., clockwise or counter-clockwise.

Benefits, other advantages, and solutions to problems are described below with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and any feature(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature of any or all the claims.

Embodiments of a refrigerator appliance, a computing device, and a method for operating an appliance are provided that may address one or more of the aforementioned issues. Embodiments of the appliance include an imaging device, such as a camera, positioned with a field of view at a surface, such as a surface of the appliance, and a computing device configured to execute instructions that cause the appliance to perform operations. The operations generate a virtual user interface at or proximate to the surface within the field of view of the imaging device. Control signals are generated based on physical user interaction at the virtual user interface.

Embodiments provided herein may allow for development and positioning of user control interfaces electrically uncoupled from physical user interfaces that generally include wired connections for power supply and/or electronic communication, and/or knobs, switches, buttons, or other physical interfaces that may be electrically or mechanically coupled to a controller. Electrical uncoupling of the physical user interface from the controller may allow for enhanced aesthetic designs and placement of user control interfaces at various surfaces. The virtual user interface may remove a need for electronic control boards, physical interfaces, wires, and sensors as may generally be associated with physical user control interfaces, which may further reduce cost and complexity, remove a need for insulation and seals associated with user interfaces, and improve durability and longevity by removing degradable components.

depict an applianceaccording to an exemplary embodiment of the present subject matter.depict the applianceas a refrigerator appliance. However, referring to, it should be appreciated that appliance, and appliance controllers and methods further described herein, may include appliances generally including an interior or exterior surfacephysically accessible to a user, an operational componentconfigured to perform a control function, an imaging deviceconfigured with a field of viewtoward the surface, and a computing deviceoperably coupled to the imaging deviceand the operational component. For instance, the operational componentcan include dispensers (e.g., water and/or ice dispenser), motors, actuators, heating and/or cooling apparatuses (e.g., heating elements, heat exchangers, etc.), microwave devices, washer and/or dryer devices, or other assemblies and components generally associated with household or commercial appliances. Control functions can include dispensing a fluid or ice, generating or removing heat, performing a cleaning, washing, or drying cycle, or other function generally associated with household or commercial appliances.

provides an exemplary perspective view of the applianceconfigured as a side-by-side refrigerator appliance including a refrigeration chamber and a freezer chamber.provides a perspective view of the applianceofwith doors,open to view refrigeration and freezer compartments,formed within a cabinet or outer case.provides a reference vertical direction V and width direction W.

Referring to, applianceincludes the refrigeration compartmentand the freezer compartment, with the compartments arranged side-by-side and contained within the outer case. Outer caseand inner linersandare generally molded from a suitable plastic or foam material. For instance, the outer case may form a molded plastic or foam outer case. Thus, applianceis generally referred to as a side-by-side style refrigerator appliance. In alternative exemplary embodiments, appliancemay include a single liner and a mullion that spans between opposite sides of the single liner to divide it into the freezer compartment, such as a compartment configured for frozen foods, and the refrigeration compartment, such as a compartment configured for fresh foods. Outer caseis normally formed by folding a sheet of a suitable material, such as pre-painted steel, into an inverted U-shape to form top and side walls of outer case. A bottom wall of outer casenormally is formed separately and attached to the case side walls and to a bottom frame that provides support for appliance. However, it should be appreciated that outer casemay be formed by other suitable manufacturing methods.

A breaker stripextends between a case front flange and outer front edges of inner linersand. Breaker stripis formed from a suitable resilient material, such as an extruded acrylo-butadiene-styrene based material (commonly referred to as ABS). The insulation in the space between inner linersandis covered by another strip of suitable resilient material, which also commonly is referred to as a mullionand may be formed of an extruded ABS material. Breaker stripand mullionmay form a front face, and extend completely around inner peripheral edges of outer caseand vertically between inner linersand.

Slide-out drawersand shelvesare normally provided in refrigeration compartmentto support items being stored therein. In addition, a shelf, a basket, or both, are generally provided in freezer compartment.

Referring to, a freezer doorand a fresh food doorclose access openings to freezer compartmentand refrigeration compartment. Freezer door and fresh food doorandare each mounted by a top hingeand a bottom hinge (not shown) to rotate about its outer vertical edge between an open position, as shown in, and a closed position. Freezer doormay include a plurality of storage shelvesand a sealing gasket, and fresh food dooralso includes a plurality of storage shelvesand a sealing gasket.

Freezer compartmentmay include an automatic ice makerand a dispenser assemblyprovided in freezer doorsuch that ice and/or chilled water can be dispensed without opening freezer door, as is well known in the art. Freezer door and fresh food doorandmay be opened by handles. It should be appreciated that the dispenser assemblymay be positioned at an exterior portion of the door, such as depicted in, or positioned in an interior portion, such as to require opening the door to access the dispenser assembly.

Appliancealso includes a machinery compartment (not shown) that includes one or more operational components. For instance, the machinery compartment may at least partially contain a cooling system including components for executing a known vapor compression cycle for cooling air. The cooling system may include a compressor, a condenser, an expansion device, and an evaporator connected in series as a loop and charged with a refrigerant. The evaporator is a type of heat exchanger which transfers heat from air passing over the evaporator to the refrigerant flowing through the evaporator, thereby causing the refrigerant to vaporize. The cooled air is used to refrigerate one or more refrigerator or freezer compartments via fans. Also, a cooling loop can be added to direct cool the ice maker to form ice cubes, and a heating loop can be added to help remove ice from the ice maker. Collectively, the vapor compression cycle components in a refrigeration circuit, associated fans, and associated compartments are conventionally referred to as a sealed system. The construction and operation of the sealed system are well known to those skilled in the art.

Although the applianceofis depicted and described as a side-by-side refrigerator appliance, it should be appreciated that present disclosure may apply to other refrigerator appliance configurations, such as, but not limited to, top-bottom configurations, two-compartment configurations, three-compartment compartment configurations, dispenser assembly within the cabinet, refrigerator-only configuration, freezer-only configuration, etc. Furthermore, as provided herein, appliancemay be configured as other commercial or household appliances, such as, but not limited to, stovetop or oven cooking appliances, microwave appliances, dishwasher appliances, laundry appliances, etc.

Referring now to, a schematic view of an exemplary embodiment of applianceis provided. Applianceincludes a computing deviceconfigured to receive or store instructions that, when executed, cause the appliance, or operational componentsthereof, to perform operations. In various embodiments, appliancefeatures are regulated with computing device. Input/output (“I/O”) signals may be routed between computing deviceand various operational componentsof appliance. The operational componentsmay be in communication with computing devicevia one or more wired or wireless signal lines or shared communication busses.

Imaging deviceis configured with a field of view toward a surface (such as field of viewtoward surfacedepicted schematically in). Imaging deviceis communicatively coupled to computing deviceto transmit images obtained by imaging device. In various embodiments, imaging deviceincludes a camera or other device configured to generate an electronic or digital image within the field of view. In some embodiments, such as depicted in, imaging devicemay be positioned at dispenser assembly. In still some embodiments, such as depicted in, imaging devicemay be positioned at handle.

Computing devicecan include any device that includes one or more processorsand memory. As an example, in some embodiments, computing devicemay be a single board computer (SBC). For example, computing devicecan be a single System-On-Chip (SOC). However, any form of computing devicemay also be used to perform the present subject matter. The processor(s)can be any suitable processing device, such as a microprocessor, microcontroller, integrated circuit, or other suitable processing devices or combinations thereof. The memorycan include any suitable storage media, including, but not limited to, non-transitory computer-readable media, RAM, ROM, hard drives, flash drives, accessible databases, or other memory devices. The memorycan store information accessible by processor(s), including instructions that can be executed by processor(s)to perform aspects of the present disclosure, such as steps of methodprovided herein.

The computing deviceincludes a virtual user interface moduleconfigured to generate a digital marker corresponding to an appliance control function. For instance, an applianceincluding an operational componentcorresponding to a dispenser assemblymay include a first appliance control function corresponding to activating dispensing water is electronically coupled to a first digital marker; a second appliance control function corresponding to activating or deactivating a light is electronically coupled to a second digital marker; a third appliance control function corresponding to dispensing ice is electronically coupled to a third digital marker, etc.

In some embodiments, computing devicefurther includes an image processor. Image processormay include any suitable processing device, and may further be configured for performing image processing tasks. For instance, image processorinclude a segmentation routine configured to partition the image captured by imaging deviceinto one or more parts or objects. The image processormay include a representation routine configured to determine regions, boundaries, or features at the image. The image processormay include a recognition or interpretation routine configured to determine an overlay of a digital representation of physical user interaction (depicted schematically by handin) at the surfaceto one or more digital markers.

Referring now to, a flowchart outlining steps of a method for operating an appliance is provided (hereinafter, “method”). Steps of methodmay generate a virtual user interface at or proximate to a surface (e.g., surface), such as may remove a need for a physical user control interface. Steps of methodmay be stored as instructions at a computing device. For instance, one or more steps of methodmay be stored at memoryof computing deviceand executed by one or more processors,and modules.

Methodincludes atgenerating a virtual user interface relative to a surface within a field of view of the imaging device. The virtual user interface includes a digital marker corresponding to an appliance control function. Generating the virtual user interface relative to the surface may include obtaining, from imaging device, an image of the surfaceupon which the virtual user interface is virtually superimposed. In some embodiments, methodatis performed after a user is detected in proximity to the appliance, such as when the user (e.g., hand) extends within the field of view. In still some embodiments, user proximity may be detected through a proximity sensor, or opening/closing of one or more doors (e.g., doors,). Selective activation of the virtual user interface, such as based on user proximity within the field of view, may reduce energy consumption in contrast to physical user control interfaces.

Referring to, perspective views of an exemplary embodiment of applianceconfigured as a refrigerator appliance are provided. Applianceincludes surfaceat door,. Dispenser assemblyis positioned at door. As described herein, surfacemay be free of physical user control interfaces. Imaging deviceis positioned with a field of viewtoward the surface.depict the applianceofhaving a virtual user interfacevirtually superimposed at or proximate to the surface, such as in accordance with method. The virtual user interfaceincludes a digital marker, depicted schematically at digital marker.

In some embodiments, methodincludes atsuperimposing the digital marker relative to the surface. Superimposing the digital marker relative to the surfacemay include obtaining an image, such as via imaging device, including a digital representation of the surfacewithin the field of view. Digital markersare imposed onto the image and correlated to the surface, such as to form the virtual user interface. Physical user interaction, such as hand, at the surfacecaptured by imaging deviceand formed into the digital representation is spatially correlated to digital markersforming the virtual user interface. It should be appreciated that physical user interaction, although depicted as a hand, may include an object generally, such as, but not limited to, a stylus, a pen, a utensil, etc. In various embodiments, superimposition of the digital marker at or proximate relative to the surface is a virtual superimposition of the digital marker, such as superimposition of the digital marker at or proximate to a digital representation of the surface.

Methodmay include atobtaining an image including a digital representation of the physical user interaction within the field of view of the imaging device. Methodmay include atdetermining a spatial correlation of the digital marker to the digital representation.

In some embodiments, methodincludes atsegmenting the image into one or more parts corresponding to the digital marker, and atdetermining, at the one or more parts, an overlay of the digital representation of the physical user interaction to the digital marker. Methodmay include atdetermining the physical user interaction relative to the digital marker based on the overlay of the digital representation of the physical user interaction to the digital marker.

For instance, referring to, one or more of processor,may be configured to determine the spatial correlation of the digital marker to the digital representation. For instance, the image may be segmented based on the digital markers. Segmenting the image may include segmenting corresponding to digital markers, such as may isolate whether the physical user interaction at the surfacecorresponds to a first, second, third, etc. digital marker.

Determining an overlay of the digital representation of the physical user interaction, such as hand, to the digital markermay be utilized determine whether the user articulated a first, second, third, etc. digital markerto perform a corresponding appliance control function. For instance, relative to the prior example, determining overlay of the digital representation of the physical user interaction (e.g., hand) to a first digital markermay generate a first control signal corresponding to a first appliance control function (e.g., dispensing water). Determining overlay of the handto a second digital markermay generate a second control signal corresponding to a second appliance control function (e.g., activating or deactivating a light). Determining overlay of the handto a third digital markermay generate a third control signal corresponding to a third appliance control function (e.g., dispensing ice), etc.

Methodincludes atgenerating a control signal based on a physical user interaction relative to the digital marker. Referring to, physical user interaction, such as depicted via handof a user, at or proximate to surfacecorresponding to a virtual location of the superimposed digital markergenerates the control signal corresponding to the appliance control function, such as described above.

In some embodiments, methodincludes attransmitting the control signal to articulate an operational component corresponding to the appliance control function. For instance, appliance computing deviceis communicatively coupled to operational componentto transmit the control signal to the operational componentto perform the appliance control function, such as described herein.

Referring to, in some embodiments, appliancemay include a physical markerpositioned at the surface. The physical markerincludes an etching, writing, groove, bump, raised surface, badge, or other physical feature visually perceptible or tactilely perceptible by a user. The physical markeris electrically uncoupled from appliance computing device. For instance, the physical markeris unwired, such as being uncoupled to a power supply, or electric or electronic device, or generally having no physical operable connection.

Referring to, in some embodiments, methodatmay include superimposing the digital marker to the physical marker at the surface. For instance, referring to, methodatmay superimpose the digital markersubstantially to physical marker. Physical markermay include a visual communication (e.g., writing, symbol, pictograph, etc.) corresponding to an appliance control function. As noted above, the physical markeris physically uncoupled to the appliance computing device. However, physical user interaction (e.g., hand) at the digital markervirtually superimposed to the physical markergenerates the control signal corresponding to the appliance control function and visual communication of the physical marker.

It should be appreciated that, in various embodiments, generation of the virtual user interface and virtual superimposition of the digital marker may be performed without requiring physical representation or projection visibly perceptible to the user. For instance, in some embodiments, visible perceptibility by the user of the user interface may be limited to the electrically uncoupled physical markeror surface, and the virtual user interface and digital markers may be visually imperceptible by the user. However, in still some embodiments, such as described herein, the virtual user interface may be visually superimposed onto the surface.

Referring to, in an embodiment, imaging devicefurther includes a projection deviceconfigured to project a light-based representation or image perceptible by a user. Referring to, in some embodiments, methodincludes atprojecting, toward the surface, the virtual user interface as a light-based representation perceptible to a user. For instance, methodatmay include projecting a visual representation of the virtual user interfaceor digital markersonto the surface. The projected visual representation may generate a physical representation or marker at which the user may touch at, or proximate to, the surfaceto generate a control signal such as described herein. For instance, projection devicemay provide a light-based representation of the digital marker of the virtual user interface superimposed at or proximate to the surface.

In still some embodiments, imaging device, projection device, or both, may include a light source configured to articulate toward the surface. For instance, the light source may facilitate acquisition of images of the surfaceor physical user interaction (e.g., hand).

Embodiments of the appliance, computing device, and methodprovided herein may allow for development and positioning of user control interfaces electrically uncoupled from physical user interfaces. Electrical uncoupling of the physical user interface may allow for enhanced aesthetic designs and placement of user control interfaces at various surfaces, remove components that may degrade or deteriorate, or remove a need for electronic control boards, physical interfaces, wires, sensors, insulation, or seals as may generally be associated with physical user control interfaces. Electrical uncoupling of the user control interface may improve durability and longevity by removing degradable components, improve design versatility, and provide replacement interfaces customizable controls without requiring physical hardware installation or changes.

Further aspects of the disclosure are provided in one or more of the following clauses:

1. An appliance computing device including a processor operably coupled to an imaging device, the processor configured to execute instructions that cause operations, the operations including generating a virtual user interface relative to a surface within a field of view of the imaging device, the virtual user interface including a digital marker corresponding to an appliance control function; generating a control signal based on a physical user interaction relative to the digital marker.

2. The appliance computing device of any one or more clauses herein, the operations including obtaining an image including a digital representation of the physical user interaction within the field of view of the imaging device; and determining a spatial correlation of the digital marker to the digital representation.