Information processing device, method, computer-readable medium, and system

The present disclosure relates to an information processing device, an information processing method, a computer-readable medium storing an information processing program, and an information processing system.

An information processing device for a user to input contents, such as characters and pictures, by hand has been known. For example, Japanese Patent Laid-Open No. 2020-024756 (hereinafter, referred to as Patent Document 1) discloses a method by which a user modifies hand-written data once stored in a file while checking how the hand-written data is displayed.

However, the method described in Patent Document 1 requires operations for checking and modifying the data, and involves a corresponding increase in a burden on the user.

The present disclosure has been conceived of in view of the above circumstances to provide an information processing device, an information processing method, a computer-readable medium storing an information processing program, and an information processing system which allow various types of information obtained through a writing operation with an input device to be automatically corrected at appropriate times.

An information processing device according to a first aspect of the present disclosure includes a processor; and a memory storing instructions that, when executed by the processor, cause the information processing device to: accept a cancel operation that cancels a first operation performed through an input device used for writing input, acquire learning information representing a result of learning about a relation between content of the first operation and content of a second operation performed through the input device after the cancel operation accepted by the operation acceptance section, and correct writing information representing a writing product corresponding to a third operation performed through the input device or writing setting information concerning a setting at a time of writing corresponding to the third operation, based on the learning information acquired.

An information processing method according to a second aspect of the present disclosure performed by one or a plurality of computers, include accepting a cancel operation that cancels a first operation performed through an input device used for writing input, acquiring learning information representing a result of learning about a relation between content of the first operation and content of a second operation performed through the input device after the accepted cancel operation, and correcting writing information representing a writing product corresponding to a third operation performed through the input device or writing setting information concerning a setting at a time of writing corresponding to the third operation, based on the learning information.

A non-transitory computer-readable medium according to a third aspect of the present disclosure stores an information processing program that, when executed by one or more computers, causes the one or more computers to: accept a cancel operation that cancels a first operation performed through an input device used for writing input, acquire learning information representing a result of learning about a relation between content of the first operation and content of a second operation performed through the input device after the cancel operation is accepted, and correct writing information representing a writing product corresponding to a third operation performed through the input device or writing setting information concerning a setting at a time of writing corresponding to the third operation, based on the learning information.

An information processing system according to a fourth aspect of the present disclosure includes the information processing device according to the first aspect of the present disclosure, and a server device that, in operation, transmits the learning information toward the information processing device in response to a request from the information processing device.

The first to fourth aspects of the present disclosure allow various types of information obtained through a writing operation with the input device to be automatically corrected at appropriate times.

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In the accompanying drawings, to make it easier to understand the following description, like constituent elements are denoted by like reference characters wherever possible, and redundant description will be omitted.

10 Overall Configuration of Information Processing System

1 FIG. 10 10 10 12 14 16 is an overall configuration diagram of an information processing systemaccording to an embodiment of the present disclosure. The information processing systemis provided to provide an “input support service” for supporting a user in performing a writing input. This information processing systemspecifically includes one or a plurality of user devices(corresponding to an “information processing device”), one or a plurality of electronic pens(corresponding to an “input device”), and a server device.

12 14 12 The user deviceis a computer owned by a user of the input support service, and has a function of detecting a position indicated by the electronic pen. The user deviceis formed by, for example, a tablet computer, a smart phone, a personal computer, or the like.

14 12 14 12 14 14 12 The electronic penis a pen-type pointing device, and is configured to be capable of one-way or two-way communication with the user device. The electronic penis, for example, a stylus of an active capacitance type (AES) or an electromagnetic induction type (EMR). The user is able to draw pictures, characters, etc., on the user deviceby holding the electronic penand moving the electronic penwith a pen point thereof put onto a touch surface of the user device.

16 16 16 16 20 22 24 The server deviceis a computer that performs centralized control related to the support in writing input, and may be either of a cloud type or an on-premises type. Here, the server deviceis depicted as a single computer in the figure, but the server devicemay alternatively be implemented as a group of computers that form a distributed system. The server devicespecifically includes a server-side communication section, a server-side control section, and a server-side storage section.

20 16 76 12 46 16 12 2 FIG. The server-side communication sectionis an interface for transmitting and receiving electrical signals to and from an external device. The server deviceis thus capable of acquiring a digital ink(see) from the user device, and of supplying a second parameter groupgenerated by the server deviceto the user device.

22 22 26 28 30 24 The server-side control sectionis formed by a processor including a central processing unit (CPU) or a graphics processing unit (GPU). The server-side control sectionfunctions as a data processing section, a learning process section, and an output process sectionby reading and executing a program and data stored in the server-side storage section.

26 76 12 42 26 2 FIG. 3 FIG. The data processing sectionprocesses the digital ink(see) generated by the user deviceto generate learning data, which will be described below. An operation of the data processing sectionwill be described in detail below with reference to a flowchart of.

42 26 28 32 34 28 4 FIG. Using the learning datagenerated by the data processing section, the learning process sectionperforms a learning process with respect to a first discriminatorand a second discriminator. An operation of the learning process sectionwill be described in detail below with reference to a flowchart of.

30 12 46 28 12 34 46 2 FIG. The output process sectionoutputs or transmits, toward the user device, the second parameter group(see) obtained as a result of the learning process performed by the learning process section. As a result, the user deviceis able to use the second discriminatorthrough the second parameter groupcustomized to suit the user thereof.

24 22 24 The server-side storage sectionstores a program and data required for the server-side control sectionto control constituent elements. The server-side storage sectionis formed by a non-transitory computer-readable storage medium. Here, examples of such a computer-readable storage medium include 1) storage devices such as a hard disk drive (HDD) and a solid-state drive (SSD), contained in a computer system, and 2) portable media such as a magneto-optical disk, a read-only memory (ROM), a compact disc (CD)-ROM, and a flash memory.

24 40 76 42 44 46 2 FIG. In the server-side storage section, a database (hereinafter referred to as a digital ink DB) related to the digital inks(see), which will be described below, is constructed, and the learning data, one or a plurality of first parameter groups, and one or a plurality of second parameter groups(corresponding to “learning information”) are stored.

42 28 42 The learning datais a collection of pieces of data used in machine learning performed by the learning process section. Examples of the learning datainclude 1) a set of feature amounts concerning a unit writing operation, 2) a state vector representing a state space when a writing operation is performed, and 3) the type of an event.

14 Here, operations are classified into a first operation, a second operation, and a third operation. Each of the first to third operations is an operation performed through an input device used for writing input. The first operation corresponds to an operation that has been canceled by a cancel operation. The second operation corresponds to an operation that has been performed after the cancel operation and which has not been canceled thereafter. The third operation corresponds to an operation that is identical or similar to the first operation and which may therefore be canceled. In the case where the input device is the electronic pen, for example, a unit operation is a whole operation performed from a pen-down operation to a pen-up operation.

12 Examples of sets of feature amounts include 1) “stroke feature amounts,” examples of which include coordinate values, pen pressure values, slant angles, etc., when a stroke is drawn, 2) “conditions set in a device driver,” examples of which include identification information (e.g., a pen identifier (ID)) and type of an input device, the type of the user device, and the shape of a pen pressure curve, and 3) “conditions set in a drawing application,” examples of which include the type of content, color information of a color palette and a brush, and setting of visual effects. In addition, the state vector includes a set of feature amounts concerning one or a plurality of writing operations performed in the past.

44 32 46 34 The first parameter groupis a collection of learning parameters for identifying an algorithm of the first discriminator. The second parameter groupis a collection of learning parameters capable of identifying an algorithm of the second discriminator. The learning parameters may include, for example, a coefficient describing an activation function of a computation unit, a weighting coefficient corresponding to strength of synaptic junction, the number of computation units constituting each of layers, the number of intermediate layers, and so on.

44 46 44 46 76 14 12 2 FIG. The first parameter groupor the second parameter groupmay be the same for all users, or may be customized on a user-by-user basis. In the latter case, it is desirable that the first parameter groupor the second parameter groupis stored in advance so as to be associated with identification information capable of identifying the user, and is read as appropriate with use of the identification information included in the digital ink(see). Examples of the identification information used here include a user account of the input support service, and various types of IDs related to the electronic penor the user device.

2 FIG. 1 FIG. 12 12 50 54 56 58 60 62 is a block diagram illustrating an example of a configuration of the user deviceillustrated in. The user devicespecifically includes a touchscreen display, a display driver integrated circuit (IC), a touch IC, a communication module, a host processor(corresponding to a “processor”), and a memory(corresponding to a “storage medium”).

50 51 52 51 52 51 51 The touchscreen displayincludes a display panelcapable of displaying visible content, and a sheet-type touch sensordisposed so as to overlap with the display panelin a plan view. In the example of this figure, the touch sensoris an “external” sensor external to and attached to the display panel, but a “built-in” (more specifically, “on-cell” or “in-cell”) sensor integrated with the display panelmay alternatively be used.

51 51 12 The display panelis capable of displaying a black-and-white image or a color image, and may be formed by, for example, a liquid crystal panel, an organic electro-luminescence (EL) panel, or an electronic paper. Note that, when the display panelis flexible, the user is able to perform various writing operations with the touch surface of the user devicebeing curved or bent.

52 52 52 The touch sensoris a capacitive sensor having a plurality of sensor electrodes disposed in a planar arrangement. The touch sensorincludes, for example, a plurality of X-line electrodes for detecting positions along an X-axis of a sensor coordinate system, and a plurality of Y-line electrodes for detecting positions along a Y-axis thereof. In this case, the X-line electrodes are each arranged to extend in the Y-axis direction, and are arranged at regular intervals in the X-axis direction. The Y-line electrodes are each arranged to extend in the X-axis direction, and are arranged at regular intervals in the Y-axis direction. Note that the touch sensormay alternatively be a self-capacitance sensor having block-type electrodes disposed in a two-dimensional grid pattern, instead of a mutual capacitance sensor as described above.

54 51 51 54 51 60 76 51 The display driver ICis an integrated circuit that is electrically connected to the display panel, and which performs drive control on the display panel. The display driver ICdrives the display panelon the basis of display signals supplied from the host processor. Content represented by the digital inkis thus displayed on the display panel.

56 52 52 56 52 60 56 14 The touch ICis an integrated circuit that is electrically connected to the touch sensor, and which performs drive control on the touch sensor. The touch ICdrives the touch sensoron the basis of control signals supplied from the host processor. The touch ICthus implements a “pen detection function” of detecting the state of the electronic pen, and a “touch detection function” of detecting a touch made by a finger of the user or the like.

52 14 14 52 52 Examples of the pen detection functions include a function of scanning the touch sensor, a function of receiving and analyzing a downlink signal, a function of estimating the state (e.g., position, posture, pen pressure, etc.) of the electronic pen, and a function of generating and transmitting an uplink signal including a command for the electronic pen. Meanwhile, examples of the touch detection functions include a function of two-dimensionally scanning the touch sensor, a function of acquiring a detection map on the touch sensor, and a function of classifying regions (e.g., classifying a finger, a palm, etc.) on the detection map.

14 52 51 Thus, a graphical user interface (GUI) is constructed by combining an input function implemented by the electronic penand the touch sensorand an output function implemented by the display panel.

58 12 76 16 46 16 The communication modulehas a communication function of performing wired communication or wireless communication with an external device. This enables the user deviceto transmit the digital inkto the server device, and to receive the second parameter groupfrom the server device.

60 60 64 66 68 70 62 The host processoris formed by a processing device including a CPU, a GPU, or a micro-processing unit (MPU). The host processorfunctions as an operation acceptance section, an information acquisition section, an event detection section, and an information correction sectionby reading and executing a program and data stored in the memory.

64 14 The operation acceptance sectionaccepts an operation performed by the user through the input device (e.g., the electronic pen) used for writing input. Through such an operation, various drawing functions can be performed, examples of the various drawing functions including 1) writing functions including generation of a stroke, 2) setting functions related to color, thickness, decoration, etc., of a stroke, and 3) editing functions including canceling of an operation, copying of an object, a change of layout, and saving of data. Examples of an operation (hereinafter referred to as a “cancel operation”) for canceling an operation performed in the past include an “undo operation” that cancels an immediately previous operation or an “erase operation” that erases at least a portion of a drawn object. Note that the cancel operation may cancel a one-time operation, or may cancel a portion or whole of multiple operations.

14 12 14 14 14 50 Operations performed by the user include 1) operations performed on the electronic pen, and 2) operations performed on the user devicewith use of the electronic pen. Examples of the former operations include a pen-down operation, a pen-move operation, a pen-up operation, and an erase operation via the electronic pen, an operation of pressing a switch provided on the electronic pen, and the like. Examples of the latter operations include an operation performed on a user control (e.g., a window, a button, a palette, etc.) displayed on the touchscreen display, and the like.

66 64 66 62 16 66 14 The information acquisition sectionacquires information (hereinafter referred to as “learning information”) representing a result of learning about a relation between the content of the first operation performed through the input device and the content of the second operation performed through the input device after the cancel operation accepted by the operation acceptance section. The information acquisition sectionmay acquire the learning information either by reading the learning information stored in the memoryor by downloading the learning information from an external device, e.g., the server device. In addition, the information acquisition sectionmay acquire learning information corresponding to the identification information (i.e., the pen ID) of the electronic pen.

68 34 66 The event detection sectiondetects whether or not an event for which the cancel operation may be performed has occurred and the type of any detected event from the current state of writing through the input device. This detection of an event is performed through the second discriminator, which is constructed according to the learning information acquired by the information acquisition section. The types of events are classified into 1) events caused by an erroneous operation, and 2) events caused by a habit, illusion, or mistaken recognition of the user. Examples of the former include miswriting of a straight line, a curve, a figure, a symbol, or the like, and inconsistency in stroke positions or pen pressure. Examples of the latter include timing error in pressing a switch, a mistaken selection of a color in a color palette, and an unwanted operation performed as a result of a habit.

68 70 72 74 72 74 8 11 FIGS.to In the case where the event detection sectionhas detected a predetermined event, the information correction sectioncorrects writing informationor writing setting informationcorresponding to the third operation according to the type of the event. That is, in the case where the third operation, which is identical or similar to the first operation and which may be canceled, has been performed, the writing informationor the writing setting informationis automatically corrected as if an operation similar to the second operation were performed. Specific examples of the correcting operation will be described in detail below with reference to.

24 62 72 74 76 46 62 1 FIG. As is the case with the server-side storage section(see), the memoryis formed by a non-transitory computer-readable storage medium. In the example of the present figure, the writing information, the writing setting information, the digital ink, and the aforementioned second parameter groupare stored in the memory.

72 72 The writing informationis information representing a writing product corresponding to an operation performed through the input device. Examples of the writing informationinclude stroke data including coordinate values, pen pressure values, slant angles, etc., when a stroke is drawn.

74 74 12 The writing setting informationis information concerning a setting at the time of writing corresponding to an operation performed through the input device. Examples of the writing setting informationinclude “conditions set in a device driver,” examples of which include a device ID, a device type, the type of the user device, and the shape of a pen pressure curve; and “conditions set in a drawing application,” examples of which include the type of content, color information of a color palette and a brush, and setting of visual effects.

76 76 76 The digital inkis ink data for representing hand-written content. Examples of the data format, i.e., what is generally called an “ink markup language,” of the digital inkinclude Wacom Ink Layer Language (WILL), Ink Markup Language (InkML), and Ink Serialized Format (ISF). The digital inkhas a data structure in which 1) document metadata (document metadata), 2) semantic data (ink semantics), 3) device data (devices), 4) stroke data (strokes), 5) grouping data (groups), and 6) context data (contexts) are arranged in order.

14 The stroke data is data for describing an individual stroke constituting content. Each stroke is described by a plurality of pieces of point data sequentially arranged in <trace> tags. The pieces of point data are each made up of at least an indicated position (X-coordinate, Y-coordinate), and are separated by a delimiter, such as a comma. For the sake of convenience in illustration, only pieces of point data that represent a starting point and an ending point of each stroke are depicted, while pieces of point data that represent a plurality of intermediate points are omitted. The pieces of point data may include, in addition to the aforementioned indicated positions, the order of writing, the pen pressure and/or posture of the electronic pen, and/or the like.

10 Operation of Information Processing System

10 10 3 7 FIGS.to The information processing systemaccording to this embodiment has the above-described configuration. Next, an operation of the information processing systemwill be described below with reference to.

76 1. Generation of Digital Ink

76 12 1 FIG. 1 2 FIGS.and First, an operation of generating the digital ink, which is performed by the user deviceillustrated in, will be described below with reference to.

14 12 64 12 72 74 64 60 76 72 74 76 62 2 FIG. The user, using the electronic penowned by the user, performs a writing input on the user device. The operation acceptance section(see) of the user deviceaccepts operations performed by the user sequentially, whereby the writing informationand the writing setting informationare generated. Then, when the operation acceptance sectionaccepts an operation for data saving, the host processorgenerates the digital inkincluding the writing informationand the writing setting information, and causes the digital inkto be stored in the memory.

12 76 12 16 22 76 20 24 76 40 Thereafter, the user devicetransmits the digital inkgenerated in the user deviceitself to the server device. Then, the server-side control sectioncauses the digital ink, received via the server-side communication section, to be stored in the server-side storage section. The new digital inkis thus added to and registered in the digital ink DB.

42 2. Generation of Learning Data

42 16 26 16 1 FIG. 3 FIG. 1 FIG. 3 FIG. Next, an operation of generating the learning data, which is performed by the server deviceillustrated in, will be described below with reference to the flowchart ofand. The acts illustrated inare performed by the data processing sectionof the server device.

10 26 16 40 76 3 FIG. At SPin, the data processing sectionof the server devicerefers to the digital ink DB, and acquires the digital inkto be analyzed.

12 26 76 10 At SP, the data processing sectionanalyzes the digital inkacquired at SP, and searches for an instruction code indicative of an undo operation.

14 26 12 14 26 16 14 26 16 At SP, the data processing sectionchecks whether or not an undo operation has been detected by the search at SP. In the case where an undo operation has not been detected (SP: NO), the data processing sectionskips SP. Meanwhile, in the case where an undo operation has been detected (SP: YES), the data processing sectionproceeds to SP.

16 26 14 42 At SP, the data processing sectionacquires and processes peripheral data relevant to the undo operation detected at SPto generate the learning data.

18 26 76 18 26 12 12 18 76 76 18 26 3 FIG. At SP, the data processing sectionchecks whether or not data search throughout the digital inkhas been completed. In the case where the data search has not been completed yet (SP: NO), the data processing sectionreturns to SP, and repeatedly performs SPto SPsequentially until the search throughout the digital inkis completed. Meanwhile, in the case where the data search throughout the digital inkhas been completed (SP: YES), the data processing sectionfinishes the procedure of the flowchart illustrated in.

34 3. Learning for Second Discriminator

16 34 1 FIG. 4 FIG. 5 6 FIGS.and Next, an operation in which the server deviceillustrated inperforms learning with respect to the second discriminatorwill be described below with reference to the flowchart ofand.

20 22 16 32 34 28 42 4 FIG. 1 FIG. At SPin, the server-side control section(see) of the server devicechecks whether or not an update request from a particular user has been accepted. In the case where the first discriminatoror the second discriminatoris customized on a user-by-user basis, the learning process sectionacquires learning datacorresponding to identification information (e.g., a pen ID) that identifies the user.

22 28 16 32 42 20 At SP, the learning process sectionof the server deviceidentifies the relation between the first operation and the second operation with use of the first discriminatorand the learning dataacquired at SP.

5 FIG. 1 FIG. 32 32 80 82 84 86 32 44 is a diagram illustrating an example network structure of the first discriminator. The first discriminatoris formed by, for example, a hierarchical neural network, and includes an input layer, an intermediate layer, and an output layer. The algorithm of the first discriminatoris determined by values of the first parameter group(see), which is a collection of learning parameters.

82 The input layeris a layer at which sets of feature amounts concerning the first operation and the second operation are inputted, and is made up of a plurality of (for example, 2·m) computation units.

84 84 The intermediate layeris made up of one or more layers, and has a dimensionality reduction function of reducing the dimensionality of an input vector. Therefore, it is desirable that the number of computation units that constitute the intermediate layeris smaller than 2·m.

86 The output layeris a layer at which a group of event labels is outputted, and, in the example of the present figure, is made up of (N+1) computation units. A label “event n” (n=1, . . . , N) represents a situation in which an nth event has occurred, and a label “normal” represents a situation in which no event has occurred. In the case where an activation function of each computation unit is a softmax function, this label group is an output vector having (N+1) components each representing the probability of occurrence of the corresponding event. That is, in the case where there is a label (hereinafter referred to as a particular label) that has the greatest label value, it is determined that the event corresponding to the particular label has occurred at the first operation.

24 28 34 42 20 22 46 4 FIG. At SPin, the learning process sectionperforms a learning process with respect to the second discriminatorwith use of the learning dataacquired at SPand the type of the event identified at SP. This learning process is performed employing any of various methods that are commonly applied to machine learning. Through this learning process, the second parameter groupcustomized on a user-by-user basis is calculated.

6 FIG. 1 FIG. 34 34 90 92 94 96 34 46 is a diagram illustrating an example network structure of the second discriminator. The second discriminatoris formed by, for example, a hierarchical neural network, and includes an input layer, an intermediate layer, and an output layer. The algorithm of the second discriminatoris determined by values of the second parameter group(see), which is a collection of learning parameters.

92 The input layeris a layer at which a state vector at the time of an operation and a set of feature amounts concerning the operation are inputted, and is made up of a plurality of (for example, (M+m)) computation units.

94 94 The intermediate layeris made up of one or more layers, and has the dimensionality reduction function of reducing the dimensionality of an input vector. Therefore, it is desirable that the number of computation units that constitute the intermediate layeris smaller than (M+m).

96 22 The output layeris a layer at which a group of event labels is outputted, and, in the example of the present figure, is made up of (N+1) computation units. A one-hot vector in which a vector component corresponding to the event identified at SPis “1” while the other vector components are “0” is used as a correct answer of teacher data.

26 20 16 46 24 12 12 46 62 34 12 16 4 FIG. 4 FIG. At SPin, the server-side communication sectionof the server devicetransmits the second parameter group, which is calculated through the learning process at SP, toward the user devicethat has made the update request. The user devicestores the received second parameter groupin the memoryto enable construction of the second discriminatorin a device environment of the user deviceitself. The server devicethus finishes the procedure of the flowchart illustrated in.

4. Automatic Correction of Operation

12 10 7 FIG. Next, an operation of the user device, which forms a portion of the information processing system, more specifically, an operation of correcting an operation, will be described below with reference to a flowchart of.

30 68 30 30 30 32 7 FIG. At SPin, the event detection sectionchecks whether or not timing of event detection has arrived. The detection timing may be, for example, at a time when one writing operation has been completed. In the case where the detection timing has not arrived yet (SP: NO), control stays at SPuntil arrival of the detection timing. Meanwhile, in the case where the detection timing has arrived (SP: YES), control proceeds to SP.

32 68 72 74 At SP, the event detection sectionacquires various types of information (here, the writing informationand the writing setting information) for identifying an event.

34 68 72 74 32 34 At SP, the event detection sectiondetects whether or not an event has occurred and the type of any detected event from the writing informationand the writing setting informationacquired at SP. This detection process is performed through the second discriminator, which has been trained for the user.

36 70 34 36 30 70 72 74 36 38 At SP, the information correction sectionrefers to a result of the detection at SPto check whether or not an event has been detected. In the case where no event has been detected (SP: NO), control returns to SPwithout the information correction sectioncorrecting the writing informationor the writing setting information. Meanwhile, in the case where an event has been detected (SP: YES), control proceeds to SP.

38 70 72 74 34 8 11 FIGS.to At SP, the information correction sectioncorrects the writing informationor the writing setting informationaccording to the type of the event detected at SP. Specific examples of the correction will be described in detail below with reference to.

30 60 30 38 7 FIG. Thereafter, returning to SP, the host processorrepeats SPto SPwhile the user is performing writing operations, and various types of information are corrected as necessary. The processes in the flowchart illustrated inare sequentially performed in the above-described manner.

14 70 74 s In the case where each of the first to third operations is a switch operation of pressing a switchfor changing a setting at the time of writing, the information correction sectionmay correct the writing setting informationsuch that an event that accompanies the switch operation will start earlier or later than a time when the switch operation is performed.

8 FIG. 2 FIG. 14 14 14 14 1 14 52 50 12 s s s is a diagram illustrating a first example concerning the correcting operation. This first example concerns an operation on the switch, which is provided on a side surface of the electronic pen. Suppose, for example, a case where a certain user has the habit of pressing the switchearly. The user has a tendency to press the switchtoo early at a position Pwhile moving the electronic penin a direction indicated by an arrow M. Moreover, because the level of the touch surface and the level of the touch sensorare slightly different from each other in the touchscreen display(see) of the user device, timing error in performing an operation can occur due to a parallax of the user.

8 FIG. 68 70 74 14 2 1 14 1 s s Thus, if the user performs a similar operation in the situation illustrated inafter learning concerning the aforementioned habit or illusion of the user is performed, an event of “timing error in pressing” is detected by the event detection section. As a result, the information correction sectioncorrects the writing setting informationsuch that the operation of pressing the switchwill be effective at a position P, which lies downstream of the position Pin the direction indicated by the arrow M, even when the switchhas been pressed at the position P. This enables the user to proceed to a next writing operation without performing the undo operation.

72 14 70 In the case where each of the first to third operations is a stroke operation of drawing a stroke, and the writing informationincludes a set of coordinate values representing a change over time of the position indicated by the electronic pen, the information correction sectionmay correct the set of coordinate values such that inconsistency in positions of the same stroke is reduced or minimized.

9 FIG. 14 100 101 102 103 14 104 is a diagram illustrating a second example concerning the correcting operation. This second example concerns an operation of drawing a plurality of strokes sequentially with use of the electronic pen. Suppose, for example, a case where a certain user is not good at drawing an objectcomposed of a plurality of parallel lines. When drawing strokes,, andin order from the top while moving the electronic pen, the user has a tendency to curve an end portionof the lowest stroke downward.

9 FIG. 68 111 112 113 110 114 113 70 72 115 113 Thus, if the user performs a similar operation in the situation illustrated inafter learning concerning the aforementioned erroneous operation by the user is performed, an event of “inconsistency in positions of a stroke” is detected by the event detection section. As a result, if the user, when drawing strokes,, andthat constitute an object, curves an end portionof the strokedownward, the information correction sectioncorrects the writing informationsuch that an end portionof the strokewill be straight. This enables the user to proceed to a next writing operation without performing the undo operation.

70 Note that, although the second example described above assumes the case where the inconsistency in positions of a stroke is reduced or minimized through a change in a locus, the change in the locus may not necessarily be involved in this process. For example, in the case where discrete data of coordinate values is converted into continuous data through interpolation, the information correction sectionmay reduce or minimize the inconsistency in positions of the stroke by using a smoother interpolation function to obtain the continuous data. Also note that a method for smoothing the stroke is not limited to using the interpolation function, and that repetitive execution of an averaging process, thinning out of coordinate values, etc., may alternatively be employed.

72 14 70 In the case where each of the first to third operations is a stroke operation of drawing strokes, and the writing informationincludes a set of pen pressure values representing a change over time of pen pressure acting on the electronic pen, the information correction sectionmay correct the set of pen pressure values such that inconsistency in the level of the pen pressure between the strokes is reduced or minimized.

10 FIG. 14 120 121 122 123 14 123 is a diagram illustrating a third example concerning the correcting operation. This third example concerns an operation of drawing a plurality of strokes sequentially with use of the electronic pen. Suppose, for example, a case where a certain user quickly writes a characterwith unstable pen pressure. When drawing strokes,, andwhile moving the electronic pen, the user has a tendency to write the last strokeentirely with reduced pen pressure, resulting in a reduced line width.

10 FIG. 68 131 132 133 130 133 70 72 133 Thus, if the user performs a similar operation in the situation illustrated inafter learning concerning the aforementioned habit of the user is performed, an event of “inconsistency in pen pressure between strokes” is detected by the event detection section. As a result, if the user, when drawing strokes,, andthat constitute a character, writes the last strokeentirely with reduced pen pressure, the information correction sectioncorrects the writing informationsuch that the pen pressure corresponding to the strokeis increased entirely. This enables the user to proceed to a next writing operation without performing the undo operation.

70 70 Note that, although the third example described above assumes the case where inconsistency in the level of pen pressure between a plurality of strokes occurs, an embodiment of the present disclosure is applicable to a case where inconsistency in the level of pen pressure occurs within a single stroke. For example, in the case where discrete data of pen pressure values is converted into continuous data through interpolation, the information correction sectionmay reduce or minimize the inconsistency in the level of pen pressure by using a smoother interpolation function to obtain the continuous data. In addition, in the case where the range of pen pressure applied by the user is relatively narrow, the information correction sectionmay change the setting of the pen pressure curve so as to enlarge the dynamic range of pen pressure values.

140 74 140 70 In the case where each of the first to third operations is a palette operation of selecting a color corresponding to a position indicated within a color palette, and the writing setting informationincludes color information regarding colors in the color palette, the information correction sectionmay correct the color information such that a color at a position different from the position indicated by the palette operation is selected.

11 FIG. 140 14 142 is a diagram illustrating a fourth example concerning the correcting operation. This fourth example concerns an operation of selecting a color to be used in drawing from within the color palette. Suppose, for example, a case where a certain user has a particular favorite color. When selecting the favorite color while moving the electronic pen, the user has a tendency to select a color patchadjacent thereto by mistake.

11 FIG. 68 140 142 70 74 144 Thus, if the user performs a similar operation in the situation illustrated inafter learning concerning the aforementioned habit of the user is performed, an event of “a mistake in color selection” is detected by the event detection section. As a result, even when the user, intending to select the particular color within the color palette, selects the different color patchby mistake, the information correction sectioncorrects the writing setting informationsuch that a color patchhaving the particular color is selected. This enables the user to proceed to a next writing operation without performing the undo operation.

140 70 70 74 Note that, although the fourth example described above assumes the case where one color is selected from within the color palette, the number of colors to be selected may be two or more. In this case, the information correction sectionchanges at least one of a plurality of colors to correct a combination of colors to be used in drawing. Alternatively, on the basis of learning information representing a result of learning concerning a combination of a color canceled and a color selected after the cancel, the information correction sectionmay correct the writing setting informationso as to add a color that the user is likely to use next upon selection of a color.

72 14 70 70 72 In the case where each of the first to third operations is a stroke operation of drawing strokes, and the writing informationincludes a set of coordinate values representing a change over time of the position indicated by the electronic pen, the information correction sectionmay correct the set of coordinate values so as to erase a stroke used for drafting after a part or whole of content is completed through the stroke operation. Alternatively, the information correction sectionmay correct the writing informationso as to add a drawing effect corresponding to “application of an eraser” to the stroke after a part or whole of the content is completed through the stroke operation.

70 72 Note that, although the fifth example described above assumes the case where a part or whole of the stroke is eliminated, an object to be eliminated is not limited to stroke. For example, the information correction sectionmay correct the writing informationso as to eliminate a portion that is not a drawn object, such as a finger or palm of the user.

70 72 74 70 72 74 14 14 Note that, although the first to fifth examples described above assume the case where the information correction sectioncorrects the writing informationor the writing setting informationwhenever an event is detected, the information correction sectionmay alternatively correct the writing informationor the writing setting informationonly when a particular condition is satisfied. Examples of such conditions include 1) a condition that the feature has been “enabled” through a user operation, and 2) a condition that the user is presumed to be irritated. The degree of irritation is estimated, for example, on the basis of the speed of movement of the electronic pen, how the electronic penis being held, and so on.

10 12 16 46 12 12 As described above, the information processing systemaccording to this embodiment includes the user device, and the server device, which transmits the learning information (here, the second parameter group) toward the user devicein response to a request from the user device.

12 64 14 66 46 14 64 70 72 14 74 46 66 In addition, the user device, which is an information processing device according to this embodiment, includes: the operation acceptance sectionwhich accepts a cancel operation for canceling the first operation performed through the input device (here, the electronic pen) used for writing input; the information acquisition sectionwhich acquires the learning information (here, the second parameter group) representing the result of learning about the relation between the content of the first operation and the content of the second operation performed through the electronic penafter the cancel operation accepted by the operation acceptance section; and the information correction sectionwhich corrects the writing informationrepresenting a writing product corresponding to the third operation performed through the electronic penor the writing setting informationconcerning a setting at the time of writing corresponding to the third operation, on the basis of the second parameter groupacquired by the information acquisition section.

14 46 14 38 72 14 74 46 7 FIG. Moreover, in each of an information processing method and an information processing program according to this embodiment, one or a plurality of computers (or processors) perform: an acceptance act of accepting a cancel operation for canceling the first operation performed through the input device (i.e., the electronic pen) used for writing input; an acquisition act of acquiring the learning information (i.e., the second parameter group) representing the result of learning about the relation between the content of the first operation and the content of the second operation performed through the electronic penafter the accepted cancel operation; and a correction act (SPin) of correcting the writing informationrepresenting a writing product corresponding to the third operation performed through the electronic penor the writing setting informationconcerning a setting at the time of writing corresponding to the third operation, on the basis of the acquired second parameter group. Furthermore, a recording medium (or a storage medium) according to this embodiment is a non-transitory computer-readable medium having the above-described program stored therein.

74 72 As described above, the writing setting informationor the writing informationconcerning the third operation is corrected on the basis of the learning information representing the result of learning about the relation between the content of the first operation and the content of the second operation performed through the input device after the accepted cancel operation, and thus, this correction reflects the cancel operation performed in the past. This enables various types of information obtained through a writing operation with use of the input device to be automatically corrected at an appropriate time.

14 70 74 14 s s In addition, in the case where each of the first to third operations is a switch operation of pressing the switchfor changing the setting at the time of writing, the information correction sectionmay correct the writing setting informationsuch that an event that accompanies the switch operation will start earlier or later than the time when the switch operation is performed. This will eliminate the need to perform a cancel operation even when the switchhas been pressed at a wrong time.

72 70 Further, in the case where each of the first to third operations is a stroke operation of drawing a stroke, and the writing informationincludes a set of coordinate values representing a change over time of the position indicated by the input device, the information correction sectionmay correct the set of coordinate values such that inconsistency in positions of the same stroke is reduced or minimized. This will eliminate the need to perform a cancel operation even when positional inconsistency has occurred at the time of drawing a stroke.

72 70 Furthermore, in the case where each of the first to third operations is a stroke operation of drawing strokes, and the writing informationincludes a set of pen pressure values representing a change over time of pen pressure acting on the input device, the information correction sectionmay correct the set of pen pressure values such that inconsistency in the level of the pen pressure between the strokes is reduced or minimized. This will eliminate the need to perform a cancel operation even when inconsistency in pen pressure has occurred at the time of drawing strokes.

140 74 140 70 140 Furthermore, in the case where each of the first to third operations is a palette operation of selecting a color corresponding to a position indicated within the color paletteand the writing setting informationincludes color information regarding colors in the color palette, the information correction sectionmay correct the color information such that a color at a position different from the position indicated by the palette operation is selected. This will eliminate the need to perform a cancel operation even when a wrong position has been indicated at the time of selecting a color within the color palette.

68 70 72 74 68 14 14 Furthermore, in the case where the event detection sectiondetects whether or not an event for which the cancel operation may be performed has occurred and the type of any detected event from a current state of writing through the input device, the information correction sectionmay correct the writing informationor the writing setting informationaccording to the type of the event detected by the event detection section. Furthermore, in the case where the input device is the electronic pen, each of the first to third operations may be a whole operation performed from a pen-down operation to a pen-up operation on the electronic pen.

66 46 14 14 Furthermore, the information acquisition sectionmay acquire the learning information (i.e., the second parameter group) corresponding to the identification information of the electronic pen. This will enable correction appropriate for the user of the electronic pento be performed.

Furthermore, the cancel operation may be an undo operation that cancels an immediately previous operation, or an erase operation that erases at least a portion of a drawn object. This will increase the likelihood that an intention of the user to cancel an operation will properly be accomplished.

It is needless to say that the present disclosure is not limited to the above-described embodiments and that various modifications can be made thereto without departing from the gist of this disclosure. Also note that various features thereof may be combined arbitrarily as long as no technical conflict arises. Also note that the order in which the acts of each flowchart are performed may be changed as appropriate as long as no technical conflict arises.

The various embodiments described above can be combined to provide further embodiments. All of the U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications and non-patent publications referred to in this specification and/or listed in the Application Data Sheet are incorporated herein by reference, in their entirety. Aspects of the embodiments can be modified, if necessary to employ concepts of the various patents, applications and publications to provide yet further embodiments.

These and other changes can be made to the embodiments in light of the above-detailed description. In general, in the following claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments along with the full scope of equivalents to which such claims are entitled. Accordingly, the claims are not limited by the disclosure.