A method for synthesizing a sequence includes defining a switching linear dynamic system (SLDS) with a plurality of dynamic systems. In a Viterbi-based method, a state transition record for a training sequence is determined. The corresponding sequence of switching states is determined by backtracking through the state transition record. Parameters of dynamic models are learned in response to the determined sequence of switching states, and a new data sequence is synthesized, based on the dynamic models whose parameters have been learned. In a variational-based method, the switching state at a particular instance is determined by a switching model. The dynamic models are decoupled from the switching model, and parameters of the decoupled dynamic model are determined responsive to a switching state probability estimate. Similar methods are used to interpolate from an input sequence.
Legal claims defining the scope of protection, as filed with the USPTO.
1. A computer implemented method for synthesizing a data sequence representing figure or human motion, comprising: defining a switching linear dynamic system (SLDS) comprising a plurality of dynamic models; associating each model with a switching state such that a model is selected when its associated switching state is true; determining a state transition record for at least one training sequence of measurements by determining and recording, for a given measurement and for each possible switching state, an optimal prior switching state, based on the at least one training sequence, wherein the optimal prior switching state optimizes a transition probability; determining, for a final measurement, an optimal final switching state; determining the sequence of switching states by backtracking, from said optimal final switching state, through the state transition record; learning parameters of the dynamic models, responsive to the determined sequence of switching states; and synthesizing a new data sequence, based on the dynamic models with learned parameters.
2. The method of claim 1 , wherein the new data sequence has characteristics which are similar to characteristics of at least one training sequence.
3. The method of claim 1 , wherein the new data sequence combines characteristics of plural training sequences.
4. The method of claim 1 , further comprising modifying the SLDS such that at least one constraint is met.
5. The method of claim 4 , wherein modifying the SLDS comprises: adding a continuous state control.
6. The method of claim 5 , wherein modifying the SLDS further comprises: adding constraints on continuous states.
7. The method of claim 5 , wherein modifying the SLDS further comprises: adding constraints on the continuous state control.
8. The method of claim 5 , wherein modifying the SLDS further comprises: adding constraints on time.
9. The method of claim 5 , further comprising: designing an optimal continuous control that satisfies the at least one constraint.
10. The method of claim 9 , further comprising: synthesizing the new data sequence using the optimal continuous control.
11. The method of claim 4 , wherein modifying the SLDS comprises: adding a switching state control.
12. The method of claim 11 , wherein modifying the SLDS further comprises: adding constraints on switching states.
13. The method of claim 12 , wherein modifying the SLDS further comprises: adding constraints on the switching state control.
14. The method of claim 12 , further comprising: designing an optimal switching control that satisfies constraints.
15. The method of claim 14 , further comprising: synthesizing the new data sequence using the optimal switching control.
16. The method of claim 4 , further comprising designing optimal switching and continuous state controls that satisfy switching and continuous constraints respectively.
17. The method of claim 16 , further comprising: synthesizing the new data sequence using the optimal switching and continuous state controls.
18. The method of claim 1 , wherein the training sequence of measurements comprises economic data.
19. The method of claim 1 , wherein the training sequence of measurements comprises image data.
20. The method of claim 1 , wherein the training sequence of measurements comprises audio data.
21. The method of claim 1 , wherein the training sequence of measurements comprises spatial data.
22. Computer apparatus for synthesizing figure motion using a switching linear dynamic system (SLDS) model, and comprising: a plurality of linear dynamic system (SLDS) models, wherein at any given instance, an LDS model is selected responsive to a switching variable; a state transition recorder which determines, for at least one training sequence of measurements, a state transition record by determining and recording, for a given measurement and for each possible switching state, an optimal prior switching state, based on the at least one training sequence, wherein the optimal prior switching state optimizes a transition probability, and which determines, for a final measurement, an optimal final switching state; a backtracker which determines a sequence of switching states corresponding to the training sequence by backtracking, from said optimal final switching state, through the state transition record; a dynamic model learner which learns parameters of the dynamic models responsive to the determined sequence of switching states; and a synthesizer which synthesizes a new data sequence, based on dynamic models with learned parameters.
23. Computer apparatus of claim 22 , wherein the new data sequence has characteristics similar to at least one training sequence.
24. Computer apparatus of claim 22 , wherein the new data sequence combines characteristics of plural training sequences.
25. A computer system for synthesizing a data sequence representing figure or human motion, comprising: means for defining a switching linear dynamic system (SLDS) comprising a plurality of dynamic models; means for associating each model with a switching state such that a model is selected when its associated switching state is true; means for determining a state transition record for at least one training sequence of measurements by determining and recording, for a given measurement and for each possible switching state, an optimal prior switching state, based on the at least one training sequence, wherein the optimal prior switching state optimizes a transition probability; means for determining, for a final measurement, an optimal final switching state; means for determining the sequence of switching states by backtracking, from said optimal final switching state, through the state transition record; means for learning parameters of the dynamic models, responsive to the determined sequence of switching states; and means for synthesizing a new data sequence, based on the dynamic models with learned parameters.
26. A computer program product for synthesizing a data sequence representing figure or human motion, the computer program product comprising a computer readable storage medium having computer executable code thereon, including program code which: defines a switching linear dynamic system (SLDS) comprising a plurality of dynamic models; associates each model with a switching state such that a model is selected when its associated switching state is true; determines a state transition record for at least one training sequence of measurements by determining and recording, for a given measurement and for each possible switching state, an optimal prior switching state, based on the at least one training sequence, wherein the optimal prior switching state optimizes a transition probability; determines an optimal final switching state for a final measurement; determines the sequence of switching states by backtracking, from said optimal final switching state, through the state transition record; learns parameters of the dynamic models, responsive to the determined sequence of switching states; and synthesizes a new data sequence, based on the dynamic models with learned parameters.
27. A computer system for performing motion synthesis and interpolation comprising: a processor; a memory system connected to the processor; and a computer program, in the memory, which; associates each of a plurality of dynamic models with a switching state such that a model is selected when its associated switching state is true; determines a state transition record for at least one training sequence of measurements by determining and recording, for a given measurement and for each possible switching state, an optimal prior switching state, based on the at least one training sequence, wherein the optimal prior switching state optimizes a transition probability; determines an optimal final switching state for a final measurement; determines a sequence of switching states by backtracking, from said optimal final switching state, through the state transition record; learns parameters of the dynamic models, responsive to the determined sequence of switching states; and synthesizes a new data sequence, based on the dynamic models with learned parameters.
28. A computer implemented method for synthesizing a data sequence representing figure motion, comprising: defining a switching linear dynamic system (SLDS) comprising a plurality of dynamic models; associating each dynamic model with a switching state such that a dynamic model is selected when its associated switching state is true, wherein the switching state at a particular instance is determined by a switching model; decoupling the dynamic models from the switching model; determining parameters of a decoupled dynamic model, responsive to a switching state probability estate; estimating a state of a decoupled dynamic model corresponding to a measurement at the particular instance, and responsive to at least one training sequence of measurements; determining parameters of the decoupled switching model, responsive to the dynamic state estimate; estimating a probability for each possible switching state of the decoupled switching model; determining a switching state sequence based on the estimated switching state probabilities; learning parameters of the dynamic models, responsive to the switching states sequence; and synthesizing a new data sequence, based on the dynamic models with learned parameters.
29. The method of claim 28 , wherein the new data sequence has characteristics similar to at least one training sequence.
30. The method of claim 28 , wherein the new data sequence combines characteristics of plural training sequences.
31. The method of claim 28 , further comprising modifying the SLDS such that at least one constraint is met.
32. The method of claim 31 , wherein modifying the SLDS further comprises: adding a continuous state control.
33. The method of claim 32 , wherein modifying the SLDS further comprises: adding constraints on continuous states.
34. The method of claim 32 , wherein modifying the SLDS further comprises: adding constraints on the continuous state control.
35. The method of claim 32 , wherein modifying the SLDS further comprises: adding constraints on time.
36. The method of claim 32 , further comprising: designing an optimal continuous control that satisfies the at least one constraint.
37. The method of claim 36 , further comprising: synthesizing the new data sequence using the optimal continuous control.
38. The method of claim 31 , wherein modifying the SLDS comprises: adding a switching state control.
39. The method of claim 38 , wherein modifying the SLDS further comprises: adding constraints on switching states.
40. The method of claim 39 , wherein modifying the SLDS further comprises: adding constraints on the switching state control.
41. The method of claim 39 , further comprising: designing an optimal switching control that satisfies constraints.
42. The method of claim 41 , further comprising: synthesizing the new data sequence using the optimal switching control.
43. The method of claim 31 , further comprising designing optimal switching and continuous state controls that satisfy switching and continuous constraints respectively.
44. The method of claim 43 , further comprising: synthesizing the new data sequence using the optimal switching and continuous state controls.
45. Computer apparatus for motion synthesis and interpolation using a switching linear dynamic system (SLDS) model, comprising: a plurality of linear dynamic system (LDS) models, wherein at any given instance, an LDS model is selected responsive to a switching variable; a switching model which determines values of the switching variable; an approximate variational state sequence inference module, which reestimates parameters of each LDS model, using variational inference, to minimize a modeling cost of current state sequence estimates, responsive to at least one training sequence of measurements; and a synthesizer which synthesizes a new data sequence, based on the reestimated LDS models.
46. Computer apparatus of claim 45 , wherein the new data sequence has characteristics similar to the at least one training sequence.
47. Computer apparatus of claim 45 , wherein the new data sequence combines characteristics of plural training sequences.
48. Computer apparatus of claim 45 , further comprising modifying the SLDS such that at least one constraint is met.
49. Computer apparatus of claim 48 , wherein modifying the SLDS comprises: adding a continuous state control.
50. Computer apparatus of claim 48 , wherein modifying the SLDS comprises: adding a switching state control.
51. Computer apparatus of claim 48 , further comprising designing optimal switching and continuous state controls that satisfy switching and continuous constraints respectively.
52. Computer apparatus of claim 51 , further comprising: synthesizing the new data sequence using the optimal switching and continuous state controls.
53. A computer implemented method for synthesizing figure motion by interpolating from an input measurement sequence, comprising: defining a switching linear dynamic system (SLDS) comprising a plurality of dynamic models; associating each model with a switching state such that a model is selected when its associated switching state is true; determining a state transition record by determining and recording, for a given measurement and for each possible switching state, an optimal prior switching state; based on at least one training measurement sequence, wherein the optimal prior switching state optimizes a transition probability; determining, for a final measurement, an optimal final switching state; determining the sequence of switching states by backtracking, from said optimal final switching state, through the state transition record; determining the sequence of continuous states based on the determined sequence of switching states; and interpolating missing motion data from the input measurement sequence, based on dynamic models and responsive to the determined sequences of continuous and switching states.
54. The method of claim 53 , further comprising modifying the SLDS such that at least one constraint is met.
55. The method of claim 54 , wherein modifying the SLDS comprises: adding a continuous state control.
56. The method of claim 54 , wherein modifying the SLDS comprises: adding a switching state control.
57. The method of claim 54 , further comprising designing optimal switching and continuous state controls that satisfy switching and continuous constraints respectively.
58. The method of claim 57 , further comprising: interpolating the new data sequence using the optimal switching and continuous state controls.
59. The method of claim 53 , further comprising: at a receiver, interpolating missing frames from transmitted model parameters and from received key frames, the key frames having been determined based on the learned parameters, wherein the input measurement sequence comprises the received key frames.
60. Computer apparatus for motion synthesis and interpolation using a switching linear dynamic system (SLDS) model, comprising: a plurality of linear dynamic system (LDS) models, wherein at any given instance, an LDS model is selected responsive to a switching variable; a state transition recorder which determines a state transition record for a training measurement sequence by determining and recording, for a given measurement and for each possible switching state, an optimal prior switching state, based on the training sequence, wherein the optimal prior switching state optimizes a transition probability, and which determines, for a final measurement, an optimal final switching state; a backtracer which determines a sequence of switching states corresponding to the training sequence by backtracking, from said optimal final switching state, through the state transition record; a dynamic model learner which learns parameters of the dynamic models responsive to the determined sequence of switching states; and an interpolator which interpolates missing motion data from an input sequence, based on the dynamic models with learned parameters.
61. A computer system for synthesizing figure motion by interpolating from an input measurement sequence, the system comprising: means for defining a switching linear dynamic system (SLDS) comprising a plurality of dynamic models; means for associating each model with a switching state such that a model is selected when its associated switching state is true; means for determining a state transition record by determining and recording, for a given measurement and for each possible switching state, an optimal prior switching state, based on at least one training sequence, wherein the optimal prior switching state optimizes a transition probability; means for determining, for a final measurement, an optimal final switching state; means for determining the sequence of switching states by backtracking, from said optimal final switching state, through the state transition record; means for learning parameters of the dynamic models, responsive to the determined sequence of switching states; and means for interpolating missing motion data from the input measurement sequence, based on dynamic models learned from training sequences.
62. A computer program product for synthesizing figure motion by interpolating from an input measurement sequence, the computer program product comprising a computer readable storage medium having computer executable code thereon, including program code which: associates each of a plurality of dynamic models with a switching state such that a model is selected when its associated switching state is true; determines a state transition record by determining and recording, for a given measurement and for each possible switching state, an optimal prior switching state, based on at least one training measurement sequence, wherein the optimal prior switching state optimizes a transition probability; determines an optimal final switching state for a final measurement; determines the sequence of switching states by backtracking, from said optimal final switching state, through the state transition record; learns parameters of the dynamic models, responsive to the determined sequence of switching states resulting; and interpolates missing motion data from the input measurement sequence, based on the dynamic models with learned parameters.
63. A computer system for motion synthesis and interpolation comprising: a processor; a memory system connected to the processor; and a computer program, in the memory, which: associates each of a plurality of dynamic models with a switching state such that a model is selected when its associated switching state is true; determines, from a set of possible switching states and responsive to a training sequence of measurements, a state transition record by determining and recording, for a given measurement and for each possible switching state, an optimal prior switching state, wherein the optimal prior switching state optimizes a transition probability; determines an optimal final switching state for a final measurement; determines a sequence of switching states corresponding to the measurement sequence by backtracking, from said optimal final switching state, through the state transition record; learns parameters of the dynamic models, responsive to the determined sequence of switching states; and interpolates missing motion data from an input sequence, based on the dynamic models with learned parameters.
64. A computer implemented method for synthesizing motion by interpolating from an input measurement sequence, comprising: defining a switching linear dynamic system (SLDS) comprising a plurality of dynamic models; associating each dynamic model with a switching state such that a dynamic model is selected when its associated switching state is true, wherein the switching state at a particular instance is determined by a switching model; decoupling the dynamic models from the switching model; determining parameters of a decoupled dynamic model, responsive to a switching state probability estimate; estimating a state of a decoupled dynamic model corresponding to a measurement at the particular instance, and responsive to at least one training sequence of measurements; determining parameters of the decoupled switching model, responsive to the dynamic state estimate; estimating a probability for each possible switching state of the decoupled switching model; determining the sequence of switching states based on the estimated switching state probabilities; learning parameters of the dynamic models, responsive to the determined sequence of switching states; and interpolating missing motion data from the input measurement sequence, based on the dynamic models with learned parameters.
65. The method of claim 64 , further comprising modifying the SLDS such that at least one constraint is met.
66. The method of claim 65 , wherein modifying the SLDS comprises: adding a continuous state control.
67. The method of claim 65 , wherein modifying the SLDS comprises: adding a switching state control.
68. The method of claim 65 , further comprising designing optimal switching and continuous state controls that satisfy switching and continuous constraints respectively.
69. The method of claim 68 , further comprising: synthesizing the new data sequence using the optimal switching and continuous state controls.
70. The method of claim 64 , wherein the measurement sequence comprises a sparsely observed image sequence.
71. The method of claim 64 , further comprising: at a receiver, interpolating missing frames from transmitted model parameters and from received key frames, the key frames having been determined based on the learned parameters.
72. Computer apparatus for motion synthesis and interpolation using a switching linear dynamic system (SLDS) model, comprising: a plurality of linear dynamic system (LDS) models, wherein at any given instance, an LDS model is selected responsive to a switching variable; a switching model which determines values of the switching variable; an approximate variational state sequence inference module, which reestimates parameters of each LDS model, using variational inference, to minimize a modeling cost of current state sequence estimates; a dynamic model learner which learns parameters of the LDS models responsive to determined sequence of switching states resulting from at least one training sequence; and an interpolator which interpolates missing motion data from an input sequence, based on the dynamic models with learned parameters.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
September 1, 2000
January 31, 2006
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.