Estimating the location of centres of rotations (CoRs) using markered optical motion capture data is useful within technique analysis for sports training; observation of asymmetries and abnormalities in rehabilitation medicine; and generation of virtual characters for films and computer games. The following video shows the marker positions of a human model as returned by a motion capture system, such as Phasespace.

In order to achieve skeletal reconstruction of any legged body, 3 markers must be available on each limb segment at each time. However, even with many cameras, there are instances where occlusion of markers by elements of the scene leads to missing data. In order to unambiguously establish its position, each marker must be visible to at least two cameras in each frame.

Locating the CoRs is a crucial step in acquiring a skeleton from raw motion capture data. To calculate the joints between two sets of markers it is helpful to have the rotation of a limb at any given time. We can estimate the orientation of a limb at time k relative to a reference frame using the Procrustes formulation. The location of the joints can be calculated using Cameron and Lasenby [1] approach. This approach takes advantage of the approximation that all markers on a segment are attached to a rigid body.

[1] Jonathan Cameron and Joan Lasenby. A real-time sequential algorithm for human joint localization. In ACM SIGGRAPH Posters, page 107, New York, USA, 2005. ACM Press.

The video below presents the marker positions and the calculated CoRs. It is obvious that, when the markers are occluded, it is impossible to establish the joint location.

Although many methods have been developed to handle the missing marker problem, most of them are not applicable in real-time and often require manual intervention or become ineffective when markers are missing for an extended period of time.

We proposed a real-time integrated framework (Unscented Kalman Filter) which predicts the occluded marker positions using a variable turn model in combination with inferred information from neighbouring markers (Neighbouring markers are considered as markers belonging to the same limb segment). The results are accurate, real-time implementable and reliable even in cases where all markers on a limb segment are occluded, or one or two markers are non-visible for a large sequence of frames. The predicted marker positions are then used to locate the human joints. The following video shows an example of marker prediction and CoR estimation with extended data occlusion. Blue represents the true positions and red the predicted positions.

Using Inverse Kinematics (without assuming any skeleton model) we ensure that the inter-joint distance is constant over time. With a continuous stream of accurate 3D data, and performing real-time CoR estimation, the skeletal information can be extracted and used for visual performance feedback. Experiments demonstrate that our method effectively recovers good estimates of the true positions of the missing markers and CoR's, even if all the markers on a limb are occluded for a long period of time. The following video shows the same example, with the extreme data occlusion, but this time when FABRIK (Forward And Backward Reaching Inverse Kinematics) was applied in order to maintain the fixed inter-joint assumption. The true positions are coloured in blue and the predicted in red.

Below you can watch an example video of implementation under cases with extended data occlusion. The left video shows results using the integrated UKF-VTM; the right video shows the results when FABRIK was applied in order to maintain the fixed inter-joint distance assumption. The true positions are coloured in blue and the predicted in red. Obviously, in the first case the skeletal structure was violated since the bones were not restricted to their original lengths; in the second case the results have been improved to a visually more natural shape.