Human Pose Matching on mobile — a fun application using Human Pose Estimation (Part 1 Intro)

Human pose estimation -computationally detecting human body posture- is rising. These technologies capable of detection the human body joints are becoming effective and accessible. And they
look astonishing, have a look:

This video is made using OpenPose and
it’s impressing

OpenPose represents the first real-time system to jointly detect human body, hand and facial keypoints (in total 130 keypoints) on single images. In addition, the system computational
performance on body keypoint estimation is invariant to the number of detected people in the image



Result using OpenPose
This OpenPose library is a wonderful example of *buzzword incoming* Deep Learning. This library is built upon
a neural network and has been developed by Carnegie Mellon University. OpenPose uses an interesting pipeline to achieve it’s robust
performance. If you want to dig into this topic, the paper “Realtime Multi-Person 2D
Pose Estimation using Part Affinity Fields” gives an overview of the inner workings of the system.
If you are looking for alternative algorithms, take a look at DeeperCut.
Here is an implementation in Python.



Result using DeeperCut

Mobile Development — The Opertunities

So, the magic of Deep Learning gives us 18 human body joints. WHAT CAN WE DO WITH IT? A lot actually. Samin does a good job listing the possibilities in his blog.



Pose Output Format OpenPose
The fact that these pose estimators work with just a normal camera, opens doors for mobile development. Imagine
using these technology with your shitty smartphone camera. OK, I admit; I want to develop for smartphone, so (ideally?) everything runs embedded on the smartphone itself. This means,
the implementation is capable of running on CPU or smartphone GPU. BUT, as these Estimator Algorithms currently run on some pretty decent GPUs , this is surely a point of discussion.
Most of the new smartphones these days have a GPU on board, but are they capable of running these frameworks? To nuance for this case, the
(multi-person) keypoint detection doesn’t need to be real-time, as there is only 1 picture (no real-time video) and a delay
time of 1 to 1,5 seconds is acceptable.
However, working embedded is not a must, it’s also possible to outsource the estimation/matching algorithm to a central
server containing a decent GPU. This particular choice, embedded or out-sourcing, is an issue that involves a lot of parameters (performance/computation power, server cost, accuracy, mobile battery
usage, delay server communication, multi platform, scalability, mobile data usage -less important- , …).

Similarity of different poses — The Application

I’m working on a project where a person must mimic a predefined pose (call it the model). A picture is made from the person that mimics this predefined pose. Then, with the help of OpenPose
the human pose of the person is extracted from this image and compared with the predefined pose. Finally a scoring mechanism decides how well the two poses match or if they match at all.
This desired project raises two big problems tough:

1. Porting OpenPose to Mobile Platform

I didn’t tell you yet, but OpenPose is build with Caffe, a Deep Learning framework. Caffe is OK, but it doesn’t
support Android, not mentioning iOS. There are plenty of other Deep Learning libraries, yet the mainstream
library is TensorFlow, a Machine Learning
framework developed by Google. And surprise surprise, TensorFlow supports Android (of course, it’s made by Google) and iOS! Great! 

So, TensorFlow it is then? I don’t know for sure yet. Still considering/researching the options … Hope to get back on this in the near future.
The community working on this topic is quite small, but it is growing. Actually Ale
Solano is working on this currently, he is trying to port the OpenPose-Caffe library to TensorFlow and he is also blogging about
it! He’s making good progress.

2. How to determine similarity between 2 sets of matching 2D points

Let’s say the previous succeeded, and we have the 2D joint points of two poses:



left: model || right: input pose

So we have 2 sets of matching points (each consisting of 18 2D coordinates);

1. The model X (which needs to be mimicked) -left hand side 

2. The input Y (needs to be checked on matching grade) -right hand side
Essentially this comes down to checking if the two poses (described by their 18 2D coordinates) are similar.

“Two geometrical objects are called similar if
they both have the same shape,
or one has the same shape as the mirror image of the other. More precisely, one can be obtained from the other by uniformly scaling (enlarging
or reducing), possibly with additional translation, rotation and
reflection. This means that either object can be rescaled, repositioned, and reflected, so as to coincide precisely with the other object.” — Wikipedia
This means, two poses have the same shape if you can transform one pose to the other one using only a combination
of translation, scaling and rotation. From Linear Algebra, we know this combination of operations is wrapped in a linear transformation, more precisely an affine
transformation(composition of linear map and a translation). 

This linear transformation is described by it’s transformation matrix [2x3]. So this is the first step: find the transformation matrix. You want this matrix? Solve the linear system
et voilà! 

You wish. There is noway two different poses are exactly described by an affine transformation on each other -ie are similar-, what’s the chance? One way or an other there will be some noise introduced,
these are people in the wild you know. Hence, we want something that gives us an approximate. Luckily, there exists something like least-squares
algorithm, which will do the job for us. Math to the rescue again!
Assuming we have the transformation matrix, the next step is to transform the second pose onto the model. Now we
can compare the image of the second pose and the model with each other. If all their corresponding
points are adjacent, we can conclude the two poses match. From the moment a distance exceeds a certain threshold, they’re different.
I’ve been quite vague in the last two steps. I know. 

That’s because I’m not sure yet about the ‘how’ precisely. I can list you all possible options I figured out, but that’s not useful I guess. I need to make some choices and try to find the best combination. 

There are many ways to determine if the image of the second pose matches with the model (first pose). An option is, as suggested above, to limit the max (Euclidean) distance between two matching points by a determined threshold. But as there are many other distance
metrics, there are some other ways to compute this transformation error. 

The same applies for finding the transformation matrix. You can split the problem in multiple smaller problems ( ie face , torso and legs) or keep the body as whole or something else. What’s the way to go? 

As always, some approaches are more effectively then others
In the weeks to come, I will try to setup a test case and test on all these different parameters.
https://becominghuman.ai/human-pose-matching-on-mobile-a-fun-application-using-human-pose-estimation-part-1-intro-93c5cbe3a096