Visual development of the Kinect v2 Sensor

In this article i will show my progression with the kinect V2 Sensor using Unity.

I will include links of the resources i used!

lets start with some early testing using the old kinect v1 sdk on the asset store:

https://assetstore.unity.com/packages/tools/kinect-with-ms-sdk-7747

using the knowledge i had of the Kinect v1 i managed to translate the detection of the skeleton to the v2 but i realised that the structure was very much outdated.

So i decided to look further on the internet for some better source material. 

 

after a long search on the web i manged to find this project:  https://github.com/camiloNemocon/KinectAvatarUnity 

The code is in spanish but after a bit of google translate i managed to understand it.
It uses the animation rigging and burst package in unity to achieve a relatively clean way of interacting with the skeleton of the kinect.

Ninja demo:

After playing with the software i decided to make interaction with objects possible with some simple box colliders.

I also found a way to work with facial recognition using Software made by brekel.

https://brekel.com/face_v2/

I asked for a sample of the unity code and found that the software uses network protocols to translate the detection of the face to unity. 

Face:

I would like to run face detection natively in unity and found this article:   https://pterneas.com/2015/06/06/kinect-hd-face/ I am currently trying to rewrite this code in unity using the kinect.face dll that was used. So far i got stuck on reassigning the events of the sensor to unity.
I will have to do more reach on c# events in order to find the solution.

Last but not least i finally managed to link my own made character from Fuse to Mixamo and send it to unity in order to do live motion capture.

Demo of Mixamo model

There is still a lot of cleaning to do!

i used a rig with less than 25 bones which gives me the problem of the twisted limbs.

the orignal demo: https://www.youtube.com/watch?v=ZbW485wlvWE&t=123s 

has better results, especialy in the head and arm area's 

Thank you for reading and i hope u found this helpfull!