A store-based smart mirror kiosk virtual fitting room
HOLLYWOOD, CA (goshrobin) 2022/07/25 – Let’s consider design process for a Unity 3D dressing room application.
Virtual Fitting Room with RealSense
Here’s a tutorial for setting up a virtual fitting room:
However, it uses Unity Realtime Tracking with an Intel Realsense (Z-depth) camera, hardware that we may not have available ($300).
Hand-tracking with iPhone LIDAR
Let’s consider this Unity hand tracking example using ARFoundation with LIDAR:
https://github.com/ogoguel/realtimehand
For LIDAR it uses an iPhone 12+, hardware we may not have available in a store kiosk. Apple LIDAR is also in the 2020 iPad Pro.
Unity Stick Figure Mo-cap with MediaPipe
A marionette-like mo-cap capture to track poses as a stick figure:
Mediapipe pose tracking
Can we make our virtual dressing room work if we access the camera from Unity, or maybe we could use MediaPipe instead to save a snapshot of what the webcam sees? We have access to the x-y-z coordinates for the hand landmarks.
Marionette joints
Currently only using one point to represent the hand (ie, points 15 and 16 in this diagram above). But we can extend it to get the full hand landmark data (ie: 15, 17, 21, 19 / 15, 18, 20, 22) from MediaPipe?
Tracking hands with MediaPipe:
https://google.github.io/mediapipe/solutions/hands.html
3D Gesture UI/UX
In a store kiosk there will be no keyboard or mouse. And controlling it using speech recognition could be distracting to others in a store, so pass on SR. How about gesture? We’re tracking body position already…
Gesture UI
When a store customer is standing in front of kiosk and being tracked, raising a hand will activate menu, with just a few buttons as choices. Poking virtual button selects.
In-Store Kiosk Design
Form factor something like a Lululemon Mirror, but really just a 4k monitor mounted sideways with a small form factor computer, such as a Mac Mini, and a webcam. For development, uses an ordinary laptop.
Intro to Unity C# Programming
Ready to try to learn Unity? Use git clone to download one of the Unity sample projects from GitHub.
Note there is no Unity Project File. There is merely a folder, with the Assets and other folders inside. Open Unity, click “open project” and click on the folder that your project is in.
Unity Hello World project:
Enjoy!
About Robin Rowe
I’m Robin Rowe. People call me Robin or Rob, sometimes professor. I work in advanced product design and innovation management.
As UN WHO Augmented Reality Group Manager and XR Games Producer, I developed a medical metaverse, an AR/VR hospital simulation to train doctors worldwide to save lives.
As chief technologist at multi-billion dollar systems integrator SAIC, I was the founding director of their AI research lab and an enterprise manager with P&L responsibility for commercial and defense divisions. I constructed the Chicago NBC-TV studios, advanced national critical infrastructure by adding Smart Cities capability to the U.S. traffic control system, created real-time AI for DoD national defense crisis management, and developed the real-time motion-capture animation system used to create digital stunt doubles in superhero films such as Disney Marvel Spider-Man.
I led the team that won the Novartis Biome innovation prize in 2019, using AI computer vision to analyze skin disease. I’ve taught C++ as a Computer Science professor at the Naval Postgraduate School and the University of Washington. Former DARPA PI for AI and digital video. Former navy research scientist for VR war gaming and vision research. I’ve chaired innovation committees at ANSI/ISO and The CFO Alliance. I’ve developed financial software for Fortune 500 companies and large NGOs. I’m a thesis advisor to graduate and doctoral students in metaverse R&D.
I founded my first start-up, a car company, when I was 16. My family is in real estate and agriculture, owns the largest organic farm in Illinois. Not wishing to run the family business, I went another way. I have 30 years experience in product design and financial systems. And, with trading stocks and now crypto on my own account, I’ve made high ROI year over year.