2024 · Case study
Blackbox for Vision Pro
Spatial puzzles, built on a new platform.
Visit project- Contribution
- Technical artist — interaction and puzzle prototyping, materials and spatial systems
- With
- Shapes & Stories
- Year
- 2024
- Built with
- RealityKit · Swift · ARKit · Reality Composer Pro · Shader Graph
- Project recognition
- Apple Design Award
The project
Custom gestures, room-aware hints and materials for a Vision Pro launch game, made in a small team led by Ryan McLeod.
Blackbox for Vision Pro was made by a small team led by Ryan McLeod, creator of the original iPhone game. I joined as technical artist while the hardware and development tools were still pre-release.
I learned RealityKit quickly and worked across materials, puzzle prototypes, custom gestures and room-aware systems. We often had to build an idea in the headset before we could tell whether it made sense.
Vision Pro launch title. Winner of the 2024 Apple Design Award for Spatial Computing in games.
01 / Process
Scissors that keep working
One puzzle asks you to cut with two fingers. As the hand turned, fingers could become hidden and the estimated joints could drift towards a relaxed pose. A recogniser that demanded the same clear gesture every frame kept losing the scissors.
I separated activation from continuation. Starting required a strong pose. Once the player had shown their intention and was working on a puzzle that expected cutting, the recognition became more tolerant. I also worked out which finger information was most useful to follow.
That let the interaction continue through imperfect tracking instead of repeatedly asking the player to prove what they were already doing.
02 / Process
A way into the grid
Blackbox's grid became a world around the player. Two pinched hands pull apart to open a tear, revealing the grid through it. Once the opening is large enough, the environment expands around you.
The same action brings you back. A reflective environment-map treatment makes the return surface appear to distort the room before the real surroundings are revealed. The gesture and the material effect had to explain the transition together.
03 / Process
Put the hint in the room
Some hints needed to appear on a real wall rather than in a floating panel. I built a selection system using the available room data, looking for a surface that was large enough, visible and usefully oriented towards the player.
Stability mattered as much as the initial choice. A slightly better candidate should not make an existing hint jump across the room. Debug views let me inspect the surfaces and the chosen placement while tuning that behaviour.
04 / Process
A material language for the puzzles
Early studies used scratched, physical-looking surfaces. I developed a softer treatment of luminous forms, noise and bright edges that we carried across the game.
The material work grew into Shader Graphs with thousands of nodes in the early Reality Composer Pro tools. It involved finding ways to express the look within the facilities the platform offered at the time.
I also made runtime controls for tuning spatial values in the headset and retaining useful changes in the project. It was a practical way to shorten the adjustment loop.
Blackbox shipped with Apple Vision Pro and received the 2024 Apple Design Award for Spatial Computing in the games category. Blackbox was created by Ryan McLeod and Shapes & Stories; this page describes my contribution to the team's work.