Services
Mixed Reality Production
Mixed reality video means filmed footage and 3D animation in the same frame, tracked and composited so the digital part belongs there. I handle the CGI side end to end: camera solve, asset build, lighting match, animation and comp. I also build 3D visuals for live shows, where the same techniques run on stage screens instead of a feed.
Written by Marco Mori, 3D artist, Lindau, Germany · Updated 2026-09-08
What does a mixed reality project include?
The deliverable is a finished, composited film, plus reformats and stills. What sits behind it is the part clients rarely see: a camera solve, a lighting match, and a comp that reproduces the imperfections of the plate. Digital elements look wrong not because they are low quality, but because they are too clean, no grain, no motion blur, no lens distortion, no bounce light from the floor.
On the Samsung Galaxy Z Flip 7 campaign, the whole job was making the animation sit inside real space without breaking immersion: precise tracking, realistic lighting, clean comp. On Fritz Kalkbrenner's "Playing Games", the same toolchain was used in the opposite direction, the digital characters are deliberately unreal, and the filmed world is what grounds them.
What do you need from the client to start?
- The footage, or the shoot plan if it has not happened yet. Getting involved before the shoot saves more time than any other single decision.
- Camera metadata: body, lens, focal length, frame rate, shutter.
- Lighting reference: HDRI, or chrome and grey ball, or at minimum wide shots of the location.
- Clean plates of the areas where digital elements will sit.
- Reference for the intended look and the degree of realism you want.
- Screen specs for live work: resolution, pixel pitch, playback system, aspect.
How does the process work?
The ordered steps are listed above the FAQ. The single most important one is the first: deciding what stays real. Mixed reality goes wrong when everything becomes negotiable in post, because then the plate is just an expensive background.
Selected work
- Samsung Galaxy Z Flip 7, mixed reality campaign: product animation embedded into filmed space through camera tracking, lighting match and compositing.
- Fritz Kalkbrenner "Playing Games", mixed reality music visuals: two pieces where abstract faceless characters inhabit filmed environments.
- Mixed reality experiments: an ongoing personal series testing compositing, tracking and hybrid direction, including DJ and live-oriented visuals.
Why work with a specialist instead of a VFX house?
| Specialist (me) | VFX house | |
|---|---|---|
| Shot volume | Single shots to short films | Long-form and feature volume |
| Pipeline | Lean, fast to change | Formal, built for many artists |
| Creative involvement | Direction and execution together | Execution against a supervisor's brief |
| Cost structure | No fixed overhead | Departmental |
| Best for | Campaign pieces, social, music, live content | Feature films, series, hundreds of shots |
Does mixed reality work for live shows?
Yes, and it is a different production discipline than film. Show content is built for a specific screen geometry and resolution, it usually has to loop cleanly, it has to survive being seen from odd angles, and it has to read at distance under stage lighting. Content that looks excellent on a laptop can disappear entirely on an LED wall, which is why screen specs come before art direction, not after.
What makes a mixed reality shot fail?
Almost always one of four things: the camera cannot be solved, the lighting does not match, the digital element does not touch anything in the frame, or the comp is too clean. Contact shadows and shared imperfections do more for believability than render quality does.
How does this relate to XR and virtual production?
XR stages and LED volumes solve the same problem in-camera by playing the digital environment on a wall behind the talent. That is a strong route when you have the stage and the budget. What I do is the post-side version: the effect is created after the shoot, which costs less to set up and gives more freedom to change the idea afterwards. A longer explanation sits in what is mixed reality video production.
Can simulation be part of a mixed reality shot?
Yes, and it is often the reason to do it at all. Cloth, fluids, particles and destruction interacting with a real environment are the shots that could not have been filmed. That work is described under Houdini FX.
Got a plate, or a shoot coming up? Send it over before the shoot day.
Frequently asked questions
What is mixed reality video production?
It is the production of video in which filmed footage and 3D animation share the same frame. The camera move in the footage is solved in 3D, digital elements are placed into that solved space, lit to match, and composited so they read as part of the shot.
How is mixed reality different from VFX?
Technically they overlap almost completely. The difference is intent: classic VFX usually hides itself inside a story, while mixed reality content is often built so the audience notices and enjoys the collision between real and digital.
Do you shoot the footage yourself?
Usually not. I work with footage you shoot or with a DP from my network. What matters is that the shoot follows a few rules so the plate can be tracked, and I supply those rules before the shoot day.
What do you need from a shoot to make tracking possible?
Known lens and focal length, a locked or deliberately moving camera rather than an unstable one, some texture in frame to track against, a clean plate, and a reference of the lighting such as a chrome and grey ball or an HDRI.
Can mixed reality be produced from existing footage?
Often yes. Send the clip and I will tell you whether it solves. Motion blur, rolling shutter, heavy grain and featureless surfaces are the usual reasons a plate resists tracking.
Can this run live on stage?
Yes for pre-rendered show content played back to LED or projection, which is what most live sets use. Fully realtime interactive systems are a different build and would be scoped separately.
Which software is involved?
Houdini for 3D and simulation, Redshift for rendering, Nuke for tracking and compositing, and standard camera solvers depending on the plate.
What formats do you deliver for social?
9:16, 1:1 and 16:9 from the same comp, plus stills. Reformatting is planned before the shoot, because a vertical crop of a horizontal plate loses the part of the frame the effect usually lives in.
Do you work with brands and with artists?
Both. The Samsung Galaxy Z Flip 7 campaign and the Fritz Kalkbrenner music pieces used the same toolchain for very different purposes.