top of page

Some wishes that about future apple product that could help 

Small Scale Live-Action Sci-fi production 

Disclaimer: All ideas are original based on first hand production work shown in the sci-fi show episodes.   Google Gemini is used for wording and English grammar checks.

 

Executive Summary

As a Senior Technical Artist/FX artist/Storyteller, I recently made a live-action sci-fi series called Trevor and Jeff’s Dimension Rift. By combining my traditional CG knowledge with new AI tools, I reached a visual quality that goes beyond normal CGI expectations for a one-person team at this period of production time.

 

I used my iPhone, iPad, MacBook Pro, and a gaming PC as my whole studio. But while working between game engines, iPhone cameras, and AI, I kept facing one main problem: physical camera motion does not connect easily with virtual 3D space, one of the reason is of course I can't afford a giant LED display to do real time virtual background so I shoot against a green screen.  Tracking a camera in post is a lot of work so I shoot on a stationary camera , and use Google Veo 3.1 to add complex camera motion like pivoting around the character in the post. But this is expensive and lowers the resolution. 

 

I love testing hardware and finding practical solutions. This document is my humble collection of "what-if" ideas to show how Apple hardware can help creators tell big stories more easily.

 

1. The Dream: A Mobile Sci-Fi Studio

I believe solely AI-generated footage will never replace real human acting or storytelling. Instead, AI should help creators who do not have big studio budgets. The key for Apple to help storytellers realize this dream is a simplified, device-based pipeline that works seamlessly and effortlessly to composite real human performances with AI-generated VFX and environments, even at the pre-visualization stage.

 

Right now, indie creators spend too much time fixing camera tracking and solving technical problems. With Apple’s integrated hardware and chips, I imagine a mobile virtual production setup. An artist can direct real actors, save sensor metadata, and use AI to render background scenes instantly—all without needing massive traditional desktop infrastructure.

 

2. Curious "What-Ifs" for Apple Hardware

I. Lightweight Spatial Metadata Saved with Video Files

  • The Problem: Every time I shoot live-action video, I must manually track 3D camera motion before adding background scenes. I can shoot on a static tripod and fake camera motion later (like using Google Veo 3.1 for pivoting), but that is expensive, takes a lot of effort, and degrades video quality.

 

  • The Idea: When I shoot on an iPhone, the device already knows its position in space. What if 6DOF camera tracking data, focal length, depth maps, and a shared "space origin" anchor point are saved directly into the video file? All takes in the same real life area would share the exact same 3D coordinate system.

 

  • The Workflow: Drop iPhone video files directly into Unreal Engine or AI tools. The virtual camera matches the real camera position instantly on frame one.

 

II. Multi-Cam Array Mocap using Ultra Wideband (UWB)

  • The Problem: Single-camera AI motion capture often fails when body parts block each other. Syncing multiple iPhones over Wi-Fi needs complex setup and has network latency, especially if Wi-Fi is slow or unstable.

 

  • The Idea: Use Apple’s Ultra Wideband (UWB) technology to pair and sync multiple iPhones wirelessly without needing a Wi-Fi network.

 

  • The Workflow: Place 2 or 3 iPhones in a room. They build a zero-latency spatial network instantly. A solo director can capture clean 3D body motion and face performance without expensive motion capture suits.

 

III. On-Device Native 3D Gaussian Splatting

  • The Problem: Scanning real environments for 3D sets currently needs slow photogrammetry or cloud rendering services.

 

  • The Idea: Use the LiDAR scanner on iPhone/iPad and Apple Silicon to create and export 3D Gaussian Splats directly on the device.

 

  • The Workflow: Walk around a space, scan it in seconds, and export a 3D set directly into Unreal Engine or Vision Pro.

 

3. How I Can Participate to Make Those Happen

If I join Apple, I want to test these ideas on real production projects. I can create short live-action sci-fi scenes to test prototype devices and measure real performance:

 

  • Testing Spatial Metadata: Measure time saved in post-production and test metadata compression limits. We can test how far we can compress this spatial data before the virtual background unlocks from the recorded actor during compositing in Unreal Engine, Nuke, After Effects, or AI models.

 

  • Stress-Testing UWB Mocap: Compare Wi-Fi sync against UWB sync in busy wireless environments. Measure setup time, latency in milliseconds, and tracking recovery when actors cross each other. We can also test if this setup can work easily with major AI mocap platforms.

 

  • Benchmarking Gaussian Splats: Test render time on Apple Silicon, battery/thermal cost, and real-time frame rates when loading the 3D splat in game engines.

 

  • Testing Local AI Models for Privacy: Help engineers benchmark generative AI models running natively on Apple Silicon. By measuring local rendering speeds, memory usage, and thermal limits, we can ensure production teams can execute these AI VFX functions entirely on-device. This protects project confidentiality by keeping sensitive video material local, without ever needing to send footage to third-party cloud vendors.


 

      Pitching a standalone camera (practicing purpose)

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

​​
 

 

 

 

 

 

 

 

 

​

Camera.png
Screenshot 2026-08-10 at 4.24.14 PM.png
Screenshot 2026-08-10 at 4.24.58 PM.png
bottom of page