Sora for Video Production: A Guide
Sora for Video Production updated on 26/04/2024
On February 15th, OpenAI announced their latest development in the Generative AI space; Sora. Sora is a text-to-video model that could have a huge impact on media, entertainment and video production companies.
The announcement for Sora was accompanied by a breakdown of its capabilities and limitations and visually supported by a series of text-to-video demonstrations. The post allowed us to preview what’s possible with the future technology in technical terms, but here, we’ll examine how this tool can be used specifically in the field of video production and filmmaking.
Lambda Films is an early adopter of Generative AI, and in our capacity as a commercial video company, we have tested and implemented many of these tools as part of the production process. Our findings on AI in the video production industry can be read here, AI Video Production.
What is OpenAI’s Sora?
In simple terms, Sora creates video clips based on a text description, or prompt, from the user. If you’ve got a visual idea in your head, write it, and Sora will turn it into a piece of video.
Some users may be familiar with the creative process of Generative AI based on existing tools such as Midjourney, Pika Labs and Runway. All these tools aimed to generate visual material based on text input. However, the actual natural language processing capabilities of these platforms is severely limited and users generally have to repeat prompt after prompt to make minor changes. Where Sora may excel, is using ChatGPT’s advanced NLP model to make the creative and generative process more organic and powerful.
Prompt: Beautiful, snowy Tokyo city is bustling. The camera moves through the bustling city street, following several people enjoying the beautiful snowy weather and shopping at nearby stalls. Gorgeous sakura petals are flying through the wind along with snowflakes.
How To Access Sora
As of March, 2024 there is no public access to the Sora platform. The technology is still undergoing assessment, and OpenAI have granted exclusive access to a select few. In their own words,
“We are granting access to a number of visual artists, designers, and filmmakers to gain feedback on how to advance the model to be most helpful for creative professionals.”
It is likely that access to Sora will be via the OpenAI website, where you can currently use ChatGPT & Dall·E. When Sora becomes publicly available, we will update this page.
What Can Sora Do?
Humans & Realistic Actions
Sora can generate video clips of humans; customise their physical appearance and clothing and produce natural looking actions. Based on the previews we’ve seen so far, it’s unlikely you can chain actions together, for example: “A stylish woman walks down a Tokyo street, looks in to her handbag and pulls out a phone”. However, that is where human creativity comes in to play, creating such a narrative through editing multiple shots together.
Human Movement Video Generated by Sora
Animation & VFX
Departing from realism, Sora is able to generate animation in a variety of styles. This includes Pixar-style, papercraft, 2D illustration or even realistic 3D for sci-fi projects. However, while this may be an incredible advancement in animation production, the old concerns around IP protection remain painfully relevant. To what extent are these character designs based on existing IP?
Animation Generated by Sora
Conceptual
Where Generative AI models have always excelled, is in the creation of conceptual imagery. At Lambda Films, we have primarily used generative AI to help produce creative and conceptual material as part of pitches and pre-vis. Sora will take this to another level.
Prompt: Photorealistic closeup video of two pirate ships battling each other as they sail inside a cup of coffee.
Exploring air head
At the end of March 2024, Sora released a slate of AI-generated films created by a selection of hand-picked video artists. The visuals for these films (an important distinction) were entirely created using the Sora platform and demonstrated the generative power of the technology. One film in particular stood out; air head, by Shy Kids. While the film itself is a narrative short, it was easy to conceive how it could be used as a piece of commercial or branded content.
While the film is impressive, we don’t know the time taken, mistakes made or the learning curve required to produce such a result. It’s also worth remembering too that production values were raised through non-AI creative elements such as the narration, colour grade, music & SFX. What this ultimately demonstrates is that Sora will go a long way to democratise the filmmaking process, but it still needs human creativity to weave the elements together to produce something remarkable.
air head by Shy Kids
Limitations of Sora
Imperfections
When viewers are confronted with a piece of video, especially video of human movements, our brain accepts that everything we see is correct and adheres to all natural laws. However, Sora is not beyond making mistakes, and the trouble is, they can be incredibly hard to spot.
In commercial production, AI generated video clips will require an increased amount of quality control time. Furthermore, creatives will have to train themselves to question and analyse all natural movements. However, should mistakes be picked up later in the process, it’s highly unlikely that the shot can be faithfully replicated. Therefore, creating a problematic ripple effect throughout the project.
Imperfections in movement
While stylish woman walking through Tokyo seems completely flawless. Closer inspection reveals rubbery limbs, sliding feet and legs that somehow cross over.
Simulating Physics
OpenAI are the first to admit that Sora doesn’t understand physics, and as my least favourite subject at school, I can see why.
Generally, the clips shared by OpenAI demonstrate a passable grasp of physics. However, Sora sometimes struggles with accurately simulating the physics of certain interactions, especially when the prompt is more ‘unusual’, leading to videos that may not fully adhere to real-world principles.
Impossible Physics From AI Video
Cause & Effects
Similar to its limitations in understanding physics Sora struggles with cause & effect. For scenes requiring an interaction between two (or more) objects, it sometimes fails to grasp sequential actions and their consequences. OpenAI use the example of “a person might take a bite out of a cookie, but afterward, the cookie may not have a bite mark” In the video example below, we can see the candles are unaffected by the blow.
Maybe they're just prank candles...?
Spatial Awareness
Sora can also struggle with spatial awareness. It can mix up left & right, randomly clone objects that occupy the same physical space and fail to understand how a movement might be generated over a long duration.
Prompt: Five gray wolf pups frolicking and chasing each other around a remote gravel road, surrounded by grass. The pups run and leap, chasing each other, and nipping at each other, playing.
Using Sora for Commercial Video Production
Stock Video
Like Midjourney and other Generative AI tools, Sora struggles with continuity and consistency. It’s highly unlikely the technology (at first) will be able to faithfully replicate a human character across multiple prompts, scenes and situations – but I can’t wait to test it!
Without continuity, it then means one of the most effective uses of Sora will be in producing highly-relevant stock video. Stock videos are already popular, especially for corporate video production, as they create a high-impact, high-production value films without the price tag. However, searching for the right piece of stock video can be a time-intensive task. Sora, in theory, could generate highly-tailored and specific stock video for your project.
In April 2024, Adobe announced that Sora (as well as Pika & Runway) will be fully-integrated in to the Premiere Pro workflow. In the announcement video below, you can clearly see how powerful Sora can be in terms of generating additional b-roll or stock footage at the click of the button. In the early stages of the technology, I foresee this as being one of the most powerful, and useful applications of Sora for commercial video production.
Sora intergration with Adobe Premiere Pro
Set Extensions & Matte Painting
Generating establishing shots or aerials of fictional locations allow filmmakers and producers to focus on building or utilising smaller sets and locations. The wider world-building can then be generated purely by AI. With enough foresight, Sora could be used to produce conceptual direction for locations, and then matching sets built after to ensure a sense of consistency.
Establishing Shots Generated by Sora
Aerial & Drone Footage
Aerial & drone footage is incredibly popular, and now even easier to source using Sora. It remains to be seen how accurately it will render well-known locations and landmarks, however, for the majority of purposes the examples seem close enough. Non-specific locations, both real and fictional will of course be where Sora can really deliver.
Aerial & Drone Video Generated by Sora
VFX
Issues with continuity may hinder a consistent approach to incorporating VFX throughout a video project. However, where Sora could really deliver is in producing an impressive, elemental VFX set piece; a planetary explosion, a portal or a monster reveal for example.
The VFX revealed as part of the Sora announcement appears weaker than the other content. But, from previous experience with Midjourney & Dall·E, imaginative VFX is where generative AI can really shine.
Prompt: A giant, towering cloud in the shape of a man looms over the earth. The cloud man shoots lighting bolts down to the earth.
Closing Notes
Through our (limited) access to OpenAI’s Sora we have attempted to explore its transformative potential in commercial video production. Yes, it is a truly innovative, and dare I say, revolutionary tool, however challenges remain, most notably in realism, continuity and physics.
It’s worth remembering that Sora is not here to take over. Instead, the power in this technology can only be harnessed by a blend of AI capabilities and human creativity and oversight. It’s another tool that can is here to support and benefit commercial production companies, if used properly. AI has already become an important video marketing trend in 2024, and we can expect that to continue once Sora becomes public access.
As Sora progresses, its refinement and application promise to redefine commercial video production, bridging the gap between conceptualisation and visualisation with unprecedented ease. As ever, attention must be paid to the ethical considerations around such a technology. In future posts, we will dive deeper into how these relate specifically to Sora. However, in the meantime you can read more about these in AI & Video Production.
