A year 3 module // Creating an Immersive experience !

Premise :

Our brief is 360° immersive experience with an Alice in Wonderland influence, and I am going to take the premise of this and apply it to the Internet. This is by taking some core parts of the Alice in Wonderland, like rabbit hole, Caterpillar, Cheshire Cat, and the Queen of hearts.

The reason I decided to twist the premise is that I believe the internet can become a rabbit hole due to the amount of information and communication available to anyone in the world. Like the rabbit hole, Alice finds herself falling into a strange and unfamiliar world. So, can the internet with its many websites, forums, and videos. I plan on intertwining aspects of Alice in Wonderland with those of the internet. 

What I will be using ?

To help speed up the creation process, I wanted to use a real time render, what I am familiar with. I will be using Unreal Engine 5.1 figure 5 (Epic Games, 2023) As unreal supports 360 video rendering, binaural audio and I can quickly view the environment in real time with a VR headset.

Alice in Wonderland Theme Mood board:

Internet Themed Mood board:

How I Will Combine These Themes:

I will combine the aspects of the Caterpillar figure 8 (John Tenniel,2023) and the search engine in figure 12 (,2020). I will do this by calling the search engine caterpillar to have a slight nod to how the caterpillar is forthright just how a search engine is. While for guiding the player, just like the Cheshire cat in figure 10. The cat helps Alice to understand her surroundings, and that’s what my adaption will be doing the same. My character will be a mouse cursor like in figure 13 that will help guide you in the changing environments.    

To show scale like in figure 1 (A.Martin,2023) where the user will be scaled down inside an Ethernet cable like in figure 11. Then for the final sequence the user will experience something like figure 9 (Kenneth Muir) where they will face a virus and escape using a potion like figure 2 (Christmas Company,2023). That will bring them back to reality.

Concept Art:

Designing With VR Safety in Mind:

To be able to create a safe experience for the viewer/player, I need to understand the possible dangers like cybersickness. To help me minimize the effects of cybersickness, I will be following the guidelines set in a report by figure 4 (Department for Business Energy & Industrial Strategy,2020) on the safety of domestic virtual reality systems. I will be following these rules, “Some evidence has shown that factors such as greater field of view and graphic realism can cause cybersickness”.  To reduce the risks of cybersickness. I will be making sure the field of view is narrower to help avoid VR sickness. As there isn’t any information on this for Unreal Engine panoramic renderer. I will do a series of test to find the best balance. As well, I will be making my experienced stylized to help avoid graphic realism, cybersickness.

Designing With Immersive Audio In Mind:

I plan on using, figure 3 Dear VR Micro by (Dear Reality ,2014) to mix immersive audio. This is because it allows me to do binaural room simulation and full 360 planning.

As I will be exporting the VR video will be exported to YouTube, I need to be aware of the supported formats if I want to export using binaural audio. I will follow, figure 7 (Google,2023) as it shows the supported formats.

As this is going to be a virtual reality experience, audio is going to be very important in making it an immersive experience. I will have to approach it differently compared to standard video, what uses a mono or stereo audio. That does not take advantage of positional audio in a 3d space. To help me design audio, I will follow figure 6 (Gajsek, 2020). It covers the 6 XR sound design principles “Envelop, Ground, Emote, Navigate, include and bridge”. I will explore these elements further in a later blog post.

Understanding Unreal Resolution Scale:

To be able to start creating an environment for VR as an immersive experience. I needed to get some data and information about how the scene would be affected by the resolution and how the scene would look, depending on asset’s word space. Then comparing that to the cameras. I decided to do these tests as there isn’t any information available about.

To help me understand how to measure 360 contents to visual acuity, I read an article by figure 5 (REDOHL, 2017) “If a 360 camera could capture 1080p video, the viewer would be experiencing something more like a 640 x 320 80s-era computer monitor. 4K video would look like 1K, etc. To achieve true 4K resolution More wherever the viewer looks, you’d actually need to capture 16K!”

I believe the 16k render in headset does look the best when viewed in headset and is held back by the resolution of my VR headset. When viewing all the images in VR, 8k and above was less straining on my eyes.

I got feedback from family and friends, and they said the 8k and 16k images were the best. So, for this project my aim is to render at 16k and then downscale that to 8k in Adobe Premier Pro, to achieve the high-quality image as YouTube does not support 16k resolution.

Embodiment :

To decide if I wanted to give the viewer a body, I tested this in VR with my project. There are benefits of embodiment, and they are “feeling embodied in a virtual body might increase the sensation of being able to control the surrounding environment” this was from figure 2 (Gall, Roth, Stauffert, Zarges and Latoschik,2021).

My Test With Both Embodiment Types :

If I developed this as a game sequence, I would include a body. However, as I will be doing a video, I believe it will be less immersive as you are not in control of the body movements. I think this would negatively impact the experience. As your movements will not match, causing a disconnect for the user. As I am tall, the scale of the body and environment is all wrong to me and for people who are smaller it would have a similar effect. What stuck out to me immediately when testing was the arm length being incorrect. I do believe this effect of wrong scale could be used for those who are disabled and are unable to experience being to walk, for example. I would like to explore this further in the future.

Safety Warning :

As there isn’t a way to know what the user is going to use when watching my immersive video, to help ease them into a new environment and ensure their safety. This is because of figure 1 (Anses, 2021) “Following a session, virtual reality can also induce a temporary change in a person’s sensory, motor, and perceptual abilities, affecting their manual dexterity or ability to orientate their body.” If I didn’t include a gradual fade into the environment with a warning, It could affect the user’s sensory ability and cause possible cybersickness or fall over.

 I wanted to add a warning at the beginning of the video. To give them a health and safety warning about the precautions they should take before watching, like in figure 7. To help me understand what possible dangers are for the user, I read through figure 4 (PlayStation 2023) Health warning for their VR Headset.

Due to this, included warnings like “The VR headset is not for use by children under age 12.” As they have higher risks of injuries and, “Review surroundings and clear obstacles before use. Take steps to prevent pets, children, or other obstacles entering the area during use.” As a precaution so, they don’t hurt themselves or someone else.  Something that wasn’t included was a warning what the experience was, moveable or a stationary experience.  

Navigation Considerations:

To help ground the player, I’m not going to move the player as this is a video sequence. They do not have control of where they would like to go, it could be dangerous. I am doing this because of“ The disconnect between what the body sees and feels is why people throw up when they feel like the room is spinning.” An article by figure 3 (Maiberg, 2016). However, not being able to move the player doesn’t mean. I cannot move the environment while keeping them static with a safety platform below them.

As the player is going to be in a static position looking around, I need to help guide them where to look. To do this, I will be using a character that will be narrating the sequences to give context but help guide them through it too.

Storyboard :

To help me get an understanding of the environment and basic story structure, I’ll be doing a mood board to help me plan out the narrative and visual structure. As this is a different to regular storyboarding and doing it different because the environment is 360. I will be talking an eagle point of view approach to this to help me make this more immersive.

I looked into different methods and found this one figure 1 by (McCurley, 2016). It would be the most affective at getting across the visual cues and placement of sound cues.  Think they could be improvements made that I will be implementing, like, what is the front facing direction ? 

Storyboard Sequences : Scene Notes : Sounds :

Animatic:

I am creating an animatic as I want to get a good idea on what the product will feel like with bare bones and no animation in the sequences. In doing the animatic, it gave me experience, rendering the scene. Giving me an idea on how long the final render may take. My estimated at 8k are around 2 hours. This sequence is broken up into all 4 environments, all having 5 seconds. Screen time. This VR video does not contain, my fades or warning at the beginning.

What I will be using :

I will be using figure 1 Adobe Audition (Adobe inc,2023) to edit the audio and fix any background noise and pop in issues. I will be using this software to try to make the experience more immersive. Then using figure 3 (Dear Reality,2014) Dear VR Micro to give audio a position in 3d space and make the experience more immersive.

For example, this environment by (Studio Afraz ,2021)

 it is visually immersive in VR but is let down by its lack of audio. Even something small like an audio soundscape of the woods, with many animals, would improve this experience and make it more believable.

This example by (Michaël Anctil ,2017) is not only visually impressive, but the audio adds a lot to the experience. However, I think the user would be moving too fast. I plan to use similar audio techniques with its use of mono audio but also stereo binaural audio. Also making sure the audio

I would like to make my experience immersive so, I’m going to use these techniques as I can get the user to look in places I want by using it effectively. It can give them information about the environment they are in without looking behind them. As figure 6 (Virsabi,2023) put it  “It tells us what happens outside the users’ field of view, and it tells us at what location those events are taking place in, and gives us detailed and nuanced information about the nature of those events. It adds additional sensory information that confirms the events, and the actions that you perform in the virtual world, making the virtual reality seems more plausible and realistic.” So doing this would be beneficial for the experience. 

Difficulty In Getting Audio Working:

One of the first issues that occurred was my recording audio was at 32000 hz and not 48000 hz. This was an issue because if I wanted to make the audio binaural.

As there isn’t much information about getting 360 audio working in VR with working headset positional audio. It was difficult to find the correct export settings because I wasn’t sure what was the issue. I had to do a step-by-step approach. Starting with Adobe Audition, getting the audio to be spatial.

Audio Before:

ID AudioBefore

0:00

/17.23

1×

Audio After:

ID AudioAfterwav

0:00

/16.060792

1×

For all the audio, I plan on removing any background noise and artefacts or popping. For this audio I used noise reduction and faded in the beginning making sure they’re both equally at 0db. This is to stop an audible pop. Then I used figure (Dear Reality,2014) Dear VR Micro, re-writing the audio with the positional data using the plugin’s UI. After this was done, It’s ready to be imported.

Testing Audio in Unreal :

Now to be able to test the audio and to make sure it will work with the 360 video I am going to test if the audio will follow the head’s position by getting it to work in unreal. This was the most difficult part, as I attempted this 9 times, all with different audio settings. These are some of the issues I came across, breaking the VR video and the most common, flat non-responsive head audio.

Broken Export 1 :

Broken Export 2 :

Working Audio :

To get the audio to work, I had missed an option in the export settings. On the sound button as there is another option below called “…More”. It only contained the 1 setting I needed, “Audio Is Ambisonics”. I found this button because I watched a video from figure 2 (Brookline Interactive Group ,2018) covering ambisonic’s audio in premier pro.

Script For Character :

Scene 1:

“Oh hi, it’s me again, can’t remember me. That’s probably for the best. I’m here to teach you about something.”

Fade 01:

“Don’t worry, I’m just shrinking you.”

Scene 2:

“See not so bad. This is a monitor; it has 5 layers, and this is a search engine. Let’s see what happens when I put this.”

Fade 02:

“Before we get to where I want to show you.”

Scene 3:

“Look at this. The internet is made of these a series of tubes.”

Fade 04:

“This shouldn’t take long!”

Scene 4:

“Annnnnnnd now we are on the internet. Let’s make sure I click that. This is an antivirus very handy if you are like me.”

“As you can see there are a lot of sites, 1.13 billion exactly. You can go shopping, talk with friends, learn new things and oooo free stuff.”

“That’s not good”

Fade 05:

“Not sure if you can hear me I’ll send you back meet you there.”

Development Approach :

To help me make an immersive environment, I needed to understand how it would look in VR. So, throughout development, I have been placing myself in the environment to get scale and object placement correct. I had to develop each environment in order because I needed the previous environment for a HDRI. Not only to save on resources but allow for me to scale the environment correctly around the HDRI like in figure 2.

To help make the first scene more immersive, I used a technique called parallax corrected cube maps. They use the cameras position and create a 3d space from that. I rendered each image at 2k. This is because you wouldn’t be getting close to them as they are distant, and it will help with render times and performance when real time editing and viewing in VR.

Assets :

To make the assets, I made them with a very simple style to allow for a quicker development and still be able to be immersed. To develop this further, I would add more assets and remake them with more detail but keep the art style to help with cybersickness. These are some of the assets I created in figure 8.

The core assets what guide the player and the story, I did in a pixelated look. To mimic a monitor’s resolution and help ground the internet, like figure 6. Theme and contrast it with Alice in Wonderland items. These assets are, figure 7 and 8.

Animation :

To make the experience more immersive, I wanted to include a character that guides you. However, It needed animation for when it speaks. I created a series of faceplates and mouth plates. That I could mix during animating in the sequencer. There are 7 faceplates and 3 mouth plates an example of this working is figure 9.

How I Am Rendering It:

For video editing, I used figure 1 (Adobe INC ,2023) Adobe Premier Pro. In figure 2, I made my workflow more efficient, by rendering my sequences at 1080p. Before replacing them with the 10k resolution images. I had to render it at a lower resolution than 16k, as I was running out of ram as the renderer took all 64gb of my ram. 

The images I used for the 1080p temp sequences didn’t include the animation sequence for the character. This effected the timing for the character’s speech, as trying to line up audio on an unresponsive timeline was difficult. It was because of the higher resolution images on the timeline.

Even though I’m rendering the video at 8k, The extra data when downscaling in my renders will help with YouTube’s video processing. It will minimize the visual quality loss. They also do not accept 16k renders yet, so it would not benefit from a much higher resolution.

The export settings I used :

Format H.264

Video Sound
Frame Size 8193 x4320 Audio Format AAC
Frame Rate 24 Audio Codec AAC
Feild Order Progressive Sample Rate 48000 Hz
Aspect Square Pixel (1.0) Channels 4
Bitrate encode VBR,2 Pass Bitrate 512
Bitrate encode 240 Ambisonics TRUE

Vr Video Monoscopic
Horizontal Fov 360
Vertical Fov 180