Computer science Professor Jonathan Ventura spent time during the COVID-19 pandemic much like the rest of us – connecting to people and places over a screen.
His screen time differed from most of us, though, as he explored approaches to empower a broader audience to create immersive virtual reality realms, recognizing that such an endeavor can foster deeper connections through shared experiences.
“The equipment to make VR content costs tens of thousands of dollars,” Ventura said. “I started exploring how everyday people could make and share immersive content in an accessible way. I wondered, ‘Can we make VR content with our phones?’”
Turns out the answer is more complicated than one might think, but with a new, pandemic-sharpened focus on virtual reality along with grant funding, Ventura is getting closer to his vision.
“It’s exciting to feel like we’re creating the future,” said Ventura, strapping on a VR headset in preparation for a demonstration of one of his projects. “And it’s motivating to work on something so cutting-edge and impactful.”
Carving out space
During his childhood on the Central Coast, Ventura was captivated by computer programming – an interest bolstered by his dad who ran a local software company.
He graduated from Arroyo Grande High School, then studied computer science at UC Santa Barbara where his path became clear after a chance conversation with his college roommate.
Ventura’s roommate was taking a course on computer vision – a field within computer science and software engineering that focuses on enabling computers to interpret and understand visual information from the world, just as humans do with their eyes – and happened to mention it during a dialogue.
“I heard about the course and thought, ‘I have to know about this!’” Ventura recalled with a laugh.
He enrolled in the class, got hooked and centered his graduate research around computer vision.
Ventura brought his expertise to Cal Poly in 2018, joining the Computer Science and Software Engineering Department and connecting with the students on topics they are passionate about: virtual reality and artificial intelligence.
The return to campus after COVID lockdowns came with a successful entry into the National Science Foundation’s Early Career Development Program and five-year grant that is funding Ventura’s work to create VR experiences from images and video.
“I’m trying to carve out my own space with technology that has made incredible advancements,” he said.
Sharing virtual experiences
Ventura works with students in a small lab in Building 52, developing virtual reality experiences using readily available equipment accessible to the public.
For the project he calls PanoSynthVR, Ventura and a team of students took 2D panoramic image from a public dataset, then fed the data into a model to produce a 3D rendering.
“Panoramic images also can be captured with a consumer VR camera or smartphone,” he added.
The panorama offers a circular perspective but doesn’t allow the viewer to see around the corner of a building, for example, so Ventura and his students employed machine learning to infer what would be around that corner, filling in the visual gaps.
Ventura published a paper on the project in 2022 and is working on the next iteration with computer science master’s student Shivam Asija to improve the clarity of the scene.
“With virtual reality, your baseline is the 3D model, then you add other elements,” said Asija, who connected with Ventura last year over their shared interest in computer vision. “The goal is to get an immersive environment.”
Resolution is the key, according to Ventura who said visual elements that are fuzzy or blurry break the feeling of immersion.
The challenge is that VR headsets can only store so much data, so Asija and Ventura are testing the most efficient, accurate method for rendering high-resolution VR graphics.
Asija sees the value of VR particularly in education and only wishes the technology had been farther along when he was enrolled in an electrical engineering lab during distance learning due to COVID.
“It would have been pretty great to participate in a VR lab with the oscilloscopes and other equipment,” he said with a grin.
Ventura is also spearheading a project focused on 3D image reconstruction, which is crucial for creating immersive environments that are convincing and engaging.
Similar to how a digital camera adapts to changing lighting conditions, Ventura aims to instruct a model to recover a high-dynamic range reconstruction, ensuring that details in both bright and dim areas are preserved.
“The ultimate goal is that the model learns the settings so that you get a consistent 3D reconstruction regardless of the input,” he said.
Aiding Ventura is Nam Nguyen, a computer science master’s student with an interest in deep learning – a subset of machine learning that uses multi-layered neural networks to process and analyze data.
Nguyen and Ventura are searching for the best model that can process data from 2D images for reliable 3D reconstruction.
Nguyen believes such research is critical in the pursuit of enabling more people to create VR content.
“Part of the reason I’m excited about this project and VR in general is that it gives us the ability to connect people through experience,” he said.
With his family still in Vietnam, where he grew up, he shared a personal example: “I would love for my mom in Vietnam to share the experience I had of seeing Yosemite.”
By Emily Slater
