AVIE

Interior view of the T_Visionarium II, with people standing in the centre looking at the various images projected around them.

AVIE: interior view showing T_Visionarium II

AVIE is the world’s first mobile and scalable 360-degree 3D immersive and interactive visualisation system. It measures 7-10 metres in diameter and 3-3.5 metres in height. It includes a motion tracking system and a multi-channel audio system. Driven by an experimental framework, it uses a suite of cutting-edge hardware and software resources that enable the development of applications in AI visualisation, immersive environments and networked collaboration. With a number of these systems available, it is deployable on-site with industry and project partners for research development and application.

AVIE has been installed as core visualisation infrastructure at: ArtLAB ÉPFL (Switzerland); Chengdu Research Institute, Chengdu (China); CityU Hong Kong (Hong Kong); China Coal Technology & Engineering Group, Fushun (China); Experimental Media Performing Arts Centre, Rensselaer University, New York (USA); ZKM Media Museum (Germany); Mines Rescue Pty Ltd; Museums Victoria; Museum of Applied Arts & Sciences; UTS Sydney (all Australia).

It has been commercialised for the mine safety training industry by: Mines Rescue Pty Ltd at its Newcastle, Wollongong, Lithgow and Singleton facilities, Australia; Shenyang Research Institute of China Coal Technology & Engineering Group (CCTEG), Fushun, China.

Exterior view of the AVIE 3D visualisation system, showing Nebula.

AVIE: exterior view showing Nebula

AVIE has been developed though three iterations, with further development on-going. AVIE1 (2004-07) facilitated audio-visual data to respond kinaesthetically to the user's body across the full breadth of the projected space; AVIE2 (2008-2010) has augmented this environment by delivering the ability to recompose digital files in real-time; AVIE3 (2011-2017) has delivered a base-line AI Database system that enables digital agents to interact with users,based on their ability to interpret and respond to the user’s behaviour in space. AVIE provides a platform in which 3D audio-visual experiences are created by a combination of multiple projectors and multiple sound sources, driven by a cluster of computers, running a highly versatile real-time 3D VR engine. It comprises custom designed technologies: a Motion Tracking system, a Mixed Reality system and an AI Reasoning system. The AI system coordinates the actions of the visualisation system by allowing it to autonomously reason about its response to user behaviour. The Motion Tracking system tracks the physical location of users in AVIE, while the Mixed Reality system allows the creation of 'scenarios' that exhibit a feedback loop between the world of the user and of the audio-visual data.

AVIE visualisation forms

High-resolution panoramic stereo images with augmented 3D graphics overlays

High-resolution panoramic videos, including stereo videos

Immersive multimedia data visualisation

Real-time interactive 3D environments supported by artificial intelligence

AVIE interaction platforms

AVIE utilises two types of interaction:

remote device: a wireless tablet device allowing real-time selection and navigation within interactive applications
motion tracking: real time software that detects, captures, tracks and interprets human movement and gestures, allowing capture and audience participation in the direct cinematic experience, interplay between users and digital characters and analysis of user behaviour

Below: Interior views showing Mining VR

Miners standing in the centre of the AVIE 3D visualisation system, lookg at a projection of an underground mine.

Miners standing in the centre of the AVIE 3D visualisation system, looking at a VR projection of an underground mine.

AVIE Components

AVIE consists of a cylindrical silvered screen 4 metres high and 10 metres in diameter on which 3D panoramic images sequences and videos are projected. The 6 projectors use active shuttering technology to deliver stereoscopic 3D to end users, and the images from the individual projectors are seamlessly blended by custom software developed in-house at iCinema. 6 high-performance image generators, capable of processing images and videos as well as rendering large virtual environments in real time, are linked to the projectors to deliver the full immersive 3D experience. It accommodates up to 30 people at any one time.

External view

Projector set up
This provides a multi-purpose sensory environment for unencumbered tracking of viewer movement and gesture, offering a large range of motion capture and interaction design functions. Covering the entire AVIE area, the system is able to accurately track the spatial position and movements of up to thirty people. When applied to smaller groups of up to five people, it generates high-resolution three dimensional voxel models of their body movements and gestures, enabling real time motion tracking and some gesture recognition.

Volumetric voxel tracking
Directly complementing the unique visualisation and interaction features of AVIE is its spatialised audio system. This is a twentyfour channel system with custom surround audio application software. This system enables fully immersive 360 degree placement of sound anywhere around the viewers. As with visual projection, the audio system can be coupled with the Intelligent Vision-Based Interaction and Motion Tracking Systems to allow voices to ‘follow’ projected characters, or to be activated only when viewers move within ‘earshot’ of the virtual audio sources, or correctly perform some task.

Spatialised Audio System

Overview
Exhibitions
Credits
Specifications

Project Directors: Jeffrey Shaw, Dennis Del Favero

Programmers: Ardrian Hardjorno, Volker Kuchelmeister, Matthew McGinity
(Additional Software and Hardware Engineering: Jared Berghold, Marc Chee, Robin Chow, Alex Kuptsov, Alex Ong & Xin Guan)

Project Funding: ARC DP0209550, ARC DP0345547, ARC LE0453517
2004–2020

AVIE has been commissioned as a curated visualisation platform by numerous premier international venues for biennales, exhibitions and festivals, including:

Fo Guang Shan Buddha Museum, Dashu, Taiwan, 2020
École Polytechnique Fédérale de Lausanne, Switzerland, 2020
CityU Galleries, City University of Hong Kong, 2020
Maritime Museum 360, Hong Kong, 2020
Centre for Applied Computing & Interactive Media, Chengdu, 2020
Art of Immersion: Virtual Frontiers, ZKM, Karlsruhe, 2017
Chengdu Research Institute, Chengdu, 2017
Digitalife, Palazzo delle Esposizione, Rome, 2017
Sydney Film Festival, Sydney, 2016
Bonlieu Scène Nationale, Annecy, 2016
International Symposium of Electronic Art 2016 (ISEA16), Hong Kong, 2016
GLOBALE, ZKM, Karlsruhe, 2016.
Seoul Mediacity Biennale, Seoul Museum of Art, Seoul, 2016
Montreal Museum of Fine Art, Montreal, 2016
Chronus Art Center, Shanghai, 2014-15
Child, Nation and World Cinema Symposium, Sydney, 2014
20th International Conference on Virtual Systems and Multimedia, Hong Kong, 2014
Data Arena, UTS, Sydney, 2014
International Symposium of Electronic Art 2013 (ISEA13), Sydney, 2013
10th Shanghai Biennale, Power Station of Art, Shanghai, 2013
SIGGRAPH Asia, Hong Kong, 2013
Hong Kong Book Fair, Hong Kong, 2013
Limited Knowledge, Hong Kong, 2013
Festival d’Avignon, Avignon, 2013
Business of Design Week, Hong Kong, 2012
NODEM, Hong Kong, 2012
Grand Opening of Run Run Shaw Creative Media Centre, Hong Kong, 2012
Smithsonian Institution, Washington DC, 2012
Matter-Light, Bethune, 2011
Sydney Film Festival, Sydney, 2011
Festival of Avignon, Gymnase Paul Giera, Avignon, 2011
National Institute of Experimental Arts double conference, Sydney, 2011
ALiVE – Applied Laboratory for Interactive Visualization & Embodiment, CityU, Hong Kong, 2011
Digitalife, La Pelanda, Rome, 2010
15th Biennial Film & History Conference, UNSW, Sydney
Dynamic Earth, Museum Victoria, Melbourne, 2010-2020
Inaugural Exhibition, ALiVE – Applied Laboratory for Interactive Visualization & Embodiment, Hong Kong, 2010
STRP Festival, Strijp R Klokgebouw, Eindhoven, 2010
APPEA Conference and Exhibition, Convention and Exhibition Centre, Brisbane, 2010
Energy Field Days, Showground, Sydney, 2010
Un Volcan Numérique, Le Volcan Scène Nationale du Havre, Le Harve, 2009
Seconde Nature, Espace Vasarély, Aix-en-Provence, 2009
International Documentary Film Festival Amsterdam, Zuiderkerk, Amsterdam, 2009
International Architecture Biennale, Erasmus Bridge Tower, Rotterdam, 2009
Imagining Media at ZKM, Karlsruhe, 2009
Immigration Museum, Melbourne, 2009
Sydney Festival, Sydney, 2008
Biennale of Seville, Carlos V Alambra Palace, Granada, 2008
eARTS Festival: eLandscapes, Zendai Museum of Modern Art, Shanghai, 2008
Mines Rescue Pty Ltd Australian Facilities in Woonona, Argenton, Lithgow & Singleton, 2008
YOUser: the Century of the Consumer, YOU_niverse & Panorama Festival, ZKM Media Museum, Karlsruhe, 2007
EMPAC, Rensselaer University, New York, 2007
i.Future Festival, Singapore Science Centre, 2007
VSMM Conference, UNSW, Sydney, 2007
Australian Council of University Art & Design Schools Conference, UNSW, Sydney, 2007
AIMEX Australian Mining Industry Conference, Sydney, 2007
From Spark to Pixel, Martin Gropius Bau, Berlin, 2007
Kaladham Museum, Karnataka, (permanent installation since 2006)
La Rivoluzione Algoritmica, Artescienza: Spazio Deformato, Casa dell’Architettura, Rome, 2006
Lille3000 Festival, Bombaysers de Lille, Rotonde de L’Opera, Lille, 2006
Festival d’Avignon, Avignon, 2005
ZKM, Karlsruhe, 2004
Cinémas du Futur, Lille 2004 Capitale Européenne de la Culture, Centre Euralille, Lille, 2004

Concept: Jeffrey Shaw

Project Directors: Jeffrey Shaw, Dennis Del Favero

Lead Software Engineer: Matthew McGinity

Software and Hardware Engineering: Ardrian Hardjono, Jared Berghold, Alex Kuptsov, Marc Chee, Robin Chow, Xin Guan

Project Co-ordination and Management: Damian Leonard, Ardrian Hardjono, Volker Kuchelmeister, Densan Obst, Kate Dennis, Sue Midgley, Joann Bowers

Projection system: F20SX+ with the generous support of Projectiondesign

This project was financially supported under the Australian Research Council’s Discovery and Linkage funding schemes, the UNSW University Capital Infrastructure Grants Scheme and the UNSW School of Mining Engineering

Dimensions

Screen Structure: 360-degrees, scalable 3.5m–4m height

Screen Diameter: Scalable 7.5m–10m

Screen Height: Scalable 3m–3.5m

Space Requirement without Projector Truss: 10.4m x 10.5m

Space Requirement with Projector Truss: 11m x 12.5m

Ceiling Height without Projector Truss Mount: 4.6m (Vibration risk if direct ceiling mount)

Ceiling Height with Projector Mount: 5.3m

Server Space: 2.5m x 3m for 2 computer stations and server racks

Hardware

Computing Software: Unity 3D Cluster

Projection system: 6 x Barco F35 projectors

Sound system: 5.1 channel sound receiver and speakers

User Interfaces: Tablet, Omnitrack full body motion capture

Axonometric view of the AVIE 3D visualisation system.

AVIE: Axonometric View

Follow

AVIE

AVIE visualisation forms

AVIE interaction platforms

AVIE Components