to interact:
- drag your mouse to rotate the frame
- or press the arrow keys to rotate the frame
if your mouse gets captured:
- hover it over a static part of the website to regain control
- or click on the menu in the top right corner to navigate to another section of the site
the videos below showcase:
- system tracking and responding to one person
- an experiment using height data to move the world in the z axis
- the chaos that can ensue as multiple people move in the space and system sensitivity is increased
- click the image below to cycle through
the media below showcases the logic layer and interaction mechanics. specifically:
- the touch designer network
- the generated world
- how changing motion and presence values affect that world
the system components belong to the following categories, described in detail below:
- sensing (input)
- processing (logic)
- content pipeline (generation)
- output (display)
- maintenance / reliability
1. sensing (input)
| component | function | source |
|---|---|---|
| ceiling-mounted azure kinect (qty depends on room size — e.g. 2 units for a 25×38 ft room) | depth tracking to detect presence/position of visitors | to be sourced — compatible with standard azure kinect hardware |
| mini-pc per kinect (1:1 in prior install; can run 1 pc per 2 kinects if a sync cable and a sufficiently long high-speed data cable are used) | runs kinect capture/processing locally; accessed via ssh from the main gallery computer for setup, coding, and monitoring during install and exhibition | windows pc with latest kinect sdk installed, able to process numpy arrays, wifi-enabled. exact specs otherwise flexible. |
| high-speed data cable, kinect to pc | ~40gbps usb-c cable for reliable data transfer between each kinect and its pc | to be sourced |
2. processing (logic)
| component | function | source |
|---|---|---|
| custom tracking software | computes presence (ratio of white-to-black pixels after masking the depth array to the range where shoulders typically sit); motion (sum of pixel-value differences between consecutive frames); and center of mass (average x/y/z position of all detected pixels, used to drive the virtual camera's position in the 3d world) | artist-built, maintained, and supplied |
| osc data bridge | sends real-time presence, motion, and position values from tracking software to touchdesigner | artist-built, maintained, and supplied |
3. content pipeline (generation)
| component | function | source |
|---|---|---|
| youtube api + automated queryer + database | pulls randomized search terms, retrieves matching videos daily, logs video ids to a database | artist-built; requires api key, supplied by artist |
| yt-dlp download routine | downloads video files from the database | artist-built |
| content refresh (current vs. planned) | prior presentation used a pre-downloaded pool — as much footage as fit on 2tb of storage, each file tagged by unique id, sequentially selected and removed from the pool in touchdesigner as it was used. a live cyclic download/delete system, enabling continuous, unbounded content refresh during the exhibition run, is in development and would be implemented for this presentation, time permitting. if implemented, this would also require a self-updating routine for yt-dlp, to keep the automated download process functional through youtube-side platform changes. | artist-built |
| storage (2tb+ external ssd or equivalent) | temporary storage for downloaded video before/during exhibition | gallery or artist, to be confirmed |
4. output (display)
| component | function | source |
|---|---|---|
| touchdesigner (pro license) | builds and renders the real-time 3d video environment | license to be secured — purchase or gallery-provided |
| 360° projection setup (projectors + edge-blending/calibration) | projection surface for the walkable 3d archive | expected from venue. this is the component the artist has the least in-house expertise in — prior presentation took place in a gallery with an existing 360° projection infrastructure already built and calibrated; the artist did not set this up. detailed specs from that prior install (projector count, placement, calibration approach) could be gathered and shared post-award if selected, but are not available ahead of the application deadline. |
| gallery/presentation computer | runs touchdesigner and tracking software in real time | gallery-provided or artist-supplied; highest available gpu/ram recommended — exact spec to be confirmed against prior installation's reference machine ahead of install |
5. maintenance / reliability
- daily operation: minimal. in the prior presentation, the kinect tracking script was automated to start each morning and shut down each evening; gallery staff only needed to open the touchdesigner file daily. no maintenance issues occurred over several weeks of exhibition.
- known failure point: occasional lag or freezing under high-presence conditions (large crowds), caused by gpu load. mitigated by either capping the maximum tracked presence value or running on a higher-spec gpu.
- platform dependency: if the live cyclic download system (see content pipeline, above) is implemented, yt-dlp would need a self-updating routine to remain functional through youtube-side platform changes. not required for the pre-downloaded-pool approach used previously.
- network: requires an isolated wifi network, primarily to ensure smooth, low-latency communication between the kinect mini-pcs and the main gallery computer. if the live cyclic download system is implemented, network isolation would additionally help prevent bandwidth competition from affecting real-time download and playback performance.