Audiovisual Processing Projects
The video below demonstrates my media processing project which uses networked cameras and different layers of analysis and feedback to gradually distort webcam streams. It was inspired by artist Quayola’s ‘paintings’ (link here).
In the software is a ‘view’ C++ class that updates images to 1) detect the closest visual keypoint to the cursor and 2) compute motion using optical flow. That data is then passed to GLSL graphics card shaders together with the livestream and two more parameters; ‘glitch’ and ‘expression,’ normalised from 0 to 1.
What is happening in the application to ensure artifacts persist is a two-frame cycle where each update is drawn onto one of two framebuffers. That framebuffer gets sent to the graphics card shader in the next update which is, in turn, drawn on the other framebuffer. The keypoint position and optical flow are incorporated in the shader to maintain visual clarity in areas of the livestream that don’t have much action, while datamoshing areas that do.
Eventually the streams end up entirely obfuscated as ‘glitch’ and ‘expression’ increase. Download the ZIP here: link
Mini Projects
Download ZIP here: link
1: AAVP-8_skew-effect, AAVP-8_video-feedback
2: AAVP-9_frame-differencing
3: AAVP_image-feature-invariance
Example output:
Comparing descriptors of input image with descriptors of...
- the input image rescaled by 0.99999
- the input image rescaled by 0.99744
- the input image rescaled by 0.93439
Average variance (lower is more robust):
- ORB: 59.9
- BRISK: 70.3
- FREAK: 65.9
- BRIEF: 61.8
//--------------------------------------------------------------
Comparing descriptors of input image with descriptors of...
- the input image rotated by 1 degrees
- the input image rotated by 2 degrees
- the input image rotated by 3 degrees
Average variance (lower is more robust):
- ORB: 61.0
- BRISK: 72.3
- FREAK: 64.8
- BRIEF: 66.0
//--------------------------------------------------------------4: AAVP_glsl-generative-video
For Advanced Audio-Visual Processing (IS53027C)