Working with openFrameworks
Coiled strip of new LEDs (APA102 chipset “hotness” link)
Last week I identified the issue of framerates from the prototype that displayed a Processing sketch on LEDs using the Raspberry Pi 3. I also identified several possible options for refactoring the code to achieve the ~30FPS performance needed for perceptually smooth animation. However I’m choosing to not go too deep into performance yet, as another consideration in this project is supporting sketches in other visual programming languages which may inform how to optimise the system later on.
One of my favourite creative coding tools is openFrameworks because it’s similar to Processing with the power of C++ behind it. On a related note: I plan to eventually build a web platform for sharing sketches in this project, and source files (not binaries) should be supported in order to keep it open and encourage learning. For that reason we have the task of compiling and running openFrameworks applications, which is the topic of this blog.
Debugging openFrameworks on a Headless Raspberry Pi 3
There are official openFrameworks guides available for ARM devices (the Raspberry Pi processor architecture) here. But I hit obstacles while following the guides, likely due to my environment being different to a RPi2 running the standard Raspbian operating system.
While researching the problems I found more up-to-date instructions here. Someone else in that forum thread mentioned a “15-20%” performance increase with the latest Debian, so I updated my container base image to install Debian Stretch:
# at the top of the Dockerfile:
FROM resin/raspberrypi3-debian:stretchHowever, executing the compiled OF example didn’t work and returned these errors:
root@bdd360b:~/openFrameworks/examples/templates/emptyExample# Xvfb :1 \
-screen 0 256x256x24+32 & export DISPLAY=":1" \
&& cd ~/openFrameworks/examples/templates/emptyExample/ \
&& make RunRelease
[4] 545
HOST_OS=Linux
checking pkg-config libraries: cairo zlib gstreamer-app-1.0 gstreamer-1.0 gstreamer-video-1.0 gstreamer-base-1.0 libudev freetype2 fontconfig sndfile openal openssl libcurl glfw3 rtaudio gtk+-3.0 libmpg123
with PKG_CONFIG_LIBDIR=
[notice ] ofAppEGLWindow: createSurface(): setting up EGL Display
[notice ] ofAppEGLWindow: createSurface(): EGL Display correctly set 0x2fb9d0
libEGL warning: DRI2: failed to open swrast (search paths /usr/lib/arm-linux-gnueabihf/dri:${ORIGIN}/dri:/usr/lib/dri)
[ error ] ofAppEGLWindow: createSurface(): eglInitialize returned EGL_FALSE
[ error ] ofAppEGLWindow: setupPeripherals(): peripherals not supported on X11
[notice ] ofAppEGLWindow: display(): eglSwapBuffers failed:After further research I identified a missing dependency and stopped the libEGL warning by installing it as so:
# install dependency containing swrast in the Dockerfile:
RUN apt-get update && \
apt-get -y install \
libgl1-mesa-driExecuting the compiled OF example then produced a new error:
[ error ] ofAppEGLWindow: createSurface(): no matching configs were found, num_configs: 0By modifying ofAppEGLWindow.cpp I saw there were 19 available ‘EGLConfigs,’ which are used to create the drawing surface and window. But none had the attributes required, the closest config (below) differed in three ways:
Desired Config Config #13
Red Size 8 Red Size 8
Green Size 8 Green Size 8
Blue Size 8 Blue Size 8
Alpha Size 8 Alpha Size 8
Luminance Size -1 Luminance Size 0 no match
Depth size 24 Depth size 24
Stencil size 8 Stencil size 8
Samples 1 Samples 0 no match
EGL_OPENGL_ES_BIT 77 no matchHence EGL returned zero configs. So I changed the desired config attributes back in ofAppEGLWindow.cpp to match those available, and finally saw successful execution:
[notice ] ofAppEGLWindow: createSurface(): surface created correctly
[notice ] ofAppEGLWindow: createSurface(): API bound correctly
[notice ] ofAppEGLWindow: createSurface(): -----EGL-----
[notice ] ofAppEGLWindow: createSurface(): EGL_VERSION_MAJOR = 1
[notice ] ofAppEGLWindow: createSurface(): EGL_VERSION_MINOR = 4
[notice ] ofAppEGLWindow: createSurface(): EGL_CLIENT_APIS = OpenGL OpenGL_ES
[notice ] ofAppEGLWindow: createSurface(): EGL_VENDOR = Mesa Project
[notice ] ofAppEGLWindow: createSurface(): EGL_VERSION = 1.4 (DRI2)
[notice ] ofAppEGLWindow: createSurface(): EGL_EXTENSIONS = EGL_KHR_create_context EGL_KHR_get_all_proc_addresses EGL_KHR_gl_colorspace EGL_KHR_no_config_context EGL_KHR_reusable_sync EGL_KHR_surfaceless_context EGL_MESA_configless_context
[notice ] ofAppEGLWindow: createSurface(): GL_RENDERER = Gallium 0.4 on llvmpipe (LLVM 3.9, 128 bits)
[notice ] ofAppEGLWindow: createSurface(): GL_VERSION = OpenGL ES-CM 1.1 Mesa 13.0.6
[notice ] ofAppEGLWindow: createSurface(): GL_VENDOR = VMware, Inc.
[notice ] ofAppEGLWindow: createSurface(): -------------
[ error ] ofAppEGLWindow: setupPeripherals(): peripherals not supported on X11
[notice ] Saved screenshot.pngThe screenshot was verified with ImageMagick using the same method used to test headless Processing two weeks ago.
What now?
I want to carry out further testing to display an OF sketch on LEDs. The changes I made to OF’s source code may have disabled anti-aliasing and hardware acceleration so perhaps different configurations will be needed elsewhere in the system before moving on.
Also, I realised the most computationally effecient approach to displaying Processing, OF and WebGL sketches interoperably would be to create their graphics framebuffers without using any window management, because they use some of the same graphics libraries. Unfortunately this would be difficult to maintain: Processing and OF specifically look for a window management system to create a new window in, so their source code would have to be modified. If performance is an issue later on in this project’s development then I’ll reconsider the approach.
Follow-up
openFrameworks framebuffer error:
root@bdd360b:/app# convert /var/tmp/Xvfb_screen0 -format '%[pixel:p{50,50}]' info:-
convert-im6.q16: no decode delegate for this image format `' @ error/constitute.c/ReadImage/504.
convert-im6.q16: no images defined `info:-' @ error/convert.c/ConvertImageCommand/3258.Compile flag difference causing OF to recompile:
-O3 -DNDEBUG -Wall -Werror=return-type -std=c++14 -DGCC_HAS_REGEX -march=armv6 -mfpu=vfp -mfloat-abi=hard -fPIC -ftree-vectorize -Wno-psabi -pipe -DOF_USING_GTK -DOF_USING_GTK -DOF_USING_MPG123 -DTARGET_RASPBERRY_PI -DSTANDALONE -DPIC -D_REENTRANT -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS -DTARGET_POSIX -DHAVE_LIBOPENMAX=2 -DOMX -DOMX_SKIP64BIT -DUSE_EXTERNAL_OMX -DHAVE_LIBBCM_HOST -DUSE_EXTERNAL_LIBBCM_HOST -DUSE_VCHIQ_ARM -I/opt/vc/include -I/opt/vc/include/IL -I/opt/vc/include/interface/vcos/pthreads -I/opt/vc/include/interface/vmcs_host/linux -pthread -D_REENTRANT -pthread -I/usr/include/gstreamer-1.0 -I/usr/include/AL -I/usr/include/arm-linux-gnueabihf -I/usr/include/rtaudio -I/usr/include/alsa -I/usr/include/gtk-3.0 -I/usr/include/at-spi2-atk/2.0 -I/usr/include/at-spi-2.0-I/usr/include/dbus-1.0 -I/usr/lib/arm-linux-gnueabihf/dbus-1.0/include -I/usr/include/gtk-3.0 -I/usr/include/gio-unix-2.0/ -I/usr/include/cairo -I/usr/include/pango-1.0 -I/usr/include/harfbuzz -I/usr/include/pango-1.0 -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/libpng16 -I/usr/include/glib-2.0 -I/usr/lib/arm-linux-gnueabihf/glib-2.0/include -I/root/openFrameworks/libs/glm/include -I/root/openFrameworks/libs/glm/include/glm -I/root/openFrameworks/libs/glm/include/glm/gtc -I/root/openFrameworks/libs/glm/include/glm/simd -I/root/openFrameworks/libs/glm/include/glm/gtx -I/root/openFrameworks/libs/glm/include/glm/detail -I/root/openFrameworks/libs/json/include -I/root/openFrameworks/libs/kiss/include -I/root/openFrameworks/libs/tess2/include -I/root/openFrameworks/libs/utf8/include -I/root/openFrameworks/libs/utf8/include/utf8 -I/root/openFrameworks/libs/openFrameworks -I/root/openFrameworks/libs/openFrameworks/sound -I/root/openFrameworks/libs/openFrameworks/app -I/root/openFrameworks/libs/openFrameworks/video -I/root/openFrameworks/libs/openFrameworks/events -I/root/openFrameworks/libs/openFrameworks/gl -I/root/openFrameworks/libs/openFrameworks/graphics -I/root/openFrameworks/libs/openFrameworks/utils -I/root/openFrameworks/libs/openFrameworks/3d -I/root/openFrameworks/libs/openFrameworks/communication -I/root/openFrameworks/libs/openFrameworks/types -I/root/openFrameworks/libs/openFrameworks/math
-O3 -DNDEBUG -Wall -Werror=return-type -std=c++14 -DGCC_HAS_REGEX -march=armv6 -mfpu=vfp -mfloat-abi=hard -fPIC -ftree-vectorize -Wno-psabi -pipe -DOF_USING_GTK -DOF_USING_GTK -DOF_USING_MPG123 -DTARGET_RASPBERRY_PI -DSTANDALONE -DPIC -D_REENTRANT -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS -DTARGET_POSIX -DHAVE_LIBOPENMAX=2 -DOMX -DOMX_SKIP64BIT -DUSE_EXTERNAL_OMX -DHAVE_LIBBCM_HOST -DUSE_EXTERNAL_LIBBCM_HOST -DUSE_VCHIQ_ARM -I/opt/vc/include -I/opt/vc/include/IL -I/opt/vc/include/interface/vcos/pthreads -I/opt/vc/include/interface/vmcs_host/linux -pthread -D_REENTRANT -pthread -I/usr/include/gstreamer-1.0 -I/usr/include/AL -I/usr/include/arm-linux-gnueabihf -I/usr/include/rtaudio -I/usr/include/alsa -I/usr/include/gtk-3.0 -I/usr/include/at-spi2-atk/2.0 -I/usr/include/at-spi-2.0 -I/usr/include/dbus-1.0 -I/usr/lib/arm-linux-gnueabihf/dbus-1.0/include -I/usr/include/gtk-3.0 -I/usr/include/gio-unix-2.0/ -I/usr/include/cairo -I/usr/include/pango-1.0 -I/usr/include/harfbuzz -I/usr/include/pango-1.0 -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/libpng16 -I/usr/include/glib-2.0 -I/usr/lib/arm-linux-gnueabihf/glib-2.0/include -I/root/openFrameworks/libs/glm/include -I/root/openFrameworks/libs/glm/include/glm -I/root/openFrameworks/libs/glm/include/glm/simd -I/root/openFrameworks/libs/glm/include/glm/gtc -I/root/openFrameworks/libs/glm/include/glm/gtx -I/root/openFrameworks/libs/glm/include/glm/detail -I/root/openFrameworks/libs/json/include -I/root/openFrameworks/libs/kiss/include -I/root/openFrameworks/libs/tess2/include -I/root/openFrameworks/libs/utf8/include -I/root/openFrameworks/libs/utf8/include/utf8 -I/root/openFrameworks/libs/openFrameworks -I/root/openFrameworks/libs/openFrameworks/app -I/root/openFrameworks/libs/openFrameworks/types -I/root/openFrameworks/libs/openFrameworks/events -I/root/openFrameworks/libs/openFrameworks/sound -I/root/openFrameworks/libs/openFrameworks/video -I/root/openFrameworks/libs/openFrameworks/3d -I/root/openFrameworks/libs/openFrameworks/math -I/root/openFrameworks/libs/openFrameworks/communication -I/root/openFrameworks/libs/openFrameworks/utils -I/root/openFrameworks/libs/openFrameworks/graphics -I/root/openFrameworks/libs/openFrameworks/glProcessing renderer error:
root@bdd360b:/app# Xvfb :1 -screen 0 256x256x24 -fbdir /var/tmp +extension GLX +render -noreset & export DISPLAY=":1"
[1] 663
root@bdd360b:/app# processing-java --sketch=./display_sketch --present
Could not parse -1 for --display
OpenGL error 1280 at bot beginDraw(): invalid enumerant
print test
GLVideo: Created compatibility symlinks
GLVideo requires the P2D or P3D renderer.
(java:1125): GLib-CRITICAL **: g_error_free: assertion 'error != NULL' failed
display_sketch.pde:7:0:7:0: RuntimeException: Could not load GStreamerError above shows that a window isn’t being created properly:
// save the current EGL context
#ifdef __APPLE__
context = gst_gl_context_get_current_gl_context (GST_GL_PLATFORM_CGL);
#elif GLES2
display = eglGetCurrentDisplay ();
surface = eglGetCurrentSurface (0);
context = eglGetCurrentContext ();
#else
display = glXGetCurrentDisplay ();
context = glXGetCurrentContext ();
#endif
if (!context) {
g_printerr ("GLVideo requires the P2D or P3D renderer.\n");
g_error_free (error);
return JNI_FALSE;
}It may be possible to read fb0 directly (e.g. in the ‘getScreenBitmap()’ function here) but this would not take advantage of hardware acceleration by default.
Seems ffmpeg can stream video from X11 and would be hardware accelerated here, and UV4L-xscreen library targetted at RPI here
Avoiding X completely (e.g. here) seems ideal but I can’t think of a way to implement it without hacking Processing/OF source code
Javascript has lots of graphics libs but needs to be run inside a compatible application such as a normal web browser or other apps here. Best suited is ‘WPE’ seen here which is hardware accelerated, but I asked how to add other packages like Xvfb and its apparently difficult. People have demonstrated adding other packages (e.g. chromecast here). Possible to make a package of WPE but no documentation on how (see here). Another WPE build guide is here and discussed here. Two related node packages that seem like they could provide WPE browser here and here. Also considering network booting 2nd RPi running WPE (guide here)