Posted on

Recording OpenGL output to H264 video

Apitrace is a tool for recording all the gl commands in a trace file. The trace file can be replay in later time, and they got a nice gui for checking all the gl call every frame, with introspection. They have a glretrace software that replay a trace file. We can use it to get the output of everyframe and push it in a gstreamer pipeline to make a video.

Why not using gtkRecordMyDesktop or other screen capture ? Sometime, the overhead of capturing and encoding video on live take too much CPU. And the application start to slow down. I didn’t see any slowdown using apitrace, and the trace file is very small compared to video output or raw video output.

So first, compile apitrace with stdout support:

$ git clone git://github.com/tito/apitrace.git
$ cd apitrace
$ git checkout snapshot-stdout
$ mkdir build
$ cd build
$ cmake ..
$ make

Take any opengl application, and make a trace file. The trace file will have the name of the binary. In my case, python is an alias to python2.7: the trace file will be python2.7.trace.

$ LD_PRELOAD=./glxtrace.so python ~/code/kivy/examples/demo/pictures/main.py
# replay for fun now
$ ./glretrace python2.7.trace

To be able to make a video from the trace file, you need to know the size of the window, and the initial framerate. Here, my example is running at 800×600, 60fps:

$ ./glretrace -sr python2.7-trace | \
  gst-launch fdsrc ! \
  videoparse width=800 height=600 format=rgbx framerate=60 ! \
  videoflip method=5 ! videorate ! ffmpegcolorspace ! \
  video/x-raw-yuv,width=800,height=600,framerate=\(fraction\)30/1 \
   x264enc pass=quant ! avimux ! filesink location=output.avi

The final video will be saved in output.avi. You can check the video output here :

If you like my work, tip me!

Posted on

Using microphone peak as input

I’m currently on a project that involve disabled peoples, audio and kinect. Theses boys and girls are doing lot of loud sounds, so the idea is to use their sound as a trigger. We can use gstreamer to make that work quite easily, cause it have everything we need: a audio source, and level calculator.

import pygst
pygst.require('0.10')
import gst, gobject
gobject.threads_init()
 
pipeline = gst.parse_launch(
    'pulsesrc ! audioconvert ! '
    'audio/x-raw-int,channels=2,rate=44100,endianness=1234,'
    'width=32,depth=32,signed=(bool)True !'
    'level name=level interval=10000000 !'
    'fakesink')
 
level = pipeline.get_by_name('level')
bus = pipeline.get_bus()
bus.add_signal_watch()
 
def show_peak(bus, message):
    # filter only on level messages
    if message.src is not level:
        return
    if not message.structure.has_key('peak'):
        return
    # read peak
    print 'peak', message.structure['peak'][0]
 
# connect the callback
bus.connect('message', show_peak)
 
# start the pipeline
pipeline.set_state(gst.STATE_PLAYING)
 
ctx = gobject.gobject.main_context_default()
while ctx:
    ctx.iteration()

The output could be something like this :

peak -35.2370719856
peak -35.0252114393
peak -10.8591229158
peak -4.6007387433
peak -4.85102463679
peak -6.45292575653
peak -6.83102903668
peak -7.39486319074
peak -13.9852340247
peak -17.423901433
peak -35.0852178272
peak -35.8725208237

Next, we can use that information to record their sound, and use it on some scenario. So, instead of use the fakesink, we can use appsink. This module allow you to read the buffer pushed by the previous module. So we can put theses buffers into a list, and use them when needed :)

The state machine will handle the 3 phases :

  1. Wait for a peak > -30db
  2. Recording the sound, stop when the peak is < -32db
  3. Replay the last sound

Note: The -30 / -32 are taken from my tests. If you have more noise, you need to adjust theses triggers.

And here is the final example:

import pygst
pygst.require('0.10')
import gst, gobject
gobject.threads_init()
 
pipeline_play = None
pipeline = gst.parse_launch(
    'pulsesrc ! audioconvert ! '
    'audio/x-raw-int,channels=2,rate=44100,endianness=1234,'
    'width=32,depth=32,signed=(bool)True !'
    'level name=level interval=10000000 !'
    'appsink name=app emit-signals=True')
 
state = 'wait'
peak = -99
buffers = []
level = pipeline.get_by_name('level')
app = pipeline.get_by_name('app')
bus = pipeline.get_bus()
bus.add_signal_watch()
 
def show_peak(bus, message):
    global peak
    # filter only on level messages
    if message.src is not level:
        return
    if not message.structure.has_key('peak'):
        return
    # read peak
    peak = message.structure['peak'][0]
 
def enqueue_audio_buffer(app):
    buffers.append(str(app.emit('pull-buffer')))
 
def play_sample(sample):
    global pipeline_play
    with open('audio.dat', 'wb') as fd:
        fd.write(sample)
    if pipeline_play is None:
        pipeline_play = gst.parse_launch(
            'filesrc location=audio.dat !'
            'audio/x-raw-int,channels=2,rate=44100,endianness=1234,'
            'width=32,depth=32,signed=(bool)True !'
            'audioamplify amplification=2 ! autoaudiosink')
    pipeline_play.set_state(gst.STATE_NULL)
    pipeline_play.set_state(gst.STATE_PLAYING)
 
# connect the callback
bus.connect('message', show_peak)
app.connect('new-buffer', enqueue_audio_buffer)
 
# start the pipeline
pipeline.set_state(gst.STATE_PLAYING)
 
# main loop
ctx = gobject.gobject.main_context_default()
while ctx:
    ctx.iteration()
    print state, peak
 
    # wait for somebody to make a sound
    if state == 'wait':
        if peak > -30:
            state = 'record'
            continue
        # discard any buffer
        buffers = []
 
    # record the current audio, until the peak is going down
    elif state == 'record':
        if peak < -32:
            state = 'replay'
            continue
 
    # replay last sound
    elif state == 'replay':
        play_sample(''.join(buffers))
        state = 'wait'

Usage: Just make a sound… and it will replay just after.