Creative programming: openFrameworks - installation and example of visualization of music
When was the last time you programmed in C ++?
Maybe this is your everyday work, and my last (until yesterday) C ++ project was back in 2000 - a diploma project based on Visual Studio 4.2 (by the way, the system was good), and since then I switched to web development - scripting languages.
That is, now I am a beginner in C ++, but this did not stop me from deploying the infrastructure for a couple of hours, building and building a multimedia application in C ++, which visualizes music with different effects. And they helped me with this:
- open framework for creating interactive applications - openFrameworks
- free IDE Code :: Blocks
See what I got
And it all started like this - after another listening to music from one composer from Samara, I thought it would be interesting to try to visualize the music, and turned to Denis Perevalov (who has been creating various interactive art / performance systems for several years) - he answered me, that this is done without problems on the basis of openFrameworks and that in the examples to his book (and he is the author of the book on openFrameworks), there is an implementation of such a task.
That is, I had to just - to establish a framework, to modify and compile the example in C ++ ... This process - installation, configuration, and a brief description will openFrameworks this article.
openFrameworks is a system with which you can program an interactive multimedia application, that is, art,performances , etc., it is free, open and cross-platform system (linux, mac, win), and there are also versions for ARM (for example for RPi), and builds for iPhone and Android.
By the way at KDPV - one of the installations based on openFrameworks (Seven Video Guides. Exhibited at VDNH in the exposition of the Polytechnic Museum. Moscow, 2014).
What is openFrameworks? This is a set of modules - for integration with Arduino, with kinekt, with OpenCV pattern recognition system, drawing 3D graphics, working with sound, cameras, etc. with which you can make an interactive application. And all this is based on C ++.
OpenFrameworks came into my field of vision when I went to the robotic installations created with its help.
The plan is this:
- 1. Setting up openFrameworks
- 2. Basic principles of openFrameworks applications
- 3. Test case
1. Setting up openFrameworks
Next steps:
- installing openFrameworks (for CodeBlocks)
- Install IDE CodeBlocks
- copy openFrameworks libraries for CodeBlocks compiler
Install openFrameworks
According to this download page, I chose the version of openFrameworks, for my OS and for the IDE on which I planned to work.
In my case, win and code :: blocks: download the archive of_v0.8.4_win_cb_release.zip
Unpack, the archive contains the following folders:
* addons
* apps
* docs
* examples
* export
* libs
* other
* projectGenerator
* scripts
These are C ++ libraries of openFrameworks, examples, addons, etc.
In order to create an openFrameworks application, it is better to use the IDE.
Install IDE CodeBlocks
As an IDE, I decided to choose code :: blocks (visual studio will still be a
bit big for me now) CodeBlocks is a free and open IDE created on the basis of the cross-platform GUI library wxWidgets . According to this page openframeworks.cc/setup/codeblocks download IDE CodeBlocks. Release 12.11 version . From here . This build comes with MinGW , an open development environment for the win platform.
This is what IDE CodeBlocks looks like
Copy openFrameworks libraries for CodeBlocks compiler
An important point is that in order for OpenFrameworks projects to be successfully assembled from the IDE CodeBlocks, you need to copy additional files to MinGW.
Here is the item .
Download Additions for Code :: Blocks to work with openFrameworks zip archive.
Unpack it in a temporary folder, and copy it to the appropriate folders in the installed CodeBlocks, according to this instruction:
Add the contents of the folder "add_to_codeblocks_mingw_include" into "... \ CodeBlocks \ MinGW \ include"
Add the contents of the folder "add_to_codeblocks_mingw_lib" into "... \ CodeBlocks \ MinGW \ lib"
That's it, now we are ready to build openFrameworks projects!
2. Basic principles of openFrameworks applications
Build test project
Let's open a test project, for this we’ll select from the start page of the IDE CodeBlocks select “Open an existing project ...” (or in File - Import Project - Dev-C ++ project ... - and select the file type *. *)
Go to the folder where we deployed openFrameworks, go to examples / empty / emptyExample, and open the project file emptyExample.
This is what the IDE looks like after opening the project:
Let's try to start the project right away - the icon is indicated by an arrow in the picture or press F9 - RUN.
If the application is not built, then the assembly will start (after your confirmation) and at the end of the assembly - the application starts.
If everything is configured correctly, then at the end of the assembly process a console window will open, and we will see this window:
Congratulations! So everything is set up correctly. And in the bin folder, the emptyExample.exe application appeared, which you can already run independently.
Files
Now let's look at the files of our emptyExample project, they are located in the src folder:
* main.cpp
* ofApp.h
* ofApp.cpp
File main.cpp :
Hidden text
#include"ofMain.h"#include"ofApp.h"//========================================================================intmain( ){
ofSetupOpenGL(1024,768, OF_WINDOW); // <-------- setup the GL context// this kicks off the running of my app// can be OF_WINDOW or OF_FULLSCREEN// pass in width and height too:
ofRunApp( new ofApp());
}
It defines the window of our application, and then an instance of the ofApp class is created.
File ofApp.h :
Hidden text
#pragma once#include"ofMain.h"classofApp :public ofBaseApp{
public:
voidsetup();
voidupdate();
voiddraw();
voidkeyPressed(int key);
voidkeyReleased(int key);
voidmouseMoved(int x, int y);
voidmouseDragged(int x, int y, int button);
voidmousePressed(int x, int y, int button);
voidmouseReleased(int x, int y, int button);
voidwindowResized(int w, int h);
voiddragEvent(ofDragInfo dragInfo);
voidgotMessage(ofMessage msg);
};
Here our class ofApp is defined, inherited from ofBaseApp. And the methods.
The main class of the application ofApp.cpp :
Hidden text
#include"ofApp.h"//--------------------------------------------------------------void ofApp::setup(){
}
//--------------------------------------------------------------void ofApp::update(){
}
//--------------------------------------------------------------void ofApp::draw(){
}
//--------------------------------------------------------------void ofApp::keyPressed(int key){
}
//--------------------------------------------------------------void ofApp::keyReleased(int key){
}
//--------------------------------------------------------------void ofApp::mouseMoved(int x, int y){
}
//--------------------------------------------------------------void ofApp::mouseDragged(int x, int y, int button){
}
//--------------------------------------------------------------void ofApp::mousePressed(int x, int y, int button){
}
//--------------------------------------------------------------void ofApp::mouseReleased(int x, int y, int button){
}
//--------------------------------------------------------------void ofApp::windowResized(int w, int h){
}
//--------------------------------------------------------------void ofApp::gotMessage(ofMessage msg){
}
//--------------------------------------------------------------void ofApp::dragEvent(ofDragInfo dragInfo){
}
As we see - nothing is implemented, we saw just an empty but working openFrameworks application.
OpenFrameworks application work cycle
The main methods of our class are:
void setup (); void update (); void draw ();
The architecture of any openFrameworks application is as follows:
The setup method prescribes settings, provisioning resources, etc. This method is executed once at application launch, before the start of the main loop.
The main cycle is update and draw, where in the first method only calculations occur, and in the second draw - drawing. And after that the cycle repeats.
Exit by pressing Esc.
3. Test case
Here we come to our task - visualization of music.
This site provides examples for Mastering openFrameworks: Creative Coding Demystified. The files themselves can be downloaded for free from the book card (after registration).
Here is a video of examples .
Basic Dancing Cloud Example
And here is the example that I wanted to take as a basis and modify - called Dancing Cloud (06-Sound / 06-DancingCloud):
I downloaded this example, and unpacked the archive in the root of my openFrameworks folder - this is important, because The project folder should be 2 levels lower.
Here is the WHOLE source code for the 06-Sound / 06-DancingCloud project:
main.cpp :
Hidden text
#include"testApp.h"#include"ofAppGlutWindow.h"//--------------------------------------------------------------intmain(){
ofAppGlutWindow window; // create a window// set width, height, mode (OF_WINDOW or OF_FULLSCREEN)
ofSetupOpenGL(&window, 1024, 768, OF_WINDOW);
ofRunApp(new testApp()); // start the app
}
testApp.h :
Hidden text
#pragma once#include"ofMain.h"/*
This example draws points cloud and plays music track.
Also it analyzes music spectrum and use this data for controlling
the radius and shuffle of the cloud.
It's the example 06-DancingCloud from the book
"Mastering openFrameworks: Creative Coding Demystified",
Chapter 6 - Working with Sounds
Music track "surface.wav" by Ilya Orange (soundcloud.com/ilyaorange)
*/classtestApp :public ofBaseApp{
public:
voidsetup();
voidupdate();
voiddraw();
voidmousePressed(int x, int y, int button);
ofSoundPlayer sound; //Sound samplevoidkeyPressed(int key);
voidkeyReleased(int key);
voidmouseMoved(int x, int y);
voidmouseDragged(int x, int y, int button);
voidmouseReleased(int x, int y, int button);
voidwindowResized(int w, int h);
voiddragEvent(ofDragInfo dragInfo);
voidgotMessage(ofMessage msg);
};
testApp.cpp
Hidden text
#include"testApp.h"constint N = 256; //Number of bands in spectrumfloat spectrum[ N ]; //Smoothed spectrum valuesfloat Rad = 500; //Cloud raduis parameterfloat Vel = 0.1; //Cloud points velocity parameterint bandRad = 2; //Band index in spectrum, affecting Rad valueint bandVel = 100; //Band index in spectrum, affecting Vel valueconstint n = 300; //Number of cloud points //Offsets for Perlin noise calculation for pointsfloat tx[n], ty[n];
ofPoint p[n]; //Cloud's points positionsfloat time0 = 0; //Time value, used for dt computing//--------------------------------------------------------------void testApp::setup(){
//Set up sound sample
sound.loadSound( "surface.wav" );
sound.setLoop( true );
sound.play();
//Set spectrum values to 0for (int i=0; i<N; i++) {
spectrum[i] = 0.0f;
}
//Initialize points offsets by random numbersfor ( int j=0; j<n; j++ ) {
tx[j] = ofRandom( 0, 1000 );
ty[j] = ofRandom( 0, 1000 );
}
}
//--------------------------------------------------------------void testApp::update(){
//Update sound engine
ofSoundUpdate();
//Get current spectrum with N bandsfloat *val = ofSoundGetSpectrum( N );
//We should not release memory of val,//because it is managed by sound engine//Update our smoothed spectrum,//by slowly decreasing its values and getting maximum with val//So we will have slowly falling peaks in spectrumfor ( int i=0; i<N; i++ ) {
spectrum[i] *= 0.97; //Slow decreasing
spectrum[i] = max( spectrum[i], val[i] );
}
//Update particles using spectrum values//Computing dt as a time between the last//and the current calling of update() float time = ofGetElapsedTimef();
float dt = time - time0;
dt = ofClamp( dt, 0.0, 0.1 );
time0 = time; //Store the current time //Update Rad and Vel from spectrum//Note, the parameters in ofMap's were tuned for best result//just for current music track
Rad = ofMap( spectrum[ bandRad ], 1, 3, 400, 800, true );
Vel = ofMap( spectrum[ bandVel ], 0, 0.1, 0.05, 0.5 );
//Update particles positionsfor (int j=0; j<n; j++) {
tx[j] += Vel * dt; //move offset
ty[j] += Vel * dt; //move offset//Calculate Perlin's noise in [-1, 1] and//multiply on Rad
p[j].x = ofSignedNoise( tx[j] ) * Rad;
p[j].y = ofSignedNoise( ty[j] ) * Rad;
}
}
//--------------------------------------------------------------void testApp::draw(){
ofBackground( 255, 255, 255 ); //Set up the background//Draw background rect for spectrum
ofSetColor( 230, 230, 230 );
ofFill();
ofRect( 10, 700, N * 6, -100 );
//Draw spectrum
ofSetColor( 0, 0, 0 );
for (int i=0; i<N; i++) {
//Draw bandRad and bandVel by black color,//and other by gray colorif ( i == bandRad || i == bandVel ) {
ofSetColor( 0, 0, 0 ); //Black color
}
else {
ofSetColor( 128, 128, 128 ); //Gray color
}
ofRect( 10 + i * 5, 700, 3, -spectrum[i] * 100 );
}
//Draw cloud//Move center of coordinate system to the screen center
ofPushMatrix();
ofTranslate( ofGetWidth() / 2, ofGetHeight() / 2 );
//Draw points
ofSetColor( 0, 0, 0 );
ofFill();
for (int i=0; i<n; i++) {
ofCircle( p[i], 2 );
}
//Draw lines between near pointsfloat dist = 40; //Threshold parameter of distancefor (int j=0; j<n; j++) {
for (int k=j+1; k<n; k++) {
if ( ofDist( p[j].x, p[j].y, p[k].x, p[k].y )
< dist ) {
ofLine( p[j], p[k] );
}
}
}
//Restore coordinate system
ofPopMatrix();
}
//--------------------------------------------------------------void testApp::keyPressed(int key){
}
//--------------------------------------------------------------void testApp::keyReleased(int key){
}
//--------------------------------------------------------------void testApp::mouseMoved(int x, int y){
}
//--------------------------------------------------------------void testApp::mouseDragged(int x, int y, int button){
}
//--------------------------------------------------------------void testApp::mousePressed(int x, int y, int button){
}
//--------------------------------------------------------------void testApp::mouseReleased(int x, int y, int button){
}
//--------------------------------------------------------------void testApp::windowResized(int w, int h){
}
//--------------------------------------------------------------void testApp::gotMessage(ofMessage msg){
}
//--------------------------------------------------------------void testApp::dragEvent(ofDragInfo dragInfo){
}
Comment from Denis (author of the book), regarding the algorithm that visualizes music:
300 points (const int n = 300;) move along the trajectories of Perlin noise, and neighboring points are connected by segments.
Cloud radius and speed are two parameters that are taken from sound analysis.
The sound analysis is this: the original sound is converted into a spectrum (using the window Fourier transform). Two spectrum values are selected, which become two parameters that control the movement of a point cloud. These two frequencies are shown in black on the spectrum.
We look at the differences from our emptyExample.
main.cpp - identical in essence.
In testApp.h, the sound attribute of the ofSoundPlayer class has been added:
ofSoundPlayer sound; //Sound sample
ofSoundPlayer is the sound docs base class .
The most interesting is in testApp.cpp.
Here are the variables that are used to implement the logic:
constint N = 256; // Число полос спектраfloat spectrum[ N ]; // массив для значений спектраfloat Rad = 500; // радиус облакаfloat Vel = 0.1; // параметр скорости точек облакаint bandRad = 2; // полоса спектра что будет модифицировать Rad параметрint bandVel = 100; // полоса спектра что будет модифицировать Vel параметрconstint n = 300; // число точек в облаке// рассчитанные смещения точке согласно шума Перлинаfloat tx[n], ty[n];
ofPoint p[n]; // координаты точек облакаfloat time0 = 0; // используется для вычисления dt - прошедшего времени между отображениями
Here is what is written in the testApp :: setup () method. Music, variables for displaying the spectrum, and cloud points are initialized:
void testApp::setup(){
//Set up sound sample
sound.loadSound( "surface.wav" );
sound.setLoop( true );
sound.play();
//Set spectrum valuesto0for (int i=0; i<N; i++) {
spectrum[i] = 0.0f;
}
//Initialize points offsets by random numbers
for ( int j=0; j<n; j++ ) {
tx[j] = ofRandom( 0, 1000 );
ty[j] = ofRandom( 0, 1000 );
}
}
We see that the music is loading, and immediately begins to play, and in a loop.
In the testApp :: update () method, all the magic happens when calculating the placement of points.
Hidden text
void testApp::update(){
//Update sound engine
ofSoundUpdate();
//Getcurrent spectrum with N bands
float *val = ofSoundGetSpectrum( N );
//We should notrelease memory of val,
//because it is managed by sound engine
//Update our smoothed spectrum,
//by slowly decreasing its valuesand getting maximum with val
//So we will have slowly falling peaks in spectrum
for ( int i=0; i<N; i++ ) {
spectrum[i] *= 0.97; //Slow decreasing
spectrum[i] = max( spectrum[i], val[i] );
}
//Update particles using spectrum values
//Computing dt as a timebetween the last
//and the current calling ofupdate()
floattime = ofGetElapsedTimef();
float dt = time - time0;
dt = ofClamp( dt, 0.0, 0.1 );
time0 = time; //Store the currenttime
//Update Rad and Vel from spectrum
//Note, the parameters in ofMap's were tuned for best result
//just for current music track
Rad = ofMap( spectrum[ bandRad ], 1, 3, 400, 800, true );
Vel = ofMap( spectrum[ bandVel ], 0, 0.1, 0.05, 0.5 );
//Update particles positions
for (int j=0; j<n; j++) {
tx[j] += Vel * dt; //move offset
ty[j] += Vel * dt; //move offset
//Calculate Perlin's noise in [-1, 1] and
//multiply on Rad
p[j].x = ofSignedNoise( tx[j] ) * Rad;
p[j].y = ofSignedNoise( ty[j] ) * Rad;
}
}
Here is the drawing method, here according to the calculated data, the spectrum, the points of the cloud, and the lines between the points are displayed (provided that they are closer than float dist = 40):
void testApp::draw(){
ofBackground( 255, 255, 255 ); //Set up the background//Draw background rect for spectrum
ofSetColor( 230, 230, 230 );
ofFill();
ofRect( 10, 700, N * 6, -100 );
//Draw spectrum
ofSetColor( 0, 0, 0 );
for (int i=0; i<N; i++) {
//Draw bandRad and bandVel by black color,//and other by gray colorif ( i == bandRad || i == bandVel ) {
ofSetColor( 0, 0, 0 ); //Black color
}
else {
ofSetColor( 128, 128, 128 ); //Gray color
}
ofRect( 10 + i * 5, 700, 3, -spectrum[i] * 100 );
}
//Draw cloud//Move center of coordinate system to the screen center
ofPushMatrix();
ofTranslate( ofGetWidth() / 2, ofGetHeight() / 2 );
//Draw points
ofSetColor( 0, 0, 0 );
ofFill();
for (int i=0; i<n; i++) {
ofCircle( p[i], 2 );
}
//Draw lines between near pointsfloat dist = 40; //Threshold parameter of distancefor (int j=0; j<n; j++) {
for (int k=j+1; k<n; k++) {
if ( ofDist( p[j].x, p[j].y, p[k].x, p[k].y )
< dist ) {
ofLine( p[j], p[k] );
}
}
}
//Restore coordinate system
ofPopMatrix();
}
My modifications
I took the music volfworks: soundcloud.com/volfworks
The author kindly agreed to my use of his composition Star.
The first thing - I replaced wav with mp3 - openFrameworks supports mp3. I also did streaming playback (otherwise all 8MB should be loaded immediately - added true as the second parameter, docs ).
sound.loadSound( "zvezda.mp3", true );
Added loading background image:
stars.loadImage("stars.jpg");
Changed the color scheme, made the effects of transparency time-dependent.
Snippet from ofApp :: draw ():
// включение использования прозрачности, и рисование квадрата поверх фоновой картинки
// с прозрачностью определяемой bg_transparent
ofEnableAlphaBlending();
ofSetColor(0, 0, 0, bg_transparent);
ofRect(0, 0, 1000, 700);
ofDisableAlphaBlending();
// рисование текста указанного цвета, в координатах
ofSetHexColor(0x606060);
ofDrawBitmapString("Music by: volfworks", 800,610);
The whole project is posted on github: github.com/nemilya/of_volfworks_example
Video creation
This caused some difficulties, and was ultimately performed using the Camtasia Recorder.
References
The main site of the project: openframeworks.cc there are quite good tutorials.
If you work with openFrameworks, or it’s interesting to try, then I invite you to the Russian-speaking group on openFrameworks .