Getting Started - C++/OpenGL Weaving

Build and run C++/OpenGL example

• Navigate to the install directory. Then execute the following commands as administrator in "Windows PowerShell"

  cd examples/opengl_weaving
  mkdir build 
  cd  build
  cmake -G "Visual Studio 15 2017 Win64" ..

  Note:
  • To run these command as administrator, open the Start menu, and scroll to "Windows PowerShell", from the right-click menu, select "Run as administrator".
  • Please take into account that you should select the correct input for -G parameter based on the installed Visual Studio on your system.
• Open Visual Studio as Administrator. To do so, open the Start menu, and scroll to Visual Studio, then from the right-click or context menu of Visual Studio, select More > Run as administrator. When Visual Studio starts, (Administrator) appears after the product name in the title bar.
• Then, open example_opengl_weaving.sln located in examples/opengl_weaving/build address in Visual Studio. Build example_opengl_weaving project and run it.
• If the Eyetracker application is running a cube object is expected to be displayed in 3D. If the Eyetracker application is not running the default eye positions will be used. These are identical for the left and right perspective such that a stereo image with zero disparity will be presented. You will only be able to look around the cube in SR if the Eyetracker application is running in the background.

C++/OpenGL example

• The expected behavior of the example is to display a cube object in 3D. This requires the eyetracker and handtracker run simultaneously.
• Class SREyePairListener is the listener that accept eye pair coordinates from the EyeTracker class.
• To start streaming eye positions SRContext has to be initialized:

   
      const double startTime = (double)GetTickCount64() / 1000.0;
   
      // Create SR context.
      while (g_srContext == nullptr)
      {
          try
          {
              g_srContext = SR::SRContext::create();
              break;
          }
          catch (SR::ServerNotAvailableException e)
          {
              // Ignore errors because SR may be starting-up.
          }
   
          std::cout << "Waiting for context" << std::endl;
   
          // Wait a bit.
          Sleep(100);
   
          // Abort if we exceed the maximum allowed time.
          double curTime = (double)GetTickCount64() / 1000.0;
          if ((curTime - startTime) > maxTime)
              break;
      }
   
      // Wait for display to be ready.
      bool displayReady = false;
      while (g_srContext && !displayReady)
      {
          // Attempt to get display.
          SR::Display* display = SR::Display::create(*g_srContext);
          if (display != nullptr)
          {
              // Get the display location, and when it is valid, the device is ready.
              SR_recti displayLocation = display->getLocation();
              int64_t width = displayLocation.right - displayLocation.left;
              int64_t height = displayLocation.bottom - displayLocation.top;
              if ((width != 0) && (height != 0))
              {
                  displayReady = true;
                  break;
              }
          }
   
          std::cout << "Waiting for display" << std::endl;
   
          // Wait a bit.
          Sleep(100);
   
          // Abort if we exceed the maximum allowed time.
          double curTime = (double)GetTickCount64() / 1000.0;
          if ((curTime - startTime) > maxTime)
              break;
      }
   

• The OpenGL library is used to draw objects. For the extensive tutorial on how to visualize objects in OpenGL follow the link. In this guide we briefly point out that OpenGL objects are represented by a buffer of vertices (surface of each object can be represented by a set of triangles. Each of these triangles is represented by three vertices in 3D space) and by a buffer of the vertices' colors. In this example a cube is drawn using 12 triangles.
• Once the object is defined in the buffers it can be further visualized in a desired location and for a desired viewpoint. Those transformations are done in the CalculateModelViewProjectionMatrix() function. Rendering is done using OpenGL function glDrawElements().
• In order to visualize 3D object both left and right eyes, images have to be rendered. Therefore projection matrix is computed for the left and for the right eyes:

   
          // Render cube into stereo view texture.
   
   
          // Set view texture as target.
  #if defined(USE_DEPRECATED_WEAVER)
          if (!g_useDeprecatedFramebuffer || (g_demoMode == eDemoMode::StereoImage))
          {
              // Use the app-provided view texture.
              glBindFramebuffer(GL_FRAMEBUFFER, g_viewFramebuffer);
          }
          else
          {
              // Use the deprecated weaver-provided view texture.
              GLuint View = g_srWeaver->getFrameBuffer();
              glBindFramebuffer(GL_FRAMEBUFFER, View);
          }
  #else
          glBindFramebuffer(GL_FRAMEBUFFER, g_viewFramebuffer);
  #endif
   
          // Clear view texture to blue
          glClearColor(GetSRGB(0.05f), GetSRGB(0.05f), GetSRGB(0.25f), GetSRGB(1.0f));
          glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
   
   
          // Render stereo views.
          for (int i = 0; i < 2; i++)
          {
              // Set viewport to render to left, then right.
              glViewport(i * g_viewTextureWidth, 0, g_viewTextureWidth, g_viewTextureHeight);
   
              // Get transform matrix.
              mat4f mvp = CalculateModelViewProjectionMatrix((i == 0) ? leftEye : rightEye, (float)elapsedTime);
   
              // Draw.
              glUseProgram(g_shaderProgram);
              glUniformMatrix4fv(g_uniformTransform, 1, GL_FALSE, (const GLfloat*) &mvp);
              glBindVertexArray(g_vao);
              glEnableVertexAttribArray(g_vaoVertexXYZAttributeIndex);
              glEnableVertexAttribArray(g_vaoVertexRGBAttributeIndex);
              glDrawElements(GL_TRIANGLES, g_vaoIndexCount, GL_UNSIGNED_INT, NULL);
              glDisableVertexAttribArray(g_vaoVertexXYZAttributeIndex);
              glDisableVertexAttribArray(g_vaoVertexRGBAttributeIndex);
          }
   

• Left and right images are stitched together by sequentially applying glViewport() function and rendered by glDrawElements() function. Subsequently, this image is passed to the weaving part of the SR SDK. During the weaving process pixels corresponding to the left and right images are distributed over the pixels of the display according to the display specifications and its calibration.

C++/OpenGL example weaving

Note:

• Weaving in our context is the process that for every subpixel (red,green,blue) selects which view (left,right) should be shown. Every subpixel can only be seen from certain angles, so for every subpixel it is determined whether the right eye or left eye of the user is closest to the direction in which this subpixel can be seen.

Loading weaving parameters and setting up a weaving shader has been encapsulated in the GLWeaver class for this example. This class provides a framebuffer that we can render our existing output to. Doing so will allow us to get weaving set up quickly and without having to compromise our existing application.

To convert the existing application which renders side-by-side output to one that uses software to weave an image for our SR devices, we need to make three small modifications.

• We need to construct a GLWeaver instance. We only have to provide the size at which we want to render our scene side-by-side:

   
      // Create weaver.
  #ifdef USE_DEPRECATED_WEAVER
      g_srWeaver = new SR::PredictingGLWeaver(*g_srContext, g_viewTextureWidth * 2, g_viewTextureHeight, GetTextureFormat(), g_hWnd);
      if (!g_useDeprecatedFramebuffer || (g_demoMode == eDemoMode::StereoImage))
          g_srWeaver->setInputFrameBuffer(g_viewFramebuffer, g_viewTexture);
  #else
      WeaverErrorCode createWeaverResult = SR::CreateGLWeaver(*g_srContext, g_hWnd, &g_srWeaver);
      if (createWeaverResult != WeaverErrorCode::WeaverSuccess)
      {
          OnError("Failed to create weaver");
          return false;
      }
      g_srWeaver->setInputViewTexture(g_viewTexture, g_viewTextureWidth, g_viewTextureHeight, GetTextureFormat());
  #endif
   

• Next our render loop needs to start by binding the framebuffer provided by the GLWeaver:

   
          // Set view texture as target.
  #if defined(USE_DEPRECATED_WEAVER)
          if (!g_useDeprecatedFramebuffer || (g_demoMode == eDemoMode::StereoImage))
          {
              // Use the app-provided view texture.
              glBindFramebuffer(GL_FRAMEBUFFER, g_viewFramebuffer);
          }
          else
          {
              // Use the deprecated weaver-provided view texture.
              GLuint View = g_srWeaver->getFrameBuffer();
              glBindFramebuffer(GL_FRAMEBUFFER, View);
          }
  #else
          glBindFramebuffer(GL_FRAMEBUFFER, g_viewFramebuffer);
  #endif
   
          // Clear view texture to blue
          glClearColor(GetSRGB(0.05f), GetSRGB(0.05f), GetSRGB(0.25f), GetSRGB(1.0f));
          glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
   

  The render loop can continue as normal, objects like the pyramid in this example can be rendered to the bound framebuffer. The application can define their own shaders to do so.
• Finally we should bind the default framebuffer again and use the weave function to generate our final output.

   
      // Set viewport to cover entire window.
      glBindFramebuffer(GL_FRAMEBUFFER, 0);
      glViewport(0, 0, g_windowWidth, g_windowHeight);
   
      // Perform weaving.
  #if !defined(USE_DEPRECATED_WEAVER)
      g_srWeaver->weave();
  #else
      // NOTE: runtimes <1.34 do not support non-fullscreen windows, nor non-fullscreen viewports!
      // NOTE: runtimes <1.34 need the fullscreen size as input to the weave function, >=1.34 will ignore it
   
      g_srWeaver->weave(/* ignored in runtime >=1.34 */ g_windowWidth, /* ignored in runtime >=1.34 */g_windowHeight);
  #endif
   

  The weave function simply combines pixels from the side-by-side left and right images in a very specific way. These combined colors are used to display an image on the LCD panel under the SR lens. Your eyes will only see colors meant for left or right after the light passes through the physical lens.