NA-MIC Project Weeks

Back to Projects List

Future of Rendering in VTK, ITK and Slicer

Key Investigators

• Rafael Palomar (NTNU, Norway)
• Jean-Christophe Fillion-Robin (Kitware, Inc., USA)
• Sankhesh Jhaveri (Kitware, Inc., USA)
• Matt McCormick (Kitware, Inc., USA)
• Lucas Gandel (Kitware SAS, USA)
• Forrest Li (Kitware, Inc., USA)

Presenter location:

Description

The goal of this breakout session is to gather all parties interested in the future of rendering in VTK and Slicer, present ongoing development by Kitware and others and discuss potential future directions and clinical and biomedical needs.

During the Breakout Session

Links and notes are organized at https://hackmd.io/j0xxip3jR_O9220uJUjJKg. It is a markdown based document we can collaboratively & interactively edit.

Once the breakout session is over, we will contribute the information back to this page. See Notes below.

Topics

• VTK C++:
  • WebGPU
  • OpenXR
• itk-viewer and itkwidgets
• vtk.js
• SlicerVirtualReality

Notes

VTK C++: WebGPU

Contact: Sankhesh Jhaveri @ Kitware

• Direct replacement of OpenGL for native desktop and webassembly for browsers
• Better separation between windowing UI and rendering API
• Implementation agnostic support via the webgpu C API that would allow switching between dawn, wgpu, and future webgpu implementations.
• What is the timeline to transition to WebGPU ?
  • this is pending funding, we are currently waiting for review of NSF grant

Notes:

• Custom shader code written in GLSL would have to be coverted to WGSL shader language
• Moving forward we will have one volume rendering implementation
• ANARI will allow to have multiple backend for ray tracing
• Sanhesh et al are active and involved with the WebGPU forum

See notes from 39th project week.

VTK C++: WebAssembly

Contact: Jaswant Panchumarti @ Kitware

See notes from 39th project week.

VTK C++: OpenXR

Contact: Lucas Gandel @ Kitware

• Hand interaction improvements for the Hololens 2:
  • Combine ‘aim/select’ and ‘grip/squeeze’ poses, mimic the Unity behavior
  • Hand joints/finger tracking
• Support occlusion from the real world in the Hololens 2:
  • Implement scene understanding extension
• Custom controller model:
  • The controller model extension is not implemented by most of the runtimes. We should provide an API for users to load their own controller model files (GLTF, OBJ, with textures).
• Registration of multiple devices:
  • Investigate spatial anchors persistency and sharing such anchors across sessions.
  • Enable QR code detection extension
  • https://publik.tuwien.ac.at/files/publik_299074.pdf

See notes from 39th project week.

SlicerVirtualReality

Contact: Jean-Christophe Fillion-Robin @ Kitware

vtk.js: Interactive, in-browser cinematic volume rendering of medical images

Contact: Stephen Aylward & Forrest Li @ Kitware

• vtk.js goals
  • support vtk-wasm as a rendering backend
  • unified WebGPU implementation (see VTK C++ WebGPU above)
• VolView goals
  • improved streaming support
  • demos with python-based processing server

itk-viewer and itkwidgets

Contact: Matt McCormick @ Kitware

• 3D Slicer on the Web via ITK-Wasm and friends
• itk-viewer: Multi-dimensional web-based image, mesh, and point set viewer
• itkwidgets: Python interface for visualization on the web platform to interactively generate insights into multidimensional images, point sets, and geometry.

Notes & Links:

• https://itkwidgets.readthedocs.io/en/latest/deployments.html
• ITKIOOMEZarrNGFF
  • https://github.com/InsightSoftwareConsortium/ITKIOOMEZarrNGFF
  • https://pypi.org/project/itk-ioomezarrngff/

Illustrations

Background and References

• 39th Project Week - Rendering breakout session
• WebGPU in VTK
• WebGPU in Slicer from Project Week 37.