3D Gaussian Splatting and Medical Image

Reconstructing deformable anatomical structures from endoscopic videos is a pivotal and promising research topic that can enable advanced surgical applications and improve patient outcomes. While existing surgical scene reconstruction methods have made notable progress, they often suffer from slow rendering speeds due to using neural radiance fields, limiting their practical viability in real-world applications. To overcome this bottleneck, we propose EndoGaussian, a framework that integrates the strengths of 3D Gaussian Splatting representations, allowing for high-fidelity tissue reconstruction, efficient training, and real-time rendering. Specifically, we dedicate a Foundation Model-driven Initialization (FMI) module, which distills 3D cues from multiple vision foundation models (VFMs) to swiftly construct the preliminary scene structure for Gaussian initialization. Then, a Spatio-temporal Gaussian Tracking (SGT) is designed, efficiently modeling scene dynamics using the multi-scale HexPlane with spatio-temporal priors. Furthermore, to improve the dynamics modeling ability for scenes with large deformation, EndoGaussian integrates Motion-aware Frame Synthesis (MFS) to adaptively synthesize new frames as extra training constraints. Experimental results on public datasets demonstrate EndoGaussian’s efficacy against prior state-of-the-art methods, including superior rendering speed (168 FPS, real-time), enhanced rendering quality (38.555 PSNR), and reduced training overhead (within 2 min/scene). These results underscore EndoGaussian’s potential to significantly advance intraoperative surgery applications, paving the way for more accurate and efficient real-time surgical guidance and decision-making in clinical scenarios. Code is available at: https://github.com/CUHK-AIM-Group/EndoGaussian.

Inthis paper, we present Cube-Cloud 3D Gaussian Splatting (Cube-Cloud 3DGS), a novel framework designed to render medical image that inherently structured with three fixed axes. The methods such as COLMAP, Neural Radiance Field (NeRF), and 3D Gaussian Splatting are unsuitable for medical image reconstruction due to the lack of diverse viewpoints. To address this challenge, we propose Cube-Cloud 3DGS that leverages Cube-based Neural Radiance Field (CuNeRF) for cube-based sampling to generate point clouds from medical data. CuNeRF in Cube-Cloud 3DGS generates point cloud and renders images through various viewpoints which can be used as camera poses. We integrate the point cloud with 3D Gaussian Splatting that is initializing 3D gaussians. By utilizing the viewpoints extracted from CuNeRF, the parameters of 3D gaussians are refined. Cube-Cloud 3DGS renders images through its 3D gaussians while traditional models fail to render based on medical images. We evaluated Cube-Cloud 3DGS on the Kidney and Kidney Tumor Segmentation (KiTS23) dataset, demonstrating that our model reconstructs 3D medical volumes effectively. Therefore, our model resolves the limitation focusing on the internal features for medical images. Our model achieves higher performance of 2.707 in PSNR and 0.0504 in SSIM over existing 3D Gaussian Splatting.

3D Gaussian Splatting (3DGS) [1] has emerged as a popular technique for neural rendering. Use of simple Gaussian primitives that enable real-time rendering, faster training, and high visual fidelity makes its use compelling over alternatives like NeRFs. However, despite its success in natural image settings, 3DGS encounters notable rendering challenges in medical imaging. Our analysis reveals that the error characteristics of sparse reconstruction pipelines (e.g., COLMAP), which serve input to the 3DGS pipeline, differ significantly between medical and natural environments. This discrepancy leads to degraded rendering performance when noise propagates from sparsely reconstructed inputs. To address this issue, we introduce a novel opacity-based regularization strategy for 3DGS that yields two key benefits: (1) enhanced rendering quality through suppression of noisy Gaussian components, and (2) substantial reduction in the number of Gaussian primitives required, improving rendering efficiency. Experiments on two medical datasets validate the effectiveness of our approach. On the C3VD dataset, our method achieves an average PSNR improvement of 0.42 dB and reduces the number of Gaussians by 14,137 per sequence. On the EndoMapper dataset, we observe an average PSNR gain of 1.60 dB with a reduction of 91,522 Gaussians per sequence. The complete source-code and the datasets are available at: https://github.com/AiH-IITD/Robust3DGS.git.

… , digital humans, and medical imaging. To address the rapid … rendering fundamentals of Gaussian splatting and summarizes … large set of anisotropic 3D Gaussian primitives and employs …

This paper introduces 2D Gaussian Splatting (GS) to Electrical Impedance Tomography (EIT), marking its first application in this field. Initially developed for computer vision tasks such as scene reconstruction, GS enables continuous representation and efficient rendering of high-resolution images. Building on these capabilities, we propose a novel GS-based EIT reconstruction framework that models conductivity distributions as a set of Gaussian kernels. These kernels act as localized basis functions, dynamically adjusting their parameters (e.g., position, covariance, and amplitude) to enhance representation accuracy. To ensure regularization and physical constraints, we integrate a threshold-adjusted ReLU activation function to filter out insignificant components and a Sigmoid function to constrain conductivity values within a valid physical range. Experimental results on both simulated and real datasets demonstrate that our approach outperforms traditional model-driven methods and is competitive with conventional neural network-based methods in reconstruction quality. Furthermore, systematic ablation studies confirm the effectiveness of the key components of our framework. This work opens new possibilities for integrating advanced rendering techniques into EIT and inverse problem solving, bridging the gap between computer vision and biomedical imaging.

3D Gaussian Splatting (3DGS) has revolutionized 3D scene representation with superior efficiency and quality. While recent adaptations for computed tomography (CT) show promise, they struggle with severe artifacts under highly sparse-view projections and dynamic motions. To address these challenges, we propose Tomographic Geometry Field (TG-Field), a geometry-aware Gaussian deformation framework tailored for both static and dynamic CT reconstruction. A multi-resolution hash encoder is employed to capture local spatial priors, regularizing primitive parameters under ultra-sparse settings. We further extend the framework to dynamic reconstruction by introducing time-conditioned representations and a spatiotemporal attention block to adaptively aggregate features, thereby resolving spatiotemporal ambiguities and enforcing temporal coherence. In addition, a motion-flow network models fine-grained respiratory motion to track local anatomical deformations. Extensive experiments on synthetic and real-world datasets demonstrate that TG-Field consistently outperforms existing methods, achieving state-of-the-art reconstruction accuracy under highly sparse-view conditions.

Sparse-view tomographic reconstruction aims to recover 3D volumes from limited projection views, but often suffers from incomplete structures and volumetric artifacts. Gaussian splatting has recently emerged as an efficient representation for continuous volumetric modeling, reducing memory cost compared to voxel grids and training time compared to implicit methods. However, existing Gaussian splatting methods for CT reconstruction struggle with needle-like artifacts in sparse-view settings. To address this, we introduce two key contributions. First, we propose a structure-aware initialization strategy that uses gradient and density magnitude from preliminary reconstructions to intelligently place Gaussian primitives in high-contrast regions. Second, we adapt the well-established Beer-Lambert law from CT physics to stabilize Gaussian splatting optimization, transforming the exponential attenuation relationship into a linear domain that mitigates vanishing gradients, and stabilizes optimization. Together, these innovations lead to sharper and more stable reconstructions, achieving average improvements of 2.32 % in PSNR and 2.41 % in SSIM while using 6.47 % fewer primitives across three standard CT datasets.

Computed Tomography (CT) enables detailed crosssectional imaging but continues to face challenges in balancing reconstruction quality and computational efficiency. While deep learning-based methods have significantly improved image quality and noise reduction, they typically require large-scale training data and intensive computation. Recent advances in scene reconstruction, such as Neural Radiance Fields and 3D Gaussian Splatting, offer alternative perspectives but are not well-suited for direct volumetric CT reconstruction. In this work, we propose Discretized Gaussian Representation (DGR), a novel framework that reconstructs the 3D volume directly using a set of discretized Gaussian functions in an end-to-end manner. To further enhance efficiency, we introduce Fast Volume Reconstruction, a highly parallelized technique that aggregates Gaussian contributions into the voxel grid with minimal overhead. Extensive experiments on both real-world and synthetic datasets demonstrate that DGR achieves superior reconstruction quality and runtime performance across various CT reconstruction scenarios. Our code is publicly available at https://github.com/wskingdom/DGR.

Accurate 3D reconstruction in surgical scenarios is essential for visualizing dynamic tissues with complex anatomical geometries. While 3D Gaussian Splatting (3D-GS) has been explored as an efficient approach to scene modeling, occlusion-induced voids and suboptimal detail optimization have limited its application in surgery. This work introduces a Complexity-Guided 3D Gaussian Splatting (CG-3DGS) framework, in which occlusion regions are globally filled by a state-of-the-art optical flow-based video inpainting method. A frequency–spatial aware refinement (FSAR) mechanism is proposed, allowing spectral signatures and spatial gradients to be jointly analyzed to enhance critical anatomical features (e.g., blood vessels). This mechanism adaptively guides Gaussian densification based on scene-specific anatomical complexity. Experimental results demonstrate that the proposed framework achieves higher reconstruction fidelity while maintaining efficient rendering speeds.

… Additionally, we adapt the 3DGS initialization to account for … is also suboptimal for anatomical data, discarding domain-… Fast auto-differentiable digitally reconstructed radiographs …

The reconstruction of monocular endoscope video scenes is essential for enhancing the application and analysis of surgical endoscopic images. However, restricted by the narrow space of endoscopic movement and the obstruction of vision within cavities, it is difficult for most conventional methods to perform high-quality reconstruction. To address these challenges, a novel dynamic growing 3D Gaussian splatting architecture is proposed to construct the 3D model of endoscopic scene without precomputed camera poses or Structure from Motion. Firstly, to establish spatial feature associations between interframes, a 2D-3D displacement fields are designed by utilizing dense feature matches and depth prediction. On this basis, a novel displacement field variational optimization is developed to obtain relative poses by minimizing the energy functional associated with field transformation. Secondly, to address the constraint of the endoscopic view, by Gaussian sequential transformation and differential gradient field optimization, a novel Sequential Gaussian Growing Module is proposed to grow the local Gaussian model sequentially. Finally, a novel Forward-Reconstruction&Backward-Optimization architecture is proposed to generate the global Gaussian model. The evaluation is conducted on two public endoscopic datasets: Scared and C3VD. The experimental results demonstrate that the proposed method outperforms state-of-the-art methods in both quantitative metrics (PSNR, SSIM, LPIPS, ATE, RMSE, MAE) and qualitative comparisons. The project page is https://iheckzza.github.io/ DG-3DGS/

… -intensive and limited in anatomical variability. … (3DGS) pipeline and emphasizes downstream medical simulation (see Figure 1). As illustrated in Figure 2, we adopt a deformable 3DGS …

Endoscopic imaging has become an indispensable tool in modern medical practice, enabling minimally invasive diagnosis and treatment. A crucial aspect of endoscopic procedures is the accurate reconstruction of the observed scene, which facilitates spatial understanding, surgical planning, and guidance. This comprehensive review delves into the latest advancements and applications of Gaussian splatting in endoscopic scene reconstruction. We explore the fundamental principles of this technique, examining how Gaussian kernels are utilized to fuse depth cues and generate detailed 3D models. Furthermore, we analyze the various refinements and extensions that have been proposed to enhance the performance and robustness of Gaussian splatting, addressing challenges such as occlusions, sparse data, and real-time processing. Through this comprehensive survey, we aim to provide researchers and medical professionals with a deep understanding of the state-of-the-art in Gaussian splatting for endoscopic scene reconstruction, highlighting its pivotal role in advancing endoscopic imaging and interventions.

AbstractDynamic 3D reconstruction with real-time rendering (D3RR) is highly regarded in the field of computer graphics. Applying D3RR to endoscopy provides surgeons with …

We develop a customized initialization method for 3D Gaussian Splatting methods, aimed at extending its application to Computed Tomography (CT) reconstruction. Initialization in 3D Gaussian Splatting is a crucial step and can be accomplished using several techniques. The official pipeline of 3D Gaussian Splatting uses the Structure-from-Motion (SfM) technique in its initialization step. While SfM works well for natural scene photographs, it is not directly applicable in the medical domain, more specifically in the CT environment. To address this limitation, we propose a customized, geometricaware initialization method that is compatible with parallel beam CT geometry. We investigated 16 simulated CT datasets along with the Shepp-Logan phantom. These simulated models were acquired from TomoPhantom toolbox that provided 2D projection images as the ground truth. These ground truth images and the 3D models were used in our customized 3D Gaussian placement strategy, ensuring accurate camera orientation and 3D point sampling for parallel-beam CT reconstruction. We obtained the rendered images corresponding to their ground truth projections that mostly preserved the true geometric structures. For the Shepp-Logan phantom, we achieved a test PSNR of 29.987 and an L1 loss of 0.015 after 30,000 iterations. Further work may extend this approach to real-time CT data with different scanner acquisition, such as cone beam or helical.

… FDM [21] are proposed to be coupled with 3DGS and facilitate superior fast reconstruction of deforming tissue surface. We keep this method for the tissue reconstruction component in \(\…

Endoscopic reconstruction plays a crucial role in surgical robotics. The dynamic lighting conditions and integrated camera-light source in endoscopic scenes create a distinct reconstruction challenge: shape ambiguity. To mitigate this, we propose a Gaussian Splatting (GS) based framework for endoscopic scene reconstruction, enhanced with reflectance regularization. We embed every 3D Gaussian point with physical reflective attributes and combine this representation with a physically based inverse rendering framework. By jointly training 3DGS for view synthesis with this reflectance regularization, we are able to attain high-quality geometry without changing the volume rendering pipeline. Our experiments demonstrate the superiority in both geometry representation and rendering performance compared to existing GS approaches, making it a practical solution for endoscopic applications. Project is available at: https://med-air.github.io/GSR2.

BACKGROUND This study evaluates the accuracy and repeatability of 3D Gaussian Splatting (3DGS)-generated 3D facial models by comparing anthropometric measurements derived from these models with those obtained using conventional 2D photogrammetry and direct anthropometry. METHODS In this cross-sectional study, 58 adults (18-25 years) underwent direct anthropometry, standardised 2D photography with calibration, and 3DGS facial reconstruction using a smartphone-based workflow. A predefined set of linear and angular facial measurements was recorded; each measurement was repeated three times by the same observer. Between-method comparisons were performed using paired tests as appropriate. Pearson correlation coefficients were reported to describe association, whereas agreement and potential interchangeability were evaluated primarily using Bland-Altman bias and 95% limits of agreement (LoA); plots were additionally inspected for proportional bias and influential outliers. RESULTS 3DGS demonstrated small mean differences relative to direct anthropometry for several measurements, with generally narrower LoA than 2D photogrammetry. Correlations were typically higher for 3DGS than for 2D photogrammetry; however, Bland-Altman analyses indicated that agreement varied by variable. For selected measurements, LoA were wide (reaching approximately 10-18 mm for some linear outcomes and exceeding 20° for certain angular outcomes), indicating limited interchangeability for individual-level clinical decisions. Perioral measures showed greater dispersion for 2D photogrammetry, particularly for N-Stm, and a small number of observations fell outside the 95% LoA in ANT vs. 2D comparisons, consistent with occasional landmark identification or reconstruction-related variability. CONCLUSION 3D Gaussian Splatting (3DGS) may offer a useful alternative workflow for selected facial analysis applications by enabling the generation of measurable three-dimensional facial models, from which anthropometric variables can subsequently be derived. Although 2D photogrammetry is readily accessible and practical to implement, its inherent limitation in representing three-dimensional anatomy may compromise measurement accuracy in certain clinical contexts. Overall, these findings indicate that 3DGS could support orthodontic practice and research as a 3D model-generation technique; however, further validation across diverse patient groups, acquisition conditions, expanded landmark-based geometric error metrics, and additional outcome measures is warranted before routine clinical adoption can be recommended.

BACKGROUND AND OBJECTIVE Robot-assisted surgery has revolutionized modern medical procedures by enhancing precision, reducing invasiveness, and providing a clearer, more controlled environment. However, it still faces challenges in fully visualizing the target tissue, particularly from multiple perspectives. This limitation is most evident in minimally invasive surgeries, Therefore, the ability to synthesize new views of the surgical scene is becoming increasingly critical. By generating multi-view visualizations, surgeons can gain a more comprehensive understanding of the target tissue, improving spatial awareness and decision-making during surgery. METHODS This article proposes an innovative novel view synthesis method for robotic surgical scenarios, which utilizes pre-trained depth estimation model to obtain global depth information and solves the scale ambiguity problem encountered in the transition region of the Gaussian distribution in the 3D Gaussian model. In addition, we introduce a multi-scale loss optimization strategy that captures features of various scales through a multi-scale loss function to regularize the Gaussian parameters while maintaining the 3D consistency of the splatting. RESULTS Our method is evaluated against current scene novel view synthesis techniques using our robotic surgery scene dataset, along with the Hamlyn and Stereo MIS public datasets. The proposed approach achieved an average PSNR of 33.45, SSIM of 0.939, LPIPS of 0.153, and an RMSE of 0.022 across the three datasets. CONCLUSIONS Our approach helps to enhance the visualization capabilities of robotic surgical systems by synthesizing novel views of surgical scenes. A deeper understanding of the target tissue will enhance patient safety during surgery and support surgeon training. These advancements will contribute to the improvement of robot-assisted surgery, making it more adaptable to diverse clinical scenarios.

Orthodontic treatment needs regular tooth alignment checks, but current methods depend on clinic visits, limiting remote care. With the emergence of 3D Gaussian Splatting (3DGS), realistic novel views can be synthesized, making it possible for clinicians to remotely monitor orthodontic conditions. However, using only five intraoral images with unknown camera poses and dynamic lighting presents major challenges in dental applications. To address these challenges, we propose DentalGS, an enhanced 3DGS framework capable of synthesizing novel intraoral views from five post-orthodontic intraoral images and pre-orthodontic intraoral scan (IOS) data as prior, without camera poses. Our method initializes a Gaussian point cloud labeled with ISO-FDI tooth classes based on the patient’s pre-orthodontic IOS data, then estimates camera poses through iterative optimization. We introduce a Progressive Pair Generation Strategy as a data augmentation method that generates damage–repair image pairs to train a RepairNet, aiming to restore degraded geometry and appearance caused by the limited number of intraoral images. Additionally, we introduce a Lighting-Aware 3DGS inspired by physical reflectance properties to mitigate the effects of dynamic lighting conditions. Experimental results show that our method produces high-quality novel views while preserving geometric structure even under extreme viewpoints, offering an efficient and reliable solution for 3D tooth visualization in remote orthodontic monitoring.

Image view correction aims to rectify non-standard perspective images to a canonical view, with critical applications in industrial inspection, autonomous driving, and clinical diagnosis. However, the field currently faces a critical trade-off: high-accuracy NeRF-based methods are hindered by high computational costs and long training/inference times, while more efficient Gaussian-based models like X-Gaussian lack a targeted “view correction” function. Furthermore, the absence of large-scale, annotated public datasets for this specific task has restricted algorithmic innovation. To address these limitations, we propose VegaNet, a view-guided efficient Gaussian alignment network based on the X-Gaussian model. Our method leverages region-of-interest (ROI) annotations to guide the generative model toward task-relevant viewpoints, while incorporating multi-view camera parameters to ensure geometric consistency and accurate view correction. To tackle the data scarcity problem, we construct a new large-scale benchmark, CTSpine3D, by re-annotating the CTSpine1K dataset. Extensive experiments on CTSpine3D demonstrate that VegaNet achieves superior accuracy and efficiency, offering a $38 \times$ faster training speed and $580 \times$ faster inference compared to NeRF-based baselines, without sacrificing visual fidelity or geometric consistency.

Cerebrovascular diseases are a leading cause of global mortality, underscoring the need for objective and quantitative 3D visualization of cerebral vasculature from dynamic imaging modalities. Conventional analysis is often labor-intensive, subjective, and prone to errors due to image noise and subtraction artifacts. This study tackles the challenge of achieving fast and accurate volumetric reconstruction from angiography sequences. We propose a multi-stage pipeline that begins with image restoration to enhance input quality, followed by neural segmentation to extract vascular structures. Camera poses and sparse geometry are estimated through Structure-from-Motion, and these reconstructions are refined by leveraging the segmentation maps to isolate vessel-specific features. The resulting data are then used to initialize and optimize a 3D Gaussian Splatting model, enabling anatomically precise representation of cerebral vasculature. The integration of deep neural segmentation priors with explicit geometric initialization yields highly detailed 3D reconstructions of cerebral angiography. The resulting models leverage the computational efficiency of 3D Gaussian Splatting, achieving near-real-time rendering performance competitive with state-of-the-art reconstruction methods. The segmentation of brain vessels using nnU-Net and our trained model achieved an accuracy of 84.21%, highlighting the improvement in the performance of the proposed approach. Overall, our pipeline significantly improves both the efficiency and accuracy of volumetric cerebral vasculature reconstruction, providing a robust foundation for quantitative clinical analysis and enhanced guidance during endovascular procedures.

… anatomical imaging such as computed tomography (CT) or magnetic resonance imaging (… on Gaussian Splatting, a recent rendering technique that projects continuous 3D Gaussian …

Accurate and reliable 3D scene reconstruction is a key component of intelligent surgery, enabling enhanced spatial understanding and data-driven analysis in minimally invasive surgery (MIS). However, existing clinical systems are often bulky and workflow-incompatible, while vision-based Structure-from-Motion methods struggle with sparse textures and specularities, leading to unstable pose estimation and high computational cost. To address these limitations, we present SurGSplat++, a progressive, pose-free Gaussian splatting framework for monocular surgical scene reconstruction that requires no auxiliary hardware or pre-computed camera poses. Experiments show that SurGSplat++ achieves improved geometric stability, reduced pose drift, and superior novel-view synthesis compared with existing approaches. By producing accurate and consistent 3D reconstructions, the proposed method provides a practical solution for post-operative analysis, pre-operative planning, and data-driven surgical modeling in clinical environments. Code will be released at https://surgsplus.github.io/.

… Alternatively, 3D Gaussian Splatting (3DGS)[10] has … in novel view synthesis and 3D reconstruction. Extensions such as … In sparseview 4DCT reconstruction experiment, we compare …