Title
题目
2D echocardiography video to 3D heart shape reconstruction for clinicalapplication
2D 超声心动图视频到 3D 心脏形状重建的临床应用
01
文献速递介绍
超声心动图是心血管医学中一种至关重要且广泛应用的影像学技术,利用超声波技术捕捉心脏及其周围结构的高分辨率图像。由于其低成本、快速采集时间以及高空间和时间分辨率,超声心动图是评估心脏形态和功能的首选非侵入性方法。然而,传统的超声心动图仅提供心脏形态在离散二维平面上的轮廓。它依赖几何假设来估算心脏容积和功能,这可能导致测量不准确。相比之下,心脏的三维模型可以直接且更准确地提供心脏容积和壁运动的测量,这对于诊断和监测心血管疾病至关重要,而心血管疾病仍然是全球死亡的主要原因(Vaduganathan et al., 2022)。此外,三维可视化可以促进临床和教育场景中医务人员之间的沟通,并改善患者对自己心脏病情的理解(Bui et al., 2021)。尽管三维超声心动图相较于二维超声心动图提供了更全面且潜在更准确的心脏解剖和功能评估,但它的可获取性较差、成本较高,且仍面临空间和时间分辨率有限、信噪比差以及复杂的噪声特性等挑战(Zhao et al., 2022)。此外,传统的经胸超声心动图(TTE)基于容积测量仅依赖于心脏周期中的两个帧,即收缩末期和舒张末期帧。尽管可以对整个心脏周期中的动态帧间容积变化进行视觉评估,但这些变化通常没有被量化并整合到常规临床TTE报告中。为了解决上述限制,我们开发了一个完全自动化的管道,用于通过4D(3D加时间)左心室(LV)形状建模来解读二维TTE视频。该框架的概述如图1所示。对于包含约100个静态帧和视频的完整TTE检查,通过适应性和微调后的ResNet-50架构(He et al., 2016),自动选择所需的超声心动图视频(回波),即心尖2腔、3腔和4腔视图(A2C、A3C、A4C)。然后,使用自训练的nnU-Net(Isensee et al., 2021)对视频中的左心室血池进行分割。最后,将回波视频时间上对齐,确保从心脏周期的舒张末期阶段开始。通过2D到3D的重建深度神经网络,在每一帧中合并视图,以推测相应的三维左心室形状。开发的方法能够修正记录单个视图时可能发生的探头定位不准确问题。由于重建网络以自监督方式进行优化,因此无需三维真实数据。通过在一个包含144例正常TTE和314例急性心肌梗死(AMI)患者的多中心数据集上进行实验,得到的4D(3D加时间)左心室形状模型(数字双胞胎)在三个不同的临床相关下游任务上进行了评估:心肌梗死定位(第5.1节)、心脏功能自动评估(第5.2节)以及使用新的3D形状衍生的回波生物标志物进行生存分析(第5.3节)。
总之,提出的框架提供了一种从二维超声心动图视频中推断个性化4D左心室数字双胞胎的解决方案,这些数字双胞胎提供了有关左心室形态和功能的临床相关信息,并有潜力改善诊断准确性和风险分层。
本研究支持结果的信息可以在论文及其附录中找到。此外,我们将144例对照组输入回波、分割图和相应的重建4D心脏公开发布。数据可以通过以下链接下载:https://polybox.ethz.ch/index.php/s/4ew0b6BGkIc9I2z
Aastract
摘要
Transthoracic Echocardiography (TTE) is a crucial tool for assessing cardiac morphology and function quicklyand non-invasively without ionising radiation. However, the examination is subject to intra- and inter-uservariability and recordings are often limited to 2D imaging and assessments of end-diastolic and end-systolicvolumes. We have developed a novel, fully automated machine learning-based framework to generate apersonalised 4D (3D plus time) model of the left ventricular (LV) blood pool with high temporal resolution.A 4D shape is reconstructed from specific 2D echocardiographic views employing deep neural networks,pretrained on a synthetic dataset, and fine-tuned in a self-supervised manner using a novel optimisationmethod for cross-sectional imaging data. No 3D ground truth is needed for model training. The generateddigital twins enhance the interpretation of TTE data by providing a versatile tool for automated analysis ofLV volume changes, localisation of infarct areas, and identification of new and clinically relevant biomarkers.Experiments are performed on a multicentre dataset that includes TTE exams of 144 patients with normalTTE and 314 patients with acute myocardial infarction (AMI). The novel biomarkers show a high predictivevalue for survival (area under the curve (AUC) of 0.82 for 1-year all-cause mortality), demonstrating thatpersonalised 3D shape modelling has the potential to improve diagnostic accuracy and risk assessment.
经胸超声心动图(TTE)是评估心脏形态和功能的关键工具,具有快速、无创且不使用电离辐射的优点。然而,该检查受限于操作者间和操作者内的变异,且影像记录通常仅限于二维成像,并且主要评估舒张末期和收缩末期容积。我们开发了一种新型的完全自动化机器学习框架,用于生成个性化的高时间分辨率4D(3D加时间)左心室(LV)血池模型。通过深度神经网络从特定的二维超声心动图视图重建4D形状,模型预先在合成数据集上训练,并使用一种新颖的优化方法对横截面影像数据进行自监督微调。模型训练不需要三维真实数据。生成的 数字双胞胎 通过提供一个多功能工具,增强了TTE数据的解读,能够自动化分析左心室容积变化、定位梗死区,并识别新的临床相关生物标志物。
实验在一个多中心数据集上进行,数据集包含144例正常TTE和314例急性心肌梗死(AMI)患者的TTE检查。新型生物标志物对生存率具有较高的预测价值(1年全因死亡率的曲线下面积(AUC)为0.82),证明个性化的3D形状建模具有提高诊断准确性和风险评估的潜力。
Method
方法
Our overall approach for the 2D to 3D reconstruction from raw TTEdata is sketched in Fig. 3. After initial data curation and preprocessingsteps (Section 4.1), we employ a 2D to 3D reconstruction network togenerate personalised digital twins of the LV blood pool (Section 4.2).The lack of a large dataset of ground truth 3D heart shapes withcorresponding echos does not permit the supervised training of a neuralnetwork that directly learns the LV shape from 2D echos. To overcomethis limitation, we first pretrain a 2D to 3D reconstruction network (𝐺)on a synthetic 3D dataset generated from the statistical shape model(SSM) (Unberath et al., 2015) introduced in the previous section. Atinference, an initial LV shape is estimated using real echo segmentationmaps. The reconstruction network and the position of the transducerare then iteratively fine-tuned for each patient by comparing the inputsegmentation maps with the rendered cross sections employing a differentiable rendering function 𝑅. The LV shape is optimised frame byframe until a full heartbeat is reconstructed.
我们从原始TTE数据进行2D到3D重建的整体方法如图3所示。在初步数据整理和预处理步骤(第4.1节)之后,我们采用2D到3D重建网络生成个性化的左心室血池数字双胞胎(第4.2节)。由于缺乏大量带有相应回波的真实三维心脏形状数据集,无法进行监督训练神经网络以直接从二维回波中学习左心室形状。为了克服这一限制,我们首先在由统计形状模型(SSM)生成的合成三维数据集上预训练一个2D到3D重建网络(𝐺)(Unberath et al., 2015),该模型在前文中已介绍。在推理过程中,使用真实的回波分割图估算初始的左心室形状。然后,通过将输入分割图与渲染的横截面进行比较,并使用可微分的渲染函数 𝑅,迭代地微调重建网络和探头位置。左心室形状在每一帧中被优化,直到完整的心跳被重建。
Conclusion
结论
The digital twin can detect and visualise impaired LV regional wallmotion due to AMI in a 3D shape and relate the affected segments tothe ECG-derived ground truth location. Quantification of wall motionabnormalities using the digital twin could potentially provide addedvalue for current clinical routine where regional hypokinesis and akinesis are only assessed semiquantitatively. Moreover, correctly locatingLV regional wall motion abnormalities may be particularly useful forinfero-lateral and lateral infarctions that are more challenging to detecton a standard 12-lead surface ECG. Left coronary artery (LCA) andRCA perfusion territories can vary between individuals, either beingleft-dominant (10% of the population), right-dominant (70% of thepopulation) or co-dominant (20% of the population) (Shriki et al.,2012; Villa et al., 2016). This diversity means that infarct size andlocation may vary depending on the individual coronary perfusion type.In general, the RCA perfuses infero-septal and the LCA (LAD and LCX)the remaining areas of the LV. We observe that inferior infarctions,apart from showing reduced wall motion in the inferior LV regions,are often associated with impaired kinetics of the remaining LV. Itis challenging to correctly label the infarcted region according to theECG (our ground-truth), due to differences between right- and leftdominant perfusion types in the context of inferior AMI. Thus, caseslabelled as inferior AMI can include both LCA and RCA infarctions,potentially blurring differentiation. In cases labelled anterior AMI, theanterior segments are most affected, with lateral segments also showingreduced wall motion. This finding overlaps with cases labelled lateralinfarctions, where the largest effect is seen in lateral segments, whilethe anterior and inferior segments also show some level of reducedkinetics. The overlap of anterior and lateral infarctions may be explained in part by the potential presence of left main coronary artery(LMCA) infarctions in either group. LMCA infarctions affect both LCAand LCX territories, i.e., both reduced wall motion may be expectedin both anterior and lateral segments. Furthermore, infarction of theproximal LAD can cause reduced wall motion in large parts of theLV, thus affecting both anterior and lateral segments. Nevertheless,differentiation between anterior, lateral and inferior infarctions, aslabelled by ECG, was still achievable through corresponding differencesin wall motion abnormalities as assessed by the digital twin.The generated digital twins model the LV volume over time. Thesevolume curves yield information on heart dynamics to define andmeasure standard (EDV, ESV, LVEF) as well as novel biomarkers.We compare the parameters measured in clinical routine (ESV, EDV)with the same parameters derived from our LV digital twin, showinggood agreement. The digital twin generally seems to overestimatethe ESV compared to the clinical measurements calculated with themodified Simpson method (biplane method of disks), which assumesa simple and symmetrical LV shape. Laumer et al. (2023) showed thatthe clinician's biplane method of disk summation (Simpson's biplanemethod) produces inflated LVEF values, most likely due to underestimating the ESV value. When the heart is fully contracted, its shapebecomes twisted, and a volume approximation through stacked discsis inadequate. Assuming such a simple shape for the LV can leadto inaccurate results, particularly in cases of ventricular remodelling,which often occurs in patients after AMI. Another reason might bethe fact that the SSM that is used to generate the synthetic dataset isunable to produce mesh videos that contract a lot, and there are no lowESV mesh videos present in the dataset. The reconstruction networkis pretrained on the synthetic mesh video dataset which biases theinitial shape prediction towards videos with a higher ESV. Furthermore,for measurements performed in the clinical routine, the LV contour isdelineated with a straight line across the valves, which could distortthe volume estimation.1-year all-cause mortality prediction based on the LVEF values derived from the digital twin was more accurate than the prediction basedon the clinical LVEF, supporting the validity of the digital twin-derivedvalues.We demonstrate the added value of the derived biomarkers by performing 1-year all-cause mortality prediction with standard echo values(EDV, ESV, LVEF) vs. standard echo values plus the novel biomarkers(contractility, relaxation, SA, RFV, plateau). Although the predictedsurvival probabilities using all biomarkers are only marginally moreaccurate compared to the standard values alone, the added value of thenew biomarkers becomes more evident when looking at the Kaplan--Meier survival probability curves of different cohorts. When patientsare divided into groups based on whether their LVEF is greater than40% or less than 40% - a condition known to result in lower survivalrates (Stewart et al., 2021) - the impact of SA, RFV, contractility, andrelaxation on survival probabilities becomes particularly pronouncedfor the cohort with lower LVEF. Patients with low LVEF (≤40%) andlow values in either of the new biomarkers have a significantly lowersurvival probability than patients with low LVEF (≤40%) but highervalues of the new biomarkers. This could be explained by the fact thatLVEF only captures the relative blood flow calculated from only twodistinct volume measurements (EDV and ESV) and does not provide iformation about contraction dynamics (e.g. contractility). Furthermore,LVEF is a measure of systolic function and does not provide information about diastolic function. The new biomarkers contain informationabout the beat-to-beat volume changes of the LV which is affected byits shape, such as LV dilation. A dilated LV after AMI is a sign of adverseremodelling and a surrogate of adverse outcome (Pfeffer et al., 1992).A weakness of our 4D LV reconstruction method is that it doesnot directly utilise gray scale information, but relies entirely on theaccuracy of view classification and 2D LV segmentation models. Forfuture work, we aim to enhance our current differentiable renderer togenerate realistic ultrasound frames instead of LV segmentation maps.This development will enable the use of raw frames to directly estimatetransducer positions and optimise 3D shape reconstruction end-toend. Another limitation of the current method is that it does not usetemporal information and reconstructs the 3D shape frame-by-frame.Our developed differentiable rendering function can be used as aversatile loss function to fine tune 2D to 3D mesh reconstruction approaches. For future work, we plan to directly model a full heart beat bycombining our differentiable rendering approach with previous workfrom Laumer et al. (2023) where temporal information from a full videois aggregated for direct 4D reconstruction. While there already existsmany works on differentiable rendering, i.e. Loper and Black (2014),Liu et al. (2019) and Ravi et al. (2020) none of these approaches workout of the box for rendering cross-section. Implementing our proposedscheme for ultrasound cross-sections still requires additional logic tocompute and render intersections. Our approach seeks to address thisspecific need, offering a novel contribution for this special case.In summary, we have developed a fully automated pipeline toextract clinically relevant diagnostic and prognostic information froma raw 2D TTE examination using a personalised 4D LV shape model.The generated digital twins provide a detailed analysis of the LVmorphology, giving rise to novel biomarkers that are proven to beinformative for survival prediction.
数字双胞胎可以检测和可视化由于急性心肌梗死(AMI)导致的左心室(LV)区域壁运动障碍,并将受影响的区域与基于心电图(ECG)的真实位置相关联。利用数字双胞胎对壁运动异常的量化可能为当前的临床常规提供额外的价值,因为当前的评估方法通常仅对区域性心脏运动减弱(低动力)和完全失动(静止)进行半定量分析。此外,正确定位LV区域壁运动异常对于识别难以通过标准12导联表面心电图检测的下侧-外侧(infero-lateral)和外侧(lateral)心肌梗死可能尤其有用。左冠状动脉(LCA)和右冠状动脉(RCA)供血区在不同个体之间可能有所不同,可能是左主导型(10%的群体)、右主导型(70%的群体)或共主导型(20%的群体)(Shriki等,2012;Villa等,2016)。这种差异意味着心肌梗死的大小和位置可能会根据个体的冠脉供血类型有所不同。一般来说,RCA供应下隔区,而LCA(包括前降支(LAD)和回旋支(LCX))供应左心室的其他区域。
我们观察到,下侧心肌梗死除了在左心室下侧区域出现壁运动减弱外,通常还伴随左心室其他部分的动力学受损。由于在下侧AMI的背景下,右主导型和左主导型供血类型存在差异,因此根据心电图(我们所用的真实数据)正确标记梗死区域是具有挑战性的。因此,标记为下侧AMI的病例可能包括LCA和RCA的梗死,可能导致这两者之间的区分模糊。在标记为前侧AMI的病例中,前壁区域受影响最为显著,外侧区域也表现出壁运动减弱。这一发现与标记为外侧心肌梗死的病例重叠,在这些病例中,外侧区域的影响最大,而前侧和下侧区域也显示出一定程度的动力学受损。前侧和外侧心肌梗死的重叠部分可能部分解释为左主冠状动脉(LMCA)梗死的潜在存在。LMCA梗死影响LCA和LCX供血区,因此前壁和外侧区域的壁运动减弱都有可能。进一步而言,前降支(LAD)近端梗死可能导致大部分LV区域的壁运动减弱,从而影响前壁和外侧区域。尽管如此,数字双胞胎通过对应的壁运动异常差异,仍能有效地区分心电图标记的前侧、外侧和下侧心肌梗死。生成的数字双胞胎模型可以模拟左心室(LV)体积随时间的变化。这些体积曲线提供了有关心脏动力学的信息,用于定义和测量标准(EDV、ESV、LVEF)以及新型生物标志物。我们将临床常规中测量的参数(ESV、EDV)与从LV数字双胞胎推导出的相同参数进行比较,结果显示良好的一致性。与使用修改的辛普森方法(双平面盘片法)计算的临床测量值相比,数字双胞胎通常会高估ESV值,辛普森方法假设LV形态简单且对称。Laumer等(2023年)显示,临床医生使用的双平面盘片法(辛普森法)可能会高估LVEF值,这很可能是因为低估了ESV值。当心脏完全收缩时,其形状会发生扭曲,使用堆叠盘片的体积估算方法不再适用。假设LV具有简单形状可能导致不准确的结果,尤其是在心室重塑的情况下,这种情况常见于急性心肌梗死(AMI)后的患者。另一个原因可能是用于生成合成数据集的SSM(形状统计模型)无法生成收缩幅度较大的网格视频,而数据集中没有低ESV网格视频。重建网络是通过合成网格视频数据集进行预训练的,因此初始形状预测偏向于具有较高ESV的视频。此外,在临床常规测量中,LV轮廓是通过一条直线划定的,穿过心脏瓣膜,这可能会扭曲体积估算。
基于从数字双胞胎推导出的LVEF值进行的1年全因死亡率预测,比基于临床LVEF值的预测更为准确,支持了数字双胞胎推导值的有效性。
我们通过使用标准回声值(EDV、ESV、LVEF)与标准回声值加上新型生物标志物(如收缩力、放松力、SA、RFV、平台值)进行1年全因死亡率预测,展示了新生物标志物的附加价值。尽管使用所有生物标志物预测的生存概率与仅使用标准值的预测相比,仅略有提高,但当查看不同队列的Kaplan--Meier生存概率曲线时,新生物标志物的附加价值变得更加明显。当患者根据其LVEF是否大于40%或小于40%进行分组时------这一条件已知与较低的生存率相关(Stewart等,2021)------SA、RFV、收缩力和放松力对生存概率的影响在LVEF较低的队列中尤为显著。低LVEF(≤40%)且新生物标志物值低的患者,其生存概率显著低于低LVEF(≤40%)但新生物标志物值较高的患者。这可以通过以下事实来解释:LVEF仅捕捉相对血流量,是通过两个不同体积测量(EDV和ESV)计算得出的,并未提供关于收缩动力学(如收缩力)的信息。此外,LVEF是一个衡量收缩功能的指标,无法提供关于舒张功能的信息。新生物标志物包含了关于LV每次心跳体积变化的信息,这些变化受到LV形状的影响,例如LV扩张。AMI后的LV扩张是不良重塑的标志,也是不良预后的替代指标(Pfeffer等,1992)。
我们4D LV重建方法的一个弱点是,它没有直接利用灰度信息,而是完全依赖于视图分类和2D LV分割模型的准确性。未来的工作中,我们计划增强现有的可微渲染器,以生成真实的超声图像帧,而不是LV分割图。这个发展将使得能够直接使用原始帧来估计换能器位置,并优化3D形状的重建。此外,当前方法的另一个限制是,它没有使用时间信息,而是逐帧重建3D形状。我们开发的可微渲染功能可以作为一种通用的损失函数,用于微调2D到3D网格重建方法。未来的工作中,我们计划通过将可微渲染方法与Laumer等(2023)先前的工作相结合,直接建模完整的心跳,其中通过整合全视频的时间信息来进行直接的4D重建。尽管已有许多关于可微渲染的研究,如Loper和Black(2014年)、Liu等(2019年)以及Ravi等(2020年),但这些方法都不能直接应用于截面渲染。实现我们提出的超声截面渲染方案仍需要额外的逻辑来计算和渲染交集。我们的方法旨在解决这一特定需求,为这一特殊案例提供新的贡献。
总之,我们开发了一个完全自动化的管道,利用个性化的4D LV形状模型从原始2D经胸超声(TTE)检查中提取临床相关的诊断和预后信息。生成的数字双胞胎提供了对LV形态的详细分析,并生成了新型的生物标志物,这些标志物已被证明对生存预测具有重要意义。
Figure
图
Fig. 1. Overview: From automated selection of 2D TTE videos to 4D LV shape generation with clinical applications. The required views, i.e. A2C, A3C, and A4C views, areautomatically selected, the LV blood pool is segmented, and the different views are temporally aligned. A digital twin is generated frame-by-frame for each patient using adeep 2D to 3D shape reconstruction network and evaluated on different clinically applicable downstream tasks such as the extraction of echo parameters, survival analysis usingnovel echo biomarkers extracted from the digital twin, and myocardial infarction localisation (example of an inferior myocardial infarction). Abbreviations: TTE, transthoracicechocardiography; LV, left ventricular; ED, end-diastolic; ES, end-systolic; EDV, end-diastolic volume; ESV, end-systolic volume, RFV, rapid filling volume
图1. 概述:从自动选择2D TTE视频到4D左心室(LV)形状生成及其临床应用。所需的视图,即A2C、A3C和A4C视图,自动选择,左心室血池进行分割,且不同视图按时间对齐。通过深度2D到3D形状重建网络逐帧生成数字双胞胎,并在不同的临床下游任务中进行评估,如提取回波参数、使用从数字双胞胎中提取的新型回波生物标志物进行生存分析以及心肌梗死定位(例如,下壁心肌梗死)。缩写:TTE, 经胸超声心动图;LV, 左心室;ED, 舒张末期;ES, 收缩末期;EDV, 舒张末期容积;ESV, 收缩末期容积;RFV, 快速充盈容积
Fig. 2. Synthetic dataset generation used for training the 2D to 3D reconstructionnetwork. Shapes are sampled from the SSM and sliced according to transducer positioncorresponding to A2C, A3C and A4C views. Note that all chambers except the LV bloodpool (in grey) are discarded for model training
图2. 用于训练2D到3D重建网络的合成数据集生成。形状从形状模型(SSM)中采样,并根据探头位置切片,分别对应A2C、A3C和A4C视图。请注意,除了左心室血池(灰色区域)外,所有心腔都被舍弃用于模型训练。
Fig. 3. Overview of the 2D TTE to 4D LV shape generation framework. TTE examinations are collected from hospitals. The data is deidentified and meta data is removed. TheA2C, A3C and A4C non-Doppler views are automatically identified, segmented, temporally aligned, and brought into a canonical position. A 2D to 3D reconstruction network isused to produce an initial shape. This shape is then iteratively refined by optimising the transducer position and network weights until the rendered cross sections of the generatedshape match the input segmentation maps of each view. A 3D shape is optimised frame by frame.
图3. 2D TTE到4D左心室形状生成框架概述。TTE检查数据来自医院,数据经过去标识化处理,并移除元数据。A2C、A3C和A4C非多普勒视图自动识别、分割、时间对齐,并置于标准位置。使用2D到3D重建网络生成初始形状。然后,通过优化探头位置和网络权重,迭代精细化该形状,直到生成的形状的渲染横截面与每个视图的输入分割图相匹配。3D形状逐帧优化。
Fig. 4. The CNN encoder extracts the feature vector 𝒍 which is assigned to each vertexof the first GNL. At each layer of the GNN decoder a GNL (𝐺𝑁𝐿𝐴𝑅𝑀𝐴) is followedby an upsampling operation UP (𝑼𝑘𝑿𝑘) that increases the number of vertices. Thetopology at each layer is fixed and given by the adjacency matrix 𝑨𝑘 . The output atthe intermediate layers does not assume a 3D shape. Only the output of the last layerpredicts the 3D coordinates at each vertex.
图4. CNN编码器提取特征向量 𝒍,该向量分配给第一个GNL(图神经网络层)中的每个顶点。在GNN解码器的每一层中,GNL(𝐺𝑁𝐿𝐴𝑅𝑀𝐴)后跟一个上采样操作 UP(𝑼𝑘𝑿𝑘),该操作增加顶点的数量。每一层的拓扑结构是固定的,由邻接矩阵 𝑨𝑘 给出。中间层的输出不假定为三维形状。仅最后一层的输出会预测每个顶点的三维坐标。
Fig. 5. The mean mesh of the SSM is used to calculate the upsampling matrices 𝑼𝑘 according to the different downsampled versions of the original mesh. The vertexconnections (topology) at each stage are captured in the adjacency matrices 𝑨𝑘 , whichare kept fixed
图5. 使用SSM的均值网格根据原始网格的不同下采样版本计算上采样矩阵 𝑼𝑘。每个阶段的顶点连接(拓扑结构)被捕捉在邻接矩阵 𝑨𝑘 中,并保持固定。
Fig. 6. Description of the transducer parameters describing the rendered image crosssection 𝐶. The origin 𝒐 describes the mid-point, the vector 𝒏 describes the plane normaland 𝒖 describes the orientation/rotation, i.e. the 'up' direction. The scale 𝑠 is a zoomfactor. These parameters vary for each video and can even change during a recording
图6. 描述渲染图像横截面 𝐶 的探头参数。原点 𝒐 描述中点,向量 𝒏 描述平面法线,𝒖 描述方向/旋转,即'向上'方向。缩放因子 𝑠 是一个变焦因子。这些参数对于每个视频都有变化,甚至在录制过程中也可能发生变化。
Fig. 7. Inference of 3D LV shape. Based on the segmentation maps, the pretrainednetwork ((𝐺) produces an initial shape estimate. This shape is then iteratively refinedby updating the network weights and transducer positions. One can see how the shapeis fitted to the contour of the segmentation maps. The final normalised negative loglikelihood of the predicted shape is 𝑛𝐺𝑀𝑀 = 0.0080.
图7. 3D左心室(LV)形状推断。基于分割图,预训练网络(𝐺)生成初始形状估计。然后,通过更新网络权重和探头位置,迭代地精细化该形状。可以看到,形状如何与分割图的轮廓拟合。最终预测形状的标准化负对数似然值为 𝑛𝐺𝑀𝑀 = 0.0080。
Fig. 8. Example where shape generation failed. In the utilised A4C view half of theLV is cut off. This affects the generated shape which has a notch at the A4C viewcross section as can be seen in the final refined shape. One can also observe an highernegative log-likelihood with value as high as 𝑛𝐺𝑀𝑀 = 0.0148 compared to the previousexample. The large negative log-likelihood results from the abnormal shape introducedby the wrong A4C view
图8. 形状生成失败的示例。在使用的A4C视图中,左心室的一半被切掉。这影响了生成的形状,在A4C视图的横截面上出现了缺口,正如最终精细化的形状所示。还可以观察到较高的负对数似然值,其值达到 𝑛𝐺𝑀𝑀 = 0.0148,相较于之前的示例更高。较大的负对数似然值是由于错误的A4C视图引入了异常形状所导致的。
Fig. 9. Left: regions of the LV predominantly perfused by the left circumflex coronaryartery (LCX), the left anterior descending coronary artery (LAD) and the right coronaryartery (RCA). Right: the 16 different heart segments plotted onto the 3D shape usinga different grey shade for each segment.
图9. 左:左心室(LV)主要由左回旋冠状动脉(LCX)、左前降冠状动脉(LAD)和右冠状动脉(RCA)灌注的区域。右:将16个不同的心脏节段绘制到3D形状上,每个节段使用不同的灰度阴影表示。
Fig. 10. Regional wall motion analysis in patients with acute myocardial infarctionin the corresponding perfusion regions (lateral-AMI, anterior-AMI, inferior-AMI) shownas bar plots and as bulls-eye plots visualising the movements in each heart segmentnormalised with respect to the control cohort.
图10. 急性心肌梗死患者在相应灌注区域(外侧-AMI、前壁-AMI、下壁-AMI)进行的区域壁运动分析,以条形图和靶心图的形式展示,靶心图可视化了每个心脏节段的运动,并与对照组进行了标准化比较。
Fig. 11. Generic heart shapes with colour-coded wall motion of the LV generatedfrom 1 control patient with a normal TTE and 3 individual patients suffering fromlateral-AMI, anterior-AMI and inferior-AMI, respectively. Red indicates no movement(akinesis), blue stands for normal movement (normokinesis)
图11. 从1名正常TTE的对照患者和3名分别患有外侧-AMI、前壁-AMI和下壁-AMI的个体患者生成的带有颜色编码壁运动的通用心脏形状。红色表示无运动(无动症),蓝色表示正常运动(正常运动)。
Fig. 12. Volume curves over time (1 heart cycle) and novel biomarkers derived fromthe LV digital twin. (a) Schematic depiction of the extracted echo biomarkers from thevolume curves. (b) Volume curves of patients from the control cohort (upper panel) andthe acute myocardial infarction (AMI) cohort (lower panel). (c) Comparison of clinical(report-based) vs. digital twin-derived measurements of EDV and ESV from patientsof the control cohort. Published range of typical variations of measurements betweendifferent clinicians is depicted as grey shaded area and the grey line shows one-to-onecorrespondence.
图12. 随时间变化的体积曲线(1个心跳周期)及从左心室数字双胞胎衍生的新的生物标志物。 (a) 从体积曲线中提取的回波生物标志物的示意图。 (b) 对照组(上图)和急性心肌梗死(AMI)组(下图)患者的体积曲线。 (c) 对照组患者的临床(基于报告)与数字双胞胎衍生的EDV和ESV测量值比较。不同临床医师之间测量的典型变化范围以灰色阴影区域表示,灰线表示一对一的对应关系。
Fig. 13. Top: hazard ratios of the echo biomarkers for 1-year all-cause mortalityprediction calculated with the Cox model. The red shaded parameters are novelbiomarkers not routinely measured in clinical practice. Bottom: the blue line showsthe ROC curve using only ESV, EDV and LVEF. The orange line shows the ROC curveusing all echo biomarkers, including the novel biomarkers (Plateau, Relaxation, RFV,SA and Contractility).
图13. 上:使用Cox模型计算的回波生物标志物对1年全因死亡预测的危险比(hazard ratios)。红色阴影部分为临床实践中不常规测量的新生物标志物。 下:蓝线表示仅使用ESV、EDV和LVEF的ROC曲线。橙线表示使用所有回波生物标志物(包括新生物标志物:Plateau、Relaxation、RFV、SA和Contractility)的ROC曲线。
Fig. 14. Survival probabilities for different patient groups based on Kaplan--Meier curves: In green, the survival probability for patients with LVEF > 40%. In blue and red, thesurvival probability of patients with LVEF ≤ 40% divided into two additional subgroups based on high and low of the respective echo biomarker
图14. 基于Kaplan-Meier曲线的不同患者组生存概率:绿色表示LVEF > 40%患者的生存概率;蓝色和红色分别表示LVEF ≤ 40%患者的生存概率,按照相应回波生物标志物的高低将患者分为两组。
Fig. A.15. Confusion matrix with absolute counts (left) and normalised values (right).The model achieves almost perfect prediction accuracy when evaluating performancewith 5-fold cross-validation
图 A.15. 混淆矩阵,左侧为绝对计数,右侧为归一化值。当使用5折交叉验证评估模型性能时,模型几乎实现了完美的预测准确度。
Fig. B.16. Areal change of the segmented LV in an A2C view. The detected peakscorrespond to the end-diastolic phase. A3C and A4C views show similar curves.
图 B.16. 在A2C视角下分割的左心室(LV)面积变化。检测到的峰值对应于舒张末期(EDP)。A3C和A4C视角显示了类似的曲线。
Fig. C.17. Generated personalised heart shapes (anterior views) of 8 randomly selectedpatients with normal TTE. We display 5 shapes over one heartbeat from diastole (phaseto systole (phase 2) and back (phase 4). Abbreviation: ZH, University Hospital Zurich.
图 C.17. 8位随机选择的正常TTE患者的个性化心脏形状(前视图)。我们展示了5个心跳周期的心脏形状,从舒张期(阶段1)到收缩期(阶段2),再到舒张期(阶段4)。缩写:ZH,苏黎世大学医院。
Fig. C.18. Generated personalised heart shapes (anterior views) of 8 randomly selectedpatients diagnosed with AMI. We display 5 shapes over one heartbeat from diastole(phase 0) to systole (phase 2) and back (phase 4). Abbreviations: GE, Geneva UniversityHospitals; ZH, University Hospital Zurich.
图 C.18. 8位随机选择的AMI(急性心肌梗死)患者的个性化心脏形状(前视图)。我们展示了5个心跳周期的心脏形状,从舒张期(阶段0)到收缩期(阶段2),再到舒张期(阶段4)。缩写:GE,日内瓦大学医院;ZH,苏黎世大学医院。
Fig. D.19. Digital twin vs. MRI-derived 3D LV shape at end-diastole and end-systole.The LV volumes are 150 ml (EDV) and 89 ml (ESV) for the MRI-derived shape, and144 ml (EDV) and 87 ml (ESV) for the digital twin, respectively. These measures giverise to LVEF values of 40% and 41%. For a fair comparison of the LV volumes, thedigital twin is cut-off at the same location as the MRI-based shape.
图 D.19. 数字双胞胎与MRI衍生的3D左心室形状在舒张末期和收缩末期的比较。对于MRI衍生的形状,左心室体积分别为150 ml(EDV)和89 ml(ESV);对于数字双胞胎,左心室体积分别为144 ml(EDV)和87 ml(ESV)。这些测量值得出的左心室射血分数(LVEF)分别为40%和41%。为了公平比较左心室体积,数字双胞胎在与MRI形状相同的位置被截断。
Fig. F.21. Scatter plots and histogram of selected biomarkers. The patients are colouredaccording to ''dead'' (red) and ''alive'' (blue). The RFV biomarker is set to zero if noplateau can be detected, and therefore RFV cannot be measured. The plateau is visiblein healthy patients, but can disappear if the atrial kick is impaired due to MI.
图 F.21. 选定生物标志物的散点图和直方图。患者根据"死亡"(红色)和"存活"(蓝色)进行着色。如果无法检测到平台期,则RFV生物标志物设置为零,因此RFV无法测量。健康患者可以看到平台期,但如果由于心肌梗死(MI)导致心房收缩受损,平台期可能会消失。
Fig. E.20. Survival curves based on LVEF alone with low LVEF group furthersubdivided into two cohorts based on median LVEF threshold of low LVEF groupr
图 E.20. 基于LVEF(左心室射血分数)值的生存曲线,其中低LVEF组根据LVEF的中位数阈值进一步细分为两个队列。
Table
表
Table 1Patient characteristics of normal controls and AMI patient cohorts collected at twodifferent centres between 2009--2020
表1 2009年至2020年间在两个不同中心收集的正常对照组和急性心肌梗死(AMI)患者队列的患者特征
Table 2Statistical significance tests to compare normalised movements in the different perfusionregions for the three AMI types. The 𝑝-value in parentheses describe the significancevalue based on the absolute (actual) movements.
表2 不同灌注区域在三种AMI类型中标准化运动的统计显著性检验。括号中的 𝑝 值描述基于绝对(实际)运动的显著性值。
Table A.3Label distribution for view prediction model training. The 'other' view class summerisesvideos corresponding to the suprasternal notch, inferior vena cava, aorta ascendens, orunknown/unclear views
表 A.3 视图预测模型训练的标签分布。'其他'视图类别汇总了对应于颈上切迹、下腔静脉、升主动脉或未知/不清晰视图的视频。