Stanford Radiology

GPU Time-Out Detection and Recovery (TDR)

Time-Out Detection and Recovery (TDR) is a Windows-intrinsic feature detecting response problems of the graphics card. If the graphics card does not respond within a certain time frame the operating system will assume a crash and reset the graphics card's driver. While this feature is beneficial in everyday use, it may lead to execution time errors (and exceptions) for GPU computation tasks, when the kernel execution takes longer than the TDR delay time. 

We have encountered this problem in the case of GPU-accelerated bilateral filtering or parallel projection forward projections using ray casting (many viewing angles). 

If you assume the errors you are getting arise from the reset of the driver, you can try increasing the TDR delay time. If you have a CUDA Development Kit installed, it may be as easy as running the Nsight Monitor and increasing the time there. More information on this topic can be found here: Opens external link in new windowlink to Nvidia Dev.  

Else, you may do so by modifying the corresponding registry entry. Therefore you will have to open the Registry Editor by running the command "regedit". In the Registry Editor navigate to HKEY_LOCAL_MACHINE -> SYSTEM -> CurrentControlSet -> Control -> GraphicsDrivers where you can create a key of type “DWORD” and name it “TdrDelay” taking the delay time as an argument.

Be considerate when changing the TDR delay time, as you may end up not detecting non-responsive graphics cards.

We do not take over any liability for disadvantages, which arise from the fixes stated here.