Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Asked Ross to provide a description of our GPU Standalone Validation Test Suite

...


Tip

Answer is yes.
Yes you can.

This involves a manual declaration of the env vars, and an adjustment of the script to comment 'CUDA_VISIBLE_DEVICES' out, so this does not over-write the UUID of the GPU of the single GPU card to be tested.
This is an applicable solution for a system admin who is comfortable working in the shell or CLI, and the Exxact GPU server or HPC is in a rack or data-center environment.


Expand the content section below to read more.


Expand

To run the GPU Stand Alone Validation tests against a single card-- we must customize the behavior of the script instead of pulling out the cards and rotating them manually.
It does involve a manual change to the GPU validation script, but I tested this in my lab and it worked as expected.

To run the test against one specific card, you will need to perform the following actions:

  1. Back-up the existing "run_test.x" shell script (just be safe, you can always re-download the entire tgz archive again)

  2. Edit the "run_test.x" using your favorite text editor (nano, vim , etc).

  3. #Comment out "CUDA_VISIBLE_DEVICES=$j",

This is seen (3) times in the run_test script. We are removing it here, because we will define this directly in the bash shell so we don't need to edit this file for each and every run.

  1. Run command, "nvidia-smi -L" to get list of all GPU UUIDs.

  2. For each card, before each run, you will set the GPU UUID for the card you wish to test.

e.g.

export CUDA_VISIBLE_DEVICES=GPU-99135ce
nohup ./run_test.x &
    ... [Test completes]

export CUDA_VISIBLE_DEVICES=GPU-13599aa
{{ repeat as needed to isolate faulty GPU }}





Expand
titleAbout Exxact's Standalone Validation Suite

Exxact's Standalone Validation Suite is a proprietary test adapted from the GPU engine within the AMBER Molecular Dynamics Software Suite. Developed by Ross Walker, the principal developer of the AMBER GPU software, the test works by repeatedly running all atom molecular dynamics simulations (MD) of varying size. There are 3 different size of test designed to stress both the GPU itself and the GPU memory. For each test size a simulation is run that consists of millions of MD steps, each comprising a large combination of single and double precision floating pointing calculations as well as fixed precision integer arithmetic. The calculation includes pair wise electrostatic and van der Waals interactions, Fourier Transforms, inverse R squared calculations, pair list sorts and integration. This computation pattern uses all parts of the GPU and also stresses the GPU memory. At the end of a fixed number of steps for each run, which averages between 15 and 30 mins the final coordinates, energies and velocities of the atoms are recorded. The calculation is then repeated from the same input parameters and again after a fixed number of steps the final coordinates, energies and velocities of the atoms are recorded. The AMBER GPU engine is designed to be bitwise reproducible which means that a simulation started from identical conditions should give identical results. Any variation in the final results is thus an indication of either a bad GPU or bad GPU memory. The test is run for a total of 24 hours and is very effective at identifying faulty GPUs. So effective in fact that it is credited with identifying design flaws and insufficient frequency margins on 5 different NVIDIA GPU models and NVIDIA now includes a variation of this code as part of their chip design testing process. In addition to checking that all GPUs give consistent results the performance of each GPU is tested using the same code. Performance between repeat runs and between GPUs is compared and determined to be within acceptable tolerances before a system is shipped. This approach effectively identifies both faulty GPUs, for example with faulty power and temperature regulators, and any GPUs that might have insufficient cooling due to air flow restrictions, fan issues etc.


Filter by label (Content by label)
showLabelsfalse
max5
spacescom.atlassian.confluence.content.render.xhtml.model.resource.identifiers.SpaceResourceIdentifier@209e85
showSpacefalse
sortmodified
reversetrue
typepage
cqllabel in ( "validation" , "test" , "gpu" ) and type = "page" and space = "ESKB"
labelsGPU validation test

...