Using the OSG
Computing resources are available through the International Gravitational-Wave Observatory Network (IGWN) Computing Grid, which is comprised of computing resources provided by member groups and external partners in partnership with the Open Science Grid (OSG). The grid relies on HTCondor for use of the resources.
Using the OSG for RIFT jobs is fast. It is a great tool, just make sure
the appropriate arguments are supplied both in the workflow and to the
environment so the jobs actually run. There are a few requirements for
submitting RIFT jobs via HTCondor on the OSG which differ from other
machines in the network. To submit a job, you must use
a flag, --import-env
, which imports all environment variables
condor_submit_dag -import_env master.dag
Ensure that the ini file you used to create your workflow included arguments indicating that your job uses the OSG. This adds the appropriate arguments in the .sub files used by your job.
Finally, you need various additional environment variables, ranging from absolutely essential to extremely helpful:
RIFT_REQUIRE_GPUS=(DeviceName=!="Tesla K10.G1.8GB")&&(DeviceName=!="Tesla K10.G2.8GB")
RIFT_LOWLATENCY=True
RIFT_GETENV=LD_LIBRARY_PATH,PATH,PYTHONPATH,*RIFT*,LIBRARY_PATH
SINGULARITY_RIFT_IMAGE=/cvmfs/singularity.opensciencegrid.org/james-clark/research-projects-rit/rift:production
RIFT_GETENV_OSG=*RIFT*,NUMBA_CACHE_DIR
A sample OSG command line might look like this:
util_RIFT_pseudo_pipe.py --gracedb-id GID_GOES_HERE --use-online-psd --online --choose-data-LI-seglen --use-ini `pwd`/tryme.ini --use-coinc `pwd`/coinc.xml