yezhengmao/nccl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Check return code for Flush operation

Browse Source 
Current NCCL code does not abort for failed Flush operations by
underlying network. This may compromise data integrity.

Signed-off-by: Rashika Kheria <rashika@amazon.com>
This commit is contained in:
Rashika Kheria 2020-03-16 18:33:48 -07:00 committed by Sylvain Jeaugey
parent c38f174bd4
commit 6c61492eba
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
src/transport/net.cc
View File

@ -378,7 +378,7 @@ ncclResult_t netRecvProxy(struct ncclProxyArgs* args) {
if (done) { if (done) {
args->head += args->sliceSteps; args->head += args->sliceSteps;
if (args->protocol == NCCL_PROTO_SIMPLE) { if (args->protocol == NCCL_PROTO_SIMPLE) {
if (resources->useGdr) ncclNetFlush(resources->netRecvComm, localBuff+buffSlot*stepSize, size, mhandle); if (resources->useGdr) NCCLCHECK(ncclNetFlush(resources->netRecvComm, localBuff+buffSlot*stepSize, size, mhandle));
resources->hostRecvMem->tail = args->head; resources->hostRecvMem->tail = args->head;
} }
args->idle = 0; args->idle = 0;