Skip to content

Fix peer access test synchronization issue #1573

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rwgk wants to merge 1 commit into from
Closed

Fix peer access test synchronization issue #1573

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions cuda_core/tests/memory_ipc/test_peer_access.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,8 @@ def child_main(self, mr, buffer):
# Test 1: Buffer accessible from resident device (dev1) - should always work
dev1 = Device(1)
dev1.set_current()
# Sync dev1 to ensure IPC import operations are complete
dev1.sync()
Comment on lines +97 to +98
Copy link
Contributor

@Andy-Jost Andy-Jost Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not be needed. For one thing, IPC import APIs are synchronous; for another, verify_buffer in the next line synchronizes device 1

See lines 65 and 82 in

cuda-python/cuda_core/tests/helpers/buffers.py

Lines 61 to 66 in eec6988

def __init__(self, device, size, stream=None):
self.device = device
self.size = size
self.stream = stream if stream is not None else device.create_stream()
self.sync_target = stream if stream is not None else device
self.pattern_buffers = {}

and

cuda-python/cuda_core/tests/helpers/buffers.py

Lines 74 to 83 in eec6988

def verify_buffer(self, buffer, seed=None, value=None):
"""Verify the buffer contents against a sequential pattern."""
assert buffer.size == self.size
scratch_buffer = DummyUnifiedMemoryResource(self.device).allocate(self.size)
ptr_test = self._ptr(scratch_buffer)
pattern_buffer = self._get_pattern_buffer(seed, value)
ptr_expected = self._ptr(pattern_buffer)
scratch_buffer.copy_from(buffer, stream=self.stream)
self.sync_target.sync()
assert libc.memcmp(ptr_test, ptr_expected, self.size) == 0

PatternGen(dev1, NBYTES).verify_buffer(buffer, seed=False)

# Test 2: Buffer NOT accessible from dev0 initially (peer access not preserved)
Expand All @@ -106,6 +108,9 @@ def child_main(self, mr, buffer):
dev1.set_current()
mr.peer_accessible_by = [0]
assert mr.peer_accessible_by == (0,)
# Sync dev1 to ensure peer access setup and any pending operations are complete
# before dev0 accesses the peer memory
dev1.sync()
Comment on lines +111 to +113
Copy link
Contributor

@Andy-Jost Andy-Jost Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is also not needed. Peer accessibility APIs are synchronous.

dev0.set_current()
PatternGen(dev0, NBYTES).verify_buffer(buffer, seed=False)

Expand Down
Loading