Open
Conversation
`cmake` throws an error if the `ThirdParty/zstd/build/cmake` git submodule is not cloned before the build: ``` CMake Error at CMakeLists.txt:96 (add_subdirectory): add_subdirectory given source "ThirdParty/zstd/build/cmake" which is not an existing directory. ``` So update the install command to avoid more users running into this. See #320 Co-authored-by: MaggieQi <chenqi871025@gmail.com>
* improve performance and memory usage * fix CLRCore build * clean code * align the cosine with production * remove the quantizer check between search head index and ssd index. * remove some freq condition check * unify AsyncReadRequest Co-authored-by: chenqi <cheqi@microsoft.com> Co-authored-by: Philip Adams <35666630+PhilipBAdams@users.noreply.github.com>
Co-authored-by: jinweizhang <jinweizhang@microsoft.com>
Fix GPU PQ build errors and added error checks
* GPU KNN Intergrate * Merged cuda hxx files * Remove extra files, fix CoreLibrary build * GPU code needs to be in cu file * break circular including * Rearranged the method * Included the files like Neighborhood Graph. h did. Only included need files in Kernel.cu * Relocate the Quey_KNN to KNN.hxx (templates cannot be in kernel.cu * Try to Instantizate before compile * Fixed compile error. Need GPU to test rest * Set Mem Failure * Adjusted the max dimension for vectors up ot 184 * Change DistCalMethod * Print Thread Start/End * Shared memory * Cannot launch Kernelr with 60% MaxSharedMemory * Root Cause: Transpose_Mem * L2 480s, Cosine 703s after Transpose * Two functions for Shared and local ThreadHeap, two launch setting with shared and local DistPair * Test version for Ben. 32 Threads defined in params.h query_KNN has transpose and all shared memory. query_KNN has transpose and heamMem in local. * Found the debugger issue, next step: fix dist calc * 1. Fixed int8 dist calc 2. Tested batch splitting on 35M 3. Moved the malloc before launching to avoid waste of memry * 45cap, monitor mem usage, track where Convert failed * Succeed on 400M 100D, Fixed int overflow, major change: int to size_t * Multi-GPU detection * Fixed the CPU mem over-usage, located the bug in updating results * Fixed multi free results * Fixed hard coded Cosine DistCalcMethod * Relocate the Point & Transposed Point to GPUKNNDistance.hxx * Merge SPTAG current changes, complaining about cuh cuda lib * Move Generate Truth to TruthSet.cpp * Fix optimization issue, add GPUCoreLibrary/GPUSSDServing to default build * Fix Error for Linux cmake. * Remove Unnecessary Changes. * restore datasets * missing s in folder name * Restore build config to Lib, restore sln Config * update tlog for lib config * Remove Wrappers * Removed not ignored log info * Remove Static in TruthSet.cpp Remove debug command, build GPU SSDServing to exe in Debug * Recover CL compile for GPU SSDServing/main.cpp Co-authored-by: diegocai <diegocai@microsoft.com> Co-authored-by: Philip Adams <philipadams@microsoft.com> Co-authored-by: Diego-Cai <103398280+Diego-Cai@users.noreply.github.com>
…tion (#339) * Fix bug with int8/cosine configuration, and enabled hardware optimization for this case. * Trigger CI Co-authored-by: diegocai <diegocai@microsoft.com>
* add nni_auto_tune * support other data format and add more result * add result picture * refine readme * refine readme * refine code style * update readme and datareader * add aml training config * update config * fix licence * refactor for data type * refactor for data type * refactor for data type * fix overflow on bruteforce * fix compute metric by index * add limits and preprocessing * change code dir Co-authored-by: Guoxin <suiguoxin@gmail.com> Co-authored-by: MaggieQi <chenqi871025@gmail.com>
* Fixed bugs with accuracy fix for TPT build with recon vectors * Accuracy fix working for GPU index build with PQ/OPQ enabled * Fix bug with accuracy fix * Trigger CI * Fix low accuracy issue with GPU index build for int8/cosine configuration (#339) * Fix bug with int8/cosine configuration, and enabled hardware optimization for this case. * Trigger CI Co-authored-by: diegocai <diegocai@microsoft.com> * Add nni_auto_tune example (#325) * add nni_auto_tune * support other data format and add more result * add result picture * refine readme * refine readme * refine code style * update readme and datareader * add aml training config * update config * fix licence * refactor for data type * refactor for data type * refactor for data type * fix overflow on bruteforce * fix compute metric by index * add limits and preprocessing * change code dir Co-authored-by: Guoxin <suiguoxin@gmail.com> Co-authored-by: MaggieQi <chenqi871025@gmail.com> * Fixed bugs with accuracy fix for TPT build with recon vectors * Accuracy fix working for GPU index build with PQ/OPQ enabled * Fix bug with accuracy fix * Trigger CI Co-authored-by: diegocai <diegocai@microsoft.com> Co-authored-by: smallv0221 <33639025+smallv0221@users.noreply.github.com> Co-authored-by: Guoxin <suiguoxin@gmail.com> Co-authored-by: MaggieQi <chenqi871025@gmail.com>
* Remove Transposed Point to save shared Memory * Add Generate GT int8 Dim to 768, add CUDA CHECK to debug, Add infty for uint32 * Fix sharedmem usage for K=100 gt Co-authored-by: Diego Cai <diegocai@microsoft.com> Co-authored-by: MaggieQi <chenqi871025@gmail.com>
* Fix build issue and windows issue with is_same_v * fix is_same_v error
* update python to python3 * use dynamic linking * enable ANNIndexTestTool code compiling * add python version in wheel package * update nuspec * enable to set different maxcheck and hashexponent * change to configure python version * trigger azurepipeline * trigger * fix python version * fix python version * fix python version in windows * fix cosine kmeans * clean avx/see header files * fix nuspec Co-authored-by: cheqi <cheqi@SRGSSD-07>
* prevent SimpleBufferIO fails to resize * Set put area for SimpleBufferIO * Update DiskIO.h
* add winrt projects * add gitignore for VS files * retarget to vc142 * make neighborCount uint32 * api takes byte[] for metadata * enable CFG and disable incremental linking to make BinSkim pass * format * remove edit and continue /ZI since it's incompatible for CFG * remove arm/arm64 platforms
* modify for thread_local context * fix initialization issue * fix ExtraWorkSpace id issue * fix workSpacePool * set thread affinity * add more affinity strategies * fix cmake compiler * fix linux libnuma compile * fix compiling and core bind * fix NumaStrategy and OrderStrategy enum type * remove space * Clear the workspace to ensure the heap size and pagebuffer size * User-overrideable worskpace implementation draft (#362) * make it possible to override workspace implementation * bool -> ErrrorCode * SPANN index should allow setting child index workspace * finish replacing by workspace factory * switch to unique_ptr * unresolved external * linux build error * windows build error --------- Co-authored-by: cheqi <cheqi@SRGSSD-07> Co-authored-by: Philip Adams <35666630+PhilipBAdams@users.noreply.github.com>
I have gone to "https://sourceforge.net/projects/boost/files/boost-binaries/1.67.0/" and downloaded "boost_1_67_0-msvc-14.1-64.exe", but encountered this failure during cmake -- Could NOT find Boost (missing: system thread serialization wserialization regex filesystem) (found suitable version "1.67.0", minimum required is "1.66") CMake Error at src/legacy/sptag/SPTAG/CMakeLists.txt:90 (message): Could not find Boost >= 1.67! I've notice that there are no precomiled libs, and had to run bootstrap + build exe to get the compiled libs. Co-authored-by: Philip Adams <35666630+PhilipBAdams@users.noreply.github.com>
* test quantization in SPFresh * enable BKT in quantization testcase --------- Co-authored-by: qiazh <qiazh@microsoft.com>
Fix testcases
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.