kv: iterators

[barraguda]

Problem

"Get me all my keys in X db"
"Get me all keys that have the prefix "nodes:"

Solution

IterTypes Requests and Responses.

// get all users (returns Vec<(String, User)>)
let all_users = kv.iter_all(None, 100)?;
for (key, user) in all_users {
    println!("Key: {}, User: {:?}", key, user);
}

// get users with prefix "user_" (batches of 50)
let user_prefix = "user_".to_string();
let filtered_users = kv.iter_all(Some(&user_prefix), 50)?;

// get all keys starting with "user_"
let keys = kv.collect_keys(Some(&"user_".to_string()))?;
for key in keys {
    println!("Found user key: {}", key);
}

// get all values (users)
let users = kv.collect_values(None)?;

Docs Update

TODO
Corresponding docs PR

Notes

Could do a full iterator impl, but might be too complex with lifetimes and an extra struct, something like

pub struct KvIterator<'a, K, V> {
    kv: &'a Kv<K, V>,                    // Reference to KV store
    iterator_id: u64,                     // Server-side iterator ID
    batch_size: u64,                    // How many items to fetch per batch
    current_batch: Vec<(K, V)>,          // Current batch of items
    done: bool,                          // Whether we've reached the end
    prefix: Option<Vec<u8>>,             // Optional prefix filter
    _marker: PhantomData<(K, V)>,        // Required for generic types
}

impl<'a, K, V> Iterator for KvIterator<'a, K, V> 
where
    K: Serialize + DeserializeOwned,
    V: Serialize + DeserializeOwned,
{
    type Item = anyhow::Result<(K, V)>;

    fn next(&mut self) -> Option<Self::Item> {
        if self.current_batch.is_empty() && !self.done {
            if let Err(e) = self.fetch_next_batch() {
                return Some(Err(e));
            }
        }
        self.current_batch.pop().map(Ok)
    }
}

// would allow for this:

let iter = kv.iter(Some(&"user_"), 100)?;
for result in iter {
    let (key, value) = result?;
    println!("Key: {}, Value: {:?}", key, value);
}

[nick1udwig]

It feels somewhat fake without an easy-to-call .next(), but given the infra added in the Kinode core PR, shouldn't be too crazy to add that method later. LGTM

[nick1udwig]

A fun thing we can do in future if we ever write a Wasm process_lib is package up this iterator in a resource