This is where you configure the proof tape, and you'll have to make some
decisions.
name: run names must be kebab-case, with a dash like/this-example.
rolling_window: every rolling_window number of tokens is considered a sample.
# register an experimentexperiment=client.experiment("your-name/your-run")# initialize your zkdv tapeproof=experiment.proof(backend="torch")proof.start(rolling_window=32)# prepare your Torch models for tracingmodel,optimizer=proof.prepare(model,optimizer)
Tap for more on rolling_window
A rolling_window of 4 means that every 4 tokens is considered a "sample."
For instance, the quick brown fox and brown fox the quick are considered
two distinct samples when rolling_window=4, but considered a shuffling of
the same sample when rolling_window=2. A longer window results in a shorter
tape, but offers less flexibility in shuffling.
We need a deterministic description of how you got from one parameter to the next.
@proof.attestdefattested_step(model,optimizer,batch):# you must keep this signaturelogits,loss=proof.forward(model,**batch)# instead of model(**batch)proof.backward(loss)# instead of loss.backward()optimizer.step()optimizer.zero_grad()
This function has to be deterministically serializable. Common culprits that prevent
this include custom kernels. Rest assured, we don't call this function that many times
(usually <1% of training), so you can definitely wire up a "slow-path" here without
loosing MFUs.
Your training loop remains largely the same, and can contain any custom logic:
for_inrange(...):batch=...withproof.transaction(batch=batch):logits,loss=model(**batch)proof.backward(loss)# instead of loss.backward()optimizer.step()optimizer.zero_grad()