+ _\i^RIHtHt^RIHtHtHt^RIHt ^RI H t H t H t ^RIHtRRltR RR lltR R R lltR#))reducewraps)AnyCallableOptionalN) tree_flattentree_maptree_unflatten)Modulec0V^8dQhR\R\/#)modelfn)r r)formats"d/Users/agent/.cache/uv/archive-v0/6ZL4-8mCcRNgGE9wi6M6J/lib/python3.14/site-packages/mlx/nn/utils.py __annotate__r s!!&!h!cpaaaVV3Rlp\P!V4o\S4VV3Rl4pV#)aTransform the passed function ``fn`` to a function that computes the gradients of ``fn`` wrt the model's trainable parameters and also its value. Args: model (mlx.nn.Module): The model whose trainable parameters to compute gradients for fn (Callable): The scalar function to compute gradients for Returns: A callable that returns the value of ``fn`` and the gradients wrt the trainable parameters of ``model`` c6<SPV4S!V/VB#Nupdate)paramsargskwargsrrs&*,rinner_fn value_and_grad..inner_fns V4"6""rcB<S!SP4.VO5/VBwr#W#3#rtrainable_parameters)rrvaluegradr value_grad_fns*, rwrapped_value_grad_fn-value_and_grad..wrapped_value_grad_fn!s*#E$>$>$@R4R6R {r)mxvalue_and_gradr)rrrr$r#sff @rr'r' s8#%%h/M 2Y ! rcFV^8dQhR\R\\,/#)r moduler)r rr)rs"rrr)s##v#8H#5#rc|aaaSfSoVV3Rlp\P!V4o\S4VV3Rl4pV#)a$Transform the passed callable to one that performs gradient checkpointing with respect to the trainable parameters of the module (and the callable's inputs). Args: module (mlx.nn.Module): The module for whose parameters we will be performing gradient checkpointing. fn (Callable, optional): The function to checkpoint. If not provided it defaults to the provided module. Returns: A callable that saves the inputs and outputs during the forward pass and recomputes all intermediate states during the backward pass. c6<SPV4S!V/VB#rr)rrrrr)s&*,rrcheckpoint..inner_fn=s f4"6""rc8<S!SP4.VO5/VB#rr)rrcheckpointed_fnr)s*,rwrapped_checkpointed_fn+checkpoint..wrapped_checkpointed_fnCs!v::<NtNvNNr)r& checkpointr)r)rrr/r.sff @rr1r1)sF z#mmH-O 2YOO #"rc V^8dQhR\R\\PP,R\ R\\P ,R\\P,/#)r gradientsgroupall_reduce_sizecommunication_typecommunication_stream)rrr& distributedGroupintDtypeStream)rs"rrrJsb[.[.[. BNN(( )[.[.!* [. #299- [.rc naaaaaaaaT;'g\PP4pVP4oS^8XdV#VVV3RlpV^8:d \ WP4#\ V4p\ V4^8XdV#VUUu.uFwrxVNK uppoVUU u.uFwrV PNK up poVUU u.uFwrV PNK up poVUU u.uFwrV PNK up po\;QJdV3RlS4F 'dK RM RM!V3RlS44'g\W^S4#Se SPMS^,Pp .p .p ^p \\ S44FHpV PV4V SV,V ,, p W8gK3V PV 4.p ^p KJ V 'dV PV 4.p .pV Fp \V3RlV ^.4p\P!V Uu.uF!qV,^,PR4NK# up4oV!S4o\P !SV^R4oVP#VVV3Rl\%V 444K \'V4#uuppiuup piuup piuup piuupi)aAverage the gradients across the distributed processes in the passed group. This helper enables concatenating several gradients of small arrays to one big all reduce call for better networking performance. Args: gradients (Any): The Python tree containing the gradients (it should have the same structure across processes) group (Optional[mlx.core.distributed.Group]): The group of processes to average the gradients. If set to ``None`` the global group is used. Default: ``None``. all_reduce_size (int): Group arrays until their size in bytes exceeds this number. Perform one communication step per group of arrays. If less or equal to 0 array grouping is disabled. Default: ``32MiB``. communication_type (Optional[mlx.core.Dtype]): If provided cast to this type before performing the communication. Typically cast to a smaller float to reduce the communication size. Default: ``None``. communication_stream (Optional[mlx.core.Stream]): The stream to usse for the communication. If unspecified the default communication stream is used which can vary by back-end. Default: ``None``. c<VPpSeVPS4MTp\PP VSR7PV4S, #)N)stream)dtypeastyper&r8all_sum)xdtNr7r6s& r_average#average_gradients.._averagelsO WW,>,JAHH' (PQ~~%%a0D%ELLRPSTTTrc3:<"TFqS^,8HxK R#5i)rN).0rDdtypess& r $average_gradients..s4Vr?VsFTc@<WR,SV,,.,#)rI)rCysizess&&r#average_gradients..s!uuQx/?.@*@rc3x<"TF/wrSV,SV,PSV,43xK1 R#5ir)reshape)rJijbig_gradkeysshapess& rrLrMs6"1DAa(1+--fQi891s7:rP)r&r8initsizer rlenshaper@allaverage_gradientsrangeappendr concatenaterVsplitextend enumerater )r3r4r5r6r7rF flat_gradsk_vitemsize grad_groups grad_groupgrad_group_sizerWnew_flat_gradsindicesrErYrKrZr[rRs&&&ff @@@@@@rraraJs_8  * *R^^((*E AAvU !,,"), z?a  ))jdaj)&01jda!''j1$./JDAJ/&01jda!''j1s4V4sss4V444$Yq:LM M"-  # #   s4y!A   a uQx(2 2O1"":. "# "    z *J%J@*qcRG~~7ABz!Aq!))"-zBH )Hxx'!B-8H  ! !"%j1" &n--W*1/1>Cs JJ <J&J,'J2 r)NiNN) functoolsrrtypingrrrmlx.corecorer&utilsrr r layers.baser r'r1rarIrrrxs3$**::!:#B[.[.r