TinyMem: learning fixed-size memoryTinyMem: learning fixed-size memoryTinyMem: learning fixed-size memory

I started TinyMem with a question: could a model learn to keep the useful parts of a history in a few hundred bytes? I built a writer that updates a small state as records arrive, then a reader that answers questions from that state. The writer never gets to see the question in advance.

I studied two related problems: whether repeated updates preserve earlier facts, and whether learned memory uses its storage budget more effectively than compact text. Controlled experiments showed that explicit state supervision could produce strong recall, but answer-only training remained unreliable.

In the final comparison on bAbI tasks 1–5, learned memory reached roughly 21% accuracy at 64, 256, and 1,024 bytes. Dictionary-coded text reached 99.7% at 64 bytes on clean inputs. All methods performed poorly under heavy distractors. That was not the result I was aiming for. The useful part was working out which failures came from the writer, which came from the reader, and what I could actually conclude from either.

I drew on recurrent memory models such as RMT^[1] and ARMT.^[2] ICAE^[3] and Compressed Context Memory^[4] informed the encoder–reader design review. These systems use different representations and storage accounting; TinyMem is not a replication of their architectures.

Writing and reading the state

The writer combines features from the current record with the previous memory state. After each update, the source text is discarded. At read time, a learned projection converts the saved state into prefix vectors for the language model; the question is introduced only at this stage.

The later studies used Qwen3-1.7B^[5] with frozen base weights. Some readers had trained rank-eight LoRA adapters. Those shared weights were separate from the per-history storage budget: the bytes that survived from one statement to the next.

From gated slots to selective updates

I first used gated slots, where a learned gate blends a candidate value into the old memory. The next design was informed by work on gated delta networks:^[6] choose an address, read what is there, and write a correction. The final comparison used attention-based slots, quantized to int8 after every write.

This is the basic update, before gating: $M_{t-1}$ is the old memory, $k_t$ is the key, and $v_t$ is the proposed value. The correction $\delta_t$ subtracts the value already stored at that key, $\hat{v}_t$. The outer product writes that correction into memory.

Overlapping keys were one possible source of write interference. The controls below test key functions, without isolating overlap from their other properties.

Reader competence

My first byte-level model, trained from scratch, scored 0 out of 500 on LongMemEval. It also failed an answer-copy control with the answer provided in its input. I could not blame the memory when the decoder could not handle that simpler task.

I therefore moved to simpler question-answering tasks and controlled readouts. The early byte-level model and the later Qwen-based system are separate experimental stages; their scores are not directly comparable.

Interference from repeated writes

A controlled test used four independent binary facts. After the writer encoded them, it received eight truthful repetitions of one fact. I measured recall of the three unmentioned facts before and after those writes.

This was the failure I found most useful to study: nothing new needed to fit, and no correct answer had changed. Yet recall fell in all three confirmation seeds, under both uniform and correction-weighted training.

A fixed linear probe recovered some facts the language model missed. Its recall also declined after repetition. This shows reduced recoverability under both readouts, without establishing that the information was irreversibly erased.

Initial recall versus retention

A subsequent delta-rule comparison used 66- and 258-byte states. Several models remained near chance and showed little dependence on their memory. Small before-and-after changes in these runs did not indicate good retention; the initial recall was already poor.

I needed a way to tell these failures apart, so I tested the components separately: reading a correct state, learning to produce that state, and retaining it through unrelated writes.

Testing the reader in isolation

I replaced the learned writer with an exact parser. Each of three fresh bridge-and-reader pairs scored 14,336/14,336 recorded answers, or 43,008 across all three readers. These records include shared-prefix duplicates and paired wording variants, rather than independent histories. Swapping in another story’s memory changed the answers to match that memory. These controls established that the reader could use the supplied state.

Stabilizing the writer

Training the writer against exact target states exposed a failure in some runs: gates or hidden activations approached zero. Adjusting write strength alone did not resolve it. Affine-free LayerNorm^[7] before the hidden activation improved recall across the paired experiment, without adding parameters or storage bytes.

The catch was that most of the improvement came from learning a better state initially. It did not mean I had solved forgetting. The result also depended on exact state targets, a stronger training signal than answers alone.

Answer-only training

On the official bAbI QA1 test, state-supervised writers reached 98.2% mean answer accuracy, close to 98.3% with oracle memory. With answer-only writer training, the same architecture performed poorly: development accuracy averaged 29.1%. Those are different evaluation splits, so they should not be read as a matched test-set comparison.

The diagnostics found useful factual gradients, but also much more interference between unrelated writes. I then held the key function fixed and trained a fresh value branch. Keys from a successful state-supervised writer raised mean development accuracy from 31.8% to 59.0%, compared with keys from a failed answer-only writer.

Six of the twelve pairs using successful keys still scored below 30%. The intervention improved mean accuracy but did not make value learning reliable. Because the reader and these keys had prior state supervision, this experiment was not a fully answer-only system.

I still needed to answer the original question: was any of this better than keeping a short piece of text? For the final study, I gave learned memory and explicit storage the same persistent byte budgets.

Matching the storage budget

It tested 64, 256, and 1,024 bytes across bAbI tasks 1–5. The learned writer and read adapter trained jointly from answer tokens. Every write was quantized, packed, and later restored from real bytes. Fixed-level scalar quantization^[8] informed the choice to avoid a learned codebook; the int8 implementation is specific to this study. The alternatives kept compressed recent text, compressed text selected for lexical diversity, or a dictionary-coded text log.

What the memory retained

The learned system stayed around 21% at every budget. Zero-memory controls averaged 17.2%, and other-story controls 16.6%. The repeated task-level scores and these controls indicated that the reader had mostly learned answer frequencies rather than a useful dependence on the stored state.

On clean inputs, the dictionary store reached 99.7% at 64 bytes, and compressed recent text reached 99.7% at 256 bytes. Under heavy distractors, every method fell to roughly 21–27%. The advantage of explicit storage was therefore specific to the clean setting.

Transfer to longer histories

On the known BABILong 1k transfer set, learned memory stayed near 20%. Compressed recent text reached 34.5%, 50.1%, and 88.4% as the budget increased. This was transfer to a benchmark the project had already encountered, not an untouched discovery set.

I started TinyMem hoping that a learned state could make better use of a small storage budget than a short text history. In the systems I tested, it did not. Compact text was much stronger on clean inputs, while heavy distractors exposed weaknesses in every method.

The experiments changed how I think about that result. Reading a correct state, learning to write one, and preserving it through later updates are separate problems. Explicit state supervision gave strong recall, but that success did not carry over to reliable learning from answers alone. It also did not establish a compression advantage: four binary facts fit in one byte, however well a larger learned state recalls them.

My main takeaway is that a convincing memory system needs more than a good final answer score. It needs controls that show what was written, what remains recoverable, and whether a simpler store would do better. These experiments concern particular architectures, tasks, and training recipes, not a general limit on learned memory. They gave me a clearer account of where this approach failed, and of what a stronger result would have to demonstrate.