Autoregressive token generation has dominated modern NLP pipelines, but forcing language models to commit to discrete, token-by-token choices introduces systemic bottlenecks. When every prediction must collapse instantly into a categorical distribution, intermediate gradient information vanishes. ELF (Efficient Language Framework) tackles this problem by shifting text generation into continuous embedding space via continuous-time flow matching. Instead of ping-ponging between discrete tokens and latent states, ELF stays inside a continuous trajectory until the final timestep, projecting to discrete tokens only at the very end.
This continuous framing changes the mechanics of language generation. Because the entire generation path operates over smooth latent vectors, techniques from continuous image diffusion, such as Classifier-Free Guidance (CFG), port directly into text workflows without awkward discrete approximations.
Core Architecture: Continuous-Time Flow Matching
Most prior diffusion language models (DLMs) stumble by constantly projecting back and forth between discrete token distributions and continuous representations. This frequent discretization injects noise into the sampling trajectory.
ELF Architecture Pipeline:
[Frozen T5 Encoder]
│
▼
[Latent Embeddings] ──► [Continuous Flow Matching Engine] ──► [Weight-Shared Projection] ──► Discrete Tokens
(Continuous Space Trajectory) (Only at final t)
ELF avoids this failure mode with a streamlined pipeline:
- Frozen Representation Base: Text inputs are mapped into high-quality continuous embeddings using a frozen T5 encoder.
- Continuous Flow Matching: The flow matching network operates entirely within this continuous embedding space across the full integration interval.
- Final Projection: A weight-shared network maps the continuous states back to discrete token outputs only at the terminal timestep.
This setup makes text generation look mathematically identical to continuous vector field transport, providing stable trajectories and enabling native support for CFG and SC-CFG.
Runtime Environment and Distributed Execution
ELF is implemented in JAX and engineered for high-throughput accelerator clusters. The canonical reference benchmarks run on Google Cloud TPU v5p-64 pods via the TPU Research Cloud (TRC).
bash# Verify your JAX runtime and TPU device visibility python -c "import jax; print('Devices:', jax.device_count(), jax.devices())"
The codebase uses automated remote fetching via HuggingFace repositories. Model checkpoints are stored under lillian039/ELF, while pre-tokenized corpora and encoder weights reside under lillian039/ELF-data. Local training environments can override these paths to read directly from local directories produced via dataset.save_to_disk(). A PyTorch implementation is also planned for future releases.
Layered Configuration System
Production workloads require hyperparameter customization without deep code modifications. ELF implements a two-tier configuration structure:
- Base Hyperparameters (
dataclass): Establishes hard defaults for all architectural, training, and sampling primitives. - Task-Specific Overrides (
YAML): Passed directly intomain.pyat runtime to modify dataset paths, training schedules, and logging sinks.
Custom experiments are defined by placing custom configuration files into configs/experiments/. For example, artifact tracking through Weights & Biases is configured by setting wandb_entity inside the target YAML file alongside your custom save_dir.
configs/
├── experiments/ # User-defined run configurations
├── samplers/ # SDE and ODE numerical sampling schedules
└── unconditional_generation/
├── elf_b.yaml # Base model configuration
├── elf_m.yaml # Medium model configuration
└── elf_l.yaml # Large model configuration
Training Workloads and Datasets
ELF supports three foundational training configurations out of the box:
bash# Launch a distributed training run python -m elf.train --config configs/unconditional_generation/elf_b.yaml
The included profiles cover both unconditional and conditional tasks:
| Config Path | Task / Target | Target Dataset |
|---|---|---|
unconditional_generation/elf_b | Unconditional Generation (ELF-B) | OpenWebText |
unconditional_generation/elf_m | Unconditional Generation (ELF-M) | OpenWebText |
unconditional_generation/elf_l | Unconditional Generation (ELF-L) | OpenWebText |
machine_translation/wmt14_de_en | Machine Translation | WMT14 De-En |
abstractive_summarization/xsum | Abstractive Summarization | XSum |
Training Optimizers and Compute Dynamics
The standard training schedule for elf_b runs on OpenWebText for 5 epochs (roughly 95,000 optimization steps) on a TPU v5p-64 slice. Each epoch completes in approximately 1.5 hours.
- Optimizer: Muon
- Base Learning Rate (blr): 0.001
- Effective Batch Size: 512 (effective learning rate scales to 0.002)
Data Preparation for Custom Corpora
To feed custom data into ELF, serialize inputs as HuggingFace Dataset instances tokenized with the T5 tokenizer:
- Unconditional Corpora: Provide a field named
text_idscontaining target token sequences. - Conditional Tasks (MT / Summarization): Provide
source_idsfor conditioning context alongsidetarget_idsfor ground truth output.
The ELF dataloader collator handles EOS token injection and dynamic attention mask generation automatically.
Sampling Paradigms and Empirical Benchmarks
Sampling parameters are isolated from training parameters, with standalone configurations stored inside configs/samplers/. This design lets engineers test different differential equation solvers without modifying training artifacts.
Solver Configurations
- Unconditional Sampling: Driven by Stochastic Differential Equation (SDE) solvers. The engine provides a 32-step solver (γ=1.5) and a 64-step solver (γ=1.0), both running with SC-CFG=3.
- Conditional Sampling: Driven by a 64-step Ordinary Differential Equation (ODE) solver operating with CFG=2 and SC-CFG=1.
Evaluation evaluates batches of 1,000 generated samples across generative perplexity, entropy, BLEU, and ROUGE:
| Task Setup | Metric | Validation Benchmark | Evaluation Notes |
|---|---|---|---|
| Unconditional (ELF-B, 32-step SDE) | Generative PPL (GPT-2 Large) | ≈ 24 | Evaluates sample fluency |
| Unigram Entropy | ≈ 5.15 | Measures lexical diversity | |
| WMT14 De-En Translation | BLEU | ≈ 26.7 | Test set reported in paper |
| XSum Summarization | ROUGE-1 | ≈ 36.3 | Test set reported in paper |
| ROUGE-2 | ≈ 12.5 | Test set reported in paper | |
| ROUGE-L | ≈ 28.1 | Test set reported in paper |
For raw evaluation workflows using JSONL inputs, the framework integrates directly with datasets.load_dataset('json', data_files=...) to tokenize on the fly.
Checkpoint Topology and Distributed IO
Checkpointing in distributed training setups requires strict IO isolation to prevent write locks and filesystem contention:
Distributed Cluster Checkpoint IO:
[Process 0 (TPU Host)] ──► Writes Disk Snapshot ──► Pushes to HF Hub (`hf_repo_id`)
[Process 1..N (TPU Pod)] ──► Compute Only (Disk IO Suppressed)
- Process-Level IO Isolation: Disk serialization is restricted solely to Process 0.
- Rolling Retention: ELF retains the 10 most recent checkpoints on disk, triggering saves at epoch boundaries or at fractions defined by
save_interval_steps. - Automated Cloud Sync: When
hf_repo_idis defined, the framework synchronizes the localsave_dirwith the remote HuggingFace model hub on every save. - Encoder Weight Decoupling: The frozen T5 encoder parameters are stored separately as MessagePack files (
flax_params.msgpack), fetched on initialization fromlillian039/t5-small-encoder.
To resume interrupted runs, omit checkpoint_path to pick up the latest state inside save_dir, or point the config to a local directory or HuggingFace repo ID such as lillian039/ELF-B-OpenWebText.
License and Attribution
The codebase is released under the MIT License. Hardware acceleration resources were supported by the Google TPU Research Cloud (TRC). Reference implementations, model weights, and datasets can be found in the upstream repository: https://github.com/lillian039/ELF.