Earth Species Project: Open-Source AI for Non-Human Communication

The Earth Species Project (ESP) is an independent non-profit research organization dedicated to developing open-source artificial intelligence and foundation models to decode and understand non-human communication. Founded in 2017 by Aza Raskin, Britt Selvitelle, and Katie Zacarian, ESP collaborates directly with field biologists and acoustic researchers globally to build machine learning tools that accelerate biological discovery.

Unlike commercial AI labs focused primarily on human language processing or industrial audio tasks, the Earth Species Project directs machine learning architectures—specifically self-supervised representations, unsupervised translation algorithms, and source-separation models—toward the unique mathematical and biological structures of animal vocalizations.

Core Mission and Strategic Research Pillars

The Earth Species Project operates on an open-source ethos, releasing its codebases, benchmark suites, and pre-trained neural network checkpoints under permissive open-source licenses to benefit the wider scientific and ecological community:

[ Earth Species Project Research Framework ]
 ├── 1. Self-Supervised Foundation Models (AVES)
 │    └── Pre-trained acoustic transformers overcoming the ecological label bottleneck
 ├── 2. Standardized Ecological Benchmarks (BEANS)
 │    └── Standard evaluation benchmark for animal sound classification and detection
 ├── 3. Bioacoustic Source Separation (Cocktail Party Problem)
 │    └── Disentangling overlapping vocalizations in wild multi-animal environments
 └── 4. Generative and Unsupervised Bioacoustic Exploration
      └── Latent space topology and unsupervised vocal morphology discovery

The AVES Foundation Model Architecture

A major challenge in non-human bioacoustics is the label bottleneck: while automated recording units (ARUs) collect millions of hours of audio, only a minuscule fraction can ever be manually annotated by expert biologists.

In 2023, Dr. Masato Hagiwara, senior research scientist at ESP, published the architecture for AVES (Animal Vocalization Encoder based on Self-Supervision) at the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP):

Model Architecture and Pre-training

AVES adapts the self-supervised HuBERT (Hidden-Unit BERT) architecture for non-human bioacoustics:

1. Convolutional Waveform Encoder: A multi-layer 1D convolutional neural network processes raw 16 kHz audio waveforms directly, extracting continuous acoustic feature representations at a 20 ms frame rate.
2. K-Means Acoustic Clustering: In the initial pre-training stage, a k-means clustering algorithm assigns discrete acoustic cluster IDs to continuous frame embeddings, generating pseudo-units for transformer modeling.
3. Masked Transformer Prediction: Similar to BERT in natural language processing, random spans of audio frames are masked out (obscured). The transformer is trained to predict the correct discrete acoustic cluster ID of the masked frames from surrounding unmasked context.

By pre-training on diverse multi-species bioacoustic datasets, AVES learns rich representations of acoustic pitch, timbre, frequency modulation, and temporal rhythm. When fine-tuned on downstream ecological tasks, AVES consistently outperforms general speech models (such as wav2vec 2.0 or HuBERT trained on human speech) and general audio models (such as Audio Spectrogram Transformer / AST).

Verified Open-Source Licensing and Repositories

In accordance with strict verification standards, the current release status and legal licensing of Earth Species Project assets on GitHub were independently confirmed:

1. AVES Codebase and Model Checkpoints:
   • Repository: github.com/earthspecies/aves
   • Verified License: MIT License (permissive open-source license permitting academic and commercial reuse with attribution).
   • Model Availability: Pretrained PyTorch weights (aves-base-bio and aves-large-bio) are freely downloadable for researchers.
2. BEANS Benchmark Suite:
   • Repository: github.com/earthspecies/beans
   • Verified License: MIT License.
   • Scope: The Benchmark of Animal Sounds (BEANS) provides standardized classification and detection datasets spanning birds, marine mammals, amphibians, and terrestrial mammals, allowing objective benchmarking of bioacoustic models.
3. Specialized Acoustic Tools:
   • Repositories such as sperm-whale-clan-and-coda-detection and cornell-birdcall-competition-starter-pack are released under the Apache-2.0 License.

Bioacoustic Cocktail Party Separation

In field environments, animals rarely vocalize in sterile silence. Dense dawn choruses of birds, frog ponds, or swimming dolphin pods produce heavily overlapping acoustic energy—known as the cocktail party problem.

ESP conducts active research into deep neural source separation, training models to isolate the vocal stream of a single individual animal from overlapping background vocalizations and wind noise. Solving this problem is a crucial prerequisite for automated sequence modeling, as language-like transition models require cleanly segmented vocal bouts from known individuals.

Conclusion and Future Horizons

The Earth Species Project represents a collaborative bridge between modern artificial intelligence and ecological conservation. By publishing open-source foundation models like AVES and rigorous benchmarks like BEANS, ESP equips marine biologists and terrestrial ecologists with the computational tools needed to protect endangered ecosystems and examine the evolutionary roots of non-human communication.

To learn more about current open-source initiatives and field collaborations, visit the Earth Species Project official website.