Tutorial 2: Contrastive Retrieval

This tutorial demonstrates contrastive learning approaches in NeuroVLM for:

1. Brain-to-Text: Labeling brain networks and activation maps

2. Text-to-Brain: Finding similar brain maps from text queries

We’ll cover:

• Network labeling with canonical atlases

• ICA component labeling (HCP, UK Biobank)

• NeuroVault map labeling

• Text-to-brain retrieval across multiple datasets

import os

os.environ["USE_TF"] = "0"
os.environ["USE_FLAX"] = "0"
os.environ["TOKENIZERS_PARALLELISM"] = "false"

from neurovlm import NeuroVLM
from neurovlm.data import load_latent

# Initialize model
nvlm = NeuroVLM()

1. Brain-to-Text: Network Labeling

Given a brain network or activation map, we can retrieve the most semantically similar text from different datasets. This is useful for:

• Automatically labeling brain networks

• Understanding what cognitive processes are associated with activation patterns

• Validating network definitions against literature

Note - The threshold serves as a relevance filter for Brain-to-text network labeling — terms appearing in the top-k results are not guaranteed to be meaningfully related to the neuroimaging data. A threshold of 0.4 is generally a reliable cutoff, as most brain-to-text scores above this value correspond to significant associations. While exceptions exist, this filter effectively removes spurious or unrelated terms.

Load Network Atlases

We’ll work with canonical brain networks from multiple atlases.

# Load pre-encoded network atlases
networks = load_latent("networks_neuro")

# Available atlases
print("Available atlases:")
for atlas_name in networks.keys():
    print(f"  {atlas_name}: {len(networks[atlas_name])} networks")

Available atlases:
  Du: 16 networks
  Glasser: 12 networks
  HCPICA: 20 networks
  Laird: 16 networks
  Shen: 8 networks
  Shirer: 13 networks
  UKBICA: 20 networks
  WashU: 30 networks
  Woodward: 12 networks
  YeoLab: 38 networks

Example: Label the Auditory Network

# Use the auditory network from Du et al. atlas
auditory_network = networks["Du"]["AUD"]

# Find related text across all datasets
result = nvlm.brain(auditory_network).to_text(head="infonce")

# Show top 5 matches per dataset per dataset within threshold
top = result.top_k(5).query("cosine_similarity > 0.4")
top

Query Specific Datasets

You can also search specific datasets like Cognitive Atlas concepts, tasks, or disorders.

# Find related cognitive concepts
result = nvlm.brain(auditory_network).to_text(head="infonce")
concepts = result.top_k(5, dataset="cogatlas")
print("\nTop Cognitive Atlas Concepts:")
concepts

# Find related scientific papers
papers = result.top_k(5, dataset="pubmed")
print("\nTop Related Papers:")
papers

Top Related Papers:

	dataset	title	description	cosine_similarity
0	pubmed	The processing of temporal pitch and melody in...	An fMRI experiment was performed to identify t...	0.526884
1	pubmed	Heschl's gyrus, posterior superior temporal gy...	A part of the auditory system automatically de...	0.504547
2	pubmed	Hierarchical processing of sound location and ...	Horizontal sound localization relies on the ex...	0.499913
3	pubmed	Brain bases for auditory stimulus-driven figur...	Auditory figure-ground segregation, listeners'...	0.490401
4	pubmed	Dichotic pitch activates pitch processing cent...	Although several neuroimaging studies have rep...	0.485120

# Find related Wikipedia entries
wiki = result.top_k(5, dataset="wiki")
print("\nTop NeuroWiki Entries:")
wiki

Top NeuroWiki Entries:

	dataset	title	description	cosine_similarity
0	wiki	Temporal masking	Temporal masking or non-simultaneous masking o...	0.489581
1	wiki	Melodic expectation	In music cognition and musical analysis, the s...	0.454363
2	wiki	Search by sound	Search by sound is the retrieval of informatio...	0.452222
3	wiki	Beat (acoustics)	In acoustics, a beat is an interference patter...	0.447247
4	wiki	Harmonic series (music)	The harmonic series (also overtone series) is ...	0.446160

Example: Label the Default Mode Network

# Default mode network from Yeo et al. atlas
dmn = networks["YeoLab"]["DefaultA"]

result = nvlm.brain(dmn).to_text(head="infonce")

# top 5 matches per dataset within threshold
top = result.top_k(5).query("cosine_similarity > 0.4")
top

Example: Label a Motor Network

# Somatomotor network
motor = networks["YeoLab"]["SomMotA"]

result = nvlm.brain(motor).to_text(head="infonce")
top = result.top_k(5).query("cosine_similarity > 0.4")
top

2. ICA Component Labeling

Independent Component Analysis (ICA) is commonly used to identify brain networks from resting-state fMRI data (e.g., HCP, UK Biobank). NeuroVLM can automatically label these components.

# Example: Label ICA components from the HCP ICA atlas
ica_component = networks["HCPICA"]["ICA10"]

result = nvlm.brain(ica_component).to_text(head="infonce")
top = result.top_k(3)
print("ICA Component 10 is most similar to:")
top

3. NeuroVault Map Labeling

For arbitrary brain activation maps (e.g., from your own study or NeuroVault), you can:

1. Load the NIfTI image

2. Query it against text datasets

# Example with visual network
visual_network = networks["YeoLab"]["VisualA"]

result = nvlm.brain(visual_network).to_text(head="infonce")
top = result.top_k(5)
print("Visual Network Labels:")
top

4. Text-to-Brain: Finding Similar Brain Maps

Given a text query, retrieve the most similar brain activation patterns from datasets.

Note - The brain-to-text filtering threshold of 0.4 does not apply to text-brain

Example: Find brain maps for “emotion processing”

# Search across all brain datasets
result = nvlm.text("emotion processing").to_brain(head="infonce")

# Get top 5 matches
top = result.top_k(5)
top.table = top.table.sort_values('cosine_similarity', ascending=False).reset_index(drop=True)
top

There are adapters available but none are activated for the forward pass.

	dataset	dataset_index	title	description	cosine_similarity
0	pubmed	10250	Acute pharmacologically induced shifts in sero...	Pharmacological manipulation of serotonin avai...	0.440253
1	pubmed	10299	Passive and motivated perception of emotional ...	Emotionally expressive faces are processed by ...	0.438481
2	pubmed	10410	Interaction of catechol O-methyltransferase an...	Imaging genetic studies showed exaggerated blo...	0.437990
3	pubmed	597	Effects of attention and emotion on repetition...	We examined whether behavioral and neural effe...	0.411406
4	pubmed	5477	Psychopathy and functional magnetic resonance ...	Comorbidity between schizophrenia and psychopa...	0.408612
5	neurovault	682	Synchrony between sensory and cognitive networ...	Individuals with autistic spectrum disorders e...	0.350966
6	neurovault	369	Affective brain patterns as multivariate neura...	This study tested whether brain activity patte...	0.346154
7	neurovault	368	Affective brain patterns as multivariate neura...	This study tested whether brain activity patte...	0.346154
8	neurovault	1484	Early childhood stress is associated with blun...	Early life stress increases risk for later psy...	0.314913
9	neurovault	661	Synchrony between sensory and cognitive networ...	Individuals with autistic spectrum disorders e...	0.308400
10	networks	167	YeoLab	SalVentAttnB	0.234777
11	networks	8	Du	LANG	0.219431
12	networks	27	Glasser	OrbitAffective	0.201329
13	networks	150	YeoLab	Language	0.198475
14	networks	15	Du	NONE	0.194798

Example: Search specific brain datasets

# Search only in canonical networks
result = nvlm.text("attention").to_brain(head="infonce", dataset="networks")
top = result.top_k(3)
print("Top attention-related networks:")
top

Top attention-related networks:

	dataset	dataset_index	title	description	cosine_similarity
0	networks	31	HCPICA	ICA4	0.347036
1	networks	70	Shen	visual association	0.325007
2	networks	75	Shirer	HighVisual	0.322428

# Search only in PubMed activation maps
result = nvlm.text("working memory").to_brain(head="infonce", dataset="pubmed")
top = result.top_k(3)
print("Top working memory studies:")
top

Top working memory studies:

	dataset	dataset_index	title	description	cosine_similarity
0	pubmed	1581	Control networks and hemispheric asymmetries i...	Neuropsychological research has consistently d...	0.405711
1	pubmed	6428	Salience maps in parietal cortex: imaging and ...	Models of spatial attention are often based on...	0.405312
2	pubmed	22999	Effects of in-Scanner Bilateral Frontal tDCS o...	Working memory is an executive memory process ...	0.399832

Example: Multi-dataset retrieval

# Search across multiple datasets
result = nvlm.text("language comprehension").to_brain(
    head="infonce",
    dataset=["networks", "neurovault"]
)
top = result.top_k(5)
top.table = top.table.sort_values('cosine_similarity', ascending=False).reset_index(drop=True)
top

	dataset	dataset_index	title	description	cosine_similarity
0	neurovault	2	Reliable individual-level neural markers of hi...	The majority of functional neuroimaging invest...	0.468160
1	neurovault	1601	Knowing what you need to know in advance: The ...	Semantic retrieval is flexible, allowing us to...	0.433621
2	neurovault	543	Functional Connectivity in the Social Brain ac...	Previous research has characterized a collecti...	0.404827
3	neurovault	2591	Oxytocin Enhances an Amygdala Circuit Associat...	Negative symptoms are core contributors to voc...	0.400779
4	neurovault	2490	Evolution of reading and face circuits during ...	Although words and faces activate neighboring ...	0.398940
5	networks	63	Laird	DivergentCog6	0.393426
6	networks	21	Glasser	Language	0.340999
7	networks	44	HCPICA	ICA18	0.338026
8	networks	123	WashU	VentAttn	0.334167
9	networks	150	YeoLab	Language	0.315106

# Plot top matches from different datasets
top.plot_row(0, threshold=0.1, title="Top match - Networks");

top.plot_row(5, threshold=0.1, title="Top match - NeuroVault");

5. Comparing Multiple Queries

You can run multiple queries at once and compare results.

# Multiple text queries
queries = [
    "visual perception",
    "motor control",
    "executive function"
]

result = nvlm.text(["vision", "default mode network"]).to_brain(head="mse")
result.to_nifti() # returns list of nib.Nifti1Image

[<nibabel.nifti1.Nifti1Image at 0x76a67844bc50>,
 <nibabel.nifti1.Nifti1Image at 0x76a678642a80>]

result.plot(0, threshold=0.25); # plot image for vision

result.plot(1, threshold=0.15); # plot image for DMN

6. Batch Network Labeling

Label multiple networks at once for systematic comparison.

# Select multiple networks to label
import torch

network_names = ["VIS-P", "AUD", "SMOT-A", "DN-A", "LANG"]
network_latents = [networks["Du"][name] for name in network_names]

# Stack into batch
batch = torch.stack(network_latents)

# Label all at once
result = nvlm.brain(batch).to_text(head="infonce")

# Show top concept for each network
results = []
for i, name in enumerate(network_names):
    print(f"\n=== {name} ===")
    top = result.top_k(3, query_index=i, dataset="cogatlas")
    results.append(top)

results[0] # top concepts for "VIS-P"

	query_index	dataset	title	description	cosine_similarity
0	0	cogatlas	visual memory	A part of memory preserving some characteristi...	0.258297
1	0	cogatlas	perception	The conscious experience or mental registratio...	0.254730
2	0	cogatlas	object perception	The process of transforming basic visual senso...	0.251840

results[1] # top concepts for "AUD"

	query_index	dataset	title	description	cosine_similarity
0	1	cogatlas	auditory stream segregation	The perceptual grouping of sounds to form cohe...	0.437669
1	1	cogatlas	auditory encoding	The process of storing auditory information in...	0.431347
2	1	cogatlas	music cognition	The processing of mental functions on auditory...	0.431245

results[2] # top concepts for SMOT-A network

	query_index	dataset	title	description	cosine_similarity
0	2	cogatlas	motor control	The function of supervising motor activities.	0.438986
1	2	cogatlas	interoceptive representation	A representation of the internal state of the ...	0.358093
2	2	cogatlas	kinesthesia	A sense mediated by receptors located in muscl...	0.316170

7. Summary

In this tutorial, you learned:

1. Brain-to-Text retrieval:

   • Network labeling using canonical atlases

   • ICA component labeling

   • Querying specific datasets (PubMed, Cognitive Atlas, NeuroWiki)

2. Text-to-Brain retrieval:

   • Finding similar brain maps from text queries

   • Searching specific datasets (networks, neurovault, pubmed)

   • Multi-dataset retrieval

3. Advanced usage:

   • Batch processing multiple queries

   • Systematic network labeling

Next: In Tutorial 3, you’ll learn how to generate brain activation maps from text using the generative approach.