Tutorial 4: Generative Brain-to-Text (LLM)#
This tutorial shows how to generate natural language descriptions of brain activation patterns using a local language model.
Pipeline (brain → text)
Brain → Autoencoder Encoder → Projection Head → Contrastive Retrieval → LLM → Text
Pipeline (text → brain, with LLM context)
Text Query → to_brain() → [internal text-to-text retrieval] → LLM → Text
You will learn:
Setting up an LLM backend (Ollama or HuggingFace)
Generating text from a brain activation map (brain-to-text mode)
Generating text from a text query with neuroscience context (text-to-brain mode)
Customising prompts and filtering the context table
Labelling multiple networks in a loop
0. LLM Backend Setup#
generate_llm_response supports two backends — pick whichever fits your environment.
Option A — Ollama (recommended, fast)#
# Install (macOS)
brew install ollama
# Or Linux
sudo apt install zstd
curl -fsSL https://ollama.com/install.sh | sh
# Install to env
pip install neurovlm[llm]
Then
# Start the server
ollama serve
# Pull a model
ollama pull llama3.2:3b # lightweight, good quality
ollama pull qwen2.5:3b # alternative
Option B — HuggingFace (no server required, works offline)#
pip install transformers accelerate
Model suggestions:
"HuggingFaceTB/SmolLM2-360M-Instruct"— very fast, ~700 MB"Qwen/Qwen2.5-1.5B-Instruct"— better quality, ~3 GB"Qwen/Qwen2.5-3B-Instruct"— best quality, ~6 GB
Note:
model_nameis always required — there is no default. Omitting it raises an error.
import os
os.environ["USE_TF"] = "0"
os.environ["USE_FLAX"] = "0"
os.environ["TOKENIZERS_PARALLELISM"] = "false"
from neurovlm import NeuroVLM
from neurovlm.data import load_latent
nvlm = NeuroVLM(device="cpu")
1. Brain-to-Text Generation#
When you run brain(...).to_text(), the contrastive model retrieves the most similar
publications, NeuroWiki concepts, and CogAtlas terms for the input brain image.
You then inspect the table — filtering it however you like — and pass it directly to
generate_llm_response via the table= argument. The LLM uses exactly those rows
as context, and is aware the terms were found through neuroimage similarity.
Load a brain network#
# Load pre-encoded network atlases
networks = load_latent("networks_neuro")
# We'll use the auditory network from Du et al.
auditory_network = networks["Du"]["AUD"]
Step 1 — Contrastive retrieval#
Run brain(...).to_text() to rank the most similar text across all datasets.
A threshold of 0.4 is a reliable cutoff for meaningful brain-to-text associations.
result = nvlm.brain(auditory_network).to_text()
# Filter to the rows you actually want the LLM to see
context_table = result.top_k(5).query("cosine_similarity > 0.4")
context_table
| dataset | title | description | cosine_similarity | |
|---|---|---|---|---|
| 0 | cogatlas | auditory stream segregation | The perceptual grouping of sounds to form cohe... | 0.437669 |
| 1 | cogatlas | auditory encoding | The process of storing auditory information in... | 0.431347 |
| 2 | cogatlas | music cognition | The processing of mental functions on auditory... | 0.431245 |
| 3 | cogatlas | acoustic phonetic processing | The cognitive ability to discriminate items on... | 0.423635 |
| 4 | cogatlas | auditory tone detection | Determining the presence of an auditory stimul... | 0.417665 |
| 5 | networks | Auditory | Auditory network (AUD; auditory cortex network... | 0.470543 |
| 10 | pubmed | The processing of temporal pitch and melody in... | An fMRI experiment was performed to identify t... | 0.526884 |
| 11 | pubmed | Heschl's gyrus, posterior superior temporal gy... | A part of the auditory system automatically de... | 0.504547 |
| 12 | pubmed | Hierarchical processing of sound location and ... | Horizontal sound localization relies on the ex... | 0.499913 |
| 13 | pubmed | Brain bases for auditory stimulus-driven figur... | Auditory figure-ground segregation, listeners'... | 0.490401 |
| 14 | pubmed | Dichotic pitch activates pitch processing cent... | Although several neuroimaging studies have rep... | 0.485120 |
| 15 | wiki | Temporal masking | Temporal masking or non-simultaneous masking o... | 0.489581 |
| 16 | wiki | Melodic expectation | In music cognition and musical analysis, the s... | 0.454363 |
| 17 | wiki | Search by sound | Search by sound is the retrieval of informatio... | 0.452222 |
| 18 | wiki | Beat (acoustics) | In acoustics, a beat is an interference patter... | 0.447247 |
| 19 | wiki | Harmonic series (music) | The harmonic series (also overtone series) is ... | 0.446160 |
Step 2 — Generate with Ollama#
Pass table=context_table so the LLM sees exactly the rows you inspected above.
response = nvlm.generate_llm_response(
backend="ollama",
model_name="qwen2.5:3b", # must be pulled first: ollama pull qwen2.5:3b
table=context_table, # use the filtered table — k is ignored when table is set
)
print(response)
### Overview of Cognitive Functions Implicated
The input brain activation pattern described by these studies primarily relates to auditory processing and perception, including pitch analysis, melody recognition, temporal processing, and spatial localization. This region encompasses primary auditory cortex (primarily Heschl's gyrus and planum temporale) as well as more posterior regions like the superior temporal gyrus (STG), planum polare (PP), inferior colliculus in the midbrain, and prefrontal cortex. These areas are crucial for various aspects of sound processing that support auditory scene analysis.
The neuroimaging experiments conducted show how different brain structures contribute to specific functions such as pitch perception, melody recognition, temporal resolution of acoustic changes, and stimulus-driven figure-ground segregation. Melodic expectation studies indicate the brain's predictive mechanisms involved in music cognition, with areas like planum temporale showing increased activation for melodic stimuli.
### Cognitive Functions and Neuroscientific Mechanisms
#### Auditory Network
The auditory network (AUD) is characterized by regions such as Heschl’s gyrus (primary auditory cortex), planum temporale, and superior temporal gyrus. These areas are involved in the analysis of acoustic features like pitch, timbre, timing, and early speech sound encoding. The auditory network operates hierarchically, with different regions playing specific roles: primary auditory cortex for basic acoustics, and more posterior regions like STG and PP for higher-level processing.
#### Melodic Expectation
Melodic expectation is a cognitive process where the brain anticipates upcoming sounds based on past inputs (esthesic level) or the actual musical content (neutral level). The planum temporale shows increased activation in response to melodic stimuli, supporting the notion of a specialized region for pitch and melody processing.
#### Auditory Scene Analysis
Auditory scene analysis involves grouping multiple auditory streams into coherent representations. This process relies on regions like Heschl's gyrus which show activity related to both tone detection and complex acoustic patterns. The planum polare (PP) also shows selective activation during motion-related auditory scenes, indicating its role in integrating dynamic binaural cues.
### Clinical and Neurological Interpretations
#### Temporal Masking
Temporal masking is a phenomenon where sudden sounds can temporarily mask preceding or following sounds. It reflects the temporal integration of acoustic signals performed by different regions within the auditory network. The study involving dichotic pitch sequences highlights this process, showing activation in Heschl's gyrus and planum temporale for melodic stimuli.
#### Acoustic Phonetics
The auditory network processes acoustic features based on various phonetic contrasts such as place of articulation, manner of articulation, voicing, etc. This is supported by experiments involving spectral matched sounds that activate different regions within the AUD, with pitch-related activations seen in Heschl's gyrus.
#### Search by Sound Applications
Applications like Shazam and Soundhound use search algorithms to match audio clips against a library database for identification. These applications leverage auditory features such as timbre, frequency content, and temporal patterns processed by different areas of the AUD including primary auditory cortex and more posterior regions.
### Summary
The brain activation pattern described encompasses multiple hierarchical levels of auditory processing from simple tone detection through melodic expectation and complex auditory scene analysis. The primary auditory cortex (Heschl's gyrus and planum temporale) plays a central role in both basic acoustic feature analysis and higher-level perception tasks like melody recognition and temporal integration. Planum polare is specifically implicated in dynamic, spatially referenced processing such as detecting moving sounds. These findings support the notion of specialized regions for different auditory functions within the AUD, emphasizing the complex nature of auditory cognition and its neural underpinnings.
For more detailed clinical or neurodevelopmental considerations, additional studies would be required to explore any potential anomalies or developmental abnormalities in these areas.
Step 2 (alt) — Generate with HuggingFace#
Same table= argument works with the HuggingFace backend.
response = nvlm.generate_llm_response(
backend="huggingface",
model_name="Qwen/Qwen3-0.6B",
table=context_table,
max_new_tokens=512,
verbose=False,
think=False # Qwen3-0.6B has a "thinking" mode
)
print(response)
============================================================
Loading HuggingFace model: Qwen/Qwen3-0.6B
============================================================
Note: First-time download may take several minutes...
Progress indicators will appear below:
Step 1/2: Loading tokenizer...
✓ Tokenizer loaded
Step 2/2: Loading model (this may take a while)...
✓ Model loaded successfully!
Device: cuda:0
Model size: Qwen3-0.6B
============================================================
The brain activation pattern, as indicated by the provided publications, suggests a hierarchical and dynamic processing of auditory information, particularly in the auditory cortex. This includes the detection of pitch and melody, as well as the processing of spatial and temporal cues, which are reflected in the use of Heschl's gyrus (HG), planum temporale (PT), and superior temporal gyrus (STG) for auditory scene analysis. The integration of binaural cues and the recognition of auditory features like pitch and melody are supported by the activation of these regions. Additionally, the results highlight the role of the intraparietal sulcus (IPS) and superior temporal sulcus (STS) in mediating stimulus-driven auditory figure-ground segregation, indicating a bottom-up mechanism in perceptual processing. The findings also suggest a general involvement of the auditory cortex in pitch processing, despite the physical attributes of the stimuli. The results emphasize the importance of automatic, bottom-up mechanisms in auditory perception and suggest that these neural regions are critical for both cognitive and clinical applications in auditory processing.
Querying a single dataset#
You can also build the context table from a single dataset — e.g. only CogAtlas concepts.
# Use only CogAtlas concepts as LLM context
cogatlas_table = result.top_k(5, dataset="cogatlas").query("cosine_similarity > 0.4")
print(cogatlas_table[["title", "cosine_similarity"]])
response = nvlm.generate_llm_response(
backend="ollama",
model_name="llama3.2:3b",
table=cogatlas_table,
)
print(response)
title cosine_similarity
0 auditory stream segregation 0.437669
1 auditory encoding 0.431347
2 music cognition 0.431245
3 acoustic phonetic processing 0.423635
4 auditory tone detection 0.417665
Based on the provided cognitive terms from the Cognitive Atlas, the brain activation pattern appears to be associated with multiple aspects of auditory processing and perception.
At its core, this brain activation pattern suggests heightened activity in areas related to auditory stream segregation, indicating an individual's ability to group sounds together into coherent representations of objects in their acoustic environment. This process is fundamental to hearing and speech perception (NeuroWiki), highlighting the brain's capacity to organize and make sense of complex auditory information.
Furthermore, this pattern may also be linked to auditory encoding, as it reveals increased activity in regions responsible for storing auditory information in memory. This could suggest that the individual is more proficient at encoding and retrieving acoustic memories, which might be particularly relevant for tasks such as music cognition (Cognitive Atlas). Music cognition encompasses mental functions on auditory encoding and emotion regulation, implying a deeper level of cognitive processing that integrates emotional experiences with auditory encoding.
Additionally, the presence of this brain activation pattern could indicate an enhanced ability to discriminate items based on acoustic phonetic contrasts. This skill is essential for tasks such as speech recognition or music perception, where distinguishing between subtly different sounds is crucial (acoustic phonetic processing).
Finally, the activity in this brain pattern may also be associated with auditory tone detection, a basic aspect of detecting the presence of an auditory stimulus like a sound or tone.
Convergent themes across these cognitive terms suggest that this brain activation pattern may be linked to:
• Enhanced auditory perception and processing capabilities
• Improved ability to encode and retrieve acoustic memories
• Integration of emotional experiences with auditory encoding
• Increased discriminative accuracy in detecting sounds based on phonetic contrasts
• Fundamental role in tasks such as speech recognition, music perception, or even more complex cognitive processes like music cognition.
These interpretations are grounded in the provided Neuroscience concepts from NeuroWiki and Cognitive Atlas terms, highlighting the intricate relationships between different aspects of auditory processing and perception.
nvlm = NeuroVLM()
result = nvlm.brain(auditory_network).to_text()
# Use only CogAtlas concepts as LLM context
cogatlas_table = result.top_k(5, dataset="cogatlas").query("cosine_similarity > 0.4")
print(cogatlas_table[["title", "cosine_similarity"]])
response = nvlm.generate_llm_response(
backend="ollama",
model_name="llama3.2:3b",
table=cogatlas_table,
)
print(response)
title cosine_similarity
0 auditory stream segregation 0.437669
1 auditory encoding 0.431347
2 music cognition 0.431245
3 acoustic phonetic processing 0.423635
4 auditory tone detection 0.417665
**Overview**
The brain activation pattern appears to be associated with multiple aspects of auditory perception and cognition. The involvement of auditory stream segregation, auditory encoding, music cognition, acoustic phonetic processing, and auditory tone detection suggests that the brain is actively processing complex auditory information.
**Neuroscientific Mechanisms**
According to NeuroWiki, the concept of auditory stream segregation (also known as auditory grouping) refers to the process by which our brains group together sounds that are perceived as coming from the same source, allowing us to identify objects and locations in the acoustic scene. This process is thought to be mediated by neural mechanisms involving the integration of information across multiple auditory cortices.
The activation pattern also shows strong connections with auditory encoding, which involves the storage of auditory information in memory. This process is crucial for learning and remembering sounds, songs, and musical pieces, and is believed to involve the formation of associations between different acoustic features.
**Cognitive Significance**
The involvement of music cognition suggests that the brain activation pattern may be associated with the processing of emotional and motivational aspects of music perception. According to the Cognitive Atlas, music cognition involves the regulation of emotion, motivation, and attention in response to auditory encoding.
Acoustic phonetic processing is also evident, which highlights the ability of the brain to discriminate between different sounds based on their acoustic properties. This process is thought to be mediated by neural mechanisms involving the perception of contrasts in sonorance, manner, place, and voicing.
**Auditory Tone Detection**
The presence of strong activation in the auditory tone detection task suggests that the brain is actively detecting the presence or absence of auditory stimuli, such as sounds or tones. This process is crucial for detecting changes in the acoustic environment and responding to threats or opportunities.
**Convergent Themes**
A common thread across these cognitive terms is the involvement of neural mechanisms involving the integration and processing of auditory information. The brain activation pattern appears to be associated with complex auditory perception, including stream segregation, encoding, music cognition, phonetic processing, and tone detection. These processes are all interconnected and contribute to our ability to perceive, process, and respond to auditory stimuli in a meaningful way.
Note: The provided Cognitive Atlas terms were used as input for the interpretation of the brain activation pattern. While not exhaustive, these terms provide a comprehensive framework for understanding the cognitive functions and neural mechanisms implicated by the activation pattern.
2. Text-to-Brain Generation#
When you run text(...).to_brain(), NeuroVLM generates (or retrieves) brain maps
for your query.
Calling generate_llm_response afterwards automatically runs a text-to-text
similarity search internally — you do not need to call to_text() yourself.
The LLM uses the resulting publications and concepts as context to:
Define the user’s query term
Explain how the top neuroscience concepts relate to it
The brain result stored on the instance is not overwritten.
Generate a brain map, then describe the concept#
# Generate brain map from text
brain_result = nvlm.text("working memory").to_brain(head="mse")
brain_result.plot(threshold=0.2, title="Working Memory");
There are adapters available but none are activated for the forward pass.
# Internal text-to-text retrieval happens automatically; k controls how many rows it fetches
response = nvlm.generate_llm_response(
backend="ollama",
model_name="qwen2.5:3b",
k=5,
)
print(response)
### Overview:
The publications provide insights into how neurocognitive processes, particularly those related to deductive reasoning, arithmetic, and attentional control mechanisms, vary with parental education levels. Additionally, they highlight the compensatory recruitment of neuronal resources during working memory processing in older adults and frontal lobe abnormalities in first-episode schizophrenia patients. These findings are contextualized using Cognitive Atlas terms such as working memory maintenance, visuospatial sketch pad, learning, and updating to provide additional perspective on cognitive functions.
### Key Findings:
#### Parental Education and Deductive Reasoning:
The studies underscore the critical role of parental education levels in shaping a child's neurocognitive basis for higher-order reasoning tasks. Higher parental education was associated with increased reliance on the left inferior frontal gyrus, which is implicated in verbal systems, when children solved set-inclusion problems. This suggests that these problems might heavily rely on verbal mechanisms in the brain (Cognitive Atlas: language network). Conversely, lower-educated children who demonstrated higher nonverbal skills used right parietal areas more extensively for solving similar tasks. These findings imply that such variations can be due to differential engagement of spatial and cognitive control mechanisms (Cognitive Atlas: frontoparietal control) in children with different educational backgrounds.
#### Function of the Left Angular Gyrus:
Research also examines the function of the left angular gyrus (lAG), which is involved in mental arithmetic. Contrary to initial speculation, lAG activity during mental arithmetic tasks does not indicate task difficulty but rather reflects automatic mapping of arithmetic problems onto memory-stored solutions (Cognitive Atlas: visuospatial sketch pad and working memory). The study using the associative confusion effect showed that increased activation in anterior lAG areas correlates with better performance on non-confusion equations, which are stored more readily from memory. This underscores the role of the lAG in symbol-referent mapping.
#### APOE ε4 Allele and Alzheimer's Disease:
The APOE ε4 allele is known to influence brain activation patterns across various cognitive domains, including working memory processing. In cognitively intact older adults with varying levels of arithmetical fluency, the presence of the ε4 allele demonstrated compensatory recruitment of neural resources in task-positive regions such as the left dorsolateral prefrontal cortex (DLPFC) to maintain functional integrity despite age-related declines (Cognitive Atlas: frontoparietal control). This suggests that individuals with a higher risk for Alzheimer's disease use additional neuronal resources even when performing tasks normally completed through memory retrieval, indicating potential compensatory mechanisms.
#### Problem Size and Arithmetic Operation:
Further research using the 1-2-AX working memory task examined how brain activation during arithmetic calculation is influenced by problem size and operation type in children with varying levels of arithmetical fluency. The study revealed differential engagement patterns: low-fluent children relied more heavily on parietal regions for smaller problems, which are typically retrieved from memory (Cognitive Atlas: working memory), suggesting a potential role of fact retrieval impairments (Cognitive Atlas: updating) in their arithmetic difficulties.
### Compensatory Recruitment and Frontoparietal Control:
The findings regarding the compensatory recruitment of neuronal resources by ε4 carriers demonstrate how such mechanisms might be part of broader frontoparietal control networks involved in goal maintenance, flexible executive control, rule implementation, planning, and rapid task switching (Cognitive Atlas: frontoparietal control). These findings extend earlier work on frontal abnormalities in schizophrenia to include compensatory processes that may mitigate cognitive deficits seen in individuals with the ε4 allele.
### Conclusion:
The studies collectively illuminate how varying levels of parental education affect neurocognitive architecture for higher-order reasoning tasks, including deductive reasoning and arithmetic. They also highlight mechanisms such as compensatory recruitment of neuronal resources in older adults or altered frontal lobe activity in schizophrenia patients, all contextualized within the framework provided by Cognitive Atlas concepts like working memory maintenance, visuospatial sketch pad, learning, and updating. These insights provide a nuanced understanding of cognitive processes that can inform educational interventions, therapeutic approaches, and neurocognitive models.
# The brain result is still available — it was not overwritten
brain_result.plot(threshold=0.2, title="Working Memory (still available)");
3. Adding a Custom User Prompt#
Pass user_prompt to steer the LLM’s focus — e.g. ask a specific question or
request a particular output format.
result = nvlm.brain(networks["YeoLab"]["DefaultA"]).to_text()
context_table = result.top_k(5).query("cosine_similarity > 0.4")
response = nvlm.generate_llm_response(
backend="ollama",
model_name="llama3.2:3b",
table=context_table,
user_prompt="What clinical conditions are associated with this network?",
)
print(response)
**Overview**
The provided brain activation pattern suggests involvement of the default mode network (DMN), particularly the medial prefrontal cortex (mPFC) and posterior cingulate/precuneus, which are primary regions associated with self-referential thought, autobiographical/episodic memory, future simulation, social inference, and narrative/semantic integration. The activation pattern may also indicate altered connectivity patterns in the DMN compared to healthy controls.
**Convergent themes**
The NeuroWiki concepts highlight the importance of the DMN in internally oriented cognition, self-referential thought, and autobiographical/episodic memory. The Cognitive Atlas terms suggest that alterations in DMN connectivity have been observed in individuals with Alzheimer's disease (AD) and autism spectrum disorder (ASD). Additionally, reduced functional coupling in the DMN during self-referential processing has been reported.
**Network associations**
The brain activation pattern may be associated with various network disruptions, including:
* Altered DMN connectivity: Decreased functional connectivity in the DMN has been observed in AD patients and individuals with reduced gray matter atrophy.
* Reduced functional coupling in the DMN during self-referential processing: This has been reported in individuals with ASD.
* Increased prefrontal connectivity: May be a compensatory mechanism to maintain cognitive efficiency in AD.
**Clinical conditions**
Based on the NeuroWiki concepts and Cognitive Atlas terms, the brain activation pattern may be associated with:
* Alzheimer's disease (AD): Altered DMN connectivity, reduced functional coupling in the DMN during self-referential processing.
* Autism spectrum disorder (ASD): Reduced functional coupling in the DMN during self-referential processing.
* Cognitively normal elderly subjects: Amyloid burden may be associated with changes in DMN connectivity.
Please note that these interpretations are based on the provided NeuroWiki concepts and Cognitive Atlas terms, and further research is necessary to confirm the specific associations between brain activation patterns and clinical conditions.
# Request a concise bullet-point summary
result = nvlm.brain(networks["YeoLab"]["SomMotA"]).to_text()
context_table = result.top_k(5).query("cosine_similarity > 0.4")
response = nvlm.generate_llm_response(
backend="ollama",
model_name="llama3.2:3b",
table=context_table,
user_prompt="Summarise in three bullet points, suitable for a non-expert audience.",
)
print(response)
**Summary in three bullet points**
The brain activation pattern discussed in these publications suggests that it is associated with motor functions and processes. Here are three key aspects:
• **Motor network activity**: The brain activation patterns suggest that the motor networks, including primary motor cortex (M1), supplementary motor area (SMA), premotor cortex, and sensorimotor integration regions, are active during motor tasks or rest. This is consistent with the concept of motor control, where these areas work together to plan and execute voluntary movements.
• **Proprioceptive feedback and context-sensitive output**: The data also suggest that proprioceptive feedback plays a crucial role in motor control. For example, the supplementary motor area (SMA) shows increased activity when proprioception is altered, indicating that it may be involved in context-sensitive output for motor control. This is supported by the concept of motor control, where proprioceptive information helps guide movements.
• **Motor unit recruitment and cortical representation**: The brain activation patterns show that different body parts are represented in distinct areas of the primary motor cortex (M1). For example, the hands have a larger representation than the legs, suggesting that precise movements require more cortical resources. This is consistent with the concept of motor unit recruitment, where smaller motor units contribute to weaker muscle contractions and larger motor units contribute to stronger contractions.
**Clinical significance**
The brain activation patterns discussed in these publications can be associated with various clinical conditions, such as stroke or neurological disorders that affect motor control. For example, the data suggest that cortical reorganization occurs after a stroke, leading to changes in motor function. This is supported by the concept of motor control, where disruptions in this process can lead to motor impairments.
**Interpretation**
The brain activation patterns discussed in these publications provide insights into the neural mechanisms underlying motor control. The findings suggest that proprioceptive feedback plays a crucial role in guiding movements, and that different body parts are represented in distinct areas of the primary motor cortex (M1). These results have implications for our understanding of motor control and may inform the development of novel treatments for motor-related disorders.
4. Labelling Multiple Networks#
Loop over networks: run contrastive retrieval, inspect the filtered table,
then pass it to generate_llm_response.
target_networks = {
"Visual": networks["YeoLab"]["VisualA"],
"Auditory": networks["Du"]["AUD"],
"Motor": networks["YeoLab"]["SomMotA"],
"Default Mode": networks["YeoLab"]["DefaultA"],
"Language": networks["Du"]["LANG"],
}
for name, latent in target_networks.items():
print(f"\n{'='*60}")
print(f"Network: {name}")
print('='*60)
result = nvlm.brain(latent).to_text()
# Build and inspect the filtered context table
context_table = result.top_k(5).query("cosine_similarity > 0.4")
if not context_table.empty:
print(context_table[["dataset", "title", "cosine_similarity"]].to_string(index=False))
else:
print("(no matches above threshold)")
continue # skip generation if nothing passed the threshold
# Generate using exactly the rows above
response = nvlm.generate_llm_response(
backend="ollama",
model_name="llama3.2:3b",
table=context_table,
verbose=False,
)
print(f"\n{response}")
============================================================
Network: Visual
============================================================
dataset title cosine_similarity
cogatlas object perception 0.452152
cogatlas animacy perception 0.447128
cogatlas visual search 0.432990
cogatlas figure ground segregation 0.413456
cogatlas feature detection 0.403889
networks Visual 0.470024
pubmed Location and spatial profile of category-specific regions in human extrastriate cortex. 0.516058
pubmed Haptic perception and body representation in lateral and medial occipito-temporal cortices. 0.512238
pubmed fMRI-adaptation studies of viewpoint tuning in the extrastriate and fusiform body areas. 0.511523
pubmed Differing causal roles for lateral occipital cortex and occipital face area in invariant shape recognition. 0.489944
pubmed Human cortical regions involved in extracting depth from motion. 0.489836
wiki Neural processing for individual categories of objects 0.463630
wiki Body transfer illusion 0.455604
wiki Extrastriate body area 0.453649
wiki Evolution of the eye 0.452901
wiki Global precedence 0.430733
The provided brain activation pattern appears to be associated with various cognitive functions and processes. Here is a brief overview summarizing the main implications:
**Object Recognition and Perception**
The activation pattern shows significant responses in regions known for object recognition and perception, including the fusiform face area (FFA) for faces, the extrastriate body area (EBA) for human body parts, and other visual areas involved in processing different types of objects. These findings support the idea that specific categories of objects are processed independently in discrete neural regions.
The pattern also reveals activation in the lateral occipital cortex (LOC) and other object-selective areas, which suggest involvement in object recognition across different viewpoints. The data from the fMRI-adaptation study indicate a high degree of view-invariance in these regions, similar to that observed in other object-selective areas.
**Multimodal Body Representation**
The EBA showed selective activation during haptic perception and visual imagery of human body parts, suggesting the existence of different multimodal body representations in the occipito-temporal cortex. The findings also reveal a comparative level of body selectivity between the left and right EBA, highlighting the role of both hemispheres in processing body information.
**Global Precedence and Gestalt Principles**
The activation pattern implies engagement with global precedence, a phenomenon related to the Gestalt principles of grouping. Global interference effects were observed when participants were directed to identify local characteristics, indicating that global aspects of stimuli can interfere with local processing.
**Object Perception and Feature Detection**
The pattern shows activity in regions involved in object perception and feature detection, such as the ventral occipital regions. These findings support the idea that basic visual sensory input is transformed into more abstract and semantically identifiable wholes through a process of object perception.
**Evolutionary Aspects**
While not directly related to the brain activation pattern, the provided neuroscience concepts from NeuroWiki highlight the evolution of the eye and diverse eyes in animal forms, which may be relevant to understanding human vision and object recognition mechanisms.
Convergent themes across these cognitive terms and neuroscience concepts suggest that:
* Object recognition and perception are complex processes involving multiple neural regions and modalities.
* Different categories of objects are processed independently, with distinct neural representations.
* Multimodal body representation involves both hemispheres and different sensory modalities.
* Global precedence and Gestalt principles play a crucial role in visual processing and object recognition.
* Object perception and feature detection are essential processes underlying human vision and cognition.
============================================================
Network: Auditory
============================================================
dataset title cosine_similarity
cogatlas auditory stream segregation 0.437669
cogatlas auditory encoding 0.431347
cogatlas music cognition 0.431245
cogatlas acoustic phonetic processing 0.423635
cogatlas auditory tone detection 0.417665
networks Auditory 0.470543
pubmed The processing of temporal pitch and melody information in auditory cortex. 0.526884
pubmed Heschl's gyrus, posterior superior temporal gyrus, and mid-ventrolateral prefrontal cortex have different roles in the detection of acoustic changes. 0.504547
pubmed Hierarchical processing of sound location and motion in the human brainstem and planum temporale. 0.499913
pubmed Brain bases for auditory stimulus-driven figure-ground segregation. 0.490401
pubmed Dichotic pitch activates pitch processing centre in Heschl's gyrus. 0.485120
wiki Temporal masking 0.489581
wiki Melodic expectation 0.454363
wiki Search by sound 0.452222
wiki Beat (acoustics) 0.447247
wiki Harmonic series (music) 0.446160
The brain activation pattern presented involves multiple brain regions and processes that are associated with various cognitive functions.
At the primary level, auditory cortical areas such as Heschl's gyrus (primary auditory cortex) and planum temporale exhibit significant activations in response to pitch-evoking sounds, melody, and changes in acoustic environment. These findings support the view of a hierarchical organization of auditory spatial processing, where the general analysis of binaural information begins early in the brainstem, while the representation of dynamic binaural cues relies on non-primary auditory fields in the planum temporale.
In addition to auditory processing, this activation pattern also reveals significant involvement of regions involved in attentional processes and figure-ground segregation. The mid-ventrolateral prefrontal cortex shows activation in response to changes in acoustic environment, suggesting a role in detecting novel or salient sounds. In contrast, the intraparietal sulcus (IPS) demonstrates activation in response to stimulus-driven auditory segregation, supporting a role for automatic, bottom-up mechanisms in mediating sensory input and perceptual organization.
The involvement of multiple brain regions highlights the complex interplay between auditory processing, attentional control, and figure-ground segregation. These processes are intricately linked, with attentional resources allocated based on the detection of salient or novel sounds, which can then be further processed through top-down mechanisms. This convergence of neural systems underscores the importance of considering both bottom-up sensory input and top-down cognitive influences in our understanding of auditory processing.
Furthermore, this activation pattern reveals specific roles for different brain regions involved in pitch processing, melody recognition, and auditory scene analysis. The general representation of pitch across multiple brain areas supports a view of a unified auditory cortical network that is sensitive to various acoustic features.
Overall, the neural mechanisms underlying auditory perception and cognition are multifaceted, involving an intricate interplay between primary sensory areas, attentional control systems, and higher-level cognitive processes.
Based on these findings, several potential clinical implications can be explored. For instance, research into individual differences in brain activation patterns could inform strategies for enhancing auditory processing abilities in individuals with hearing impairments or other auditory disorders.
============================================================
Network: Motor
============================================================
dataset title cosine_similarity
cogatlas motor control 0.471245
networks Motor 0.503942
pubmed The hand motor hotspot for seed-based functional connectivity of hand motor networks at rest 0.522147
pubmed Convergence of human brain mapping tools: neuronavigated TMS parameters and fMRI activity in the hand motor area. 0.499828
pubmed The role of the contralesional motor cortex for motor recovery in the early days after stroke assessed with longitudinal FMRI. 0.496707
pubmed High-sensitivity TMS/fMRI of the Human Motor Cortex Using a Dedicated Multichannel MR Coil. 0.496136
pubmed Re-thinking the role of motor cortex: context-sensitive motor outputs? 0.489319
wiki Nonprimary motor cortex 0.482699
wiki Motor evoked potentials 0.465895
wiki Motor unit recruitment 0.462458
wiki Supplementary motor area 0.460516
wiki Primary motor cortex 0.457346
**Overview**
The input brain activation pattern is associated with the functional connectivity of hand motor networks at rest. This analysis reveals that the individual hand motor hotspot (hMHS) as a seed region showed FC patterns mainly including sensorimotor regions, indicating the involvement of primary motor cortex (M1), premotor cortex, and supplementary motor area (SMA). The brain activation pattern may be associated with movement planning, initiation, and execution, as well as proprioceptive/tactile feedback.
**Motor Cortex and Motor Control**
The primary motor cortex (M1) is the primary region of the motor system and works in association with other motor areas to plan and execute voluntary movements. The supplementary motor area (SMA) is thought to play a role in postural stabilization, coordination of bilateral movements, and control of sequences of movements. Motor control refers to the function of supervising motor activities, which involves the coordinated activation of multiple neural systems, including the primary motor cortex, premotor cortex, SMA, and sensorimotor areas.
**Motor Evoked Potentials**
The analysis of motor evoked potentials (MEPs) is relevant to understanding the brain activation pattern. MEPs are recorded from muscles following direct stimulation of exposed motor cortex or transcranial stimulation of motor cortex. This technique can provide insights into the neural mechanisms underlying motor control and has been used in various studies on motor learning, neurorehabilitation, and motor control.
**Supplementary Motor Area**
The supplementary motor area (SMA) is a part of the motor cortex that contributes to the control of movement. It contains a rough map of the body, but this map is not apparent in humans. Neurons in the SMA project directly to the spinal cord, suggesting a role in direct control of movement.
**Primary Motor Cortex**
The primary motor cortex (M1) is defined anatomically as the region of cortex that contains large neurons known as Betz cells. The primary motor cortex is orderly arranged from the toe to mouth along a fold in the cortex called the central sulcus, but some body parts may be controlled by partially overlapping regions of cortex.
**Convergence across Cognitive Terms**
The brain activation pattern and cognitive terms from the Cognitive Atlas suggest that motor control is closely related to movement planning, initiation, and execution. The involvement of primary motor cortex (M1), premotor cortex, and supplementary motor area (SMA) in this analysis supports the idea that these regions are critical for supervising motor activities.
============================================================
Network: Default Mode
============================================================
dataset title cosine_similarity
networks Default Mode 0.467490
pubmed The Self-Concept Is Represented in the Medial Prefrontal Cortex in Terms of Self-Importance 0.504036
pubmed Resting state fMRI in Alzheimer's disease: beyond the default mode network. 0.481533
pubmed Reduced functional coupling in the default-mode network during self-referential processing. 0.479043
pubmed Regional amyloid burden and intrinsic connectivity networks in cognitively normal elderly subjects. 0.473489
pubmed Associations and dissociations between default and self-reference networks in the human brain. 0.472962
wiki Default mode network 0.469827
wiki TheBrain 0.422660
wiki Oxygen 0.410480
wiki 2C-O 0.408183
Here's an overview of the main cognitive functions, processes, or states implicated by the brain activation pattern:
The brain activation patterns presented are related to various aspects of self-awareness, cognition, and neural function. The Self-Concept Is Represented in the Medial Prefrontal Cortex study suggests that the medial prefrontal cortex (mPFC) is involved in representing one's sense of self, specifically in terms of self-importance. This region is also implicated in default mode network (DMN) activity, which is thought to be related to introspection and self-referential processing.
The Resting state fMRI in Alzheimer's disease study reveals alterations in connectivity patterns within the DMN and other brain networks, including the frontal networks and salience networks. The DMN is also involved in cognitively normal elderly subjects with amyloid burden, but its functional connectivity is reduced compared to those without amyloid burden.
The Reduced functional coupling in the default-mode network during self-referential processing study highlights the specialization of areas within the DMN for different subfunctions of self-referential processing. The Regional amyloid burden and intrinsic connectivity networks in cognitively normal elderly subjects study shows that amyloid burden is associated with alterations in functional connectivities of the DMN and other networks.
Convergent themes across these studies include:
* The involvement of the mPFC in representing one's sense of self, specifically in terms of self-importance
* Alterations in DMN connectivity patterns in various cognitive states, including Alzheimer's disease and normal aging
* Specialization of areas within the DMN for different subfunctions of self-referential processing
* Associations between amyloid burden and alterations in functional connectivities of the DMN and other networks
These findings suggest that the brain activation patterns are related to complex cognitive processes, including self-awareness, introspection, and neural function.
============================================================
Network: Language
============================================================
dataset title cosine_similarity
pubmed FMRI study of emotional speech comprehension. 0.483236
pubmed What you say versus how you say it: Comparing sentence comprehension and emotional prosody processing using fMRI. 0.458775
pubmed Prosodic and narrative processing in American Sign Language: an fMRI study. 0.449186
pubmed Understanding the internal states of others by listening to action verbs. 0.446219
pubmed Observing Others Speak or Sing Activates Spt and Neighboring Parietal Cortex. 0.444839
wiki Nonverbal autism 0.443231
wiki Stance (linguistics) 0.438382
wiki Phatic expression 0.432939
wiki Emotional prosody 0.432630
wiki Verbal Behavior 0.428893
Based on the input brain activation patterns related to emotional speech comprehension, what your say versus how you say it, prosodic and narrative processing in American Sign Language, understanding the internal states of others by listening to action verbs, observing Others Speak or Sing Activates Spt and Neighboring Parietal Cortex, as well as the provided NeuroWiki concepts, particularly nonverbal autism, stance (linguistics), phatic expression, emotional prosody, and verbal behavior, I can provide a summary of the main cognitive functions, processes, or states implicated by the brain activation.
The brain activation patterns suggest that processing affective semantic content, including emotional prosody, requires cooperation between semantics, affective prosody, and mind-reading neural networks. This is evident in the increased activation of left inferior frontal area involved in semantic knowledge retrieval, posterior part of the left superior temporal sulcus (STS), and medial prefrontal cortex during affective classification tasks. The presence of additional rightward activations in areas corresponding to human-selective voice area and STS when affective prosody was present further highlights the importance of emotional prosody processing.
In contrast, verbal communication and narrative processing, as seen in American Sign Language (ASL), rely on similar left hemisphere-dominant patterns for brain organization. However, the right hemisphere is involved in processing narrative-level information, including prosody, affect, facial expression, and discourse structure. This suggests that both spoken and signed languages share common neural mechanisms for language processing, with the right hemisphere playing a more significant role in processing narrative linguistic functions.
The investigation into auditory vitality forms while listening to action verbs reveals specific activation of the central part of insula during processing of human voice conveying specific vitality forms. This finding is consistent with previous studies on vitality forms encoding and suggests that the central part of insula is a key region for understanding internal states, regardless of the modality by which they are conveyed.
Observing Others Speak or Sing Activates Spt and Neighboring Parietal Cortex highlights the involvement of posterior parietal cortex in action observation and planning. The distinct activation of area Spt (supramarginal gyrus) specifically for vocal communication actions, as well as its distinction from the parietal region involved in processing other's visual and auditory signals related to vocal communication, underscores the complex neural mechanisms underlying action observation.
In the context of nonverbal autism, the brain activation patterns suggest that individuals with this condition may have altered neural responses to emotional prosody, social interactions, or verbal behavior. This could be due to differences in stance-taking, phatic expression, or other linguistic and cognitive processes.
The concepts of stance (linguistics) and phatic expression are particularly relevant here, as they describe the ways in which speakers position themselves relative to the ongoing interaction, including evaluation, intentionality, epistemology, or social relations. Phatic expressions serve to establish or maintain social relationships and can be observed in everyday conversational exchanges.
In terms of emotional prosody, the findings highlight its importance in conveying emotion and interacting with verbal content. This is particularly evident in the improved perception of anger and sadness compared to other emotions.
Finally, the concept of verbal behavior by B. F. Skinner provides a framework for understanding verbal communication as a system of controlling elements, including echoics, mands, tacts, autoclitics, and others. This framework can be used to analyze linguistic processes, including emotional prosody processing, in the context of human communication.
In summary, the brain activation patterns suggest that emotional speech comprehension, narrative processing in signed languages, auditory vitality forms processing, action observation, and verbal behavior all rely on complex neural mechanisms involving multiple brain regions. The concepts of stance (linguistics), phatic expression, emotional prosody, and nonverbal autism provide a framework for understanding the cognitive functions and processes implicated by these brain activation patterns.
5. Querying a Specific Dataset Before Generating#
Pass datasets to to_text() to retrieve only from one source, then build
the context table from that restricted result.
# Retrieve only PubMed papers for the visual network
visual = networks["YeoLab"]["VisualA"]
result = nvlm.brain(visual).to_text(datasets=["pubmed"])
context_table = result.top_k(3, dataset="pubmed")
print("Papers used as context:")
print(context_table[["title", "cosine_similarity"]].to_string(index=False))
Papers used as context:
title cosine_similarity
Location and spatial profile of category-specific regions in human extrastriate cortex. 0.516058
Haptic perception and body representation in lateral and medial occipito-temporal cortices. 0.512238
fMRI-adaptation studies of viewpoint tuning in the extrastriate and fusiform body areas. 0.511523
# LLM will only have paper context (no wiki or cogatlas)
response = nvlm.generate_llm_response(
backend="ollama",
model_name="llama3.2:3b",
table=context_table,
)
print(response)
The brain activation patterns presented in these publications suggest a complex interplay between various cognitive functions, including category-specific perception, body representation, and viewpoint tuning. Here is a brief overview of the main cognitive functions implicated:
The extrastriate cortex, particularly the fusiform face area (FFA), extrastriate body area (EBA), parahippocampal place area (PPA), lateral occipital complex (LOC), and MT+/V5, are regions that show category-specific responses to different visual stimuli. The FFA and EBA, in particular, are involved in the perception of faces and body parts, respectively.
The haptic exploration of body parts also activates these regions, suggesting a multimodal representation of the body that integrates visual and somatosensory information. Furthermore, the activation patterns reveal different body representations in the occipito-temporal cortex, which are activated during perception and imagery of human body parts.
The viewpoint tuning experiments provide insight into how the extrastriate and fusiform body areas (EBA and FBA) process views of the human body. These regions exhibit view-invariance, meaning they are less responsive to changes in viewing angle, similar to other object-selective areas. The results suggest that later waves of neuronal responses drive fMRI adaptation effects.
Here is a synthesis of the terms and their implications:
* **Category-specific perception**: The activation patterns reveal category-specific responses to different visual stimuli, such as faces, body parts, objects, and scenes.
* **Body representation**: The FBA and EBA are involved in the perception of human bodies, while the PPA is implicated in spatial memory and navigation. Multimodal representations of the body integrate visual and somatosensory information.
* **Viewpoint tuning**: The extrastriate and fusiform body areas exhibit view-invariance, meaning they are less responsive to changes in viewing angle. This suggests that these regions process views of the human body in a more abstract or categorical manner.
* **Multimodal processing**: The activation patterns reveal different body representations in the occipito-temporal cortex, which are activated during perception and imagery of human body parts.
These findings highlight the complex neural mechanisms underlying category-specific perception, body representation, and viewpoint tuning. They provide insights into how the brain processes visual information from multiple modalities and how these processes are organized across different brain regions.
6. Summary#
In this tutorial you learned:
LLM backends: Ollama (fast, requires local server) and HuggingFace (offline, no server).
model_nameis always required — there is no default.Brain-to-text mode (
brain(...).to_text()→generate_llm_response):Contrastive retrieval ranks the most similar publications, NeuroWiki concepts, and CogAtlas terms for the input brain map.
You inspect and filter the table yourself (e.g.
top_k(5).query("cosine_similarity > 0.4")).Pass
table=togenerate_llm_response— the LLM sees exactly those rows.The LLM frames the terms in the context of neuroimage similarity.
Text-to-brain mode (
text(...).to_brain()→generate_llm_response):An internal text-to-text retrieval is run automatically — no extra call needed.
Use
k=to control how many rows are fetched internally.The LLM defines the user’s query term and explains how the top neuroscience concepts relate to it. The stored brain result is preserved.
user_prompt: steer the LLM towards a specific question or output format.Batch labelling: loop over networks — build the filtered
context_tableper network and pass it togenerate_llm_response.Dataset filtering: pass
datasets=["pubmed"](or any subset) toto_text()to restrict what context the LLM receives.
Key tip: larger models (≥ 7B parameters) produce noticeably better responses.
For production use, prefer qwen2.5:7b-instruct or llama3.1:8b via Ollama.